AWS Certified Data Engineer – AssociateData Storage and ManagementMedium
A data engineer is designing a data lake solution that ingests raw log files from various sources. These logs contain sensitive customer information and must be encrypted at rest and in transit. The solution needs to support querying the data directly using SQL-like queries without loading it into a database. Which combination of AWS services best meets these requirements?
- AAWS DataSync for ingestion, Amazon S3 for storage with SSE-KMS encryption, and Amazon Athena for querying.
- BAWS Glue for ETL, Amazon DynamoDB for storage, and AWS QuickSight for reporting.
- CAmazon Kinesis Data Firehose for ingestion, Amazon RDS for storage, and Amazon Redshift for querying.
- DAWS Glue for ETL, Amazon S3 for storage with SSE-S3 encryption, and Amazon Athena for querying.
Show answer & explanationAnswer & explanation
Correct answer: A. AWS DataSync for ingestion, Amazon S3 for storage with SSE-KMS encryption, and Amazon Athena for querying.
AWS DataSync can securely ingest data. Amazon S3 with SSE-KMS provides encryption at rest with KMS key management. Amazon Athena allows direct SQL querying of data in S3, making it suitable for serverless analytics on log files.
Why the other options are wrong
- B. DynamoDB is a NoSQL database, not suitable for storing raw log files for SQL-like querying, and QuickSight is a BI tool, not a query engine for raw data.
- C. RDS and Redshift require data loading and are not ideal for direct querying of raw S3 logs.
- D. SSE-S3 uses S3-managed keys; SSE-KMS provides more control with customer-managed keys (CMKs) or AWS managed keys, which is often preferred for sensitive data.
Data Lake Querying with Athena
Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. It's serverless, so there's no infrastructure to manage, and you pay only for the queries you run.
- Serverless interactive query service
- Queries data directly in S3
- Uses standard SQL
- Pay-per-query pricing model
Memory trick: Athena Answers Any Analytics on S3.