AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A retail company collects point-of-sale (POS) data from thousands of stores. Each store sends daily batch files (CSV format) to a central S3 bucket. The data needs to be aggregated by product, anonymized for customer information, and then partitioned by date and store ID into Parquet format for efficient querying in a data lake. The processing should be scheduled nightly and be cost-optimized, paying only for compute capacity used during processing. Which AWS service is most suitable for this batch transformation?
- AAmazon Kinesis Data Streams
- BAmazon EMR with Apache Hive
- CAWS Data Pipeline
- DAWS Glue ETL
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Glue ETL
AWS Glue ETL is a serverless data integration service that is well-suited for batch processing of large datasets. It can read CSV files from S3, perform complex transformations like aggregation and anonymization, and write the output in Parquet format, partitioned, to another S3 location. Its serverless nature means you pay only for the duration of the ETL job.
Why the other options are wrong
- A. Amazon Kinesis Data Streams is for real-time streaming data, not for daily batch file processing.
- B. Amazon EMR with Apache Hive could perform the transformations but requires managing EMR clusters, which is not as cost-optimized as a serverless solution like Glue ETL for batch jobs where resources are provisioned on demand.
- C. AWS Data Pipeline is an orchestration service but typically relies on other compute resources (like EC2 or EMR) for the actual transformation, making it less integrated and potentially more expensive than Glue ETL for the transformation itself.
AWS Glue ETL Batch Processing
A serverless service that allows for scheduled, cost-effective batch transformations of data from various sources (like S3) into optimized formats (like Parquet) for analytical workloads.
- Serverless and pay-per-use
- Handles large batch datasets
- Supports diverse data formats and complex transformations
Memory trick: Glue processes nightly batches, making data fit for matches.