AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A retail company collects daily sales reports from thousands of point-of-sale (POS) systems across various stores. Each report is a CSV file, typically 10-100 MB, and is uploaded to a centralized SFTP server hosted on-premises. The company needs to move these files to Amazon S3, combine them into a single Parquet file per day, and partition the data by date for efficient querying. This process must run daily after all reports are received. Which transformation service is most appropriate for this task?
- AAmazon EMR with Apache Spark
- BAmazon Kinesis Data Analytics for Apache Flink
- CAWS Glue Streaming ETL
- DAWS Glue ETL (Batch)
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Glue ETL (Batch)
AWS Glue ETL (Batch) is a serverless data integration service that is well-suited for scheduled, batch processing tasks. It can read CSV files from S3, perform transformations like combining and converting to Parquet, and partition the output, aligning perfectly with the daily batch nature of this requirement.
Why the other options are wrong
- A. Amazon EMR with Apache Spark is a powerful and flexible option, but for a recurring, scheduled batch job like this, AWS Glue ETL offers a more fully managed, serverless experience with less operational overhead, making it generally more cost-effective and simpler to manage.
- B. Kinesis Data Analytics for Apache Flink is for real-time streaming data processing, not daily batch processing of files.
- C. AWS Glue Streaming ETL is designed for continuous processing of streaming data, not for daily batch jobs on static files.
AWS Glue ETL (Batch)
A serverless data integration service that makes it easy to discover, prepare, and combine data for analytics. It's ideal for scheduled, batch-oriented Extract, Transform, and Load (ETL) jobs.
- Serverless: no infrastructure to manage.
- Spark-based: scalable for large datasets.
- Integrated with Data Catalog for metadata management.
Memory trick: Glue's batch jobs make files pristine.