AWS Certified Data Engineer – AssociateData Storage and ManagementHard
A data engineering team is building a serverless data processing pipeline using AWS Lambda and Amazon S3. The pipeline processes billions of small JSON files (average 10KB each) daily. To improve the efficiency and cost-effectiveness of downstream analytics with Amazon Athena, the team wants to consolidate these small files into larger, optimized files and convert them to a columnar format. Which AWS service is best suited for orchestrating and executing this compaction and conversion process in a serverless manner?
- AAWS Data Pipeline
- BAWS Step Functions
- CAWS Glue ETL jobs
- DAmazon EMR
Show answer & explanationAnswer & explanation
Correct answer: C. AWS Glue ETL jobs
AWS Glue ETL jobs are serverless, scale automatically, and are designed for data transformation, including compaction and format conversion (e.g., to Parquet), directly integrating with the Glue Data Catalog for schema management. This makes it ideal for the described scenario.
Why the other options are wrong
- A. AWS Data Pipeline is an older orchestration service, less flexible and powerful for modern data transformation tasks than Glue ETL jobs, and not considered serverless in the same context as Glue.
- B. Step Functions orchestrates workflows but doesn't perform the data processing itself; it would need to invoke other services like Lambda or Glue.
- D. Amazon EMR is a managed Hadoop cluster service, which is powerful but not serverless and requires cluster management, making it less ideal for a purely serverless pipeline compared to Glue.
AWS Glue ETL Jobs
AWS Glue ETL jobs are serverless Apache Spark-based jobs that enable data engineers to extract, transform, and load data at scale, automatically handling provisioning, setup, and scaling of compute resources.
- Serverless Spark environment
- Automates ETL processes
- Integrates with Glue Data Catalog
- Ideal for data transformation, compaction, format conversion
Memory trick: Glue ETL jobs transform data, serverless and grand.