AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global e-commerce company needs to process customer order data in near real-time. Each order record needs to be validated, enriched with customer demographic information from a DynamoDB table, and then stored in an Amazon S3 data lake in Parquet format. The solution must be serverless, cost-effective, and scale automatically with fluctuating order volumes. Which AWS service is the MOST appropriate for this transformation?
- AAWS Step Functions with AWS Lambda
- BAWS Glue Streaming ETL
- CAWS Glue ETL (Batch)
- DAmazon EMR with Apache Spark
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Glue Streaming ETL
AWS Glue Streaming ETL is designed for continuous, near real-time processing of streaming data. It can read from sources like Kinesis Data Streams (implied for near real-time order data), perform transformations like validation and enrichment, and write to S3 in optimized formats like Parquet, all in a serverless and auto-scaling manner.
Why the other options are wrong
- A. AWS Step Functions orchestrates workflows, and while Lambda can perform transformations, it's not optimized for continuous, high-volume streaming ETL and would require more custom code and orchestration than Glue Streaming ETL for this specific use case.
- C. AWS Glue ETL (Batch) is for batch processing of historical data, not suitable for near real-time requirements.
- D. Amazon EMR with Apache Spark is powerful for large-scale data processing but requires managing EMR clusters, which contradicts the 'serverless' requirement.
AWS Glue Streaming ETL
A serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development, specifically for streaming data.
- Serverless and auto-scaling
- Apache Spark-based for powerful transformations
- Supports continuous data streams (Kinesis, Kafka)
- Integrates with various data sources and targets (S3, DynamoDB)
Memory trick: Glue Streaming ETL sticks your data together and shapes it as it flows.