AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global e-commerce company needs to process customer orders in real-time. Each order record contains various attributes like customer ID, product details, quantity, and payment information. The company anticipates up to 10,000 orders per second during peak sales events. The data needs to be ingested, validated, and transformed before being stored in an analytical data store. Which AWS service combination should be used to achieve this real-time ingestion and transformation with high throughput and low latency?
- AAmazon Kinesis Data Streams + AWS Glue Streaming ETL
- BAWS DataSync + Amazon Redshift
- CAmazon MQ + AWS Lambda
- DAmazon S3 + AWS Glue ETL (Batch)
Show answer & explanationAnswer & explanation
Correct answer: A. Amazon Kinesis Data Streams + AWS Glue Streaming ETL
Amazon Kinesis Data Streams is designed for real-time ingestion of high-throughput data like customer orders. AWS Glue Streaming ETL can then consume this stream, perform real-time validation and transformation, and deliver the processed data to an analytical store with low latency.
Why the other options are wrong
- B. AWS DataSync is for large-scale data transfers between on-premises and AWS, not for real-time streaming ingestion from an application. Amazon Redshift is a data warehouse, not an ingestion or transformation service itself.
- C. Amazon MQ is a message broker and AWS Lambda can process messages, but Kinesis Data Streams is specifically optimized for high-throughput, ordered, and replayable real-time data ingestion, which is more suitable for this scale and use case than a generic message queue for streaming analytics.
- D. Amazon S3 and AWS Glue ETL (Batch) are suitable for batch processing, not real-time ingestion and transformation at 10,000 orders per second.
Kinesis Data Streams + Glue Streaming ETL
A powerful combination for real-time data ingestion and continuous transformation, where Kinesis Data Streams captures high-volume, real-time data and Glue Streaming ETL processes it on the fly.
- Kinesis provides durable, scalable real-time ingestion.
- Glue Streaming ETL offers serverless, continuous data transformation.
- Ideal for analytics on live operational data, IoT, or clickstreams.
Memory trick: Kinesis streams the data, Glue transforms it with a gleam.