AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global e-commerce company needs to process customer order data in near real-time. Each order record, averaging 1 KB in size, arrives concurrently from various regional microservices, with peak rates reaching 50,000 records per second. The company requires the data to be immediately available for fraud detection and personalized recommendations, and also stored in Amazon S3 for historical analysis. Which combination of AWS services should be used for this ingestion and initial processing?
- AAWS DataSync and AWS Lambda
- BAmazon SQS and AWS Glue Streaming ETL
- CAmazon Kinesis Data Streams and Amazon Kinesis Data Analytics for Apache Flink
- DAWS Glue ETL (Batch) and Amazon S3
Show answer & explanationAnswer & explanation
Correct answer: C. Amazon Kinesis Data Streams and Amazon Kinesis Data Analytics for Apache Flink
Kinesis Data Streams provides a highly scalable and durable real-time data ingestion layer, capable of handling high throughput. Kinesis Data Analytics for Apache Flink can then process this stream in near real-time for fraud detection and recommendations, while also delivering to S3.
Why the other options are wrong
- A. AWS DataSync is for large-scale file transfers, not real-time streaming, and Lambda is a compute service, not a primary streaming ingestion or processing layer for this scale.
- B. Amazon SQS is a message queuing service, not optimized for high-throughput stream processing like Kinesis Data Streams. AWS Glue Streaming ETL could process a stream, but SQS isn't the best ingestion layer here.
- D. AWS Glue ETL (Batch) is for batch processing, not near real-time, and Amazon S3 is a storage layer, not an ingestion service for real-time streams.
Kinesis Data Streams + Kinesis Data Analytics
A powerful combination for real-time data ingestion and processing. Kinesis Data Streams captures data, and Kinesis Data Analytics (with Flink) performs sophisticated real-time analytics.
- Kinesis Data Streams: highly scalable, durable, real-time data capture.
- Kinesis Data Analytics: executes SQL or Apache Flink applications on streaming data.
- Ideal for real-time dashboards, anomaly detection, and interactive analytics.
Memory trick: Streams flow, Flink thinks, insights grow.