AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global gaming company collects player interaction data (e.g., achievements, in-game purchases, session durations) from millions of concurrent players. This data needs to be ingested in real-time, aggregated into hourly summaries, and then loaded into an Amazon Redshift data warehouse for analytical reporting. The solution must be serverless and handle fluctuating data volumes efficiently. Which AWS service combination is best suited for the real-time ingestion and aggregation before loading to Redshift?
- AAmazon MQ + Amazon EMR
- BAmazon Kinesis Data Streams + Amazon Kinesis Data Analytics for Apache Flink
- CAWS Step Functions + AWS Lambda
- DAmazon Kinesis Data Firehose + AWS Glue ETL (Batch)
Show answer & explanationAnswer & explanation
Correct answer: B. Amazon Kinesis Data Streams + Amazon Kinesis Data Analytics for Apache Flink
Amazon Kinesis Data Streams provides the highly scalable, real-time ingestion layer for millions of events. Amazon Kinesis Data Analytics for Apache Flink is ideal for performing real-time aggregations (like hourly summaries) on this stream in a serverless manner, before the aggregated data is loaded into Redshift.
Why the other options are wrong
- A. Amazon MQ is a message broker, not primarily for high-throughput real-time ingestion for analytics. Amazon EMR offers powerful processing but is cluster-based and not serverless for real-time stream aggregation without significant configuration.
- C. AWS Step Functions orchestrates workflows, and AWS Lambda executes serverless functions. While they could be used, they are not optimized for continuous, high-throughput stream processing and stateful aggregations like Kinesis Data Analytics for Apache Flink.
- D. Amazon Kinesis Data Firehose is for direct delivery to destinations, with limited in-flight transformation capabilities, not for complex aggregations. AWS Glue ETL (Batch) is for batch processing, not real-time aggregation.
Kinesis Data Streams + Kinesis Data Analytics (Flink)
A serverless combination for real-time data ingestion and advanced stream processing, enabling complex aggregations, filtering, and analysis on high-volume data streams.
- Kinesis Data Streams: high-throughput, low-latency ingestion.
- Kinesis Data Analytics (Flink): serverless, real-time stream processing.
- Ideal for complex event processing, aggregations, and real-time dashboards.
Memory trick: Kinesis streams, Flink aggregates, Redshift reports.