AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global gaming company collects player interaction data (e.g., achievements, in-game purchases, chat messages) from its mobile game. This data is generated by millions of players concurrently, resulting in high-volume, low-latency streams. The company needs to perform real-time analytics on this data to detect fraudulent activities and provide personalized player experiences. The solution must be fully managed, scalable, and support SQL-based queries on the incoming streams. Which combination of AWS services is the MOST appropriate for this real-time analytics use case?
- AAWS Glue Streaming ETL to Amazon S3 and Amazon Athena
- BAmazon DynamoDB Streams to AWS Lambda for processing
- CAmazon Kinesis Data Firehose to Amazon Redshift
- DAmazon Kinesis Data Streams to Amazon Kinesis Data Analytics for Apache Flink
Show answer & explanationAnswer & explanation
Correct answer: D. Amazon Kinesis Data Streams to Amazon Kinesis Data Analytics for Apache Flink
Amazon Kinesis Data Streams provides the necessary high-throughput, low-latency ingestion for millions of concurrent players. Amazon Kinesis Data Analytics for Apache Flink is a fully managed service that allows running SQL queries or Apache Flink applications directly on streaming data, enabling real-time analytics for fraud detection and personalization without managing servers.
Why the other options are wrong
- A. AWS Glue Streaming ETL is for complex data transformations on streams, and Athena is for querying S3 data. This setup would introduce more latency and is not primarily designed for direct SQL querying of live streams.
- B. DynamoDB Streams is for capturing changes to DynamoDB tables. While Lambda can process these, it's not designed for high-volume, generic player interaction streams for SQL-based real-time analytics.
- C. Kinesis Data Firehose is for delivery to destinations, and Redshift is a data warehouse. This combination is less suitable for true real-time, SQL-based stream analytics directly on incoming data.
Kinesis Data Streams + Kinesis Data Analytics (Flink)
A powerful combination for real-time data ingestion and immediate SQL-based or Apache Flink-based analytics on high-volume, low-latency streaming data.
- Kinesis Data Streams for scalable, real-time ingestion
- Kinesis Data Analytics for serverless, real-time processing
- Supports SQL queries directly on streaming data
- Enables immediate insights for fraud detection, personalization, etc.
Memory trick: Kinesis Streams the data, and Kinesis Analytics instantly analyzes it like a super-fast brain.