AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A gaming company collects telemetry data from millions of players, generating petabytes of event data daily. This data is ingested into Amazon Kinesis Data Streams. The company needs to perform real-time aggregations (e.g., calculating average session duration per game, total active players) and detect anomalies in player behavior. The results of these aggregations and anomaly detections should be continuously published to a dashboard for operational monitoring. Which AWS service is best suited for this real-time stream processing and analytics?
- AAmazon EMR with Apache Spark Streaming
- BAmazon Kinesis Data Analytics for Apache Flink
- CAWS Glue streaming ETL
- DAmazon Kinesis Data Firehose
Show answer & explanationAnswer & explanation
Correct answer: B. Amazon Kinesis Data Analytics for Apache Flink
Amazon Kinesis Data Analytics for Apache Flink is a fully managed service for real-time processing of streaming data. It supports complex operations like aggregations, windowing, and anomaly detection using powerful Apache Flink applications. Being fully managed, it handles scaling and operational overhead, making it ideal for continuous, low-latency analytics on Kinesis Data Streams for dashboarding.
Why the other options are wrong
- A. EMR with Spark Streaming requires cluster management and is less serverless and potentially more costly for continuous, low-latency stream analytics compared to Kinesis Data Analytics for Flink.
- C. AWS Glue streaming ETL can perform transformations but is typically more suited for ETL tasks (e.g., joining, cleaning) before landing data, and Flink offers more powerful stream processing primitives for analytics.
- D. Kinesis Data Firehose is for delivering raw streams to destinations, not for complex real-time aggregations and anomaly detection.
Amazon Kinesis Data Analytics for Apache Flink
A fully managed service that allows you to process and analyze streaming data in real time using Apache Flink.
- Supports complex stream processing, aggregations, and windowing.
- Automatically scales to handle varying data throughput.
- Integrates with Kinesis Data Streams and Firehose for input/output.
Memory trick: Flink analyzes streams fast, finding patterns and anomalies.