AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A gaming company is developing a new mobile game and anticipates millions of concurrent players. The game generates high-velocity, low-latency telemetry data (player actions, scores, in-game events) that needs to be captured for real-time leaderboards, fraud detection, and analytics. The data needs to be available for processing within seconds of being generated. Which AWS service is best suited for ingesting this real-time event data?
- AAmazon DynamoDB
- BAmazon Kinesis Data Firehose
- CAmazon Kinesis Data Streams
- DAmazon SQS
Show answer & explanationAnswer & explanation
Correct answer: C. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is designed for continuous ingestion and storage of high-throughput, low-latency data streams. It ensures data ordering within shards and provides data availability within seconds, making it ideal for real-time applications like leaderboards, fraud detection, and analytics where immediate processing of individual records is crucial.
Why the other options are wrong
- A. Amazon DynamoDB is a NoSQL database, suitable for storing and retrieving individual key-value pairs with low latency, but not for ingesting and fanning out a continuous stream of event data for real-time analytics.
- B. Kinesis Data Firehose is primarily for simplified delivery of streaming data to destinations like S3 or Redshift, often with buffering. While real-time, Kinesis Data Streams offers more control over stream processing, retention, and is better suited when multiple consumers need to process the stream in real-time for different purposes (leaderboards, fraud, analytics).
- D. Amazon SQS is a message queuing service, not optimized for the continuous ingestion and real-time processing requirements of a high-velocity data stream with ordered records.
Amazon Kinesis Data Streams
A real-time data streaming service that can continuously capture gigabytes of data per second from thousands of sources.
- Highly scalable for high-throughput data ingestion.
- Provides low-latency data availability (seconds).
- Data is immutable and ordered within a shard.
- Supports multiple consumers for diverse real-time applications.
Memory trick: Kinesis Streams are the 'Nerve Center' for game events.