AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global logistics company needs to track the real-time location of its delivery vehicles. Each vehicle sends GPS coordinates and status updates every 5 seconds. The data volume is expected to be extremely high, potentially millions of messages per second during peak times, with each message being small (around 200 bytes). The company needs to store this data for at least 7 days for immediate operational monitoring and analysis, and then archive it in Amazon S3 for long-term historical analysis. The solution must be highly scalable, durable, and provide ordered delivery of records within a partition. Which AWS service is most appropriate for ingesting this real-time, high-volume data stream?
- AAmazon Kinesis Data Firehose
- BAmazon Kinesis Data Streams
- CAWS IoT Core
- DAmazon SQS (Standard Queue)
Show answer & explanationAnswer & explanation
Correct answer: B. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is designed for real-time ingestion of high-throughput data streams, offering high scalability, durability, and ordered delivery of records within a shard. It provides data retention for up to a year, making it suitable for both immediate operational monitoring and subsequent archival to S3.
Why the other options are wrong
- A. Kinesis Data Firehose is for delivering streaming data to destinations, but Kinesis Data Streams provides more granular control over the stream, including ordered delivery within a shard and longer retention for direct consumption by analytical applications.
- C. AWS IoT Core is for connecting IoT devices and applying rules, but for generic high-volume, ordered real-time stream ingestion and analysis, Kinesis Data Streams is the more direct and powerful service.
- D. Amazon SQS is a message queuing service, not optimized for high-throughput, ordered stream processing required for real-time analytics at this scale.
Amazon Kinesis Data Streams
A highly scalable and durable real-time data streaming service that continuously captures gigabytes of data per second from hundreds of thousands of sources.
- High throughput and low latency data ingestion.
- Ordered delivery of records within a shard.
- Data retention up to 365 days, ideal for real-time and near real-time processing.
Memory trick: Kinesis Streams, data gleams, in real-time dreams.