AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global IoT company collects sensor data from millions of devices worldwide. The data is high-volume, low-latency, and needs to be ingested into AWS for real-time processing and durable storage. The company has a strict requirement for ordered delivery of events within each device's stream and needs to replay historical data for backfilling and reprocessing when new analytics models are deployed. Which AWS service is best suited to ingest and store this raw stream data?
- AAmazon Kinesis Data Streams
- BAmazon SQS
- CAmazon Kinesis Data Firehose
- DAWS IoT Core Rules Engine
Show answer & explanationAnswer & explanation
Correct answer: A. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is designed for high-volume, real-time streaming data ingestion. It guarantees ordered delivery of records within a shard, and its durable storage allows for data retention (up to 365 days) enabling reprocessing and backfilling. This fits the requirements for ordered delivery and replayability for IoT sensor data.
Why the other options are wrong
- B. SQS is a message queuing service, not optimized for high-volume, ordered, and replayable stream ingestion.
- C. Kinesis Data Firehose is for delivering streams to destinations, not for retaining data for reprocessing or guaranteeing strict order within a stream for multiple consumers.
- D. AWS IoT Core Rules Engine is for processing and routing IoT messages, but Kinesis Data Streams is the underlying service for durable, ordered stream storage and replay for analytics.
Amazon Kinesis Data Streams
A massively scalable and durable real-time data streaming service capable of continuously capturing gigabytes of data per second from hundreds of thousands of sources.
- Guarantees ordered delivery of records within a shard.
- Data is stored for up to 365 days, enabling reprocessing.
- Multiple consumers can read from the same stream concurrently.
Memory trick: Kinesis Streams keeps order and lets you rewind the data movie.