AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global IoT company collects sensor data from millions of devices worldwide. The data is small, JSON-formatted, and arrives continuously at a high velocity. The company needs to ingest this data, apply immediate filtering (e.g., remove malformed records), and route valid records to an Amazon S3 data lake for long-term storage and a separate Amazon Kinesis Data Stream for real-time anomaly detection. The solution must be fully managed, scalable, and minimize operational overhead. Which AWS service combination should be used for this ingestion and initial routing?
- AAWS IoT Core with IoT Rules Engine
- BAmazon Kinesis Data Firehose with AWS Lambda transformation
- CAWS IoT Core and AWS Lambda
- DAmazon Kinesis Data Streams and AWS Glue Streaming ETL
Show answer & explanationAnswer & explanation
Correct answer: A. AWS IoT Core with IoT Rules Engine
AWS IoT Core is specifically designed for connecting IoT devices and routing their messages. Its IoT Rules Engine can ingest high-velocity data, apply immediate filtering based on message content, and then route the filtered data to multiple destinations simultaneously, such as an S3 data lake and an Amazon Kinesis Data Stream, all in a fully managed and scalable manner.
Why the other options are wrong
- B. Kinesis Data Firehose is for delivering streams to destinations, but it's not explicitly designed for IoT device connectivity and its built-in filtering/routing capabilities are less powerful and flexible for IoT messages than the IoT Rules Engine when combining multiple destinations and conditions.
- C. While IoT Core can ingest, and Lambda can process, relying solely on Lambda for filtering and routing high-velocity streams to multiple destinations adds significant operational complexity compared to the declarative nature of the IoT Rules Engine.
- D. Kinesis Data Streams is an ingestion layer, and Glue Streaming ETL is for complex transformations. While they could process IoT data, IoT Core is the specialized service for managing device connectivity and initial routing/filtering of IoT messages.
AWS IoT Core with IoT Rules Engine
A fully managed service that enables connected devices to easily and securely interact with cloud applications and other devices, with a Rules Engine to filter, transform, and route messages.
- Connects millions of IoT devices securely.
- IoT Rules Engine allows filtering, transforming, and routing messages.
- Supports multiple destinations (S3, Kinesis, Lambda, DynamoDB, etc.).
- Fully managed and highly scalable for high-velocity IoT data.
Memory trick: IoT Core is the 'Traffic Cop' for device data.