AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A manufacturing company operates thousands of IoT sensors on its factory floor, generating high-volume, low-latency telemetry data (e.g., temperature, pressure, vibration) at a rate of millions of events per second. The company needs to ingest this data, apply simple filtering and aggregation in real-time, and then route the processed data to both an S3 data lake for historical analysis and Amazon DynamoDB for real-time dashboards. Which AWS service should be used for the ingestion and real-time processing of this IoT data?
- AAWS IoT Core with IoT Rules Engine
- BAWS Glue Streaming ETL
- CAmazon Kinesis Data Streams
- DAmazon Kinesis Data Firehose
Show answer & explanationAnswer & explanation
Correct answer: A. AWS IoT Core with IoT Rules Engine
AWS IoT Core is designed specifically for connecting IoT devices and ingesting their data. Its Rules Engine provides real-time filtering, transformation, and routing capabilities to multiple AWS services like S3 and DynamoDB, making it the most direct and efficient solution for this IoT-specific scenario.
Why the other options are wrong
- B. AWS Glue Streaming ETL is excellent for continuous, serverless data transformation from streaming sources, but it typically consumes from Kinesis Data Streams or Kafka, not directly from IoT devices. AWS IoT Core handles the initial ingestion and routing more directly for IoT specific use cases.
- C. Amazon Kinesis Data Streams is a general-purpose real-time streaming service. While it can ingest high-volume data, AWS IoT Core is purpose-built for IoT and offers more direct integration and features for device management and rules-based routing of IoT data.
- D. Amazon Kinesis Data Firehose can ingest streaming data and deliver it to S3, Redshift, etc., but it's not specifically designed for IoT device connectivity or the rich real-time rules engine capabilities of AWS IoT Core.
AWS IoT Core with IoT Rules Engine
A managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices, with a Rules Engine for real-time processing and routing of device data.
- Securely connect millions of IoT devices.
- Ingest high-volume device telemetry.
- Rules Engine for real-time filtering, transformation, and routing to AWS services.
Memory trick: IoT Core takes device data, rules route it far.