AWS Certified Machine Learning – SpecialtyData EngineeringHard
A data engineer is designing a data ingestion pipeline for an IoT application that generates millions of small sensor readings per second. These readings need to be processed in near real-time for anomaly detection and then archived to Amazon S3 for long-term analytics. The solution must be highly scalable, fault-tolerant, and cost-effective. Which AWS service combination is best suited for this scenario?
- AAWS DataSync to transfer data from IoT devices to S3, followed by AWS Batch for processing.
- BAmazon Kinesis Data Firehose for direct ingestion to S3, and Amazon Athena for querying the raw data.
- CAmazon Kinesis Data Streams for ingestion, AWS Lambda for real-time processing, and Amazon Kinesis Data Firehose for S3 archiving.
- DAWS IoT Core for device connectivity, Amazon SQS for message queuing, and Amazon EC2 instances for processing.
Show answer & explanationAnswer & explanation
Correct answer: C. Amazon Kinesis Data Streams for ingestion, AWS Lambda for real-time processing, and Amazon Kinesis Data Firehose for S3 archiving.
Kinesis Data Streams is ideal for ingesting millions of records per second and provides near real-time processing capabilities. AWS Lambda offers serverless, scalable real-time processing of stream data. Kinesis Data Firehose can then efficiently deliver the processed data to S3 for archiving and long-term analytics, handling batching and compression for cost-effectiveness.
Why the other options are wrong
- A. AWS DataSync is for large-scale data transfers, not real-time streaming ingestion from IoT devices. AWS Batch is for batch processing, not near real-time.
- B. Kinesis Data Firehose is good for direct-to-S3 delivery but lacks the real-time processing capabilities needed for anomaly detection before archiving. Athena is for querying, not ingestion or real-time processing.
- D. While AWS IoT Core handles device connectivity, SQS is not designed for high-throughput real-time stream processing, and managing EC2 instances for this scale adds operational overhead and cost.
Real-time IoT Data Pipeline
An architecture for ingesting, processing, and storing high-volume, low-latency data from IoT devices using AWS managed services.
- Kinesis Data Streams for scalable, real-time ingestion.
- AWS Lambda for serverless, event-driven real-time processing.
- Kinesis Data Firehose for efficient, managed delivery to S3 for archiving.
Memory trick: IoT Streams flow to Lambda, then Firehose to S3 storage!