AWS Certified Machine Learning – SpecialtyData EngineeringMedium
A data engineer is designing a data ingestion pipeline for real-time sensor data from thousands of IoT devices. Each device sends small JSON payloads every few seconds. The data needs to be processed, transformed, and then stored in a data lake for analytics and machine learning. The solution must be highly scalable, serverless, and cost-effective. Which combination of AWS services should be used for this pipeline?
- AAWS IoT Core -> Amazon Kinesis Data Firehose -> Amazon S3
- BAmazon Kinesis Data Streams -> AWS Lambda -> Amazon S3
- CAmazon MSK -> AWS Glue Streaming ETL -> Amazon Redshift
- DAmazon SQS -> AWS Batch -> Amazon DynamoDB
Show answer & explanationAnswer & explanation
Correct answer: A. AWS IoT Core -> Amazon Kinesis Data Firehose -> Amazon S3
AWS IoT Core is designed for connecting IoT devices and ingesting their data. Kinesis Data Firehose is a serverless service for delivering streaming data to various destinations, including S3, with built-in buffering, compression, and basic transformations. This combination provides a scalable, serverless, and cost-effective solution for real-time IoT data ingestion to a data lake.
Why the other options are wrong
- B. Kinesis Data Streams requires managing shards and can be more complex for simple ingestion to S3 compared to Firehose. Lambda would be needed for transformation, adding complexity.
- C. Amazon MSK (Managed Streaming for Apache Kafka) is a good streaming solution but requires more management than Firehose. AWS Glue Streaming ETL can transform data, but Firehose can handle basic transformations directly. Amazon Redshift is a data warehouse, not a raw data lake storage.
- D. Amazon SQS is a message queuing service, not optimized for real-time streaming ingestion from thousands of devices. AWS Batch is for batch processing, not real-time, and DynamoDB is a NoSQL database, not typically a data lake storage.
Real-time IoT Data Pipeline
A system for ingesting, processing, and storing high-volume, low-latency data from IoT devices using serverless AWS services.
- AWS IoT Core for device connectivity and message broker.
- Amazon Kinesis Data Firehose for serverless ingestion, buffering, and delivery.
- Amazon S3 as the scalable and cost-effective data lake storage.
- Supports basic transformations and data format conversions.
Memory trick: IoT Core connects, Firehose streams, S3 stores the data dreams.