AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A large manufacturing company needs to ingest operational sensor data from thousands of industrial machines located globally. Each machine generates small (approx. 100-byte) telemetry records every second. The data must be ingested with extremely low latency, processed in real-time to detect anomalies, and then stored in an S3 data lake for historical analysis. The solution needs to be highly scalable to handle millions of records per second and provide strong durability guarantees. Which AWS service should be used for the initial ingestion layer?
- AAmazon SQS
- BAmazon Kinesis Data Firehose
- CAWS IoT Core
- DAmazon Kinesis Data Streams
Show answer & explanationAnswer & explanation
Correct answer: D. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a highly scalable and durable real-time data streaming service capable of ingesting millions of records per second with low latency. Its ability to buffer data for up to a year and integrate with real-time processing services like Kinesis Data Analytics makes it ideal for the ingestion layer of such a critical, high-volume, low-latency pipeline.
Why the other options are wrong
- A. Amazon SQS is a message queuing service, not optimized for high-throughput, real-time streaming ingestion and processing of a continuous stream of records.
- B. Kinesis Data Firehose is primarily for delivering streams to destinations like S3 with minimal transformation. While it's real-time, Kinesis Data Streams offers more control over retention, ordering, and integration with real-time analytics for complex anomaly detection.
- C. AWS IoT Core is for connecting IoT devices and routing messages, but Kinesis Data Streams is the dedicated service for ingesting raw, high-throughput, low-latency data streams for real-time analytics and storage, even if the data originates from IoT devices.
Amazon Kinesis Data Streams
A real-time data streaming service that can continuously capture gigabytes of data per second from thousands of sources.
- Highly scalable for high-throughput data ingestion.
- Provides low-latency data availability.
- Data is immutable and ordered within a shard.
- Integrates with other services for real-time processing and analytics.
Memory trick: Kinesis Streams are like 'digital highways' for fast data.