AWS Certified Data Engineer – AssociateData Storage and ManagementHard
A data engineer is designing a data processing pipeline that involves ingesting large volumes of streaming data (e.g., IoT sensor readings, clickstreams) from various sources. This data needs to be temporarily stored, processed in real-time, and then loaded into a data lake for further analysis. The solution must be highly scalable, durable, and capable of handling fluctuating data ingestion rates without data loss. Which AWS service is best suited for reliably collecting and buffering this streaming data?
- AAmazon SQS
- BAWS Step Functions
- CAmazon S3
- DAmazon Kinesis Data Streams
Show answer & explanationAnswer & explanation
Correct answer: D. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a fully managed service for real-time processing of large streams of data. It is specifically designed to handle high-throughput data ingestion, buffer data for a configurable retention period (up to 365 days), and allow multiple consumers to process the data concurrently, making it ideal for the described streaming data collection and buffering requirements.
Why the other options are wrong
- A. Amazon SQS is a message queuing service, suitable for decoupling microservices, but not optimized for high-throughput, real-time streaming data ingestion and multi-consumer access patterns like Kinesis Data Streams.
- B. AWS Step Functions is a serverless workflow orchestrator, not a service for collecting and buffering streaming data.
- C. Amazon S3 is object storage, not a streaming ingestion service. While data can be written to S3, it doesn't provide real-time buffering or stream processing capabilities.
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a massively scalable, durable, real-time data streaming service that continuously captures gigabytes of data per second from hundreds of thousands of sources.
- Real-time data ingestion and processing.
- Scalable to handle high-throughput data streams.
- Data retention up to 365 days.
- Supports multiple consumers reading from the same stream.
- Used for IoT, clickstreams, log aggregation, and real-time analytics.
Memory trick: Kinesis streams, for real-time dreams, no data's lost, it always gleams.