AWS Certified Data Engineer – AssociateData Ingestion and TransformationEasy
A marketing team needs to collect customer clickstream data from a high-traffic website. The data, in JSON format, needs to be ingested in real-time and delivered to an Amazon S3 bucket for long-term storage and subsequent batch analysis. The solution must be fully managed, highly scalable, and minimize operational overhead. Which AWS service is the MOST appropriate for this ingestion task?
- AAWS IoT Core
- BAmazon Kinesis Data Firehose
- CAmazon Kinesis Data Streams
- DAmazon SQS
Show answer & explanationAnswer & explanation
Correct answer: B. Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose is ideal for direct, fully managed, real-time ingestion of streaming data to various destinations, including Amazon S3. It handles scaling, buffering, and delivery automatically, minimizing operational overhead.
Why the other options are wrong
- A. AWS IoT Core is designed for IoT device connectivity and data ingestion, not for general website clickstream data.
- C. Amazon Kinesis Data Streams provides a real-time data stream that requires consumers (e.g., Lambda, KCL applications) to pull data, which adds operational overhead compared to Firehose for direct S3 delivery.
- D. Amazon SQS is a message queuing service, suitable for decoupling application components, but not optimized for direct high-volume, real-time data ingestion to S3 for analytics.
Amazon Kinesis Data Firehose
A fully managed service for delivering real-time streaming data to destinations like Amazon S3, Amazon Redshift, OpenSearch, and Splunk.
- Fully managed, serverless.
- Automatic scaling, buffering, compression, encryption.
- Direct delivery to S3, Redshift, etc.
Memory trick: Firehose takes the stream and delivers to S3, no fuss.