AWS Certified Solutions Architect – Associate (SAA-C03)Design High-Performing ArchitecturesHard
A data engineering team is building a pipeline to ingest continuously flowing clickstream data from a website. The data needs to be processed in real-time for immediate analytics and then stored durably for long-term historical analysis. The volume of data can fluctuate significantly, reaching terabytes per hour during peak times. Which AWS service is best suited for ingesting and processing this real-time stream at scale?
- AAmazon Kinesis Data Streams
- BAWS Batch
- CAmazon SQS Standard
- DAmazon Kinesis Data Firehose
Show answer & explanationAnswer & explanation
Correct answer: A. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is designed for real-time processing of streaming data at massive scale. It allows for custom processing of records as they arrive (e.g., with Lambda or Kinesis Data Analytics) before being stored. Given the 'continuously flowing', 'real-time analytics', and 'terabytes per hour' requirements, Kinesis Data Streams provides the necessary throughput and real-time processing capabilities.
Why the other options are wrong
- B. AWS Batch is for running batch computing workloads, not for real-time ingestion and processing of continuously flowing data.
- C. Amazon SQS Standard is a message queue, suitable for decoupling, but not optimized for continuous, high-throughput streaming data ingestion for real-time processing and fan-out to multiple consumers.
- D. Amazon Kinesis Data Firehose is primarily for delivering streaming data to destinations like S3, Redshift, or Splunk, with optional transformations. While it ingests data, Kinesis Data Streams offers more control and flexibility for custom real-time processing logic before delivery.
Amazon Kinesis Data Streams
A real-time data streaming service that enables you to build custom applications to process or analyze streaming data for various use cases.
- Ingests and stores data records for up to 365 days.
- Allows multiple consumers to process the same stream concurrently.
- Provides ordering of records within a shard.
- Scales by adjusting the number of shards.
Memory trick: Real-time flow, Kinesis will go, Streams for code, Firehose's load.