AWS Certified Solutions Architect – Associate (SAA-C03)Design High-Performing ArchitecturesMedium
A startup is building a real-time analytics dashboard that ingests thousands of events per second from various sources. Each event needs to be processed, transformed, and then stored in a data warehouse for immediate visualization. The solution must be highly scalable, durable, and capable of handling fluctuating ingestion rates without data loss.
- AIngest events using Amazon SQS queues and process with EC2 instances.
- BUse Amazon Kinesis Data Streams to ingest events, followed by AWS Lambda for processing.
- CStore events directly into Amazon S3 and then use AWS Glue to process.
- DWrite events directly to an Amazon Redshift cluster.
Show answer & explanationAnswer & explanation
Correct answer: B. Use Amazon Kinesis Data Streams to ingest events, followed by AWS Lambda for processing.
Amazon Kinesis Data Streams is designed for real-time ingestion of large streams of data records, providing high throughput and durability. Integrating it with AWS Lambda allows for serverless, event-driven processing of each record as it arrives, making it a highly scalable and resilient solution for real-time analytics without managing servers.
Why the other options are wrong
- A. SQS is a message queuing service, not optimized for streaming thousands of events per second in a continuous real-time fashion like Kinesis Data Streams.
- C. Storing directly to S3 and using Glue is suitable for batch processing, not for real-time ingestion and immediate visualization.
- D. Writing directly to Redshift for every single incoming event is inefficient, expensive, and can overload the data warehouse, which is optimized for analytical queries, not high-volume transactional writes.
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a massively scalable and durable real-time data streaming service that continuously captures gigabytes of data per second from hundreds of thousands of sources.
- Enables real-time processing of streaming data.
- Data is available for processing within milliseconds.
- Highly durable and scalable to handle varying throughput.
- Integrates with various AWS services like Lambda, S3, and Redshift.
Memory trick: Kinesis for Continuous, Kafka-like Streams.