An analytics startup processes large volumes of streaming data from IoT devices. Their existing data ingestion pipeline uses Apache Kafka on self-managed EC2 instances, and they perform real-time analytics using a custom application that consumes from Kafka. They want to improve the reliability and scalability of their analytics processing, reduce operational overhead, and gain deeper insights from the streaming data without managing underlying infrastructure. Which AWS service should the Solutions Architect recommend for processing this real-time streaming data?
- AMigrate Apache Kafka to Amazon Managed Streaming for Apache Kafka (MSK) and continue using the custom application for analytics.
- BAmazon Kinesis Data Firehose to deliver data to Amazon S3 for batch processing with AWS Glue.
- CAmazon Kinesis Data Streams for data ingestion, followed by Amazon Kinesis Data Analytics for Apache Flink for real-time processing.
- DUse AWS Step Functions to orchestrate data processing jobs on Amazon EMR clusters.
Show answer & explanationAnswer & explanation
Correct answer: C. Amazon Kinesis Data Streams for data ingestion, followed by Amazon Kinesis Data Analytics for Apache Flink for real-time processing.
Amazon Kinesis Data Streams provides a highly scalable and durable data ingestion service for real-time data. Amazon Kinesis Data Analytics for Apache Flink offers a fully managed service for processing streaming data with Apache Flink, reducing operational overhead and enabling real-time insights without managing servers.
Why the other options are wrong
- A. Migrating to MSK reduces Kafka management overhead but still requires managing the custom analytics application, which doesn't fully address the desire for reduced operational overhead for analytics processing or leverage a managed Flink service for deeper insights.
- B. Kinesis Data Firehose is for delivery to destinations like S3, Redshift, or Splunk, typically for near real-time or batch processing, not for complex real-time analytics with Flink directly. AWS Glue is for batch ETL, not real-time streaming analytics.
- D. Amazon EMR is suitable for large-scale batch processing or interactive analytics, but it's not primarily designed for continuous, low-latency real-time streaming analytics as effectively as Kinesis Data Analytics for Apache Flink, and managing EMR clusters still involves operational overhead.
Kinesis Data Streams + Kinesis Data Analytics for Flink
Amazon Kinesis Data Streams is a highly scalable, durable real-time data streaming service. Amazon Kinesis Data Analytics for Apache Flink is a fully managed service that allows you to easily process and analyze streaming data in real time using Apache Flink.
- Kinesis Data Streams ingests data at high throughput.
- Kinesis Data Analytics for Flink provides serverless real-time processing.
- Reduces operational overhead and scales automatically.
Memory trick: Kinesis Keeps Kicking Knowledge.