AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global e-commerce company wants to centralize customer clickstream data from various regional websites. The data arrives as JSON objects, approximately 1KB each, with a peak ingestion rate of 50,000 records per second. The company needs to transform this data by filtering out bot traffic and enriching it with customer demographic information from a DynamoDB table before storing it in an Amazon S3 data lake in Parquet format. The solution must provide near real-time processing with low latency. Which combination of AWS services should be used?
- AAWS IoT Core and AWS Lambda
- BAmazon Kinesis Data Streams and AWS Glue Streaming ETL
- CAmazon Kinesis Data Firehose with built-in data transformation
- DAmazon Kinesis Data Streams and AWS Glue batch ETL
Show answer & explanationAnswer & explanation
Correct answer: B. Amazon Kinesis Data Streams and AWS Glue Streaming ETL
Amazon Kinesis Data Streams can handle the high ingestion rate of 50,000 records/second with low latency. AWS Glue Streaming ETL is ideal for near real-time transformations like filtering and enrichment, and it can write the processed data to S3 in Parquet format. This combination provides a robust, scalable, and low-latency solution for the specified requirements.
Why the other options are wrong
- A. AWS IoT Core is for IoT device connectivity and management, not general clickstream ingestion. While Lambda could perform transformations, managing the streaming pipeline and state for 50,000 records/second with complex enrichment using just Lambda would be significantly more complex and resource-intensive than Glue Streaming ETL.
- C. Kinesis Data Firehose has limited built-in transformation capabilities (e.g., format conversion, simple record transformation with Lambda) and might not be sufficient for complex filtering and enrichment from an external source like DynamoDB. Its maximum record size is 1MB and it's append-only.
- D. AWS Glue batch ETL is not suitable for near real-time processing with low latency requirements. Kinesis Data Streams is appropriate for ingestion but batch ETL is the bottleneck.
Kinesis Data Streams + Glue Streaming ETL
A powerful combination for real-time data ingestion and complex, serverless streaming transformations, delivering processed data to various destinations like S3.
- Kinesis Data Streams handles high-throughput, low-latency ingestion.
- AWS Glue Streaming ETL performs complex, stateful transformations.
- Supports various data formats (e.g., Parquet) for output.
- Serverless and scalable for processing large volumes of streaming data.
Memory trick: Kinesis Streams the data, Glue Transforms it, S3 stores it.