AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global e-commerce company wants to centralize customer clickstream data from various regional websites for real-time analytics and personalized recommendations. The data arrives continuously and needs to be transformed (e.g., anonymize PII, enrich with product catalog data) before being loaded into a data lake in Amazon S3 and a data warehouse in Amazon Redshift. The solution must be fully managed and support both real-time ingestion and transformation. Which AWS service combination is most suitable?
- AAmazon Kinesis Data Streams with AWS Glue Streaming ETL
- BAWS DataSync with AWS Glue Batch ETL
- CAmazon MSK with AWS Step Functions
- DAmazon Kinesis Data Firehose with AWS Lambda
Show answer & explanationAnswer & explanation
Correct answer: A. Amazon Kinesis Data Streams with AWS Glue Streaming ETL
Amazon Kinesis Data Streams provides real-time ingestion of continuous data. AWS Glue Streaming ETL can consume data directly from Kinesis Data Streams, perform real-time transformations (anonymization, enrichment), and then deliver the processed data to S3 and Redshift. This combination offers a fully managed, scalable, and real-time solution.
Why the other options are wrong
- B. AWS DataSync is for batch file transfers, not real-time streaming. AWS Glue Batch ETL is for periodic batch processing, not real-time streaming transformations.
- C. Amazon MSK (Managed Streaming for Apache Kafka) is a good ingestion service but requires managing Kafka clusters. AWS Step Functions is an orchestration service, not an ETL engine for streaming data transformation.
- D. Kinesis Data Firehose has limited transformation capabilities (Lambda for basic record transformations), and complex enrichment is better handled by a dedicated ETL engine like Glue Streaming ETL. Lambda can also be less cost-effective for continuous high-volume streaming transformations compared to Glue Streaming ETL.
Kinesis Data Streams + Glue Streaming ETL
A powerful combination for real-time data ingestion and serverless, continuous transformation of streaming data, enabling delivery to various AWS destinations.
- Kinesis Data Streams for real-time ingestion
- Glue Streaming ETL for serverless, continuous transformations
- Supports multiple destinations like S3 and Redshift
Memory trick: Kinesis streams flow, Glue transforms, data glows.