AWS Certified Data Engineer – AssociateData Ingestion and TransformationEasy
A marketing analytics team needs to combine customer interaction data from various sources, including CRM systems (PostgreSQL), website logs (S3), and social media feeds (API endpoints). The data needs to be cleaned, de-duplicated, and aggregated daily before being loaded into Amazon Redshift for reporting. The solution requires a serverless approach for cost efficiency and ease of management. Which AWS service is best suited for building this daily batch transformation pipeline?
- AAWS Step Functions with Lambda
- BAWS Glue ETL
- CAmazon Kinesis Data Analytics
- DAmazon Kinesis Data Firehose
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Glue ETL
AWS Glue ETL is a fully managed, serverless ETL service that is ideal for batch data processing. It can connect to various data sources (S3, PostgreSQL), perform complex transformations like cleaning, de-duplication, and aggregation using Apache Spark, and then load the processed data into Amazon Redshift. Its serverless nature aligns with the cost-efficiency and ease of management requirements.
Why the other options are wrong
- A. AWS Step Functions orchestrates workflows, but Lambda functions are generally not suitable for large-scale, long-running data transformations that require Spark's power.
- C. Amazon Kinesis Data Analytics is for real-time stream processing, not for daily batch transformations from multiple disparate sources.
- D. Amazon Kinesis Data Firehose is for streaming data delivery, not for complex batch transformations involving multiple sources and de-duplication.
AWS Glue ETL
A serverless data integration service that makes it easy to discover, prepare, move, and combine data for analytics, machine learning, and application development.
- Fully managed, serverless, and scales dynamically.
- Uses Apache Spark for powerful data transformations.
- Integrates with a wide range of AWS data sources and targets.
Memory trick: Glue: Your serverless helper for daily data transformations.