AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A financial services company needs to process large batches of credit card transaction data (hundreds of gigabytes per batch) daily. These batches arrive as CSV files in an S3 bucket. The data must be validated, transformed (e.g., PII masking, currency conversion), and then aggregated before being loaded into a data warehouse for fraud detection and regulatory reporting. The solution needs to be robust, scalable, and support custom Python code for complex transformation logic. Which AWS service is most appropriate for this batch transformation process?
- AAmazon Redshift Spectrum
- BAWS Lambda
- CAmazon Kinesis Data Analytics
- DAWS Glue ETL
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Glue ETL
AWS Glue ETL is a serverless data integration service that uses Apache Spark for robust, scalable batch processing. It can read large CSV files from S3, execute custom Python (PySpark) code for complex transformations like PII masking and aggregation, and then load the processed data into a data warehouse, perfectly matching the requirements.
Why the other options are wrong
- A. Amazon Redshift Spectrum allows querying data in S3 using Redshift, but it's a query service, not an ETL service for complex transformations and loading aggregated data.
- B. AWS Lambda has execution limits (memory, duration) that make it unsuitable for processing hundreds of gigabytes of data per batch.
- C. Amazon Kinesis Data Analytics is for real-time stream processing, not for daily batch transformations of hundreds of gigabytes.
AWS Glue ETL
A serverless data integration service that makes it easy to discover, prepare, move, and combine data for analytics, machine learning, and application development.
- Fully managed, serverless, and scales dynamically using Apache Spark.
- Supports Python (PySpark) and Scala for custom transformation logic.
- Integrates with AWS Glue Data Catalog for metadata management.
Memory trick: Glue: The powerful, custom-scripted solution for your batch data.