AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A data engineering team needs to build a highly available and fault-tolerant pipeline to process customer interaction data from various social media platforms. The data is ingested as raw JSON files into an Amazon S3 landing zone. They need to perform complex transformations, enrichments, and aggregations using Apache Spark, requiring a highly scalable and resilient processing environment. Which AWS service is the MOST appropriate for implementing this transformation solution?
- AAWS Lambda
- BAWS Glue ETL
- CAmazon EMR with Apache Spark
- DAmazon Athena
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Glue ETL
AWS Glue ETL provides a serverless Apache Spark environment, which is ideal for complex transformations, enrichments, and aggregations on large datasets stored in S3. It is highly available and fault-tolerant by design, aligning with the requirements for a robust data pipeline without managing infrastructure.
Why the other options are wrong
- A. AWS Lambda is a serverless compute service for running short-lived functions. While it can process data, it's not suitable for large-scale, complex Apache Spark-based transformations that require significant memory and processing power over longer durations.
- C. Amazon EMR with Apache Spark can perform these transformations, but it typically requires managing clusters, which goes against the preference for a fully managed, serverless solution that AWS Glue ETL offers out-of-the-box for Spark workloads.
- D. Amazon Athena is an interactive query service for S3 data using SQL. It's not designed for complex programmatic transformations, enrichments, or aggregations using Apache Spark.
AWS Glue ETL (Spark)
A serverless data integration service that provides a fully managed Apache Spark environment for performing complex ETL (Extract, Transform, Load) operations on large datasets.
- Serverless Apache Spark environment.
- Handles complex transformations, enrichments, aggregations.
- Highly available and fault-tolerant by design.
Memory trick: Glue ETL runs Spark, serverless and smart.