AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global IoT company collects sensor data from millions of devices worldwide. The data is generated continuously, with high velocity and varying schemas depending on the device type. The company needs to transform this raw, semi-structured data into a structured format (e.g., JSON to Parquet) and enrich it with device metadata (e.g., location, ownership) stored in a DynamoDB table. The transformation must occur near real-time, be serverless, and automatically adapt to schema variations without manual intervention. Which AWS service is most appropriate for this real-time, schema-aware transformation?
- AAWS Lambda with custom Python scripts
- BAWS Glue Streaming ETL
- CAmazon Kinesis Data Firehose with built-in transformations
- DAmazon EMR with Apache Spark Streaming
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Glue Streaming ETL
AWS Glue Streaming ETL is a serverless, Apache Spark-based service designed for continuous data transformation. It can automatically infer schema from streaming data, handle schema evolution, perform complex enrichments (e.g., with DynamoDB lookups), and convert data to optimized formats like Parquet in near real-time, making it ideal for high-velocity, varying-schema IoT data.
Why the other options are wrong
- A. AWS Lambda can perform transformations, but it's typically better suited for event-driven, smaller-scale, or less stateful processing. For continuous, high-volume streaming with schema inference and evolution, Glue Streaming ETL is more robust and cost-effective.
- C. Kinesis Data Firehose offers basic transformations (e.g., format conversion, Lambda for simple record transformations) but lacks the advanced schema inference, complex enrichment capabilities, and schema evolution handling of Glue Streaming ETL.
- D. Amazon EMR with Apache Spark Streaming can do this, but it's not serverless; it requires managing EMR clusters, which adds operational overhead and cost compared to Glue Streaming ETL.
AWS Glue Streaming ETL for IoT
A serverless, Apache Spark-based service tailored for continuous, real-time transformation, enrichment, and schema-aware processing of high-volume streaming data from sources like IoT devices.
- Serverless and scalable for real-time streams
- Automatic schema inference and evolution handling
- Supports complex transformations and data enrichment
Memory trick: Glue streams IoT data, transforming schema-aware.