AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global e-commerce company needs to process customer orders in near real-time. Each order record, averaging 1 KB in size, needs to be ingested, transformed to a standardized format, and then loaded into a data warehouse for immediate business intelligence reporting. The company anticipates peak traffic of 5,000 orders per second. Which AWS data ingestion and transformation services should be combined to meet these requirements most efficiently?
- AAmazon S3 for ingestion, followed by Amazon EMR with Apache Spark for transformation.
- BAWS DataSync for ingestion, followed by AWS Glue ETL (batch processing) for transformation.
- CAmazon Kinesis Data Streams for ingestion, followed by AWS Glue Streaming ETL for transformation.
- DAWS Transfer Family (SFTP) for ingestion, followed by Amazon Athena for transformation.
Show answer & explanationAnswer & explanation
Correct answer: C. Amazon Kinesis Data Streams for ingestion, followed by AWS Glue Streaming ETL for transformation.
Amazon Kinesis Data Streams is ideal for ingesting high-throughput, real-time data like customer orders. AWS Glue Streaming ETL is designed to continuously process and transform data from streaming sources like Kinesis Data Streams, making this combination efficient for near real-time requirements.
Why the other options are wrong
- A. Amazon S3 is an object storage service, not a real-time ingestion service for high-velocity data. Amazon EMR for batch processing would introduce latency not suitable for near real-time.
- B. AWS DataSync is for large-scale file transfers, not real-time streaming. AWS Glue batch processing would not meet near real-time needs.
- D. AWS Transfer Family is for file transfer protocols, not real-time streaming. Amazon Athena is a query service, not a transformation engine for streaming data.
Kinesis Data Streams + Glue Streaming ETL
A powerful combination for building real-time data pipelines, where Kinesis Data Streams ingests high-velocity data, and Glue Streaming ETL continuously processes and transforms it.
- Kinesis Data Streams provides real-time, scalable data ingestion.
- AWS Glue Streaming ETL processes data continuously from streaming sources.
- Ideal for low-latency data processing and analytics.
- Supports various data formats and transformations.
Memory trick: Kinesis Streams the data, Glue Transforms it Live, for Instant Insights.