AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A media company needs to process large volumes of video files (averaging 500 MB to 5 GB each) uploaded daily by content creators to an Amazon S3 bucket. The processing involves extracting metadata, generating thumbnails, and transcoding the videos into multiple formats. This is a batch process triggered by new file uploads. The solution must be highly scalable, cost-effective, and provide robust error handling and retry mechanisms. Which AWS service is most suitable for orchestrating and executing these transformations?
- AAWS Step Functions with AWS Lambda and Amazon Elastic Transcoder
- BAmazon Kinesis Data Analytics for Apache Flink
- CAmazon EMR with Apache Spark
- DAWS Glue Streaming ETL
Show answer & explanationAnswer & explanation
Correct answer: A. AWS Step Functions with AWS Lambda and Amazon Elastic Transcoder
AWS Step Functions provides a serverless workflow orchestrator, allowing you to define complex, stateful workflows. It can trigger AWS Lambda functions for metadata extraction and thumbnail generation, and Amazon Elastic Transcoder for video transcoding, all in response to S3 object creation events, with built-in error handling and retries.
Why the other options are wrong
- B. Kinesis Data Analytics is for real-time streaming data processing, not batch processing of large video files.
- C. Amazon EMR with Apache Spark can perform large-scale data processing but is typically used for general-purpose big data analytics, and can be more complex to set up and manage for a specific event-driven media workflow than Step Functions.
- D. AWS Glue Streaming ETL is for continuous data streams and typically used for structured or semi-structured data, not for orchestrating multi-step media processing workflows.
AWS Step Functions + Lambda + Elastic Transcoder
A serverless architecture for orchestrating complex, multi-step workflows, combining Lambda for custom logic and Elastic Transcoder for media processing.
- Step Functions: Visual workflow orchestration, state management, error handling.
- Lambda: Serverless compute for event-driven functions.
- Elastic Transcoder: Cloud-based media transcoding service.
Memory trick: Steps orchestrate media, Lambda and Transcoder free.