AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global media company receives large video files (up to 50 GB each) from content creators worldwide. These files often arrive via SFTP. The company needs to automatically process these files by extracting metadata, transcoding them into multiple formats, and storing the processed versions in Amazon S3. Which combination of AWS services provides a robust, scalable, and serverless solution for this workflow?
- AAWS DataSync + AWS Glue ETL
- BAWS Storage Gateway (File Gateway) + AWS Batch + Amazon S3
- CAmazon Kinesis Data Firehose + AWS Lambda + Amazon S3
- DAWS Transfer Family (SFTP) + AWS Lambda + Amazon S3 + AWS Step Functions
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Transfer Family (SFTP) + AWS Lambda + Amazon S3 + AWS Step Functions
AWS Transfer Family for SFTP provides a fully managed SFTP endpoint for receiving files. An S3 event notification (triggered by file upload) can invoke an AWS Lambda function to extract metadata and initiate transcoding. AWS Step Functions can orchestrate the complex transcoding workflow, ensuring reliability and retries, with processed files stored in S3.
Why the other options are wrong
- A. DataSync is for bulk data transfer/sync; it doesn't provide an SFTP endpoint for external users or workflow orchestration.
- B. Storage Gateway is for connecting on-premises storage to S3; it doesn't provide an SFTP endpoint for external content creators or workflow orchestration for transcoding.
- C. Kinesis Data Firehose is for streaming data, not large individual files via SFTP.
Serverless SFTP to S3 ETL Pipeline
A serverless architecture for ingesting files via SFTP, triggering automated processing, and storing results in S3 without managing servers.
- Uses AWS Transfer Family for SFTP endpoint.
- Leverages S3 event notifications for triggering processing.
- AWS Lambda for serverless compute and metadata extraction.
- AWS Step Functions for workflow orchestration and error handling.
Memory trick: SFTP needs 'Transfer', files in 'S3' trigger 'Lambda', and 'Steps' orchestrate the 'Functions'.