AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global media company needs to process large volumes of video metadata (XML files) generated hourly by various content partners. These files are typically 10-50 MB each, and arrive via SFTP. The company requires a fully managed and scalable solution to ingest these files into Amazon S3, transform them into a structured format (Parquet), and store them in a data lake for analytics. What is the most appropriate architecture for this ingestion and transformation pipeline?
- AUse an on-premises SFTP server to push files to S3, then use Amazon EMR for transformation.
- BUse Amazon SQS for file arrival notifications, then AWS Lambda for transformation and S3 storage.
- CUse AWS DataSync to transfer files to S3, then Amazon Kinesis Data Analytics for processing.
- DUse AWS Transfer Family (SFTP) to ingest files to S3, then AWS Glue ETL to transform and store in S3.
Show answer & explanationAnswer & explanation
Correct answer: D. Use AWS Transfer Family (SFTP) to ingest files to S3, then AWS Glue ETL to transform and store in S3.
AWS Transfer Family (SFTP) provides a fully managed SFTP endpoint that directly writes incoming files to S3. AWS Glue ETL is a serverless data integration service that can read from S3, perform transformations (like XML to Parquet), and write the results back to S3 for the data lake.
Why the other options are wrong
- A. Using an on-premises SFTP server introduces operational overhead. Amazon EMR is powerful but might be overkill for this scenario and less cost-effective than Glue for scheduled batch transformations.
- B. Amazon SQS can notify of file arrivals but doesn't handle SFTP ingestion. AWS Lambda can transform files, but for 10-50MB XML files hourly, Glue ETL is generally more robust and better suited for larger data volumes and complex transformations.
- C. AWS DataSync is for bulk data transfers, not a managed SFTP endpoint. Kinesis Data Analytics is for streaming data, not batch processing of files that arrive hourly.
AWS Transfer Family + AWS Glue ETL
A common pattern for ingesting files via standard protocols into S3, followed by serverless ETL processing to transform and prepare data for analytics.
- AWS Transfer Family: fully managed SFTP/FTP/FTPS endpoints to S3.
- AWS Glue ETL: serverless Spark-based ETL for data transformation.
- Ideal for external partner data ingestion and data lake population.
Memory trick: Transfer files to S3, Glue transforms with glee.