AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global logistics company receives shipment tracking updates from thousands of partners in various formats (CSV, JSON, XML) via SFTP. The data needs to be collected, validated, and transformed into a standardized Parquet format for analysis in an S3 data lake. The solution must provide a fully managed SFTP endpoint and integrate seamlessly with serverless ETL services. Which AWS service combination should be used for data ingestion and initial transformation?
- AAmazon S3 Transfer Acceleration for ingestion, Amazon EMR for transformation
- BAWS DataSync for ingestion, AWS Lambda for transformation
- CAmazon Kinesis Data Firehose for ingestion, Amazon Kinesis Data Analytics for transformation
- DAWS Transfer Family (SFTP) for ingestion, AWS Glue ETL for transformation
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Transfer Family (SFTP) for ingestion, AWS Glue ETL for transformation
AWS Transfer Family provides fully managed SFTP endpoints, allowing partners to upload files directly to S3. AWS Glue ETL can then be triggered to read these raw files from S3, perform schema inference, validate, transform them into Parquet format, and write them back to the S3 data lake. This offers a serverless and scalable solution for both ingestion and transformation.
Why the other options are wrong
- A. Amazon S3 Transfer Acceleration speeds up S3 uploads but doesn't provide an SFTP endpoint. Amazon EMR is not serverless and would require cluster management.
- B. AWS DataSync is for large-scale, one-time or periodic transfers, not typically for continuous SFTP ingestion from external partners. Lambda is suitable for small, simple transformations, but complex validation and format conversion of diverse file types is better handled by Glue ETL.
- C. Kinesis Data Firehose and Kinesis Data Analytics are for real-time streaming data, not for file-based SFTP transfers.
AWS Transfer Family + AWS Glue ETL
A common pattern for ingesting files from external partners via SFTP/FTPS/FTP directly to S3, followed by serverless batch ETL using AWS Glue to transform and standardize the data.
- AWS Transfer Family for managed file transfer protocols (SFTP, FTPS, FTP)
- AWS Glue ETL for serverless batch data transformation
- Directly integrates with Amazon S3 for source and target
Memory trick: Transfer Family gets files, Glue transforms them to golden piles.