AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A global logistics company receives shipment tracking updates from thousands of partners in various formats (CSV, JSON, XML) via SFTP. These updates arrive continuously throughout the day, but do not require immediate real-time processing. The company needs to consolidate these files into a centralized Amazon S3 bucket, convert them to Parquet format, and then catalog their schemas in the AWS Glue Data Catalog for subsequent analysis. Which combination of AWS services would be most efficient for this ingestion and initial transformation pipeline?
- AAmazon Kinesis Data Streams -> Amazon Kinesis Data Analytics -> Amazon S3
- BAWS DataSync -> Amazon Kinesis Data Firehose -> Amazon S3
- CAWS Transfer Family (SFTP) -> Amazon S3 -> AWS Lambda -> AWS Glue Data Catalog
- DAWS Transfer Family (SFTP) -> AWS Glue ETL -> AWS Glue Data Catalog
Show answer & explanationAnswer & explanation
Correct answer: D. AWS Transfer Family (SFTP) -> AWS Glue ETL -> AWS Glue Data Catalog
AWS Transfer Family (SFTP) is ideal for receiving files via SFTP. Once in S3, AWS Glue ETL can efficiently read files from S3, handle various formats, convert them to Parquet, and then update the AWS Glue Data Catalog with the new schema, fulfilling all requirements for ingestion and initial transformation.
Why the other options are wrong
- A. Kinesis Data Streams and Kinesis Data Analytics are for real-time streaming, which is explicitly stated as 'do not require immediate real-time processing' in the scenario.
- B. DataSync is for bulk transfer, not SFTP ingestion. Kinesis Data Firehose is for streaming delivery, not complex batch transformation like format conversion and cataloging from diverse file types.
- C. While Lambda can be triggered by S3 events, it's not well-suited for large-scale file format conversion and schema cataloging at the scale and complexity implied, which Glue ETL handles more efficiently.
AWS Transfer Family + AWS Glue ETL
A common pattern for ingesting files via standard protocols into S3, followed by serverless batch ETL for transformation and cataloging.
- AWS Transfer Family supports SFTP, FTPS, and FTP for secure file transfer.
- AWS Glue ETL can process diverse file formats in S3 and convert them.
- AWS Glue Data Catalog stores metadata for data in S3, enabling analytics.
Memory trick: Transfer + Glue: Your two-step for file intake and transformation.