AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium

A media company needs to process large volumes of video metadata (XML files) generated hourly from its content management system. These files are stored in an on-premises SFTP server. The company wants to automate the ingestion of these files into Amazon S3, parse the XML, extract specific metadata fields, and store the extracted data in a structured format (Parquet) in a separate S3 bucket for analytics. The process should be fully managed and serverless. Which combination of AWS services should be used?

  1. AAWS Direct Connect + Amazon EMR
  2. BAWS Transfer Family (SFTP) + AWS Glue ETL
  3. CAWS Storage Gateway (File Gateway) + Amazon Redshift
  4. DAWS DataSync + AWS Glue ETL
Show answer & explanation

Correct answer: B. AWS Transfer Family (SFTP) + AWS Glue ETL

AWS Transfer Family (SFTP) provides a fully managed SFTP endpoint to ingest files directly into S3. AWS Glue ETL is a serverless service that can read various formats (including XML from S3), transform the data, and write it to S3 in Parquet format, fulfilling the serverless transformation requirement.

Why the other options are wrong

  • A. AWS Direct Connect provides a dedicated network connection, but not an SFTP endpoint. Amazon EMR offers powerful processing (e.g., Spark), but it requires managing clusters, which goes against the serverless requirement of the transformation step.
  • C. AWS Storage Gateway (File Gateway) provides file-based access to S3 but doesn't offer a direct SFTP endpoint for external systems. Amazon Redshift is a data warehouse, not a transformation engine for files in S3.
  • D. AWS DataSync is suitable for large-scale migrations and ongoing replication, but AWS Transfer Family is specifically designed for SFTP access, which is the exact source system. AWS Glue ETL is appropriate for transformation.

AWS Transfer Family + AWS Glue ETL

A serverless solution for ingesting files via SFTP/FTPS/FTP into S3, followed by automated, serverless transformation of that data using AWS Glue ETL.

  • Transfer Family provides managed SFTP/FTPS/FTP endpoints.
  • Files land directly in S3.
  • Glue ETL offers serverless data transformation (parsing, cleaning, format conversion).

Memory trick: Transfer Family brings files, Glue ETL makes them shine.

More Data Ingestion and Transformation questions