AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A global media company needs to ingest petabytes of video and audio files daily from various content providers. These files, often several gigabytes in size, arrive asynchronously and must be stored in Amazon S3 for subsequent processing and archiving. The company requires a highly scalable and cost-effective solution that can handle unpredictable peaks in data volume without manual intervention. Which AWS service is most appropriate for ingesting this data?
- AAWS Transfer Family (SFTP)
- BAWS Snowball Edge
- CAmazon Kinesis Data Firehose
- DAmazon Kinesis Data Streams
Show answer & explanationAnswer & explanation
Correct answer: A. AWS Transfer Family (SFTP)
AWS Transfer Family (SFTP) provides fully managed support for transferring files directly into and out of Amazon S3 using standard file transfer protocols. It is well-suited for large, asynchronous file transfers from external partners, offering scalability and integration with S3 without requiring custom application development or server management.
Why the other options are wrong
- B. AWS Snowball Edge is for offline data transfer of extremely large datasets (petabytes) where network transfer is impractical, not for ongoing daily asynchronous ingestion.
- C. Kinesis Data Firehose is for real-time stream delivery to S3, Redshift, etc., but is optimized for smaller records, not large files, and often has size limitations per record.
- D. Kinesis Data Streams is designed for real-time streaming data, typically smaller records, not large multi-gigabyte files.
AWS Transfer Family (SFTP)
A fully managed service that enables secure file transfers directly into and out of Amazon S3 using standard file transfer protocols (SFTP, FTPS, FTP).
- Supports SFTP, FTPS, and FTP protocols.
- Directly integrates with Amazon S3 for storage.
- Managed service, eliminating server provisioning and maintenance.
- Ideal for external partners to exchange files securely.
Memory trick: Transfer Family makes big files fly to S3.