AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A biotechnology company generates large genomic sequencing files (tens of gigabytes each) from its on-premises lab equipment. These files need to be securely and efficiently transferred to Amazon S3 for archival and processing. The transfers occur periodically, and the company wants to optimize network utilization and ensure data integrity during transit. Which AWS service is best suited for this task?
- AAWS Direct Connect
- BAWS DataSync
- CAWS Storage Gateway (File Gateway)
- DAWS Transfer Family (SFTP)
Show answer & explanationAnswer & explanation
Correct answer: B. AWS DataSync
AWS DataSync is designed for secure, efficient, and optimized transfer of large files between on-premises storage and AWS services like S3. It handles network optimization, data integrity, and can be scheduled, making it ideal for periodic transfers of large genomic files.
Why the other options are wrong
- A. AWS Direct Connect provides a dedicated network connection, improving bandwidth, but it's an infrastructure service. DataSync would still be needed on top of Direct Connect to manage the actual file transfer process efficiently.
- C. AWS Storage Gateway (File Gateway) provides file-based access to S3 from on-premises, acting as a cache. It's more for hybrid cloud storage access than for a one-time or periodic bulk transfer of large files.
- D. AWS Transfer Family (SFTP) provides managed SFTP endpoints for file transfers, but AWS DataSync is optimized for large-scale, high-performance data transfers between on-premises systems and S3, including network optimization and integrity checks.
AWS DataSync
An online data transfer service that simplifies, automates, and accelerates moving data between on-premises storage systems and AWS storage services, and between AWS storage services.
- Optimized for large-scale data transfers.
- Handles data integrity verification and network optimization.
- Supports various storage types (NFS, SMB, S3, EFS).
Memory trick: DataSync moves big files, fast and secure.