AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A research laboratory generates massive genomic datasets, with individual files often exceeding 100 GB. These files are stored on an on-premises Network Attached Storage (NAS) appliance. The lab needs to regularly transfer these files to Amazon S3 for long-term archival and analysis. The transfers must be reliable, resumable, and optimized for high bandwidth utilization over a dedicated network connection (AWS Direct Connect). What is the most efficient and cost-effective service for this data ingestion requirement?
- AAWS Transfer Family (SFTP)
- BS3 Transfer Acceleration
- CAWS DataSync
- DAWS Storage Gateway (File Gateway)
Show answer & explanationAnswer & explanation
Correct answer: C. AWS DataSync
AWS DataSync is specifically designed for high-performance, secure, and automated data transfers between on-premises storage and AWS storage services like S3. It handles large files efficiently, provides reliability, and optimizes network bandwidth.
Why the other options are wrong
- A. AWS Transfer Family (SFTP) is for secure file transfers using standard protocols, but DataSync offers better performance and optimization for very large datasets and bulk transfers.
- B. S3 Transfer Acceleration optimizes uploads over the internet, but the scenario specifies a dedicated network connection (Direct Connect) and on-premises storage, where DataSync is more appropriate.
- D. AWS Storage Gateway (File Gateway) provides NFS/SMB access to S3, but DataSync is more optimized for bulk, one-time or scheduled transfers of large datasets.
AWS DataSync
A data transfer service that simplifies, automates, and accelerates moving data between on-premises storage systems and AWS storage services, and between AWS storage services.
- Optimized for large-scale data transfers.
- Supports various on-premises storage types (NFS, SMB, HDFS, object stores).
- Automates, encrypts, and validates data transfers.
Memory trick: DataSync makes your data zoom to the cloud.