AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A research institution collects high-resolution satellite imagery, with individual image files ranging from 500 MB to 5 GB. These files are generated on-premises and need to be regularly transferred to Amazon S3 for long-term archival and further processing by AWS Lambda functions. The transfers must be reliable, secure, and optimized for large file sizes over a WAN connection. Which AWS service would be most effective for this data ingestion task?
- AAmazon S3 Transfer Acceleration
- BAWS Transfer Family
- CAWS DataSync
- DAWS Storage Gateway (File Gateway)
Show answer & explanationAnswer & explanation
Correct answer: C. AWS DataSync
AWS DataSync is designed for fast, simple, and secure online data transfer between on-premises storage and AWS storage services like S3. It optimizes transfers over WANs, handles large files efficiently, and provides built-in encryption and integrity checks, making it ideal for moving large satellite imagery files.
Why the other options are wrong
- A. Amazon S3 Transfer Acceleration speeds up transfers to S3 over long distances but requires clients to use the S3 API directly, and does not provide the comprehensive management and optimization features of DataSync for on-premises sources.
- B. AWS Transfer Family provides fully managed support for SFTP, FTPS, and FTP, primarily for transferring files using standard protocols, not optimized for massive, high-performance data migration like DataSync.
- D. AWS Storage Gateway (File Gateway) provides file-based access to S3 from on-premises, acting as a cache, but it's not primarily a high-performance one-time or recurring bulk transfer service like DataSync.
AWS DataSync
A data transfer service that makes it easier for you to automate moving data between on-premises storage and AWS storage services.
- Optimizes transfers over WANs with parallelization and sparse file detection.
- Provides built-in encryption, data integrity verification, and automatic retries.
- Supports NFS, SMB, HDFS, and object storage locations.
Memory trick: DataSync: Your express lane for on-premises data to AWS.