AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A research institution collects high-resolution satellite imagery, with individual image files often exceeding 500 GB. These files are generated on-premises and need to be regularly transferred to Amazon S3 for long-term archival and processing by AWS analytics services. The institution has a 1 Gbps internet connection, but transferring these large files takes too long and frequently fails due to network interruptions. The solution must be reliable, secure, and minimize transfer time. Which AWS service should the institution use for data ingestion?
- AAWS Transfer Family (SFTP)
- BAWS DataSync
- CAWS Snowball Edge
- DAmazon S3 Multipart Upload API
Show answer & explanationAnswer & explanation
Correct answer: B. AWS DataSync
AWS DataSync is designed for large-scale, online data transfers between on-premises storage and AWS storage services like S3. It provides robust features such as automatic retry, in-transit encryption, data integrity verification, and network optimization, making it ideal for reliably transferring very large files over a potentially unstable or limited internet connection.
Why the other options are wrong
- A. AWS Transfer Family is for file transfer via standard protocols (SFTP/FTPS/FTP) to S3, but it does not offer the same level of network optimization, reliability, and large-file transfer resilience as DataSync for on-premises to S3 migration over a WAN.
- C. AWS Snowball Edge is for offline data transfer of extremely large datasets (petabytes) where network transfer is entirely impractical or too slow, not for regular, online transfers over an existing internet connection, regardless of its speed or stability.
- D. The S3 Multipart Upload API is a low-level API for uploading large objects to S3. While effective for large objects, it requires custom development and does not inherently provide the network optimization, resume capabilities, or managed service reliability of DataSync for on-premises transfers.
AWS DataSync
An online data transfer service that simplifies, automates, and accelerates moving data between on-premises storage and Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server.
- Accelerates data transfers over WANs.
- Provides automatic retry, encryption, and integrity verification.
- Managed service, eliminating custom scripting and infrastructure.
- Ideal for large files and datasets, ensuring reliable delivery.
Memory trick: DataSync is the 'Guardian' of your large files moving to S3.