AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A research institution collects high-resolution satellite imagery, with individual image files often exceeding 50 GB. These files are generated on-premises and need to be securely and efficiently transferred to Amazon S3 for long-term archival and subsequent processing. The institution has limited internet bandwidth (100 Mbps upload speed) at its remote data collection sites. Which AWS service is best suited for ingesting this data?
- AAWS Snowball Edge Storage Optimized
- BAmazon Kinesis Data Firehose
- CAWS Transfer Family (SFTP)
- DAWS DataSync
Show answer & explanationAnswer & explanation
Correct answer: A. AWS Snowball Edge Storage Optimized
Given the large file sizes (over 50 GB), limited internet bandwidth (100 Mbps), and the need for secure transfer to S3 for archival, AWS Snowball Edge Storage Optimized is the most appropriate service. It allows for physical transfer of petabyte-scale data, bypassing network limitations.
Why the other options are wrong
- B. Amazon Kinesis Data Firehose is for streaming data, typically smaller records, not large individual files like 50 GB+ satellite imagery.
- C. AWS Transfer Family provides SFTP/FTPS/FTP access to S3 but relies entirely on internet bandwidth, which is a major constraint in this scenario.
- D. AWS DataSync is excellent for online data transfers but would be severely bottlenecked by the 100 Mbps link for 50 GB+ files, making the transfer time unacceptably long.
AWS Snowball Edge Storage Optimized
A petabyte-scale data transfer device with on-board storage and compute capabilities, used to move large amounts of data into and out of AWS when network transfer is impractical.
- Physical appliance for massive data transfers (PB scale).
- Bypasses internet bandwidth limitations.
- Offers local storage and optional compute for edge processing.
- Ideal for remote locations with poor connectivity or extremely large datasets.
Memory trick: Snowball Edge: Too much data, too little internet, just ship it!