AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A financial institution needs to process large volumes of historical market data, approximately 500 TB, stored in various on-premises network attached storage (NAS) devices. The data needs to be moved to Amazon S3 for long-term archival and subsequent analysis. The institution has limited internet bandwidth and strict security requirements, preventing direct internet transfers of such a large volume. Which AWS service is the MOST appropriate for this data ingestion task?
- AAWS DataSync
- BAWS Snowball Edge Storage Optimized
- CAWS Direct Connect
- DAWS Storage Gateway File Gateway
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Snowball Edge Storage Optimized
For ingesting large volumes of data (hundreds of TBs to PBs) with limited internet bandwidth and strict security, AWS Snowball Edge Storage Optimized is the most suitable service. It provides a physical device for offline data transfer, bypassing internet limitations and offering strong security features.
Why the other options are wrong
- A. AWS DataSync is excellent for online data transfer between on-premises and AWS, but it still relies on existing network bandwidth, which is a limitation in this scenario.
- C. AWS Direct Connect provides dedicated network connectivity but does not address the challenge of initial bulk data transfer for hundreds of TBs with limited existing internet bandwidth.
- D. AWS Storage Gateway File Gateway provides cloud-backed storage for on-premises applications but is not designed for a one-time bulk migration of 500 TB with limited network bandwidth.
AWS Snowball Edge Storage Optimized
A physical device with significant storage and compute capabilities for transferring large amounts of data into and out of AWS, especially when network bandwidth is limited or non-existent.
- Designed for petabyte-scale data transfers.
- Offers local storage and compute for edge processing.
- Bypasses internet for data transfer, shipped physically.
Memory trick: Snowball rolls data fast when the internet is slow or gone.