AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A financial institution needs to process large volumes of historical market data, approximately 500 TB, currently stored on-premises across various network-attached storage (NAS) devices. The data needs to be moved to Amazon S3 for long-term archival and eventual analysis. The transfer must be highly secure, reliable, and optimized for large datasets over a standard internet connection, minimizing manual effort. Which AWS service is the MOST suitable for this one-time bulk migration?
- AAmazon Kinesis Data Firehose
- BAWS Snowball Edge
- CAWS DataSync
- DAWS Storage Gateway (File Gateway)
Show answer & explanationAnswer & explanation
Correct answer: C. AWS DataSync
AWS DataSync is designed for online data transfers between on-premises storage and AWS storage services like S3. It automates and accelerates data transfer over the internet, handling large datasets reliably, securely, and efficiently, making it ideal for this 500 TB one-time bulk migration.
Why the other options are wrong
- A. Amazon Kinesis Data Firehose is for real-time streaming data ingestion, not for bulk transfer of existing files.
- B. AWS Snowball Edge is suitable for petabyte-scale migrations or environments with limited network connectivity, but for 500 TB over a standard internet connection, DataSync is generally more cost-effective and efficient for an online transfer.
- D. AWS Storage Gateway (File Gateway) provides file-based access to S3 for on-premises applications, but it's more for ongoing access/hybrid storage rather than a one-time bulk migration of 500 TB.
AWS DataSync
An online data transfer service that simplifies, automates, and accelerates moving data between on-premises storage and AWS storage services, and between AWS storage services.
- Supports NFS, SMB, HDFS, S3, EFS, FSx
- Optimizes network utilization and accelerates transfers
- Provides encryption in transit and at rest
- Ideal for one-time migrations, recurring transfers, or data archiving
Memory trick: DataSync is like a super-fast, secure moving truck for your data.