AWS Certified Data Engineer – AssociateData Ingestion and TransformationHard
A financial institution needs to process massive volumes of historical market data, approximately 500 TB, stored in an on-premises data center. The data is currently on Network Attached Storage (NAS) and needs to be moved to Amazon S3 for long-term archival and analytics. The institution has a 10 Gbps direct connection to AWS (AWS Direct Connect). Which AWS service is the most cost-effective and efficient for this one-time bulk migration?
- AAWS DataSync
- BAWS Transfer Family (SFTP)
- CAmazon Kinesis Data Firehose
- DAWS Snowball Edge Storage Optimized
Show answer & explanationAnswer & explanation
Correct answer: A. AWS DataSync
AWS DataSync is designed for online data transfer between on-premises storage (NFS shares, SMB shares, self-managed object storage) and AWS storage services like S3. With a 10 Gbps Direct Connect, DataSync can efficiently move 500 TB of data over the network, making it more cost-effective and faster than physical appliance-based transfers like Snowball Edge for this volume and connectivity.
Why the other options are wrong
- B. Transfer Family is for SFTP/FTPS/FTP access to S3, not a bulk migration tool from NAS to S3.
- C. Kinesis Data Firehose is for real-time streaming data ingestion, not a one-time bulk migration of historical files.
- D. Snowball Edge is ideal for disconnected or extremely large transfers (petabytes) where network transfer isn't feasible or cost-effective; for 500 TB with 10 Gbps Direct Connect, DataSync is usually faster and cheaper.
AWS DataSync
AWS DataSync is an online data transfer service that simplifies, automates, and accelerates moving data between on-premises storage and AWS storage services.
- Supports NFS, SMB, self-managed object storage, HDFS.
- Can transfer to S3, EFS, FSx for Windows File Server.
- Optimized for network transfers, ideal with Direct Connect.
- Handles encryption, data integrity, and task scheduling.
Memory trick: For 'huge' data, 'Direct Connect' needs 'DataSync' to efficiently 'Sync' it to the 'Cloud'.