AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A healthcare provider receives patient data from various on-premises systems, including electronic health records (EHR) and laboratory information systems (LIS). This data, typically in structured formats like CSV or flat files, needs to be ingested into Amazon S3 daily. The total daily volume is around 5 TB, and the transfer must be reliable, secure, and easily scheduled. The provider wants to minimize operational overhead. Which AWS service is most appropriate for ingesting this data?
- AAWS Storage Gateway (File Gateway)
- BAWS DataSync
- CAWS Snowball Edge
- DAmazon S3 Transfer Acceleration
Show answer & explanationAnswer & explanation
Correct answer: B. AWS DataSync
AWS DataSync is designed for large-scale, online data transfers between on-premises storage and AWS storage services like S3. It automates and accelerates the transfer, providing reliability, integrity verification, and scheduling capabilities, making it ideal for regular, high-volume data ingestion from on-premises systems to S3 with minimal operational overhead.
Why the other options are wrong
- A. AWS Storage Gateway (File Gateway) provides a file interface to S3 for on-premises applications, making S3 appear as a local file share. While it can store data in S3, it's more for hybrid cloud storage access than scheduled, large-volume, one-way data ingestion for analytics. It acts as a cache.
- C. AWS Snowball Edge is for offline petabyte-scale data migration, not for regular, online daily transfers.
- D. Amazon S3 Transfer Acceleration speeds up transfers to/from S3 over long distances but requires clients to use the S3 API directly. It's a feature of S3, not a managed service for automating on-premises transfers with scheduling and integrity checks like DataSync.
AWS DataSync
An online data transfer service that simplifies, automates, and accelerates moving data between on-premises storage and Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server.
- Accelerates data transfers over WANs.
- Provides automatic retry, encryption, and integrity verification.
- Managed service, eliminating custom scripting and infrastructure.
- Ideal for large files and datasets, ensuring reliable delivery.
Memory trick: DataSync is the 'Bridge' for on-prem to S3 data.