AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A healthcare provider needs to ingest patient data from various on-premises systems, including electronic health records (EHR) and laboratory information systems (LIS). These systems generate large, structured CSV and XML files (up to 50 GB each) nightly. The data needs to be moved to Amazon S3 for secure archiving and subsequent processing. The transfer must be highly secure, encrypted, and provide comprehensive logging and auditing capabilities for compliance reasons. Which AWS service is most suitable for this secure and automated data ingestion?
- AAWS Transfer Family (SFTP)
- BAWS Storage Gateway (File Gateway)
- CAWS DataSync
- DAWS Database Migration Service (DMS)
Show answer & explanationAnswer & explanation
Correct answer: C. AWS DataSync
AWS DataSync is ideal for secure, automated, and high-performance data transfers between on-premises storage and AWS storage services like S3. It supports encryption in transit and at rest, provides comprehensive logging, and is built for compliance-heavy environments, making it suitable for large, nightly transfers of sensitive healthcare data.
Why the other options are wrong
- A. AWS Transfer Family (SFTP) is for standard SFTP/FTP/FTPS file transfers, but DataSync is more optimized for bulk, automated, and high-performance transfers from on-premises file systems.
- B. AWS Storage Gateway (File Gateway) provides NFS/SMB access to S3, which might be an option, but DataSync is specifically designed for transfers, not for presenting cloud storage as a local file share.
- D. AWS DMS is for migrating databases, not for transferring large files from file systems like EHR/LIS exports.
AWS DataSync
A data transfer service that simplifies, automates, and accelerates moving data between on-premises storage systems and AWS storage services, and between AWS storage services.
- Optimized for large-scale data transfers.
- Supports various on-premises storage types (NFS, SMB, HDFS, object stores).
- Automates, encrypts, and validates data transfers, with compliance features.
Memory trick: DataSync secures and sends health data to S3.