AWS Certified Data Engineer – AssociateData Ingestion and TransformationMedium
A cybersecurity firm needs to ingest security logs from various on-premises firewalls and intrusion detection systems (IDS). These logs are generated continuously and need to be analyzed in near real-time for threat detection. The firm requires a solution that can reliably collect data from on-premises, encrypt it in transit, and deliver it to an Amazon S3 bucket for long-term storage, and also to an Amazon OpenSearch Service domain for real-time analysis. The solution should be simple to deploy and manage. Which AWS service is the MOST appropriate for ingesting this data?
- AAmazon Kinesis Data Firehose
- BAWS DataSync
- CAWS Transfer Family (SFTP)
- DAWS Storage Gateway (File Gateway)
Show answer & explanationAnswer & explanation
Correct answer: A. Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to various AWS destinations, including S3 and OpenSearch Service. It handles data buffering, compression, transformation, and encryption in transit, making it ideal for ingesting continuous security logs from on-premises sources via agents.
Why the other options are wrong
- B. AWS DataSync is for bulk data transfers between on-premises and AWS, not for continuous, real-time streaming to multiple destinations like S3 and OpenSearch Service.
- C. AWS Transfer Family (SFTP) is for secure file transfers, not for continuous, real-time ingestion of streaming logs to multiple destinations.
- D. AWS Storage Gateway (File Gateway) provides on-premises applications file-based access to cloud storage, but it is not designed for real-time streaming ingestion to multiple analytics services.
Amazon Kinesis Data Firehose
A fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk.
- Fully managed and serverless
- Supports various streaming data sources (Kinesis Data Streams, custom applications)
- Automatic scaling, buffering, compression, and encryption
- Integrates with S3, Redshift, OpenSearch Service, Splunk
Memory trick: Firehose sprays your logs to all your storage and analytics destinations.