A security engineer needs to establish a centralized logging solution for a new application deployed on Amazon EKS. The application consists of multiple microservices, each running in its own pod. All application logs, standard output, and standard error from containers must be collected, processed, and sent to a centralized Amazon OpenSearch Service domain for analysis and visualization. The solution should be robust, scalable, and handle high volumes of log data. Which logging agent and delivery mechanism should the engineer choose?
- AMount an Amazon EFS volume to each pod for log storage, and use a separate EC2 instance with Logstash to process and forward logs from EFS to OpenSearch Service.
- BUse Fluentd or Fluent Bit deployed as a DaemonSet in the EKS cluster to collect logs and send them to Kinesis Data Firehose, which then delivers to OpenSearch Service.
- CDeploy a custom Python script in each pod to push logs directly to OpenSearch Service.
- DConfigure EKS to send control plane logs to CloudWatch Logs, then use a Lambda function to forward these to OpenSearch Service.
Show answer & explanationAnswer & explanation
Correct answer: B. Use Fluentd or Fluent Bit deployed as a DaemonSet in the EKS cluster to collect logs and send them to Kinesis Data Firehose, which then delivers to OpenSearch Service.
Fluentd or Fluent Bit are lightweight log processors commonly used in Kubernetes environments. Deploying them as a DaemonSet ensures they run on every node to collect logs from all pods. Kinesis Data Firehose provides a fully managed, scalable, and reliable way to deliver high volumes of streaming data to destinations like Amazon OpenSearch Service without needing to manage underlying infrastructure.
Why the other options are wrong
- A. Using EFS for log storage and a separate Logstash instance introduces unnecessary complexity, latency, and single points of failure. It's not as efficient or scalable as direct streaming with Fluent Bit/Firehose.
- C. Deploying custom scripts in each pod adds significant overhead, complexity, and maintenance, and is not robust or scalable for high volumes.
- D. This option only addresses EKS control plane logs, not application logs from microservices running in pods, which is the primary requirement.
EKS Container Logging with Fluent Bit/Firehose
A common architecture for collecting logs from Amazon EKS containers using Fluent Bit (deployed as a DaemonSet) to aggregate logs, and Amazon Kinesis Data Firehose to reliably deliver them to a centralized destination like Amazon OpenSearch Service.
- Fluent Bit/Fluentd are lightweight log collectors for Kubernetes.
- DaemonSet ensures a collector runs on every node.
- Kinesis Data Firehose provides managed, scalable, and reliable delivery.
- Enables centralized logging for EKS applications in OpenSearch Service.
Memory trick: Fluent Bit collects from every pod, Firehose pipes them to OpenSearch.