A security engineer has deployed a new web application on Amazon EKS. The application consists of several microservices, and the engineer needs to ensure that all application logs, regardless of the pod they originate from, are collected, centralized, and available for real-time monitoring and troubleshooting. The solution should be resilient to pod restarts and scaling events. Which approach should the engineer take?
- AUse AWS CloudTrail to capture EKS API events and store them in Amazon S3 for analysis.
- BConfigure each microservice to directly send logs to Amazon CloudWatch Logs using the AWS SDK.
- CMount an Amazon EFS volume to each pod for log storage and then periodically copy logs to Amazon S3.
- DDeploy a Fluent Bit DaemonSet on the EKS cluster to collect logs and forward them to Amazon CloudWatch Logs.
Show answer & explanationAnswer & explanation
Correct answer: D. Deploy a Fluent Bit DaemonSet on the EKS cluster to collect logs and forward them to Amazon CloudWatch Logs.
Deploying Fluent Bit as a DaemonSet ensures that a log collector agent runs on every node in the EKS cluster. Fluent Bit can then collect logs from all pods on that node, process them, and reliably forward them to a centralized logging destination like Amazon CloudWatch Logs. This approach is resilient to pod restarts and scaling because the DaemonSet ensures the collector is always present, and Fluent Bit handles buffering and forwarding, making it ideal for real-time monitoring.
Why the other options are wrong
- A. AWS CloudTrail captures EKS API events (control plane activities), not application logs (data plane activities) generated by the microservices within the pods.
- B. Directly sending logs from each microservice via the AWS SDK adds complexity and overhead to the application code and can be less resilient to network issues or service outages compared to a dedicated log forwarder.
- C. Mounting EFS for log storage is not a real-time monitoring solution and introduces complexities for aggregation and analysis. Periodically copying to S3 is for archival, not real-time troubleshooting.
EKS Container Logging with Fluent Bit
Utilizing Fluent Bit as a DaemonSet in Amazon EKS to collect and forward application logs from containers to a centralized logging service.
- Fluent Bit runs on each EKS node.
- Collects logs from stdout/stderr of pods.
- Forwards logs to destinations like CloudWatch Logs or Kinesis Firehose.
Memory trick: Fluent Bit is the reliable 'log truck' on every EKS node, picking up logs from all containers and driving them to CloudWatch.