A development team has deployed a new microservice on Amazon ECS. They need to collect application logs from the containers and centralize them for analysis and long-term storage. The solution must be cost-effective and scalable without requiring manual intervention for log rotation or storage management. Which logging configuration meets these requirements?
- AUse the 'awslogs' log driver for ECS tasks to send logs directly to Amazon CloudWatch Logs, with a CloudWatch Logs subscription filter sending to Amazon S3.
- BConfigure containers to write logs to a shared Amazon EFS volume, then use a Lambda function to periodically transfer logs to Amazon S3.
- CWrite logs to standard output/error and use a sidecar container running a log agent (e.g., Fluent Bit) to forward logs to an Amazon Kinesis Data Firehose delivery stream, which then stores them in Amazon S3.
- DMount an Amazon EBS volume to each ECS instance and configure a log agent (e.g., Fluentd) to push logs from the volume to Amazon S3.
Show answer & explanationAnswer & explanation
Correct answer: C. Write logs to standard output/error and use a sidecar container running a log agent (e.g., Fluent Bit) to forward logs to an Amazon Kinesis Data Firehose delivery stream, which then stores them in Amazon S3.
Writing logs to standard output/error (stdout/stderr) is a best practice for containerized applications. Using a sidecar container with a log agent like Fluent Bit is a robust and scalable pattern to collect these logs and forward them to Kinesis Data Firehose, which then delivers them to Amazon S3 for cost-effective long-term storage, handling buffering, compression, and delivery automatically.
Why the other options are wrong
- A. 'awslogs' sends to CloudWatch Logs, which is good for real-time monitoring. While CloudWatch Logs can eventually go to S3 via a subscription filter, using Kinesis Data Firehose offers more flexible and cost-effective buffering and delivery options for large volumes of logs destined for S3, especially for long-term storage.
- B. EFS is a shared file system, but managing log rotation, retention, and transfer to S3 with Lambda adds complexity and potential cost overhead for log processing.
- D. EBS volumes are attached to instances, not directly to containers in a scalable way for log collection across multiple containers/tasks, and managing log agents and storage on EBS adds operational burden inconsistent with 'no manual intervention'.
Container Log Centralization with Kinesis Firehose
A common pattern for container logging involves writing logs to stdout/stderr, using a sidecar log agent (e.g., Fluent Bit) to collect them, and forwarding them to a Kinesis Data Firehose delivery stream for centralized, scalable, and cost-effective storage in Amazon S3.
- Containers log to stdout/stderr.
- Sidecar log agent collects logs.
- Kinesis Data Firehose buffers and delivers logs.
- Amazon S3 provides cost-effective long-term storage.
Memory trick: Containers stream logs to Firehose for S3 storage.