A financial services company is migrating its legacy monolithic application to a microservices architecture on AWS. They require a robust solution for collecting and analyzing highly sensitive audit logs from all microservices, which are deployed as Docker containers on Amazon ECS. The solution must ensure data encryption at rest and in transit, support long-term retention for 7 years, and provide granular access controls over who can view specific log data. Performance and scalability are critical due to high log volumes. Which combination of AWS services is the most appropriate for these requirements?
- AUtilize Amazon Kinesis Data Firehose for ingestion, Amazon S3 for long-term storage with SSE-KMS, and Amazon OpenSearch Service (with fine-grained access control) for real-time analysis.
- BAmazon CloudWatch Logs for collection and storage, with KMS encryption and IAM policies for access control. Use CloudWatch Logs Insights for analysis.
- CConfigure AWS CloudTrail to capture all API calls, store them in Amazon S3 with SSE-KMS, and use Amazon GuardDuty for anomaly detection.
- DImplement a custom Fluentd agent on each ECS container to send logs directly to Amazon S3 with server-side encryption, and use Amazon Athena for querying.
Show answer & explanationAnswer & explanation
Correct answer: A. Utilize Amazon Kinesis Data Firehose for ingestion, Amazon S3 for long-term storage with SSE-KMS, and Amazon OpenSearch Service (with fine-grained access control) for real-time analysis.
This solution provides a comprehensive, scalable, and secure logging pipeline. Kinesis Data Firehose efficiently ingests high volumes of log data. OpenSearch Service offers powerful real-time analysis, search, dashboards, and crucial fine-grained access control for sensitive data. S3 with SSE-KMS ensures encrypted, durable, and cost-effective long-term retention for 7 years. Data in transit is secured via Firehose and OpenSearch's HTTPS endpoints. This meets all requirements for encryption, retention, granular access, performance, and scalability for sensitive audit logs.
Why the other options are wrong
- B. CloudWatch Logs supports KMS encryption and IAM, but its query capabilities with Logs Insights are less powerful for complex, high-volume real-time analysis and dashboarding compared to OpenSearch. Granular access control within CloudWatch Logs is primarily at the log group level, not as fine-grained as OpenSearch's document-level security.
- C. CloudTrail is for AWS API call logging, not application audit logs from microservices. GuardDuty is for threat detection, not a logging and analysis platform for application logs. This option does not meet the core requirement for collecting and analyzing microservice audit logs.
- D. While Fluentd to S3 is viable for storage, querying with Athena is not 'real-time analysis' and dashboards are not natively supported. It also lacks the fine-grained access control and advanced analytical capabilities of OpenSearch Service.
Amazon OpenSearch Service (Fine-Grained Access Control)
A managed service that makes it easy to deploy, operate, and scale OpenSearch clusters. Its fine-grained access control feature allows for highly specific permissions on indices, types, documents, and fields.
- Provides search, analysis, and visualization capabilities.
- Built on OpenSearch (formerly Elasticsearch).
- Fine-grained access control is crucial for sensitive data.
Memory trick: Firehose Feeds OpenSearch Securely with S3.