A large enterprise uses AWS Organizations to manage hundreds of AWS accounts. They need to ensure that all security-related logs, such as VPC Flow Logs, DNS query logs (Route 53 Resolver query logs), and AWS WAF logs, are centrally collected, retained for 7 years, and made immediately searchable for incident response and forensic analysis. The solution must be cost-effective and scalable to petabytes of data without operational overhead. Which architecture should the DevOps team implement?
- AConfigure each log source to send logs directly to Amazon S3, then use AWS Glue and Amazon Athena for querying.
- BSend all logs to Amazon CloudWatch Logs, configure cross-account log subscription filters to stream to a central Lambda function, and then write to a custom log analysis platform.
- CStream all logs from their respective sources (VPC Flow Logs, Route 53, WAF) to Amazon Kinesis Data Firehose, which then delivers them to a central Amazon OpenSearch Service domain.
- DEnable AWS CloudTrail organization trail to a central S3 bucket and use CloudWatch Logs Insights for analysis.
Show answer & explanationAnswer & explanation
Correct answer: C. Stream all logs from their respective sources (VPC Flow Logs, Route 53, WAF) to Amazon Kinesis Data Firehose, which then delivers them to a central Amazon OpenSearch Service domain.
This scenario requires centralized collection, petabyte-scale searchability, long-term retention, and minimal operational overhead. Streaming logs via Kinesis Data Firehose (a fully managed service) to a central Amazon OpenSearch Service domain (also managed) directly addresses these needs. OpenSearch Service provides immediate searchability and analysis capabilities for petabytes of data, Firehose handles ingestion and delivery, and data can be tiered to S3 within OpenSearch for cost-effective long-term retention. This architecture minimizes operational overhead while providing powerful analytics.
Why the other options are wrong
- A. While S3 and Athena are cost-effective for storage and ad-hoc queries, 'immediately searchable for incident response and forensic analysis' implies a dedicated indexing/search engine like OpenSearch, not just S3/Athena which can have query cold-start times and are not designed for real-time indexing.
- B. Streaming logs through Lambda to a 'custom log analysis platform' introduces significant operational overhead for platform management and scaling, which is explicitly to be avoided. CloudWatch Logs Insights is not designed for petabytes of data across hundreds of accounts for real-time forensic search.
- D. AWS CloudTrail only captures API calls and management events, not VPC Flow Logs, DNS query logs, or WAF logs directly. CloudWatch Logs Insights is for log analysis within CloudWatch Logs, which may not scale to petabytes with the same search efficiency as OpenSearch for 'immediately searchable' forensic analysis.
Kinesis Firehose to OpenSearch for Centralized Logging
A managed architecture for centralizing and analyzing petabytes of security logs (VPC Flow, DNS, WAF) from multiple accounts using Kinesis Data Firehose for ingestion and Amazon OpenSearch Service for real-time search and analytics.
- Kinesis Firehose for scalable, managed log ingestion.
- OpenSearch Service for real-time, petabyte-scale search and analysis.
- Supports long-term retention via OpenSearch data tiers (UltraWarm, Cold Storage) or S3 integration.
- Minimizes operational overhead with fully managed services.
Memory trick: Firehose to OpenSearch: All security logs, instantly searchable.