AWS Certified Data Engineer – AssociateData Operations and MonitoringMedium
A financial services company uses AWS Step Functions to orchestrate a complex data ingestion and processing pipeline. This pipeline involves multiple Lambda functions, Glue jobs, and S3 operations. Due to the sensitive nature of the data and strict compliance requirements, the company needs to ensure that all execution history, including input and output data for each step, is retained for a minimum of five years for auditing purposes. Standard Step Functions execution history retention is 90 days. Which is the MOST cost-effective and operationally efficient solution to meet this long-term retention requirement?
- AManually export the execution history from the Step Functions console every 90 days and upload it to an S3 bucket with appropriate lifecycle policies.
- BUtilize AWS Backup to create automated backups of Step Functions execution history and define a retention policy of five years.
- CConfigure Amazon CloudWatch Logs for Step Functions to capture all execution events and stream them to Amazon S3 for long-term storage.
- DCreate a custom Lambda function that periodically queries Step Functions execution history and stores it in Amazon S3, then use S3 Lifecycle policies.
Show answer & explanationAnswer & explanation
Correct answer: C. Configure Amazon CloudWatch Logs for Step Functions to capture all execution events and stream them to Amazon S3 for long-term storage.
Amazon CloudWatch Logs can capture detailed Step Functions execution events, including input and output. By configuring CloudWatch Logs to stream these logs to an S3 bucket, and then applying S3 Lifecycle policies, the company can achieve long-term retention beyond the default 90 days in a cost-effective and automated manner.
Why the other options are wrong
- A. Manual export is not scalable, error-prone, and highly inefficient for a daily pipeline with a five-year retention requirement.
- B. AWS Backup is primarily for backing up AWS resources like EC2, EBS, RDS, etc. It does not directly back up Step Functions execution history in a way that meets this specific requirement efficiently.
- D. While feasible, this requires custom development and management of a Lambda function, increasing operational overhead compared to native CloudWatch integration.
Step Functions Long-Term Auditing
To retain AWS Step Functions execution history beyond 90 days for auditing, stream CloudWatch Logs (which capture execution details) to Amazon S3 and manage retention using S3 Lifecycle policies.
- Default Step Functions retention is 90 days.
- CloudWatch Logs capture detailed execution events.
- S3 Lifecycle policies manage long-term data retention cost-effectively.
Memory trick: CloudWatch logs everything, S3 stores it forever, cheap.