A data platform team needs to implement a data retention policy for their Amazon S3 data lake. The policy states that all log data should be moved to a cost-effective archival storage class after 30 days and then permanently deleted after 5 years. However, specific log files related to security incidents must be retained indefinitely. The team wants an automated solution that minimizes operational overhead. How should the data engineer design this retention strategy?
- AApply a bucket-level lifecycle policy for 30-day transition and 5-year deletion, and manually move security incident logs to a separate, untagged bucket.
- BImplement an AWS Lambda function triggered by S3 events to move standard logs to Glacier and delete them, while moving security incident logs to a separate bucket with Object Lock.
- CUse S3 Lifecycle Policies with object tags: one tag for standard logs (30-day transition, 5-year deletion) and another tag for security incident logs (never expire).
- DConfigure S3 Object Lock on the entire bucket for indefinite retention, and use S3 Intelligent-Tiering to manage standard log costs.
Show answer & explanationAnswer & explanation
Correct answer: C. Use S3 Lifecycle Policies with object tags: one tag for standard logs (30-day transition, 5-year deletion) and another tag for security incident logs (never expire).
S3 Lifecycle Policies with object tags allow for flexible, automated retention. By tagging standard logs for transition and deletion and tagging security incident logs with a 'never expire' policy, the team can manage diverse retention rules within the same bucket with minimal operational overhead.
Why the other options are wrong
- A. Manually moving logs is inefficient and prone to error, contradicting the 'automated solution' requirement.
- B. While Lambda could achieve this, it introduces custom code and operational overhead compared to the native S3 Lifecycle Policies, which are designed for this exact purpose.
- D. Configuring S3 Object Lock on the entire bucket for indefinite retention would prevent deletion of standard logs after 5 years, violating that part of the policy. Intelligent-Tiering is for optimizing costs based on changing access patterns, not for indefinite retention for specific files or complex deletion rules.
S3 Lifecycle Policies with Object Tags
S3 Lifecycle Policies automate the management of objects over their lifespan. Object tags enable applying different lifecycle rules to specific subsets of objects within a bucket, providing fine-grained retention and transition control.
- Automates transitions to different storage classes.
- Automates object expiration (deletion).
- Object tags allow multiple, distinct policies within one bucket.
Memory trick: Tags Tailor Timely Transitions & Terminations.