A data engineering team manages a complex data pipeline using AWS Step Functions, which orchestrates various AWS Glue jobs and Lambda functions. The pipeline processes sensitive financial data, and there is a strict requirement to encrypt all temporary data generated during Glue job execution. This temporary data includes shuffle data, spill data, and intermediate results stored on disk or in S3. The team needs to implement a solution that ensures this temporary data is encrypted both at rest and in transit, with customer-managed keys (CMKs) from AWS Key Management Service (KMS), applied consistently across all Glue jobs in the pipeline. Which approach should they use?
- ACreate an AWS Glue Security Configuration that specifies the KMS CMK for S3 and CloudWatch Logs, and apply it to all relevant Glue jobs.
- BConfigure the S3 bucket policies for the Glue temporary directory to enforce KMS encryption for all objects.
- CEnable 'encryption at rest' for the Glue job's security configuration, specifying the KMS CMK for S3 and CloudWatch Logs.
- DModify each Glue job script to explicitly encrypt temporary files using the AWS KMS SDK before writing them.
Show answer & explanationAnswer & explanation
Correct answer: A. Create an AWS Glue Security Configuration that specifies the KMS CMK for S3 and CloudWatch Logs, and apply it to all relevant Glue jobs.
An AWS Glue Security Configuration provides a centralized way to manage encryption settings for Glue jobs. By creating a security configuration that specifies a KMS CMK for S3 (for temporary data) and CloudWatch Logs, and then applying this configuration to all relevant Glue jobs, the team can ensure consistent encryption of temporary data both at rest and in transit (via S3 and CloudWatch interactions) with minimal effort.
Why the other options are wrong
- B. S3 bucket policies enforce encryption for S3 objects, but a Glue Security Configuration provides a more comprehensive and integrated way to manage encryption for *all* temporary data, including local disk spills and CloudWatch Logs, within the Glue job context.
- C. This option is partially correct but incomplete. While 'encryption at rest' is part of it, the most effective way to apply it consistently across multiple jobs and include both S3 and CloudWatch Logs with CMKs is through a dedicated Glue Security Configuration.
- D. Modifying each job script is highly inefficient, error-prone, and adds significant operational overhead, especially for ensuring consistent CMK usage and managing encryption for all types of temporary data (e.g., Spark shuffle files).
Glue Security Configuration
An AWS Glue Security Configuration centralizes encryption settings for Glue jobs, allowing consistent application of KMS CMKs for temporary S3 data, CloudWatch Logs, and Job Bookmarks.
- Centralized encryption management for Glue jobs.
- Applies to S3 temporary data, CloudWatch Logs, Job Bookmarks.
- Ensures consistent use of KMS CMKs.
Memory trick: Glue's Security Config is the 'Key Master' for temporary data.