A data engineering team manages a critical data pipeline that processes sensitive customer information. The pipeline involves several AWS services, including Amazon S3 for storage, AWS Glue ETL jobs for transformations, and Amazon Redshift for analytical queries. The company has a strict compliance requirement to encrypt all data at rest and in transit. The team has already configured server-side encryption for S3 buckets and Amazon Redshift clusters. What is the most effective and secure way to ensure that data processed by AWS Glue ETL jobs is encrypted at rest and in transit within the Glue environment?
- AImplement a Virtual Private Cloud (VPC) endpoint for S3 and Redshift, and ensure all Glue network traffic is routed through the VPC to enforce encryption.
- BUse AWS Key Management Service (KMS) directly within Glue ETL job scripts to encrypt and decrypt data chunks as they are processed, ensuring end-to-end encryption.
- CConfigure a Security Configuration in AWS Glue that specifies encryption keys for S3 data, CloudWatch logs, and Job bookmarks, and associate it with the Glue ETL jobs.
- DEnable client-side encryption for all data processed by AWS Glue by modifying the ETL job scripts to use custom encryption libraries before writing to S3 or Redshift.
Show answer & explanationAnswer & explanation
Correct answer: C. Configure a Security Configuration in AWS Glue that specifies encryption keys for S3 data, CloudWatch logs, and Job bookmarks, and associate it with the Glue ETL jobs.
AWS Glue Security Configurations provide a centralized and managed way to enforce encryption for data at rest (S3, CloudWatch, Job bookmarks) and data in transit within the Glue environment. This approach is more secure and manageable than implementing custom encryption in job scripts or relying solely on VPC routing for encryption.
Why the other options are wrong
- A. VPC endpoints enhance security by keeping traffic within the AWS network but do not inherently enforce encryption for data at rest or within Glue's processing environment; they secure the network path.
- B. While KMS is used by Glue Security Configurations, directly integrating KMS calls into ETL scripts is overly complex and bypasses the managed encryption features provided by Glue.
- D. Implementing client-side encryption within job scripts is complex, error-prone, and not the recommended managed approach for Glue's internal encryption.
AWS Glue Security Configurations
AWS Glue Security Configurations allow you to specify encryption settings for data at rest (S3, CloudWatch logs, Job bookmarks) and data in transit within the Glue environment, using AWS KMS keys.
- Centralized encryption management for Glue.
- Encrypts S3 data at rest, CloudWatch logs, and Job bookmarks.
- Encrypts data in transit within Glue processing.
- Uses AWS KMS for key management.
Memory trick: Glue's Secure Config: Key to Protecting Data's Journey.