AWS Certified Data Engineer – AssociateData Operations and MonitoringMedium
A data engineering team is responsible for a critical data pipeline that extracts data from an on-premises database, transforms it using AWS Glue, and loads it into an Amazon Redshift data warehouse. The pipeline runs daily. Recently, stakeholders have reported that the reports generated from Redshift are occasionally missing data, but the pipeline status in AWS Glue always shows 'Succeeded'. The team needs to implement a solution to detect these data anomalies proactively and receive notifications when they occur. Which AWS service should the team use to address this requirement efficiently?
- AAWS Lambda functions to periodically query Redshift and compare row counts with source data.
- BAWS Glue Data Quality (Deequ) rules integrated into the Glue jobs to validate data.
- CAWS CloudWatch Alarms to monitor Glue job logs for specific error patterns.
- DAmazon EventBridge rules to trigger a notification when a Glue job completes successfully.
Show answer & explanationAnswer & explanation
Correct answer: B. AWS Glue Data Quality (Deequ) rules integrated into the Glue jobs to validate data.
AWS Glue Data Quality (Deequ) is specifically designed for data quality assessment within data pipelines, allowing the definition of rules to validate data and detect anomalies. Integrating this directly into the Glue jobs will proactively identify missing data or other quality issues.
Why the other options are wrong
- A. Lambda functions querying Redshift and comparing row counts is a reactive and less efficient approach. It requires custom development and might not catch all types of data quality issues beyond simple row counts, and it runs after the data is already in Redshift.
- C. CloudWatch Alarms can monitor logs but are less effective at detecting data content anomalies than dedicated data quality tools. They'd primarily catch execution errors, not data integrity issues when the job 'succeeds'.
- D. EventBridge rules triggering on successful job completion would only confirm the job ran, not that the data it processed was correct or complete.
AWS Glue Data Quality (Deequ)
A feature within AWS Glue that allows users to define and run data quality rules directly within their Glue ETL jobs to validate data at various stages of the pipeline.
- Open-source Deequ library integrated into Glue.
- Supports various data quality checks (e.g., completeness, uniqueness, consistency, validity).
- Can be configured to fail jobs or trigger alerts based on rule violations.
Memory trick: Deequ Detects Data Discrepancies, Delivering Dependable Data.