Professional Data EngineerDesigning data processing systemsHard
A data engineering team is responsible for maintaining several critical data pipelines that feed a central data warehouse. They need a solution to ensure data quality and track data lineage across various transformations. Specifically, they want to automatically discover data assets, understand their relationships, and identify potential data quality issues before data reaches the production data warehouse. Which Google Cloud service is designed to address these data governance and data quality requirements?
- ADataplex
- BCloud Monitoring
- CCloud Data Catalog
- DDataflow
Show answer & explanationAnswer & explanation
Correct answer: A. Dataplex
Dataplex is a fully managed data lake management service that helps organizations curate, secure, monitor, and manage their data across data lakes, data warehouses, and data marts. It provides automated data discovery, metadata management, data quality monitoring, and data lineage tracking capabilities, making it ideal for comprehensive data governance.
Why the other options are wrong
- B. Cloud Monitoring is for observing metrics, logs, and events, not for data governance, data quality, or lineage tracking within data pipelines.
- C. Cloud Data Catalog is a metadata management service for data discovery and understanding, but it doesn't provide the comprehensive data quality and automated data lineage tracking of Dataplex.
- D. Dataflow is a data processing service for transformations and aggregations, not for data governance, quality, or lineage tracking as its primary function.
Dataplex for Data Governance
Google Cloud Dataplex is an intelligent data fabric that enables organizations to unify distributed data, manage, monitor, and govern data across data lakes, data warehouses, and data marts, providing capabilities for data discovery, quality, and lineage.
- Unified data management across diverse data stores.
- Automated data discovery and metadata management.
- Built-in data quality monitoring and remediation.
- Provides data lineage tracking for compliance and understanding.
Memory trick: Dataplex: Data's Perfect Ledger.