Professional Data EngineerManaging and securing dataHard
A financial institution is building a data warehouse in BigQuery. They need to ensure that data lineage is tracked for all transformations, from ingestion to final aggregated reports, to meet strict auditing and compliance requirements (e.g., BCBS 239). The lineage information must include schema changes, transformation logic, and responsible users. Which Google Cloud service should you integrate to achieve this comprehensive data lineage tracking?
- ACloud Data Catalog with Data Lineage enabled.
- BCloud Monitoring to alert on schema changes in BigQuery.
- CDataflow Flex Templates for all data transformations.
- DCloud Audit Logs to track all BigQuery API calls.
Show answer & explanationAnswer & explanation
Correct answer: A. Cloud Data Catalog with Data Lineage enabled.
Cloud Data Catalog with its Data Lineage feature is specifically designed to automatically discover and visualize data lineage across various Google Cloud services, including BigQuery. It tracks schema changes, transformation logic, and links to jobs/users, providing the comprehensive audit trail required for compliance.
Why the other options are wrong
- B. Cloud Monitoring tracks metrics and creates alerts, which can be useful for operational monitoring but does not provide a comprehensive data lineage solution.
- C. While Dataflow Flex Templates can be used for transformations, they don't inherently provide a centralized, automatically tracked data lineage system across all BigQuery operations and other services.
- D. Cloud Audit Logs track API calls, which is part of auditing, but it does not automatically construct or visualize data lineage (transformation logic, schema evolution) across multiple steps in a pipeline.
Cloud Data Catalog Data Lineage
Cloud Data Catalog's Data Lineage feature automatically tracks and visualizes the flow of data through various Google Cloud services, including BigQuery, enabling transparency and auditability of data transformations.
- Automatically discovers lineage across BigQuery, Dataflow, Dataproc, etc.
- Visualizes data flow, schema changes, and transformation logic.
- Helps meet regulatory compliance requirements (e.g., BCBS 239).
- Integrates with the broader Data Catalog for metadata management.
Memory trick: For lineage so clear, Data Catalog is here, have no fear!