Professional Data EngineerDesigning data processing systemsMedium
A healthcare provider is building a new data lake on Google Cloud to store patient health records (PHR), medical images, and research data. The data lake must enforce strict data governance policies, including data quality checks, metadata management, and lineage tracking, to comply with HIPAA and other healthcare regulations. The provider needs a unified way to discover, manage, and govern diverse data assets across different Google Cloud services. Which Google Cloud service is best suited to meet these data governance requirements?
- ADataplex for unified data management and governance
- BBigQuery for schema enforcement and data cataloging
- CCloud Storage with custom Python scripts
- DDataproc for data processing and metadata extraction
Show answer & explanationAnswer & explanation
Correct answer: A. Dataplex for unified data management and governance
Dataplex is specifically designed as an intelligent data fabric that unifies distributed data, automates data quality, and provides centralized metadata management and lineage tracking across data lakes, data warehouses, and data marts. This makes it ideal for enforcing strict governance and compliance in a healthcare data lake.
Why the other options are wrong
- B. BigQuery is a data warehouse and can enforce schemas for its own tables, but it doesn't provide unified governance, cataloging, or lineage across an entire data lake spanning various services.
- C. Cloud Storage is a storage service; custom scripts would require significant development and maintenance to achieve comprehensive governance.
- D. Dataproc is a managed Apache Spark/Hadoop service for processing, not a dedicated data governance solution for metadata, quality, and lineage across a data lake.
Dataplex for Data Governance
A unified data fabric that centralizes the management, monitoring, and governance of distributed data across data lakes, data warehouses, and data marts, ensuring data quality and compliance.
- Unifies data across multiple Google Cloud services.
- Automates data quality checks and profiling.
- Provides centralized metadata management and data catalog.
- Enables data lineage tracking.
- Facilitates compliance with regulations like HIPAA.
Memory trick: To manage a complex data city, you need a central 'plex' that handles all the rules, maps, and quality control.