Professional Data EngineerManaging and securing dataMedium
A research institution is building a data lake on Google Cloud using Cloud Storage. They frequently receive new datasets from various partners, which are stored in different buckets. To ensure data governance, they need to apply consistent metadata (e.g., data source, department, sensitivity level) to all incoming data files for easier discovery and regulatory compliance. How should the institution implement this metadata management?
- AImplement Cloud Data Catalog to centralize metadata management and discovery.
- BUse Cloud Functions triggered by object creation events to write metadata to object names.
- CManually add custom metadata to each object using gsutil or the Cloud Console.
- DStore metadata in a separate BigQuery table and link it to Cloud Storage objects using file paths.
Show answer & explanationAnswer & explanation
Correct answer: A. Implement Cloud Data Catalog to centralize metadata management and discovery.
Cloud Data Catalog is a fully managed, scalable metadata management service that allows organizations to discover, manage, and understand all their data. It automatically ingests technical metadata and allows for custom business metadata, making it ideal for consistent governance and discovery across a data lake.
Why the other options are wrong
- B. Writing metadata to object names is restrictive in length, format, and searchability, and doesn't provide rich, structured metadata for governance or discovery.
- C. Manually adding metadata is prone to errors, inconsistent, and not scalable for large volumes of incoming data.
- D. Storing metadata in a separate BigQuery table requires custom integration, potential synchronization issues, and doesn't offer the native discovery and governance features of Data Catalog.
Cloud Data Catalog
Cloud Data Catalog is a fully managed, scalable metadata management service that helps organizations discover, manage, and understand all their data across Google Cloud and hybrid environments.
- Automatically ingests technical metadata.
- Allows adding custom business metadata (tags).
- Enables data discovery and searchability.
- Supports data governance and compliance efforts.
Memory trick: Data Catalog is your map, for data discovery, metadata's trap!