A pharmaceutical company is processing clinical trial data, which is highly sensitive and subject to strict regulatory audits. They need to ensure that all data transformations performed by their Dataflow pipelines are fully traceable and immutable. Any change to the data schema or transformation logic must be recorded and auditable. You need to design a solution that provides this level of traceability and immutability for Dataflow pipelines. Which approach is best?
- AVersion control Dataflow pipeline code using Cloud Source Repositories and implement CI/CD for deployments.
- BUse Cloud Logging to capture Dataflow job logs and store them in Cloud Storage.
- CStore input and output data for each Dataflow job in immutable Cloud Storage buckets.
- DImplement custom metadata tracking within the Dataflow pipeline to record transformation details and store it in BigQuery.
Show answer & explanationAnswer & explanation
Correct answer: A. Version control Dataflow pipeline code using Cloud Source Repositories and implement CI/CD for deployments.
Version controlling Dataflow pipeline code in Cloud Source Repositories ensures that every change to the transformation logic is tracked, auditable, and immutable. Implementing CI/CD (Continuous Integration/Continuous Deployment) ensures that only approved, versioned code is deployed, creating a traceable history of all schema and logic changes, which is critical for regulatory audits.
Why the other options are wrong
- B. Cloud Logging captures execution logs, but it doesn't track changes to the pipeline's code or schema, which is the source of transformation logic.
- C. Storing input/output data in immutable buckets ensures data immutability, but it doesn't provide traceability or immutability for the transformation logic *itself* or schema changes, which is the core of the question.
- D. Custom metadata tracking can record details, but it's prone to human error, can be bypassed, and doesn't inherently provide the immutability and auditability for the *logic changes* themselves, unlike version control systems.
Data Transformation Traceability
The ability to track and audit all changes to data transformation logic and schema, ensuring immutability and compliance.
- Version control for pipeline code.
- CI/CD for controlled deployments.
- Auditable history of logic changes.
Memory trick: To audit your data transformations, treat your code like a legal document: version it, sign it (CI/CD), and keep every draft.