A financial institution processes sensitive customer transaction data daily. The data is generated in various on-premises systems and needs to be ingested into Google Cloud for analysis in BigQuery. Due to strict regulatory compliance requirements, the data must be encrypted at rest and in transit, and access must be tightly controlled and auditable. Furthermore, the ingestion process must be highly available and resilient to network disruptions. Which Google Cloud services and practices should the data engineer prioritize for a secure and reliable ingestion strategy?
- ASet up a Cloud VPN tunnel from on-premises to Google Cloud, use Cloud Pub/Sub for ingestion with default encryption, and configure BigQuery to use customer-managed encryption keys (CMEK).
- BUse `gcloud storage cp` with customer-managed encryption keys (CMEK) to upload files to Cloud Storage, then load into BigQuery.
- CImplement a Data Transfer Service job for scheduled transfers, ensuring all data is encrypted before leaving on-premises, and store in Cloud Storage with customer-supplied encryption keys (CSEK).
- DDeploy a custom application on Compute Engine to push data over public internet, encrypting data at the application layer, and store in BigQuery with default encryption.
Show answer & explanationAnswer & explanation
Correct answer: A. Set up a Cloud VPN tunnel from on-premises to Google Cloud, use Cloud Pub/Sub for ingestion with default encryption, and configure BigQuery to use customer-managed encryption keys (CMEK).
A Cloud VPN tunnel ensures secure, encrypted data transfer in transit from on-premises. Cloud Pub/Sub provides highly available and resilient ingestion, with default encryption in transit and at rest. Configuring BigQuery with CMEK ensures customer control over encryption keys for data at rest, meeting regulatory compliance for sensitive data and auditable access.
Why the other options are wrong
- B. `gcloud storage cp` is a manual or script-based process, not highly available or resilient for continuous ingestion from multiple sources. While CMEK is good for at-rest encryption, the transit part needs more robust solution.
- C. Data Transfer Service is for scheduled, bulk transfers, not real-time or near real-time ingestion from various systems. CSEK requires the customer to manage keys, which is fine, but the ingestion mechanism itself is not as robust or real-time as Pub/Sub, and the transit security needs explicit mention.
- D. Pushing data over the public internet, even with application-layer encryption, is less secure and less reliable than a private interconnect like Cloud VPN. Default encryption in BigQuery is not sufficient for requirements demanding customer control over keys (CMEK/CSEK).
Secure and Reliable On-Premises to GCP Ingestion
A strategy for transferring sensitive on-premises data to Google Cloud, emphasizing encrypted transit, highly available ingestion, and controlled encryption at rest to meet regulatory compliance.
- Cloud VPN/Interconnect for secure transit.
- Cloud Pub/Sub for resilient, scalable ingestion.
- CMEK/CSEK for controlled encryption at rest.
- Auditable access control (IAM).
Memory trick: VPN for the journey, Pub/Sub for the delivery, CMEK for the rest.