Professional Data EngineerBuilding and operationalizing data processing systemsHard
A data engineering team is developing a new streaming pipeline to process real-time sensor data. The pipeline runs on Dataflow, and the processed data is eventually stored in BigQuery. The team needs to ensure that the Dataflow workers can securely communicate with BigQuery and other Google Cloud services without exposing credentials directly in the code or to the internet. They also want to restrict access based on the principle of least privilege. Which authentication and networking approach should they implement?
- AConfigure VPC Service Controls around the Dataflow job and BigQuery.
- BAssign a custom service account to the Dataflow job with specific IAM roles and enable Private Google Access.
- CEmbed service account keys directly in the Dataflow job code.
- DUse SSH tunneling from Dataflow workers to access BigQuery.
Show answer & explanationAnswer & explanation
Correct answer: B. Assign a custom service account to the Dataflow job with specific IAM roles and enable Private Google Access.
Assigning a custom service account with least privilege IAM roles to the Dataflow job ensures secure, managed authentication. Enabling Private Google Access (or using Private Service Connect) allows workers on a private network to reach Google Cloud services like BigQuery without traversing the public internet, enhancing security.
Why the other options are wrong
- A. VPC Service Controls enhance data exfiltration protection but do not primarily handle the core authentication for Dataflow workers to services; it's a perimeter security layer.
- C. Embedding service account keys is a security anti-pattern and highly discouraged as it exposes credentials.
- D. SSH tunneling is a complex, manual, and inefficient solution for Dataflow workers to communicate with managed Google Cloud services.
Service Account-based Auth with Private IP (Dataflow)
Using a Google Cloud service account with specific IAM roles for Dataflow worker authentication, combined with Private Google Access or Private Service Connect for secure, private network communication to GCP services.
- Managed identity for GCP services
- Principle of least privilege (IAM roles)
- Private network access to GCP APIs (no public internet)
Memory trick: Service accounts on private roads, keep data safe.