Professional Data EngineerBuilding and operationalizing data processing systemsHard
A data engineering team is developing a new streaming pipeline to process real-time sensor data from IoT devices using Dataflow. The pipeline needs to perform complex aggregations and transformations, and then write the results to a BigQuery table. The Dataflow job is running in a private network, and BigQuery is also configured with Private Google Access. To ensure secure and private communication between the Dataflow workers and BigQuery, what authentication and networking setup is required?
- APublic IP addresses for Dataflow workers with API Key authentication.
- BDataflow workers with public IP addresses and a VPN to BigQuery.
- CService account-based authentication for Dataflow workers with Private Google Access enabled for the VPC network.
- DStatic IP addresses for Dataflow workers and whitelisting BigQuery IP ranges.
Show answer & explanationAnswer & explanation
Correct answer: C. Service account-based authentication for Dataflow workers with Private Google Access enabled for the VPC network.
Service account-based authentication provides secure, granular access control for Dataflow workers to BigQuery. Private Google Access enabled on the VPC network allows Dataflow workers in a private subnet to securely communicate with BigQuery APIs without traversing the public internet.
Why the other options are wrong
- A. Public IP addresses and API Keys are insecure and violate the private networking requirement.
- B. VPNs are for connecting to on-premises networks; Dataflow workers should use Private Google Access for GCP services within Google's network.
- D. Static IP addresses are not scalable for Dataflow workers, and whitelisting IP ranges is less secure and manageable than Private Google Access.
Service Account Auth with Private Google Access
This setup enables secure and private communication between GCP resources (like Dataflow workers in a private VPC) and Google APIs (like BigQuery), without exposing traffic to the public internet.
- Service accounts provide identity for applications to authenticate to GCP services.
- Private Google Access allows instances in a private subnet to reach Google APIs.
- Ensures data privacy and security.
- Eliminates the need for public IPs or VPNs for GCP-to-GCP communication.
Memory trick: Service accounts unlock the private access path.