Professional Cloud ArchitectAnalyze and optimize technical and business processesMedium
A large retail company is migrating its on-premises data warehouse to Google Cloud BigQuery. They have petabytes of historical sales data stored in various formats (CSV, JSON, XML) across multiple on-premises databases and file systems. The data needs to be cleaned, transformed, and loaded into BigQuery on a daily basis. The company requires a fully managed, serverless solution that can handle large volumes of data and complex transformations without requiring extensive operational overhead. Which Google Cloud service should they use for this ETL process?
- ACloud Dataproc
- BA custom Compute Engine instance running Apache Spark
- CCloud SQL with federated queries to Cloud Storage
- DCloud Dataflow
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Dataflow
Cloud Dataflow is a fully managed, serverless service for executing Apache Beam pipelines, making it ideal for large-scale data processing, ETL, and complex transformations without operational overhead. It can handle diverse data formats and integrate with BigQuery.
Why the other options are wrong
- A. Cloud Dataproc is a managed Apache Hadoop and Spark service, but it's not fully serverless and requires more operational management than Dataflow.
- B. Running a custom Spark cluster on Compute Engine involves significant operational overhead and is not a fully managed, serverless solution.
- C. Cloud SQL is a relational database and federated queries are for querying external data, not for complex, large-scale ETL processes and transformations into BigQuery.
Cloud Dataflow
Cloud Dataflow is a fully managed, serverless service for executing Apache Beam pipelines, enabling reliable and expressive data processing for ETL, batch, and stream analytics.
- Fully managed and serverless, reducing operational overhead.
- Supports both batch and stream processing with Apache Beam.
- Scales automatically to handle large datasets and complex transformations.
Memory trick: Extract, Transform, Load: Data's Journey.