Professional Data EngineerBuilding and operationalizing data processing systemsMedium
A data analytics team is migrating an on-premises data warehouse to Google Cloud. They have petabytes of historical data stored in various formats (CSV, JSON, Parquet) that need to be loaded into BigQuery for analysis. The team wants to perform initial data cleaning and transformation before loading, but they also need to support incremental daily loads. They prefer a serverless approach that minimizes operational overhead. Which Google Cloud service combination would be most suitable for this migration and ongoing data loading?
- ACloud SQL and Cloud Functions
- BCloud Storage (as a data lake) and Dataflow (batch and streaming)
- CCloud Spanner and Data Catalog
- DCloud Pub/Sub and Bigtable
Show answer & explanationAnswer & explanation
Correct answer: B. Cloud Storage (as a data lake) and Dataflow (batch and streaming)
Cloud Storage serves as a cost-effective data lake for storing raw historical data in various formats. Dataflow, in both batch mode for initial historical loads and streaming mode for incremental daily loads, provides serverless, scalable data cleaning and transformation before loading into BigQuery.
Why the other options are wrong
- A. Cloud SQL is a relational database, not suited for petabyte-scale data lake storage or complex transformations. Cloud Functions are for event-driven microservices, not large-scale data processing.
- C. Cloud Spanner is a globally distributed relational database, not a data lake. Data Catalog is for metadata management, not data processing or storage.
- D. Cloud Pub/Sub is for real-time messaging ingestion, not for storing petabytes of historical data. Bigtable is a NoSQL database for operational workloads, not a data warehouse or general-purpose data lake.
Data Lake with Cloud Storage & Dataflow
A common pattern for building a scalable data lake on Google Cloud using Cloud Storage for raw data storage and Dataflow for flexible, serverless ETL/ELT processing.
- Cloud Storage provides cost-effective, scalable object storage for raw data.
- Dataflow supports both batch and streaming transformations.
- Serverless operations minimize overhead.
Memory trick: Storage for the lake, Dataflow for the transformation stream.