Google Cloud Digital LeaderGeneral knowledge of Google CloudHard
A retail company wants to use Google Cloud to analyze customer purchasing behavior to recommend products. They have a vast amount of historical transaction data in various formats (CSV, JSON, Avro) stored in Cloud Storage. They need a serverless service that can perform complex transformations and aggregations on this data before loading it into BigQuery for analysis, without managing any clusters. Which Google Cloud product should they use for the data transformation step?
- ACloud Pub/Sub
- BDataflow
- CDataproc
- DCloud Composer
Show answer & explanationAnswer & explanation
Correct answer: B. Dataflow
Dataflow is a fully managed, serverless service for executing Apache Beam pipelines for both batch and stream processing. It's ideal for complex data transformations and aggregations on large datasets from various sources, without requiring cluster management, perfectly fitting the scenario.
Why the other options are wrong
- A. Cloud Pub/Sub is a messaging service for real-time data ingestion, not for complex batch data transformations.
- C. Dataproc is a managed Apache Hadoop and Spark service, requiring cluster management, which goes against the 'without managing any clusters' requirement.
- D. Cloud Composer is a managed Apache Airflow service for orchestrating workflows, not for actual data transformation.
Google Cloud Dataflow
Dataflow is a fully managed, serverless service for executing Apache Beam pipelines for ETL, batch, and stream processing.
- Serverless and auto-scaling for both batch and streaming data.
- Supports complex data transformations and aggregations.
- Integrates with other Google Cloud data services like Cloud Storage and BigQuery.
Memory trick: Processing Data in GCP: Dataflow for Serverless, Dataproc for Hadoop, Composer for Orchestration.