Google Cloud Digital LeaderData and AI with Google CloudHard
A data engineering team needs to ingest data from various sources, including on-premises databases, SaaS applications, and streaming sources, into BigQuery for analysis. They require a fully managed, serverless service that can orchestrate and automate complex data pipelines, including transformation and loading. Which Google Cloud service is the most suitable for building these ETL/ELT pipelines?
- ACloud Pub/Sub
- BDataproc
- CDataflow
- DCloud Data Fusion
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Data Fusion
Cloud Data Fusion is a fully managed, cloud-native data integration service built on open-source CDAP. It provides a graphical interface for building and managing ETL/ELT pipelines from various sources to BigQuery and other destinations, making it ideal for complex orchestrations.
Why the other options are wrong
- A. Cloud Pub/Sub is a messaging service for ingesting and delivering real-time event streams, not for orchestrating complex ETL/ELT pipelines.
- B. Dataproc is a managed service for Apache Hadoop and Spark, suitable for big data processing but requires more management for complex ETL orchestration compared to Data Fusion.
- C. Dataflow is excellent for stream and batch processing, but Cloud Data Fusion offers a more comprehensive and visual approach to orchestrating diverse ETL/ELT pipelines from varied sources.
Cloud Data Fusion
A fully managed, cloud-native data integration service built on open-source CDAP for building and managing ETL/ELT pipelines.
- Graphical interface for pipeline development
- Connects to a wide range of data sources and sinks
- Supports ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) patterns
Memory trick: Data Fusion brings all data pieces together.