Professional Data EngineerBuilding and operationalizing data processing systemsEasy
A global e-commerce company uses BigQuery for its analytical data warehouse. They have a daily batch job that processes customer order data, performs complex transformations, and loads the results into another BigQuery table. This job is part of a larger pipeline that includes data ingestion from multiple sources, data quality checks, and report generation. The company wants to ensure reliable, scheduled execution of this entire pipeline with dependencies between tasks. Which Google Cloud service would be most appropriate for orchestrating this workflow?
- ACloud Pub/Sub
- BCloud Composer
- CDataflow
- DBigQuery Data Transfer Service
Show answer & explanationAnswer & explanation
Correct answer: B. Cloud Composer
Cloud Composer, built on Apache Airflow, is designed for orchestrating complex workflows with dependencies, scheduling, and monitoring capabilities, making it ideal for managing the described data pipeline.
Why the other options are wrong
- A. Cloud Pub/Sub is a messaging service for real-time data ingestion, not workflow orchestration.
- C. Dataflow is a data processing service for batch and streaming data, not an orchestration tool for entire pipelines.
- D. BigQuery Data Transfer Service automates data movement from SaaS applications to BigQuery, not general workflow orchestration.
Cloud Composer
A fully managed workflow orchestration service built on Apache Airflow, enabling users to author, schedule, and monitor pipelines.
- Managed Apache Airflow
- Supports complex DAGs (Directed Acyclic Graphs)
- Integrates with many Google Cloud services
Memory trick: Composer conducts the data orchestra.