Professional Cloud ArchitectAnalyze and optimize technical and business processesHard
A global logistics company wants to build a recommendation engine for optimizing delivery routes. This engine needs to process historical delivery data (petabytes in BigQuery), real-time traffic updates (streaming via Pub/Sub), and driver location data (from IoT devices). The goal is to generate optimal routes with minimal latency for drivers, requiring complex graph algorithms and machine learning models. Which Google Cloud service is best suited for orchestrating and executing this complex, hybrid (batch and streaming) data processing and machine learning pipeline?
- ACloud Dataflow for its unified stream and batch processing capabilities.
- BCloud Dataproc for running managed Apache Spark clusters at scale.
- CVertex AI Workbench for developing and deploying machine learning models.
- DCloud Composer for orchestrating Apache Airflow workflows.
Show answer & explanationAnswer & explanation
Correct answer: D. Cloud Composer for orchestrating Apache Airflow workflows.
Cloud Composer (managed Apache Airflow) is specifically designed for orchestrating complex workflows, including hybrid batch/streaming data processing and machine learning pipelines. It can schedule, monitor, and manage dependencies across various Google Cloud services (BigQuery, Pub/Sub, Dataflow, Vertex AI), making it ideal for the end-to-end route optimization engine.
Why the other options are wrong
- A. Cloud Dataflow is excellent for executing stream and batch processing jobs, but it is a processing engine, not an orchestrator for an entire end-to-end pipeline involving multiple services and dependencies.
- B. Cloud Dataproc is for running managed Spark/Hadoop clusters, suitable for processing data, but not for orchestrating the entire workflow across disparate services and managing dependencies like Airflow.
- C. Vertex AI Workbench is for ML model development and deployment, not for orchestrating the entire data ingestion, processing, and model serving pipeline.
Cloud Composer (Apache Airflow)
A fully managed workflow orchestration service built on Apache Airflow, enabling users to author, schedule, and monitor complex pipelines programmatically.
- Uses Directed Acyclic Graphs (DAGs) to define workflows.
- Integrates deeply with other Google Cloud services.
- Ideal for ETL, machine learning pipelines, and complex data processing.
- Provides robust scheduling, monitoring, and dependency management.
Memory trick: Composer orchestrates the whole symphony, batch and stream!