Professional Data EngineerOperationalizing machine learning modelsEasy
A global e-commerce company is building an automated ML pipeline to generate personalized product recommendations. The pipeline involves data ingestion, preprocessing (feature engineering), model training, evaluation, and deployment. Each stage needs to be orchestrated, allowing for retry logic, conditional execution based on previous stage outcomes, and easy visualization of the pipeline's progress. Which Google Cloud service is specifically designed for building and managing such complex, multi-step ML workflows?
- ACloud Composer
- BCloud Run
- CVertex AI Workbench
- DCloud Functions
Show answer & explanationAnswer & explanation
Correct answer: A. Cloud Composer
Cloud Composer, based on Apache Airflow, is a fully managed workflow orchestration service. It is ideal for defining, scheduling, and monitoring complex, multi-step ML pipelines with features like retry logic, conditional execution, and DAG visualization.
Why the other options are wrong
- B. Cloud Run is a serverless platform for deploying containerized applications, not an orchestration service for pipelines.
- C. Vertex AI Workbench provides a Jupyter-based environment for ML development, not for orchestrating production pipelines.
- D. Cloud Functions are serverless functions for event-driven computing, not a full-fledged workflow orchestration tool.
Cloud Composer
A fully managed workflow orchestration service built on Apache Airflow, used to author, schedule, and monitor pipelines.
- Ideal for complex, multi-step data and ML pipelines.
- Provides DAG (Directed Acyclic Graph) visualization.
- Supports retry logic, conditional execution, and parallelization.
Memory trick: The 'Composer' conducts the ML pipeline like a symphony.