Professional Data EngineerOperationalizing machine learning modelsMedium
A data science team is building a pipeline to train a new time-series forecasting model daily. The pipeline involves data extraction from Cloud Storage, preprocessing with Dataflow, model training using custom Python code on Vertex AI Training, and model deployment to Vertex AI Endpoints. They need a robust way to orchestrate these steps, handle dependencies, and manage failures. Which Google Cloud service is most suitable for orchestrating this end-to-end ML pipeline?
- ACloud Build
- BCloud Functions
- CCloud Composer
- DCloud Scheduler
Show answer & explanationAnswer & explanation
Correct answer: C. Cloud Composer
Cloud Composer, based on Apache Airflow, is designed for orchestrating complex workflows, managing dependencies between tasks, retries, and monitoring, making it ideal for multi-step ML pipelines across various Google Cloud services.
Why the other options are wrong
- A. Cloud Build automates software builds and deployments, not complex data/ML pipelines.
- B. Cloud Functions are for executing single-purpose, event-driven code, not for orchestrating multi-step pipelines.
- D. Cloud Scheduler is for scheduling recurring jobs, but lacks workflow orchestration capabilities like dependency management.
Cloud Composer
A fully managed workflow orchestration service built on Apache Airflow, enabling users to create, schedule, and monitor pipelines across cloud and on-premises environments.
- Uses Directed Acyclic Graphs (DAGs) for workflows.
- Manages task dependencies and retries.
- Integrates with various Google Cloud services.
Memory trick: Composer's your conductor, making all pipeline instruments play in harmony.