Professional Data EngineerDesigning data processing systemsHard
A data analytics company has developed a proprietary machine learning model for fraud detection. The model needs to be updated daily with new training data, which amounts to several terabytes. The training process is computationally intensive and takes about 6 hours. The company wants to run this training job efficiently, ensuring that resources are provisioned only when needed and scaled appropriately for the workload, minimizing idle costs. Which Google Cloud service is most suitable for orchestrating and executing this daily batch ML training job?
- AKubernetes Engine (GKE)
- BCloud Composer
- CCloud Run
- DCloud Functions
Show answer & explanationAnswer & explanation
Correct answer: B. Cloud Composer
Cloud Composer (managed Apache Airflow) is an excellent choice for orchestrating complex, scheduled workflows like daily ML model training. It allows defining DAGs (Directed Acyclic Graphs) to manage dependencies, retry logic, and conditional execution. It integrates well with other Google Cloud services (like Dataflow, BigQuery, Vertex AI) for executing the actual training steps and ensures resources are provisioned on demand for the duration of the job, minimizing idle costs.
Why the other options are wrong
- A. GKE provides a managed Kubernetes environment, which is powerful for deploying containerized applications, but it requires more operational overhead for workflow orchestration compared to a dedicated service like Cloud Composer.
- C. Cloud Run is for deploying stateless containers that scale to zero, suitable for web services or APIs, but not designed for orchestrating complex, stateful batch workflows with dependencies.
- D. Cloud Functions are for short-lived, event-driven tasks, not for orchestrating long-running, complex ML training workflows.
Cloud Composer for Workflow Orchestration
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow, enabling users to author, schedule, and monitor pipelines programmatically.
- Uses Python DAGs for workflow definition.
- Provides robust scheduling and monitoring.
- Integrates with various Google Cloud services.
- Ideal for complex ETL, ML pipelines, and batch jobs.
Memory trick: Composer 'conducts' your data orchestra.