Professional Data EngineerOperationalizing machine learning modelsMedium
A retail company wants to develop a recommendation engine using machine learning. The data scientists have built a TensorFlow model that needs to be trained on a very large dataset (petabytes) stored in BigQuery. The training process is computationally intensive and requires distributed processing. After training, the model will be deployed for online predictions. Which Google Cloud service combination should the data engineering team use to efficiently train and deploy this model?
- AVertex AI Training for training, Vertex AI Endpoints for deployment, Cloud Storage for data
- BCompute Engine for training, Kubernetes Engine for deployment, Cloud SQL for data
- CCloud Composer for orchestration, Dataflow for training, Cloud Functions for deployment
- DDataproc for training, App Engine for deployment, Bigtable for data
Show answer & explanationAnswer & explanation
Correct answer: A. Vertex AI Training for training, Vertex AI Endpoints for deployment, Cloud Storage for data
Vertex AI Training is optimized for large-scale, distributed ML model training, especially with TensorFlow and BigQuery data. Vertex AI Endpoints provides a managed service for deploying models for online predictions. Cloud Storage is a common choice for storing training data artifacts.
Why the other options are wrong
- B. While possible, Compute Engine requires significant manual setup for distributed training. Kubernetes Engine can deploy, but Vertex AI Endpoints is more streamlined for ML models. Cloud SQL is not suitable for petabyte-scale ML training data.
- C. Dataflow is for data processing, not direct ML training. Cloud Functions are for serverless functions, not ideal for ML model deployment at scale.
- D. Dataproc is for Apache Spark/Hadoop, not the primary choice for TensorFlow distributed training. App Engine is for web applications, not model deployment. Bigtable is a NoSQL database, not ideal for relational BigQuery data for training.
Vertex AI Platform
Google Cloud's unified platform for machine learning development, encompassing data preparation, model training, deployment, and monitoring.
- Offers managed services for ML lifecycle.
- Supports various ML frameworks.
- Scales for large datasets and complex models.
Memory trick: From vast data to live predictions, Vertex AI's got the full ML pipeline covered.