AWS Certified Machine Learning – SpecialtyMachine Learning Implementation and OperationsMedium
A machine learning engineer needs to deploy a new model that accurately predicts customer churn. Before fully replacing the existing model, the engineer wants to compare the performance of the new model against the current production model using real-time traffic, but only for a small percentage of users, without impacting the majority. Which SageMaker deployment strategy should the engineer use?
- ABlue/Green Deployment
- BBatch Transform
- CShadow Deployment
- DA/B Testing
Show answer & explanationAnswer & explanation
Correct answer: D. A/B Testing
A/B testing on SageMaker endpoints allows routing a small percentage of live traffic to a new model (variant) while the majority of traffic goes to the current model (production), enabling direct comparison of performance metrics with minimal user impact.
Why the other options are wrong
- A. Blue/Green deployment switches all traffic at once or gradually, but typically aims for a full cutover, not a controlled comparison with a small percentage of users for performance metrics.
- B. Batch Transform is for offline, large-scale inference and is not suitable for real-time traffic splitting and comparison.
- C. Shadow deployment sends production traffic to a new model in the background without affecting actual user responses, primarily for observation, not for actively serving a small percentage of users to compare live performance metrics.
A/B Testing (ML Deployment)
A deployment strategy where different versions of a machine learning model are served to different segments of live user traffic to compare their performance metrics directly.
- Routes a percentage of real-time traffic to new model.
- Allows direct comparison of model performance.
- Minimizes risk by limiting exposure of new model.
Memory trick: Comparing models is like a traffic light, directing users to different paths.