AWS Certified Machine Learning – SpecialtyMachine Learning Implementation and OperationsMedium
A retail company wants to test two different versions of a recommendation model (Model A and Model B) in a production environment to determine which one generates higher customer engagement before fully rolling out the superior model. They need to direct a small percentage (e.g., 10%) of live traffic to Model B and the remaining traffic to Model A, collect performance metrics for both, and compare the results. Which deployment strategy is best suited for this scenario?
- ABlue/Green Deployment
- BShadow Deployment
- CCanary Deployment
- DRolling Update
Show answer & explanationAnswer & explanation
Correct answer: C. Canary Deployment
Canary deployment is ideal for this scenario. It allows routing a small percentage of live traffic to a new model (the 'canary') while the majority of traffic still goes to the stable version. This enables real-world testing and performance comparison without impacting most users, and if issues arise, the small percentage of traffic can be quickly reverted.
Why the other options are wrong
- A. Blue/Green deployment involves switching all traffic at once or gradually, but it's primarily for risk mitigation and rollback, not for A/B testing with a small percentage of traffic.
- B. Shadow deployment runs the new model in parallel but does not serve its predictions to live users, making it unsuitable for directly measuring customer engagement.
- D. Rolling updates sequentially update instances, which doesn't provide a clear A/B testing mechanism with controlled traffic split for comparison.
Canary Deployment (ML)
A deployment strategy where a new version of a model is released to a small subset of users or traffic, allowing its performance to be monitored and compared against the existing version before a full rollout.
- Minimizes risk by exposing new features to a small audience.
- Enables A/B testing and real-world performance validation.
- Allows for quick rollback if issues are detected.
Memory trick: Canary is like sending a scout bird to test the air.