AWS Certified Machine Learning – SpecialtyMachine Learning Implementation and OperationsMedium
A financial institution is deploying a new machine learning model to predict fraudulent transactions. To minimize the risk of a faulty model impacting live operations, they want to test the new model (Model B) by sending copies of live production traffic to it, while the existing production model (Model A) continues to serve all actual responses to users. The predictions of Model B will be logged and compared against Model A's predictions and actual outcomes offline. Which deployment strategy should they use?
- AA/B Testing
- BShadow Deployment
- CCanary Deployment
- DBlue/Green Deployment
Show answer & explanationAnswer & explanation
Correct answer: B. Shadow Deployment
Shadow deployment involves routing a copy of live production traffic to a new model version (Model B) while the existing production model (Model A) continues to handle all actual user requests. This allows the new model's performance to be evaluated with real-world data without affecting user experience, as its predictions are not used in production. This perfectly matches the requirement of comparing predictions offline without impacting live operations.
Why the other options are wrong
- A. A/B testing typically involves routing different *user segments* to different models, where each model's predictions are used in production for its segment.
- C. Canary deployment routes a small percentage of *actual* live traffic to the new model, meaning its predictions are used in production for that subset.
- D. Blue/Green deployment switches all live traffic to the new model at once after testing, potentially impacting all users if issues arise.
Shadow Deployment (ML)
A deployment strategy where a new model version receives a copy of live production traffic, but its predictions are not used in production, allowing for risk-free evaluation.
- Zero impact on live user experience.
- Evaluates new model with real-world data.
- Predictions are logged for offline analysis and comparison.
Memory trick: Shadow's Whisper, No Real Touch, Test in Stealth, Means So Much.