AWS Certified Machine Learning – SpecialtyMachine Learning Implementation and OperationsMedium
A financial institution is deploying a new credit risk assessment model. Before fully rolling it out to all customers, they want to compare its performance against the existing legacy model in a production environment using a small subset of live traffic. They need to ensure that the new model's predictions are captured for analysis without impacting the customer experience of the existing model. Which A/B testing strategy should they use?
- AMulti-armed bandit
- BBlue/Green deployment
- CShadow deployment
- DCanary release
Show answer & explanationAnswer & explanation
Correct answer: C. Shadow deployment
Shadow deployment routes live traffic to both the old and new models simultaneously, but only the old model's predictions are used for the actual response. The new model's predictions are logged for comparison, allowing for non-disruptive testing.
Why the other options are wrong
- A. Multi-armed bandit algorithms optimize traffic distribution based on performance, which is for exploration/exploitation, not initial comparison without impact.
- B. Blue/Green deployment switches all traffic at once to a new environment, which is too risky for initial comparison without impact.
- D. Canary release gradually rolls out the new model to a small percentage of users, directly impacting a subset of live traffic.
Shadow Deployment
A model deployment strategy where a new model runs in parallel with the current production model, processing live traffic, but its predictions are not used to respond to users.
- Allows non-disruptive testing and performance comparison.
- New model's predictions are logged for analysis.
- Minimizes risk by not impacting user experience.
Memory trick: Shadows watch, canaries test, blue/green flips, bandits learn.