AWS Certified Machine Learning – SpecialtyMachine Learning Implementation and OperationsMedium
A retail company uses a machine learning model to predict customer churn. The model was trained on historical data and deployed as a SageMaker real-time endpoint. Over time, the company observes that the model's accuracy on new data has significantly degraded, even though the input data schema remains consistent. The distribution of customer behavior patterns has subtly shifted due to new market trends. What type of model monitoring issue is the company most likely experiencing?
- AFeature Skew
- BData Quality Drift
- CModel Performance Degradation (due to infrastructure issues)
- DConcept Drift
Show answer & explanationAnswer & explanation
Correct answer: D. Concept Drift
Concept drift occurs when the underlying relationship between input features and the target variable changes over time, leading to degraded model performance even with consistent data schema. The shift in customer behavior patterns is a classic example.
Why the other options are wrong
- A. Feature Skew typically refers to differences in feature distributions between training and serving datasets, or between features themselves, but 'concept drift' more accurately captures the change in the underlying target relationship.
- B. Data Quality Drift refers to issues like missing values, incorrect data types, or outliers, which are not described here as the schema is consistent.
- C. Model Performance Degradation due to infrastructure issues would manifest as latency or availability problems, not a degradation in prediction accuracy due to changes in underlying patterns.
Concept Drift
A phenomenon in machine learning where the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways.
- Causes model accuracy to degrade over time
- Often requires model retraining with newer data
- Can be gradual or sudden
- Different from data drift (input feature distribution changes)
Memory trick: Drifting Concepts Degrade Accuracy.