Microsoft Azure AI Fundamentals (AI-900)Describe fundamental principles of machine learning on AzureMedium
An AI developer is building a model to predict the likelihood of a customer churning (canceling their subscription). The model's output is a probability score between 0 and 1. Which evaluation metric would be most appropriate to assess the model's ability to distinguish between churning and non-churning customers across various probability thresholds?
- AR-squared
- BPrecision
- CMean Squared Error (MSE)
- DArea Under the Receiver Operating Characteristic Curve (AUC-ROC)
Show answer & explanationAnswer & explanation
Correct answer: D. Area Under the Receiver Operating Characteristic Curve (AUC-ROC)
AUC-ROC is specifically designed for binary classification problems where the output is a probability. It measures the model's ability to distinguish between positive and negative classes across all possible classification thresholds, making it robust and widely used for imbalanced datasets common in churn prediction.
Why the other options are wrong
- A. R-squared is also for regression problems, indicating the proportion of variance in the dependent variable predictable from the independent variables.
- B. Precision measures the proportion of true positive predictions among all positive predictions, but doesn't capture performance across thresholds or overall discriminative power as comprehensively as AUC-ROC.
- C. MSE is used for regression problems, measuring the average squared difference between predicted and actual continuous values.
AUC-ROC
Area Under the Receiver Operating Characteristic Curve; an evaluation metric for binary classification models that measures the model's ability to distinguish between classes across all possible thresholds.
- Ranges from 0 to 1; higher is better.
- 1.0 indicates perfect classification, 0.5 indicates random guessing.
- Robust to imbalanced datasets.
- Considers both true positive rate (sensitivity) and false positive rate (1-specificity).
Memory trick: AUC-ROC curves show 'how much' a model 'Rocks' at distinguishing.