Microsoft Azure AI Fundamentals (AI-900)Describe fundamental principles of machine learning on AzureHard
An AI developer is building a model to predict the likelihood of a customer churning (canceling their subscription). The model outputs a probability score between 0 and 1. To evaluate the model's ability to distinguish between churners and non-churners across all possible classification thresholds, which metric is most appropriate?
- AMean Absolute Error (MAE)
- BAccuracy
- CArea Under the Receiver Operating Characteristic (ROC) Curve (AUC-ROC)
- DF1-Score
Show answer & explanationAnswer & explanation
Correct answer: C. Area Under the Receiver Operating Characteristic (ROC) Curve (AUC-ROC)
AUC-ROC is specifically designed to evaluate the performance of binary classification models across all possible classification thresholds. It measures the model's ability to distinguish between positive and negative classes, regardless of the chosen threshold, which is crucial when dealing with probability outputs.
Why the other options are wrong
- A. Mean Absolute Error (MAE) is a regression metric used for continuous predictions, not for evaluating binary classification probabilities' discriminative power.
- B. Accuracy is threshold-dependent and can be misleading with imbalanced datasets.
- D. F1-Score is also threshold-dependent and provides a balance between precision and recall at a specific threshold.
AUC-ROC
Area Under the Receiver Operating Characteristic (ROC) Curve. It's a performance measurement for classification problems at various threshold settings.
- Measures the ability of a model to distinguish between classes.
- Evaluates performance across all possible classification thresholds.
- A value of 1.0 indicates a perfect classifier; 0.5 indicates a random classifier.
Memory trick: Beyond Simple Accuracy: ROC Reveals True Power