Microsoft Azure AI Fundamentals (AI-900)Describe fundamental principles of machine learning on AzureHard
A machine learning engineer is evaluating a binary classification model for predicting a rare disease. The model has an accuracy of 99%, but further analysis reveals that the disease prevalence in the dataset is only 1%. This high accuracy is primarily due to the model correctly identifying healthy individuals, while its performance on actual disease cases is poor. Which concept explains why accuracy alone is misleading in this scenario?
- ABias-Variance Tradeoff
- BOverfitting
- CAccuracy Paradox
- DRecall
Show answer & explanationAnswer & explanation
Correct answer: C. Accuracy Paradox
The Accuracy Paradox occurs when a high accuracy score is achieved by a model that simply predicts the majority class, especially in highly imbalanced datasets. In this case, a 99% accuracy can be achieved by always predicting 'healthy', completely missing the 1% rare disease cases.
Why the other options are wrong
- A. Bias-Variance Tradeoff relates to model complexity and generalization, not specifically misleading accuracy in imbalanced datasets.
- B. Overfitting implies good performance on training data but poor on unseen data, whereas here the accuracy is high even on healthy cases, but fails for the minority class.
- D. Recall (or sensitivity) is a metric that would highlight the model's failure for the minority class, but it is not the concept describing why accuracy is misleading.
Accuracy Paradox
A phenomenon in machine learning where a high overall accuracy score can be misleading, particularly in datasets with highly imbalanced classes. A model can achieve high accuracy by simply predicting the majority class, while performing poorly on the minority class.
- Occurs in datasets where one class significantly outnumbers others.
- A simple 'always predict majority' model can yield high accuracy.
- Requires other metrics like Precision, Recall, F1-score, or AUC-ROC for proper evaluation.
Memory trick: When the scales are tipped, accuracy can lie.