Microsoft Azure AI Fundamentals (AI-900)Describe fundamental principles of machine learning on AzureHard
A machine learning engineer is evaluating a binary classification model for predicting a rare disease. The dataset is highly imbalanced, with only 1% of samples belonging to the positive class (disease present). The model achieves 99% accuracy. However, upon further inspection, it is discovered that the model simply predicts 'no disease' for all cases. What term best describes this model's performance?
- AHigh Precision, Low Recall
- BHigh Accuracy, but a poor model
- CLow Precision, High Recall
- DLow F1-Score, High Accuracy
Show answer & explanationAnswer & explanation
Correct answer: B. High Accuracy, but a poor model
In highly imbalanced datasets, a model can achieve deceptively high accuracy by simply predicting the majority class. While numerically 99% accurate, a model predicting 'no disease' for all cases is useless for identifying the rare disease and therefore a poor model, highlighting accuracy's weakness with imbalance.
Why the other options are wrong
- A. This model would have undefined precision (division by zero) if it never predicts positive, or very low precision if it makes some incorrect positive predictions. Recall would be 0.
- C. This is incorrect; recall would be 0 as it finds no positive cases.
- D. While the F1-Score would indeed be low (0) because recall is 0, the more encompassing and critical description is that despite high accuracy, it's a poor model due to its inability to detect the target class.
Accuracy Paradox
A phenomenon where a model can achieve high accuracy on an imbalanced dataset by simply predicting the majority class, making the model practically useless for the minority class.
- Occurs with highly skewed class distributions.
- Accuracy alone is a misleading metric in such cases.
- Requires other metrics like Precision, Recall, F1-Score, AUC-ROC to assess true performance.
Memory trick: Accuracy Tricks: Don't Fall for the Majority