AWS Certified Machine Learning – SpecialtyModelingEasy
A financial institution is developing a machine learning model to predict loan default. The dataset is highly imbalanced, with only 2% of loans resulting in default. The team initially trained a model that achieved 98% accuracy. However, upon closer inspection, they found that the model simply predicts 'no default' for all cases. The business stakeholders are most concerned about identifying as many actual defaulting loans as possible to intervene early. Which evaluation metric should the team prioritize to address the stakeholders' primary concern?
- AAccuracy
- BRecall (Sensitivity)
- CF1-Score
- DPrecision
Show answer & explanationAnswer & explanation
Correct answer: B. Recall (Sensitivity)
The stakeholders' primary concern is to identify as many actual defaulting loans as possible. This directly corresponds to maximizing the True Positive Rate, which is the definition of Recall (Sensitivity). The initial model's high accuracy is misleading because of the class imbalance.
Why the other options are wrong
- A. Accuracy is misleading in imbalanced datasets as it can be high even if the model predicts only the majority class.
- C. F1-Score is the harmonic mean of precision and recall, providing a balance. While useful, it doesn't prioritize finding 'as many actual defaulting loans as possible' as directly as recall does.
- D. Precision focuses on the accuracy of positive predictions (minimizing false positives), but the goal is to find all actual positives, not just accurate ones.
Recall (Sensitivity) in Imbalanced Data
The proportion of actual positive cases that are correctly identified by the model. It is crucial when the cost of False Negatives (missing a positive case) is high, especially in imbalanced datasets.
- Calculated as True Positives / (True Positives + False Negatives).
- Prioritizes minimizing False Negatives.
- Essential for fraud detection, disease diagnosis, and anomaly detection.
Memory trick: Don't Miss a Single Default: Recall is Your Call.