AWS Certified Machine Learning – SpecialtyModelingEasy
A machine learning engineer is developing a model to predict the probability of equipment failure in a factory. The dataset is highly imbalanced, with very few instances of actual failures compared to normal operation. The initial model, a Logistic Regression, achieves a high accuracy of 99.5% but completely fails to predict any failures, resulting in zero recall for the positive class. Which evaluation metric should the engineer prioritize to get a more meaningful assessment of the model's ability to detect failures?
- APrecision
- BRecall
- CF1-Score
- DAccuracy
Show answer & explanationAnswer & explanation
Correct answer: B. Recall
Recall (also known as sensitivity) measures the proportion of actual positive cases that were correctly identified. In the context of predicting rare equipment failures, it is crucial to maximize the detection of actual failures, making recall the most important metric.
Why the other options are wrong
- A. Precision measures the proportion of predicted positive cases that were actually positive. While important, a high precision with zero recall means the model detected no failures, making precision irrelevant in this scenario.
- C. F1-Score is the harmonic mean of precision and recall. While useful, if recall is zero, the F1-score will also be zero, indicating no true positives were found. Recall is a more direct focus when the goal is to find all positive cases.
- D. Accuracy is misleading with imbalanced datasets because the model can achieve high accuracy by simply predicting the majority class (no failure) all the time.
Recall (Sensitivity)
The proportion of actual positive instances that were correctly identified by the model. It measures the model's ability to find all positive samples.
- Calculated as True Positives / (True Positives + False Negatives).
- Crucial for imbalanced datasets where missing positive cases is costly.
- Also known as sensitivity or True Positive Rate.
Memory trick: When positives are rare, recall's the flair, to catch every failure, you must be aware.