AWS Certified AI PractitionerResponsible AIHard
A research team is developing an AI model to detect early signs of a rare disease. Due to the extreme scarcity of positive cases, the training dataset is heavily imbalanced, with very few examples of the disease. Simply maximizing overall accuracy results in a model that almost always predicts 'no disease.' To build a Responsible AI system, which metric should they primarily focus on to ensure the model can effectively identify positive cases?
- ARecall (Sensitivity)
- BAccuracy
- CF1-Score
- DPrecision
Show answer & explanationAnswer & explanation
Correct answer: A. Recall (Sensitivity)
In a scenario with a rare disease where failing to detect positive cases (false negatives) has severe consequences, Recall (Sensitivity) is the most critical metric. It measures the proportion of actual positive cases that were correctly identified, ensuring that few sick individuals are missed.
Why the other options are wrong
- B. Accuracy can be misleading with imbalanced datasets, as a model predicting 'no disease' for almost all cases would still have high accuracy if positive cases are rare.
- C. F1-Score is the harmonic mean of precision and recall, providing a balance, but for very rare and critical cases, maximizing recall is often prioritized, even at the cost of some precision.
- D. Precision measures the proportion of positive identifications that were actually correct, but a high precision model might miss many actual positive cases if it's too conservative.
Recall (Sensitivity)
Recall, also known as sensitivity, measures the proportion of actual positive cases that were correctly identified by an AI model. It is crucial when the cost of false negatives is high.
- Calculated as True Positives / (True Positives + False Negatives).
- High recall means fewer actual positive cases are missed.
- Important for rare disease detection, fraud detection, etc.
Memory trick: P.R.A.F. for Balanced Views