AWS Certified Machine Learning – SpecialtyModelingHard
A healthcare provider is developing a machine learning model to diagnose a specific medical condition from patient data. The dataset is highly imbalanced, with only 1% of patients having the condition. The goal is to ensure that the model is fair across different demographic groups (e.g., age, gender, ethnicity) and does not disproportionately misdiagnose any particular group. Which fairness metric is most appropriate for assessing the model's performance in this high-stakes, imbalanced scenario, focusing on equalizing true positive rates across groups?
- APredictive Parity
- BDemographic Parity
- COverall Accuracy
- DEqual Opportunity
Show answer & explanationAnswer & explanation
Correct answer: D. Equal Opportunity
Equal Opportunity focuses on ensuring that the true positive rate (recall) is equal across different groups. In a medical diagnosis scenario, this means that the model is equally good at identifying actual cases of the condition for all demographic groups, which is critical to avoid disproportionately missing diagnoses for certain populations, especially with a rare condition where false negatives are highly costly.
Why the other options are wrong
- A. Predictive Parity (or Positive Predictive Value parity) requires the precision to be equal across groups, meaning that when the model predicts positive, it's equally likely to be correct for all groups. This is different from ensuring equal detection of actual cases.
- B. Demographic Parity requires the positive prediction rate to be equal across groups, which can be problematic if the base rates of the condition differ between groups.
- C. Overall Accuracy can be misleading in imbalanced datasets and does not account for fairness across subgroups, which is the primary concern here.
Equal Opportunity (Fairness Metric)
A fairness metric that requires the true positive rate (recall) to be equal across different protected groups.
- Focuses on ensuring that the model is equally effective at identifying positive outcomes for all groups.
- Relevant when the cost of false negatives is high and fairness in 'being served' is paramount.
- Aims to prevent discrimination in access to benefits or detection of conditions.
Memory trick: Equal Opportunity means equal chances for true positives.