AWS Certified AI PractitionerAI/ML and Generative AI FundamentalsMedium
A financial institution is implementing a machine learning model to detect fraudulent transactions. The model is trained on historical data where only a very small percentage of transactions are actually fraudulent. During evaluation, the model achieves 99.9% accuracy. However, a review by human experts reveals that many actual fraudulent transactions are still being missed. Which evaluation metric should the team prioritize to address this issue?
- AF1-Score
- BPrecision
- CAccuracy
- DRecall
Show answer & explanationAnswer & explanation
Correct answer: D. Recall
The problem states that many *actual fraudulent transactions are still being missed*. This indicates a high number of false negatives, meaning the model fails to identify positive cases. Recall (also known as Sensitivity) is the metric that measures the proportion of actual positive cases that are correctly identified by the model (True Positives / (True Positives + False Negatives)). Increasing recall will reduce missed fraudulent transactions.
Why the other options are wrong
- A. F1-Score is the harmonic mean of precision and recall. While a good overall measure, in this specific scenario where missing actual fraud is the primary concern, recall is the most direct metric to optimize.
- B. Precision measures the proportion of positive identifications that were actually correct (True Positives / (True Positives + False Positives)). While important, it doesn't directly address missing actual fraud.
- C. Accuracy can be misleading in imbalanced datasets; 99.9% accuracy might just mean the model correctly identifies most non-fraudulent transactions while missing fraud.
Recall (Sensitivity)
The proportion of actual positive cases that are correctly identified by the model. It measures the model's ability to find all relevant instances.
- Formula: TP / (TP + FN).
- High recall means fewer false negatives (missed positives).
- Important when the cost of missing a positive case is high (e.g., fraud, disease detection).
- Also known as Sensitivity or True Positive Rate.
Memory trick: Metrics tell what's true, false, and missed.