A machine learning engineer is deploying a model that predicts manufacturing defects. The model outputs a probability score for each product. To decide which products to send for inspection, a threshold needs to be set on this probability score. Changing this threshold will affect the trade-off between identifying actual defects and incorrectly flagging good products. Which graphical tool is most appropriate for visualizing and choosing an optimal threshold based on this trade-off?
- AHistogram
- BROC Curve
- CConfusion Matrix
- DScatter Plot
Show answer & explanationAnswer & explanation
Correct answer: B. ROC Curve
An ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (Recall) against the False Positive Rate at various threshold settings. This directly visualizes the trade-off between identifying actual defects (True Positives) and incorrectly flagging good products (False Positives) as the decision threshold changes, making it ideal for selecting an optimal threshold.
Why the other options are wrong
- A. A histogram shows the distribution of a single variable, not the trade-off between error types.
- C. A confusion matrix provides a snapshot of performance at a single threshold, but doesn't visualize the trade-off across varying thresholds.
- D. A scatter plot shows the relationship between two variables, not suitable for threshold selection in this context.
ROC Curve (Receiver Operating Characteristic)
An ROC curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold settings.
- TPR is Recall; FPR is (1 - Specificity).
- Used to evaluate classifier performance and select optimal thresholds.
- Area Under the Curve (AUC-ROC) measures overall model performance.
- A curve closer to the top-left corner indicates better performance.
Memory trick: ROC: Reveals the 'Range Of Choices' for your threshold, showing the trade-off.