A machine learning engineer is training a neural network for a multi-class image classification task. The model is prone to making overconfident predictions, assigning very high probabilities to incorrect classes, especially when the input is ambiguous. This behavior makes the model unreliable for critical applications. To encourage the model to produce more calibrated and less overconfident probability estimates, which technique should the engineer apply during training?
- ABatch Normalization
- BLabel Smoothing
- CEarly Stopping
- DWeight Decay (L2 Regularization)
Show answer & explanationAnswer & explanation
Correct answer: B. Label Smoothing
Label Smoothing is a regularization technique that prevents a model from becoming overconfident by replacing hard labels (0 or 1) with soft labels during training. Instead of assigning a probability of 1 to the true class and 0 to others, it assigns a slightly lower probability (e.g., 0.9) to the true class and a small, non-zero probability (e.g., 0.1 / N-1) to the other classes, encouraging better calibration.
Why the other options are wrong
- A. Batch Normalization stabilizes and speeds up training by normalizing layer inputs, but it doesn't directly address overconfident predictions or improve calibration in the way label smoothing does.
- C. Early Stopping prevents overfitting by halting training when validation performance plateaus, but it doesn't inherently make the model's probability estimates less overconfident.
- D. Weight Decay (L2 Regularization) penalizes large weights to prevent overfitting, but it does not directly adjust the target probabilities to reduce overconfidence in predictions.
Label Smoothing
A regularization technique that replaces hard labels (0 or 1) with smoothed probability distributions during training, preventing the model from becoming overconfident and improving calibration.
- Reduces overconfidence in predictions.
- Improves model generalization.
- Adds a small amount of noise to the target labels.
Memory trick: Label smoothing, like a gentle hand, guides the model to understand, that certainty's a spectrum, not just a command.