AWS Certified Machine Learning – SpecialtyModelingHard
A computer vision team is training a deep neural network for object detection in autonomous vehicles. They are using a large dataset of annotated images. During training, they observe that the model's performance on the validation set is significantly worse than on the training set, indicating overfitting. They also notice that the model makes very confident but incorrect predictions on some validation images. Which hyperparameter or technique, if tuned or applied correctly, is most likely to specifically address the issue of overconfident predictions and improve generalization by smoothing the model's output distribution?
- AApply Label Smoothing.
- BIncrease the number of training epochs.
- CDecrease the learning rate.
- DIncrease the batch size.
Show answer & explanationAnswer & explanation
Correct answer: A. Apply Label Smoothing.
Overfitting and overconfident predictions often go hand-in-hand. Label smoothing is a regularization technique that prevents the model from becoming too confident in its predictions by replacing hard labels (0 or 1) with softer targets (e.g., 0.1 and 0.9). This encourages the model to be less 'certain' and improves generalization, especially when labels might be noisy or ambiguous.
Why the other options are wrong
- B. Increasing epochs would likely exacerbate overfitting, leading to more confident incorrect predictions.
- C. Decreasing the learning rate might help convergence but doesn't directly address overconfidence or act as a primary regularization against overfitting.
- D. Increasing batch size can sometimes lead to flatter minima and better generalization, but it's not as direct a solution for overconfident predictions as label smoothing.
Label Smoothing
A regularization technique that replaces hard target labels (0 or 1) with a weighted average of the hard label and a uniform distribution, making the model's output probabilities less extreme.
- Prevents models from becoming overconfident in predictions.
- Improves generalization and robustness.
- Acts as a form of regularization, mitigating overfitting.
Memory trick: Smooth labels, smooth predictions.