A machine learning engineer is training a deep neural network for a computer vision task. During training, the model's performance on the validation set initially improves but then starts to degrade, while the training set performance continues to improve. This suggests the model is overfitting. Which technique would be most effective in mitigating this issue without significantly increasing the model's complexity?
- AIncreasing the number of layers in the neural network
- BDecreasing the learning rate
- CCollecting more training data
- DApplying L1 regularization to the model weights
Show answer & explanationAnswer & explanation
Correct answer: D. Applying L1 regularization to the model weights
Overfitting occurs when a model learns the training data too well, including its noise, leading to poor generalization. L1 regularization (Lasso) adds a penalty proportional to the absolute value of the coefficients, encouraging sparsity and effectively performing feature selection by driving some weights to zero, thereby reducing model complexity and mitigating overfitting without adding layers or requiring new data. Increasing layers would increase complexity, more data is often not feasible, and decreasing the learning rate primarily affects convergence speed, not overfitting directly.
Why the other options are wrong
- A. Increasing layers would increase model complexity, exacerbating overfitting.
- B. Decreasing the learning rate primarily affects convergence, not necessarily overfitting directly, though an extremely high learning rate could prevent convergence to a good minimum.
- C. Collecting more data is an effective solution but often not feasible or 'without significantly increasing model complexity'.
L1 Regularization (Lasso)
L1 regularization, or Lasso regularization, adds a penalty term to the loss function that is proportional to the absolute value of the magnitude of the coefficients.
- Encourages sparsity in weights, effectively performing feature selection.
- Can drive some feature weights to exactly zero, simplifying the model.
- Helps mitigate overfitting by reducing model complexity.
Memory trick: To stop overfitting, 'Lasso' the complexity before it 'runs away' with your data.