Lecture 5. Regularization#
Regularization: Importance, Reasons, and Approaches#
Reasons for Regularization:
Presence of irrelevant features and collinearity in the dataset.
Can lead to:
Ill-posed optimization problems, especially in linear regression.
Broken interpretability due to inflated or unreliable coefficients.
Importance:
Regularization introduces constraints to prevent overfitting.
Ensures stability in predictions and maintains model simplicity.
Helps in feature selection, especially with L1 regularization.
Approaches:
L2 Regularization (Ridge Regression):
Aims to re-condition the optimization problem.
Equivalently, it’s the Maximum a Posteriori (MAP) estimation in a Bayesian context with a Gaussian prior on weights.
L1 Regularization (The Lasso):
Induces sparsity in the model, effectively leading to feature selection.
Especially favored in situations with high-dimensional features but limited data points.
Multiple Intuitions:
Regularization can be understood through various lenses, including algebraic and geometric interpretations.