All posts

Machine Learning

•

30 April 2025

Elastic Net Regression

Implementing a combination of Lasso and Ridge regression in Python

Introduction

Elastic Net Regression is a combination of Lasso Regression and Ridge Regression.1 It combines the penalties of both Lasso and Ridge Regression to create a model that is more robust to outliers and multicollinearity.2

Mathematics Behind Elastic Net Regression

min⁡β(1N∑i=1n(y^i−yi)2+λ1∑j=1p∣θj∣+λ2∑j=1p∣θj2∣)\min _\beta\left( \frac{1}{N} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2 + \lambda_1 \sum_{j=1}^{p} |\theta_j| + \lambda_2 \sum_{j=1}^{p} |\theta_j^2| \right)

where

  1. 1/N∑i=1n(y^i−yi)21/N \sum_{i=1}^{n} (\hat{y}_i - y_i)^2 is the avg. Mean Squared Error.
  2. λ1\lambda_1 is the regularization parameter for Lasso penalty.
  3. λ2\lambda_2 is the regularization parameter for Ridge penalty.
  4. θj\theta_j is the coefficient of the jthj^{th} feature.
  5. λ1∑j=1p∣θj∣\lambda_1 \sum_{j=1}^{p} |\theta_j| is Lasso Penalty
  6. λ2∑j=1p∣θj2∣\lambda_2 \sum_{j=1}^{p} |\theta_j^2| is Ridge Penalty