2.17 Weight Regularization in Machine Learning

-

Weight Regularization in Machine Learning

  • Regularization is a technique used to prevent overfitting in machine learning models.
  • Regularization is an important technique used to control model complexity and prevent overfitting.
  • Regularization helps machine learning models perform better on new data rather than memorizing training data.
  • Sometimes a model performs very well on training data but poorly on test data.
  • This happens because the model memorizes the training data instead of learning general patterns. This problem is called overfitting.
  • Regularization helps solve this problem by adding a penalty to the model so that it does not become too complex. 

Regularization is Needed . In machine learning, models may become too complex when:

  • There are many features

  • The model tries to perfectly fit the training data

  • Noise is present in the dataset

As a result:

  • Training accuracy becomes very high

  • Test accuracy becomes low

Regularization reduces this problem by controlling the size of model coefficients (weights).

Regularization keeps all features but reduces the impact of less important features.


Types of Regularization

There are mainly two regularization techniques:

  1. Ridge Regression (L2 Regularization)

  2. Lasso Regression (L1 Regularization)

Ridge RegressionShrinks coefficients but keeps all features
Lasso RegressionShrinks coefficients and removes unnecessary features

1. Ridge Regression (L2 Regularization): -

  • Ridge regression is a regularization technique that reduces model complexity by shrinking coefficient values.
  • It introduces a penalty based on the square of coefficients.
  • It is also called as L2 regularization.

Cost Function : Cost = SSE + λ Σ (β²)

Where:

  • SSE
  • sum of squared errors
  • λ (lambda)- regularization parameter
  • β - coefficient weights

Lambda controls the strength of regularization

Lambda ValueEffect
  • λ = 0
  • Normal linear regression
  • Small λ    Slight regularization
  • Large λ    Strong regularization

Important Characteristics

  • Reduces coefficient values

  • Does not eliminate features

  • Keeps all features in the model

  • Helps reduce multicollinearity


Example

Suppose we are predicting house prices using features:

  • Size

  • Number of rooms

  • Location

  • Age of house

If some features are less important, Ridge regression reduces their coefficients, but does not remove them.

2. Lasso Regression (L1 Regularization): -

  • Lasso regression is a regularization technique that reduces model complexity and performs feature selection.
  • It adds a penalty based on the absolute value of coefficients.
  • It is also called as L1 regularization.

Cost Function : Cost = SSE + λ Σ |β|

Key Characteristics

  • Shrinks coefficients toward zero

  • Some coefficients become exactly zero

  • Performs automatic feature selection


Example

Suppose we have 10 features for predicting salary.

Lasso regression may reduce some coefficients to zero, meaning those features are removed from the model. This helps simplify the model.


Example of Regularization in Real Life

Suppose we build a model to predict student performance.

Features include:

  • Study hours

  • Attendance

  • Social media usage

  • Sleep time

  • Class participation

Some features may not strongly affect performance.

Regularization reduces the impact of unnecessary features, improving model performance.


Advantages of Regularization

  • Prevents overfitting

  • Improves model generalization

  • Handles multicollinearity

  • Reduces model complexity

  • Improves prediction accuracy on unseen data





























Popular posts from this blog

operators in c programming

-
Image
v Operators: - An operator is a symbol that tells or intimates to the compiler to perform specific calculations or manipulations on one or more operands. Operators are essential for performing computations and making decisions in programs.   a + b is an operation a & b are operands + is an operator   Based on number of operands involved: 1.      Unary operators: - unary operators can be applied on only one operator. Example: -   ++ , -- , - (unary minus) , sizeof , ~ (compliment) , ! (not).   2.      Binary operators: - binary operators can be applied on two operands. Example: -   + , -,   * , / , etc.   3.    Ternary operators: - Ternary operator can be applied on three operands. Example: -   ?: (ternary conditional operator).   C supports several operators, can be classified into following...
Read more

2.4 Arrays in c programming

-
Image
  Arrays: - An array is a collection of elements of the same data type stored in contiguous memory locations. It is a derived data type and a homogenous data structure. It is a variable which can store multiple values of the same type under a single variable name, making it easier to manage and manipulate related data. The size of an array is fixed. To access the elements of an array, we use indexes or subscripts accordingly. array index or subscript is used to identify array elements address. Array index or subscript starts with 0 to n-1.   Declaration of an array: - The declaration of an array means telling the compiler to create an array variable and reserving memory for a fixed number of elements of the same data type. Initialization of an array: - Initializing an array means assigning initial values to its elements. Types of arrays: - Arrays are classified based on the number of dimensions they have:             ...
Read more

Variables in c

-
Image
  Variables: - A variable is a data name or an identifier which stores a value that can be changed during the program execution.  Every variable might belong to any of the data types, like int, float, char, etc.  The name of a variable can be composed of letters, digits, and the underscore character. It must begin with either a letter or an underscore.  Variables are case-sensitive because uppercase and lowercase letters are distinct. C programming language also allows defining various other types of variables, like numeric array, character array, pointer, Structure, Union, Enumeration and etc.               Example: - sum, height, _value Declaring a variable in C: - Variable declaration means telling the compiler about the name and type of a variable before using it in the program. Variable declaration is the process of reserving a memory location and defining the type of data that will be stored in that locatio...
Read more