2.3 Perceptron Learning Rule in Machine Learning

Perceptron Learning Rule in Machine Learning

The Perceptron Learning Rule is the weight update rule used to train a single layer perceptron for binary classification.

It adjusts the weights whenever the model makes a mistake.

The goal is to move the decision boundary so that misclassified points are correctly classified. 

Perceptron Model Structure

Mathematical Model

$$z = w_1x_1 + w_2x_2 + ... + w_nx_n + b$$
$$\widehat{y}=f(z)$$

Where:

• $x_1, x_2, ..., x_n$ → input features

• $w_1, w_2, ..., w_n$ → weights

• $b$ → bias

• $f(z)$ → Step activation function

Step Function

$$f(z) = \begin{cases} 1 & z \ge 0 \\ 0 & z < 0 \end{cases}$$

When prediction is wrong, update weights using:

$$w_i = w_i + \eta (y - \hat{y}) x_i$$ $$b = b + \eta (y - \hat{y})$$

Where:

• $y$ → actual output

• $\hat{y}$ → predicted output

• $\eta$ → learning rate

If prediction is correct → no change in weights.

If prediction is wrong → weights are adjusted.​

Working Process (Step by Step)

Step 1: Initialize

• Set weights = 0

• Set bias = 0

• Choose learning rate (say η = 1)

Step 2: For each training sample

1. Compute weighted sum

2. Apply step function

3. Compare predicted output with actual output

4. Update weights if error exists

5. Repeat for multiple epochs until no error

Problem:

Classify points into two classes.

it works like an AND gate.

Initial Values:

• w1 = 0 , w2 = 0, b = 0 , η = 1

Iteration 1

Input: (0,0), y=0

z = 0
Output = 1 (since z ≥ 0)

Error = 0 − 1 = -1

Update:

w1 = 0 + (-1)(0) = 0
w2 = 0 + (-1)(0) = 0
b = 0 + (-1) = -1

Input: (0,1), y=0

z = (0)(0) + (0)(1) + (-1) = -1
Output = 0

Correct → No update

Input: (1,0), y=0

z = -1
Output = 0

Correct → No update

Input: (1,1), y=1

z = (0)(1) + (0)(1) + (-1) = -1
Output = 0

Error = 1 − 0 = 1

Update:

w1 = 0 + (1)(1) = 1
w2 = 0 + (1)(1) = 1
b = -1 + 1 = 0

After multiple passes, final weights become:

w1 = 1
w2 = 1
b = -1

Final Decision Rule:

$$x_1+x_2-1\ge 0$$

Which correctly models AND logic.

Learning Rule Works

If a positive sample is misclassified:

• $y - \hat{y} = 1$
  

• Weights increase

• Boundary shifts toward correct side

If a negative sample is misclassified:

• $y - \hat{y} = -1$
  

• Weights decrease

• Boundary shifts away

So the boundary gradually adjusts until all samples are correctly classified.

The perceptron converges only if data is: Linearly separable

It will not converge for: XOR problem

Example:

Understanding the AND Gate

An AND gate is a digital logic gate that outputs true (1) only when all its inputs are true (1). Otherwise, it outputs false (0). The truth table for an AND gate with two inputs (A and B) is:

Perceptron Basics:

A perceptron takes inputs, applies weights, sums them up, and passes the result through an activation function to produce an output. For binary classification, the activation function is typically a step function:

Initiating values:

                                Inputs: A and B.

                                Weights: w1=1.2 , w2=0.6 (randomly initialized).

                                Threshold (θ): 1.

                                Learning Rate (η): 0.5.

 Perceptron Simulation

Case 1: A = 0, B = 0, Target = 0

·        Weighted Sum = w1 * A + w2 * B.

·        Weighted Sum = (1.2 * 0) + (0.6 * 0) = 0.

·        Since 0 < 1, the output is 0.

·        The output matches the target, so no weight update is needed.

 

Case 2: A = 0, B = 1, Target = 0

·        Weighted Sum = w1 * A + w2 * B.

·        Weighted Sum = (1.2 * 0) + (0.6 * 1) = 0.6 .

·        Since 0.6 < 1 , so the output is 0.

·        The output matches the target, so no weight update is needed.

 

Case 3: A = 1, B = 0, Target = 0

·        Weighted Sum = w1 * A + w2 * B.

·        Weighted Sum = (1.2 * 1) + (0.6 * 0) =1.2.

·        Since 1.2 ≥ 1, the output is 1.

·        The output does not match the target (0), so weights need to be updated.

 

Weight Update Rule : If the output is incorrect, update the weights using: 

                             w_i=w_i+η*(Target−Actual)*x_i

For this case:

·        w1 = 1.2 + 0.5 * (0 − 1) * 1

       =  1.2 − 0.5

       =  0.7.

·        w2 = 0.6 + 0.5 * (0−1) * 0

       = 0.6  + 0 

       = 0.6.

Now the new weights are w1 = 0.7 , w2 = 0.6

 

Case 4: A = 1, B = 1, Target = 1

·        Weighted Sum = 0.7*1+0.6*1=1.3

·        Since 1.3 ≥ 1, the output is 1.

·        The output matches the target, so no weight update is needed.

 

Final Weights

After training, the final weights are:

·        w1=0.7.

·        w2=0.6.

 

Verification

Using the final weights, verify the perceptron's output for all input combinations: