2.3 Perceptron Learning Rule in Machine Learning

Perceptron Learning Rule in Machine Learning

• The Perceptron Learning Rule is the weight update rule used to train a single layer perceptron for binary classification.
• It explains how a perceptron updates its weights when it makes a mistake.
• It adjusts the weights whenever the model makes a mistake.

                If the prediction is correct, do nothing.
                If the prediction is wrong, adjust the weights to reduce the error.

• The goal is to move the decision boundary so that misclassified points are correctly classified. 

Perceptron Model Structure

Formula of Perceptron Learning Rule    

It adjusts the weights so that the perceptron gives a more correct prediction next time.

Mathematical Model

$$\mathrm{z }= w_1{x}_{\mathrm{1 }}+ w_2{x}_{\mathrm{2 }}+\mathrm{.}\mathrm{.}\mathrm{.}+ w_n{x}_{\mathrm{n }}+b$$
$$\stackrel{^ }{y}= f(z)$$

Where:

• $x_1, x_2, ..., x_n$ → input features

• $w_1, w_2, ..., w_n$ → weights

• $b$ → bias

• $f(z)$ → Step activation function

Working Process (Step by Step)

Step 1: Initialize

• Set weights = 0

• Set bias = 0

• Choose learning rate (say η = 1)

Step 2: For each training sample

1. Compute weighted sum

2. Apply step function

3. Compare predicted output with actual output

4. Update weights if error exists

5. Repeat for multiple epochs until no error

Problem:

Classify points into two classes.

it works like an AND gate.

Initial Values:

• w1 = 0 , w2 = 0, b = 0 , η = 1

Iteration 1

Input: (0,0), y=0

z = 0
Output = 1 (since z ≥ 0)

Error = 0 − 1 = -1

Update:

w1 = 0 + (-1)(0) = 0
w2 = 0 + (-1)(0) = 0
b = 0 + (-1) = -1

Input: (0,1), y=0

z = (0)(0) + (0)(1) + (-1) = -1
Output = 0

Correct → No update

Input: (1,0), y=0

z = -1
Output = 0

Correct → No update

Input: (1,1), y=1

z = (0)(1) + (0)(1) + (-1) = -1
Output = 0

Error = 1 − 0 = 1

Update:

w1 = 0 + (1)(1) = 1
w2 = 0 + (1)(1) = 1
b = -1 + 1 = 0

After multiple passes, final weights become:

w1 = 1
w2 = 1
b = -1

Final Decision Rule:

$$x_1+x_2-1\ge 0$$

Which correctly models AND logic.

Learning Rule Works

If a positive sample is misclassified:

• $y - \hat{y} = 1$
  

• Weights increase

• Boundary shifts toward correct side

If a negative sample is misclassified:

• $y - \hat{y} = -1$
  

• Weights decrease

• Boundary shifts away

So the boundary gradually adjusts until all samples are correctly classified.

The perceptron converges only if data is: Linearly separable

It will not converge for: XOR problem

Example:

Understanding the AND Gate

An AND gate is a digital logic gate that outputs true (1) only when all its inputs are true (1). Otherwise, it outputs false (0). The truth table for an AND gate with two inputs (A and B) is:

Perceptron Basics:

A perceptron takes inputs, applies weights, sums them up, and passes the result through an activation function to produce an output. For binary classification, the activation function is typically a step function:

Initiating values:

                                Inputs: A and B.

                                Weights: w1=1.2 , w2=0.6 (randomly initialized).

                                Threshold (θ): 1.

                                Learning Rate (η): 0.5.

 Perceptron Simulation

Case 1: A = 0, B = 0, Target = 0

·        Weighted Sum = w1 * A + w2 * B.

·        Weighted Sum = (1.2 * 0) + (0.6 * 0) = 0.

·        Since 0 < 1, the output is 0.

·        The output matches the target, so no weight update is needed.

 

Case 2: A = 0, B = 1, Target = 0

·        Weighted Sum = w1 * A + w2 * B.

·        Weighted Sum = (1.2 * 0) + (0.6 * 1) = 0.6 .

·        Since 0.6 < 1 , so the output is 0.

·        The output matches the target, so no weight update is needed.

 

Case 3: A = 1, B = 0, Target = 0

·        Weighted Sum = w1 * A + w2 * B.

·        Weighted Sum = (1.2 * 1) + (0.6 * 0) =1.2.

·        Since 1.2 ≥ 1, the output is 1.

·        The output does not match the target (0), so weights need to be updated.

 

Weight Update Rule : If the output is incorrect, update the weights using: 

                             w_i=w_i+η*(Target−Actual)*x_i

For this case:

·        w1 = 1.2 + 0.5 * (0 − 1) * 1

       =  1.2 − 0.5

       =  0.7.

·        w2 = 0.6 + 0.5 * (0−1) * 0

       = 0.6  + 0 

       = 0.6.

Now the new weights are w1 = 0.7 , w2 = 0.6

 

Case 4: A = 1, B = 1, Target = 1

·        Weighted Sum = 0.7*1+0.6*1=1.3

·        Since 1.3 ≥ 1, the output is 1.

·        The output matches the target, so no weight update is needed.

 

Final Weights

After training, the final weights are:

·        w1=0.7.

·        w2=0.6.

 

Verification

Using the final weights, verify the perceptron's output for all input combinations: