Skip to content
BENIBIX Learn

Activation Function

AI📈 Machine LearningNeural Networks

Also known as Activation, ReLU, Sigmoid Function

A rule inside each neuron. It decides what number gets passed on.

Think of it like

Think of it like a gate. It only opens when the push is hard enough.

Example

A neuron in Jack's model adds up to -3.2. ReLU turns that into 0, so the neuron stays quiet. Another neuron adds up to 4.1, and ReLU passes 4.1 on.

Why it matters

It is what lets a neural network learn curves and twists in data. Pick the wrong one, and training can stall.

Definition

An activation function decides what a neuron sends on after it adds up its inputs. A common one, called ReLU, keeps any total above zero and turns the rest into zero. Another, called sigmoid, squeezes the total into a number from 0 to 1. These bends let a network learn curves, not just straight lines. Without them, a deep network would act like one straight line.
ReLU keeps any number above zero and blocks the rest. Sigmoid squeezes every number into 0 to 1.

Common mistake

Leaving it out to keep things simple. Layers stacked with no activation function add up to one plain straight line.

Key takeaways

  1. 1
    After the sum. Used on each neuron's total.
  2. 2
    Adds bends. Lets the network learn curves.
  3. 3
    ReLU is common. Below zero becomes zero.

Learn it properly