Skip to content
BENIBIX Learn

Softmax Function

AI📈 Machine LearningNeural Networks

Also known as Softmax, Softmax Activation, Normalized Exponential

Turns raw scores into chances that add up to 100%.

Think of it like

It is like cutting one pizza for three friends. The most hungry gets the big slice.

Example

A photo model scores cat 4, dog 2 and fox 1. Softmax turns those into about 84% cat, 11% dog and 4% fox.

Why it matters

It lets a network say how sure it is of each answer. Temperature works by changing the scores first.

Definition

Softmax turns a list of raw scores into chances that add up to 100%. A network's last layer gives one score for each possible answer. The scores can be any size, even below zero. Softmax makes each one positive and scales them so they sum to 1. A big lead in score becomes a big lead in chance. A language model uses it to choose its next token.

Example code

python
import torch
torch.softmax(torch.tensor([4.0, 2.0, 1.0]), dim=0)
# tensor([0.8438, 0.1142, 0.0420])
Three raw scores in, three chances out. Together they add up to 1.
Softmax turns scores of any size into shares of one whole.

Common mistake

Reading its output as true confidence. A model can give 99% and still be wrong.

Key takeaways

  1. 1
    Scores in. One raw score per answer.
  2. 2
    Chances out. All positive, summing to 1.
  3. 3
    Big gaps grow. A lead in score becomes a bigger lead.

Learn it properly