Skip to content
BENIBIX Learn

AI Training and Inference

Level 1 · What AI is

How an AI model learns from examples in training, then answers in inference. Why training costs so much, and why each answer costs too.

Course contentsLesson 2 of 10

After this lesson you can

  • Tell training from inference
  • Explain what training changes inside a model
  • Say why AI needs so much computer power

11 mincovers 5 glossary entries

Worth reading first: Artificial Intelligence (AI)

How a model learns, then answers

Every AI model has two stages. First comes training, when it learns from examples. Then comes inference, when it is used to answer. Training happens once and is slow. A famous photo model from 2012 took five to six days on two graphics chips. Inference happens each time someone asks, often in under a second.

Training starts with training data: thousands or millions of examples, each with the right answer. The model makes a guess for one example. The guess is checked against the answer. Then the model's numbers are nudged a tiny bit, so the next guess is closer. This repeats, again and again.

Those numbers are called parameters. They start out random, so the first guesses are nonsense. No one sets them by hand. Training does, one small nudge at a time. A small model may have a few thousand parameters. The largest have hundreds of billions.

Once training ends, the parameters are locked. Now the model is used for inference. A new photo goes in, and a label comes out. The model learns nothing from this. Asking it a question does not teach it your answer, which surprises a lot of people.

All of this needs compute: the raw power of the chips that do the sums. Training uses huge amounts of it at once. The compute used to train the top models has grown about four to five times each year since 2010. That is one reason AI labs spend so much on chips and power.

Inference costs far less per answer, but it never stops. A chat app may answer millions of questions a day, and each one runs the model once. So a company pays for training once and for inference all day long. For a busy app, that daily bill adds up fast.

The data shapes the model as much as the compute does. A model can only learn patterns that are in its examples. If the examples are wrong, too few or one-sided, the model will be too. More power does not fix poor data.

Training builds the model once. Inference then uses the same file again and again.

Worked example

Jack trains an apple sorter

Jack's farm shop trains a model on 8,000 photos of apples, each marked ripe or not ripe. Training takes 3 hours on one rented graphics chip.

After that, each new photo takes a tenth of a second to check. His camera checks 600 apples a day, and the model itself never changes.

The words for this lesson5

Check yourself

Does an AI model learn from the questions you ask it?

Finished this lesson?

Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.