Chain-of-Thought Prompting
Level 3 · Harder and bigger tasks
Why asking for the steps makes hard answers better, with the real numbers. When to vote across runs or branch, and when it is all a waste.
Course contentsLesson 7 of 16
Level 1 · What a prompt is
Level 2 · Writing a better prompt
Level 3 · Harder and bigger tasks
Level 4 · Prompts inside an app
After this lesson you can
- Explain why written steps help a hard answer
- Hide the working inside tags
- Choose between one chain, a vote and a tree
12 mincovers 4 glossary entries
Worth reading first: Iterative and Meta-Prompting
Why asking for the steps makes answers better
Chain-of-thought prompting asks a model to work through the steps before it answers. In the 2022 paper that named it, a large model solved 17.9 in 100 school maths problems with a plain prompt. Shown worked steps, it solved 56.9 in 100. Writing steps gives the model more to build on.
The reason is how a model writes. Each word it writes becomes part of what it reads next. A model that has written step one has a better start for step two. A model that jumps straight to the answer gets one pass at the whole problem and no second look.
It pays on sums, on logic and on tasks with several parts. It costs on everything else. The steps are output tokens, which are the slow kind and the dear kind. For a simple question, asking for steps just makes the answer longer and the bill bigger.
Self-consistency is the heavier version. You ask the same question several times with some randomness on, and each run works it out its own way. Then you keep the answer that comes up most often. In the 2022 paper that named it, the vote added 17.9 points on the same maths test.
Tree of thoughts goes further. The model writes a few possible next steps, judges which look good and drops the weak ones. It can back up and try another branch. On a number puzzle in the 2023 paper, GPT-4 solved 4 in 100 with a chain of thought and 74 in 100 with a tree.
Both cost far more. Self-consistency with five runs costs five times as much. A tree can cost many more calls, and it needs a program around the model to run it. Save them for questions where a wrong answer is costly and a right one can be checked.
Two cautions. The steps a model writes are not a true record of how it got the answer, so do not read them as proof. And reasoning models now think before they answer on their own. Asking one to think step by step again adds cost and little gain.
Chain of thought, self-consistency and tree of thoughts
| Method | How it works | What it costs |
|---|---|---|
| Chain of thought | One line of written steps, then the answer | One call, with a longer answer |
| Self-consistency | Several runs, then a vote on the answer | One call per run |
| Tree of thoughts | Branches of steps, judged and pruned | Many calls, plus a program to run them |
Which one to pick
- 1
Pick it for sums, logic and many-part questions. Skip it on a model that thinks first by itself.
- 2
Pick it when one answer must be right and you can pay for five runs.
- 3
Pick it for puzzles where you must try a path, back up and try another.
A pack of 3 pens costs $2. How much do 12 pens cost?
Work it out step by step inside <working> tags.
Then give only the final amount inside <answer> tags.Worked example
Jack's bot works out delivery fees
Jack's shop bot works out delivery fees from the weight and the distance. Asked for the fee alone, it gets 6 of 25 test orders wrong.
With the working inside tags first, it gets 1 wrong. Each answer costs about 150 more output tokens, and he decides the fix is worth it.
The words for this lesson4
Check yourself
Why does asking an AI to think step by step help on hard questions?
Finished this lesson?
Marking it complete also marks its 4 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.