Skip to content
BENIBIX Learn

ASCII, Unicode and UTF-8

Level 4 · Characters and escapes

How a computer stores letters as numbers. What ASCII, Unicode and UTF-8 each do, why text turns into odd signs, and why lines end in two ways.

Course contentsLesson 8 of 11

After this lesson you can

  • Tell Unicode from UTF-8
  • Fix text that shows odd signs
  • Explain LF and CRLF line endings

13 mincovers 5 glossary entries

Worth reading first: Markdown and Front Matter

How letters become bytes, and why text comes out garbled

Computers store only numbers, so each letter needs a number. A character encoding is the rule that turns letters into bytes and back. ASCII is the old one, with 128 codes for basic English. Unicode gives a number to more than 150,000 characters. UTF-8 stores those numbers as bytes, and 99 in 100 websites use it.

ASCII dates from the 1960s. Capital A is 65, small a is 97 and a space is 32. Some codes are not letters but signals, like a new line. It has no room for é, for Persian letters or for emoji.

Unicode fixed that by giving every character in every language its own number, called a code point. It is written like U+0041 for A. Its first 128 numbers match ASCII, so old English text still works.

Unicode only picks the numbers. UTF-8 is the rule that stores them as bytes. Plain English letters take 1 byte each, just like ASCII. Other letters take 2, 3 or 4 bytes. So é takes 2 bytes, and an emoji takes 4.

Text turns into odd signs when it is saved with one encoding and read with another. The word café saved as UTF-8 and read the old way comes out as café. The bytes are fine; the reader is wrong. Opening the file as UTF-8 fixes it.

A line ending is the hidden character that marks where a line stops. Linux and macOS use one, the line feed, written \n. Windows uses two, a carriage return then a line feed, written \r\n. The two styles are called LF and CRLF.

Most editors show both styles the same, so the trouble hides. A shell script saved with CRLF can fail on Linux with a strange error. Set your editor to LF for code. Git can also turn line endings into one style for the whole team.

python
print(len("A".encode("utf-8")))   # 1
print(len("é".encode("utf-8")))   # 2
print(len("😀".encode("utf-8")))  # 4
In UTF-8, an English letter takes 1 byte, é takes 2 and an emoji takes 4.
console
$ printf 'hi\r\n' | od -c
0000000   h   i  \r  \n
0000004
A Windows line ending, laid bare: the two letters, then \r and \n, 4 bytes in all.
Unicode picks the number and UTF-8 stores it. Only the letter A is also in ASCII.

Worked example

Lucy's customer list shows odd signs

Lucy opens a list of 800 customers, and every name with an accent is garbled. Zoë shows up as Zoë.

The file was saved as UTF-8 and her app read it the old way. She opens it again with UTF-8 picked, and all 800 names look right.

The words for this lesson5

Check yourself

How is Unicode not the same as UTF-8?

Finished this lesson?

Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.