ASCII, Unicode and UTF-8
Level 4 · Characters and escapes
How a computer stores letters as numbers. What ASCII, Unicode and UTF-8 each do, why text turns into odd signs, and why lines end in two ways.
Course contentsLesson 8 of 11
Level 1 · Files and formats
Level 2 · Data formats
Level 3 · Config and docs files
Level 4 · Characters and escapes
Level 5 · IDs, dates and patterns
After this lesson you can
- Tell Unicode from UTF-8
- Fix text that shows odd signs
- Explain LF and CRLF line endings
13 mincovers 5 glossary entries
Worth reading first: Markdown and Front Matter
How letters become bytes, and why text comes out garbled
Computers store only numbers, so each letter needs a number. A character encoding is the rule that turns letters into bytes and back. ASCII is the old one, with 128 codes for basic English. Unicode gives a number to more than 150,000 characters. UTF-8 stores those numbers as bytes, and 99 in 100 websites use it.
ASCII dates from the 1960s. Capital A is 65, small a is 97 and a space is 32. Some codes are not letters but signals, like a new line. It has no room for é, for Persian letters or for emoji.
Unicode fixed that by giving every character in every language its own number, called a code point. It is written like U+0041 for A. Its first 128 numbers match ASCII, so old English text still works.
Unicode only picks the numbers. UTF-8 is the rule that stores them as bytes. Plain English letters take 1 byte each, just like ASCII. Other letters take 2, 3 or 4 bytes. So é takes 2 bytes, and an emoji takes 4.
Text turns into odd signs when it is saved with one encoding and read with another. The word café saved as UTF-8 and read the old way comes out as café. The bytes are fine; the reader is wrong. Opening the file as UTF-8 fixes it.
A line ending is the hidden character that marks where a line stops. Linux and macOS use one, the line feed, written \n. Windows uses two, a carriage return then a line feed, written \r\n. The two styles are called LF and CRLF.
Most editors show both styles the same, so the trouble hides. A shell script saved with CRLF can fail on Linux with a strange error. Set your editor to LF for code. Git can also turn line endings into one style for the whole team.
print(len("A".encode("utf-8"))) # 1
print(len("é".encode("utf-8"))) # 2
print(len("😀".encode("utf-8"))) # 4$ printf 'hi\r\n' | od -c
0000000 h i \r \n
0000004Worked example
The words for this lesson5
Check yourself
How is Unicode not the same as UTF-8?
Finished this lesson?
Marking it complete also marks its 5 glossary entries learned, which updates your roadmap and your saved page. 0 are already marked.