Character Encoding

Characters and bytes

Memory and files hold only bytes — numbers from 0 to 255 (see Number Systems). Text is not bytes: it is a sequence of characters, such as A, é, € or 中. To store or send text, every character has to be turned into one or more bytes, and when you read the text back, the bytes have to be turned into characters again.

A character encoding is the agreement that defines this mapping. Turning text into bytes is called encoding; turning bytes back into text is decoding. The crucial point: bytes do not carry a label that says which encoding produced them. If the reader uses a different agreement than the writer, the text comes out wrong — without any error.

ASCII

ASCII (American Standard Code for Information Interchange, 1963, revised 1967) is the classic encoding. It uses 7 bits, so it has 2⁷ = 128 codes, numbered 0 to 127:

• 0–31 and 127: control characters. They are not printed but control a device or mark the structure of text. The ones you still meet: 9 (tab), 10 (line feed, LF — a new line on Linux and macOS) and 13 (carriage return, CR — Windows ends lines with CR followed by LF).
• 32–126: printable characters. Space (32), digits (0 is 48), uppercase letters (A is 65), lowercase letters (a is 97) and punctuation.

The layout is deliberate: lowercase and uppercase letters differ by exactly 32, one bit, and digits and letters are in order, so sorting and case conversion are simple. Python's ord() gives a character's number and chr() the character for a number:

for ch in ["A", "a", "0", " "]:
    print(repr(ch), ord(ch))
print(chr(72) + chr(105))
print(ord("a") - ord("A"))

'A' 65
'a' 97
'0' 48
' ' 32
Hi
32

ASCII was designed for English. It has no é, ü, ß or €, let alone Greek, Arabic or Chinese.

Extended ASCII and code pages

A byte has 8 bits, and ASCII uses only 7, so the values 128 to 255 were free. Many "extended ASCII" encodings, called code pages, used them for extra characters — but each for different ones:

• ISO 8859-1 (Latin-1) adds the accented letters of Western European languages, such as é (233).
• Windows-1252 is Latin-1 plus extra printable characters in the range 128–159, which Latin-1 reserves for control codes: € (128), curly quotes, the dash. Because the two are so often confused, web browsers read text labeled Latin-1 as Windows-1252.
• Other code pages cover Cyrillic (Windows-1251), Greek, Hebrew, and so on; the original IBM PC used code page 437, with line drawing symbols.

The problem is plain when you decode one and the same byte with three code pages:

data = bytes([0xE9])
for codec in ["latin-1", "cp1251", "cp437"]:
    print(codec, data.decode(codec))

latin-1 é
cp1251 й
cp437 Θ

A file written in one code page and opened with another shows the wrong letters. And no single code page can hold French and Russian at the same time, let alone Chinese, which needs tens of thousands of characters. One byte per character is simply too little.

Unicode and code points

Unicode is the standard that solves this. It is a single, universal list of characters for every writing system in use — Latin, Cyrillic, Arabic, Chinese, Hindi — plus historic scripts, mathematical symbols and emoji: more than 150,000 characters, with room for over a million.

Unicode gives each character a number called a code point, written as U+ followed by at least four hexadecimal digits:

The first 128 code points are identical to ASCII, and the first 256 to Latin-1, which made the move to Unicode easier. Code points run from U+0000 to U+10FFFF.

Note what Unicode does not say: how a code point is stored as bytes. It is a list of numbers, not an encoding. That is the job of the Unicode encodings — UTF-8, UTF-16 and UTF-32. In Python, a str is a sequence of code points; see Characters and code points.

UTF-8

UTF-8 is a Unicode encoding that stores each code point in 1 to 4 bytes: the lower the code point, the fewer bytes. It is by far the most used encoding — on the web, in Linux and macOS, in JSON, and as the default for source files in most programming languages.

The x positions hold the bits of the code point. For é, U+00E9 is 00011 101001 in 11 bits, which becomes 11000011 10101001 = C3 A9. Python's .encode() shows the real bytes:

for ch in ["A", "é", "€", "😀"]:
    data = ch.encode("utf-8")
    print(ch, f"U+{ord(ch):04X}", len(data), data.hex(" "))

A U+0041 1 41
é U+00E9 2 c3 a9
€ U+20AC 3 e2 82 ac
😀 U+1F600 4 f0 9f 98 80

This design has three big advantages:

• ASCII-compatible. Every ASCII file is already valid UTF-8, byte for byte, and English text takes no extra space.
• Self-synchronizing. The first bits tell whether a byte starts a character (0, 110, 1110, 11110) or continues one (10), so a program can always find the start of the next character, even in the middle of a stream.
• No byte-order problem. UTF-8 is read one byte at a time, so it is the same on every processor.

The consequence: the number of characters and the number of bytes are different things. A str counts characters; the encoded bytes count bytes:

word = "café"
data = word.encode("utf-8")
print(len(word), len(data))
print(data)
print(data.decode("utf-8"))

4 5
b'caf\xc3\xa9'
café

UTF-16 and the byte order mark

UTF-16

UTF-16 stores code points in units of 16 bits (2 bytes). Code points up to U+FFFF take one unit; higher ones, such as most emoji, take two units, called a surrogate pair. UTF-16 is used inside Windows, Java, .NET and JavaScript. Because it uses 2 bytes per unit, the order of those bytes matters: little-endian (-le) puts the low byte first.

print("A€".encode("utf-16-le").hex(" "))
print("😀".encode("utf-16-le").hex(" "))

41 00 ac 20
3d d8 00 de

Here A becomes 41 00 — so ASCII text in UTF-16 is full of zero bytes — and 😀 becomes the surrogate pair D83D DE00. That is why, in JavaScript, whose strings are UTF-16, an emoji has a length of 2:

console.log("😀".length);                          // UTF-16 units
console.log([..."😀"].length);                     // code points
console.log(new TextEncoder().encode("😀").length); // UTF-8 bytes

2
1
4

The byte order mark

A byte order mark (BOM) is the invisible character U+FEFF placed at the very start of a file. In UTF-16 it tells the reader which byte order was used. UTF-8 has no byte order, but some Windows programs still write the UTF-8 BOM EF BB BF as a signature. Other programs then see it as an unwanted character at the start of the text. Python's "utf-8-sig" codec writes it, and removes it when reading:

data = "hi".encode("utf-8-sig")
print(data.hex(" "))
print(repr(data.decode("utf-8")))
print(repr(data.decode("utf-8-sig")))

ef bb bf 68 69
'\ufeffhi'
'hi'

Mojibake: decoding with the wrong encoding

Mojibake (Japanese for "character transformation") is garbled text that appears when bytes are decoded with a different encoding than the one used to encode them. The classic case: UTF-8 bytes read as Latin-1 or Windows-1252. Each byte of a two-byte UTF-8 character is then shown as a separate character.

text = "café"
garbled = text.encode("utf-8").decode("latin-1")
print(garbled)
print(garbled.encode("latin-1").decode("utf-8"))   # undo the mistake

café
café

The mistake can be undone, as the last line shows, as long as the garbled text has not been changed or saved yet. The opposite mistake — Latin-1 bytes decoded as UTF-8 — usually fails loudly instead, because a lone byte such as E9 is not valid UTF-8:

data = "café".encode("latin-1")
print(data)
print(data.decode("utf-8"))

b'caf\xe9'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data

On a web page, the same byte problem shows up as a black diamond with a question mark, � (U+FFFD, the replacement character), which a browser shows where bytes are not valid in the encoding it assumed.

Practical advice

Encoding bugs happen at the borders of a program, where text is turned into bytes or back: reading and writing files, network traffic, databases. The rule that prevents almost all of them: use UTF-8 everywhere, and say so explicitly.

• Files: save source code and text files as UTF-8 (without BOM). In code, always pass the encoding when you open a text file — in Python, open(path, encoding="utf-8"), because before Python 3.15 the default depends on the operating system (see Encoding).
• HTML: put <meta charset="utf-8"> at the very start of the <head>.
• HTTP: send the encoding in the header, as in Content-Type: text/html; charset=utf-8.
• Databases: set the database, tables and connection to UTF-8. In MySQL and MariaDB, use utf8mb4; the older utf8 there stores at most 3 bytes per character and cannot hold emoji.
• Inside the program: decode bytes to text as soon as they come in, work with text, and encode only when it goes out.