Character Encodings

For numbers, representation rules could be derived from the structure of the hardware: positional value, two’s complement, and floating point were all designed around the requirements of arithmetic. Text has no such natural necessity. Which number the letter A maps to is a design decision, and that decision has to be shared by whoever writes and whoever reads.

This lesson covers how that sharing is established. There are two separate questions, and confusing them is a common source of error: which number a given character receives, and how that number is written into bytes.

Character Set and Encoding

A character set states which characters exist and which number each one is assigned. This number is called a code point.

An encoding states how a code point is written into bytes.

In small sets these two layers coincide: if the character count does not exceed 256, every code point fits into a single byte and the distinction becomes invisible. Once the set grows, the distinction becomes unavoidable — there is more than one way to write a number that will not fit in a single byte, and which one is used must be stated.

ASCII

ASCII is a 7-bit character set: it defines $2^7 = 128$ code points. Its layout is not arbitrary; it is arranged for arithmetic convenience:

Range (hex)	Content
00–1F	Control characters (newline, tab, end of file, etc.)
20	Space
30–39	Digits 0–9
41–5A	Uppercase letters A–Z
61–7A	Lowercase letters a–z

The digits being consecutive reduces going from a digit character to its numeric value to a single subtraction: '7' - '0' = 7. The difference between uppercase and lowercase letters is exactly 0x20, that is, a single bit: A = 0x41 = 0100 0001, a = 0x61 = 0110 0001. Upper- and lowercasing, within ASCII’s range, is a single bit operation.

The course’s shared example reaches its third interpretation here. The four bytes of the 0x41424344 pattern are 0x41, 0x42, 0x43, 0x44; their ASCII counterparts:

$$\texttt{0x41424344} \longrightarrow \texttt{"ABCD"}$$

The same thirty-two bits read as $1{,}094{,}861{,}636$ as an unsigned integer, $12.141422271728515625$ as binary32, and ABCD as text. Reading the contents of a file or a network packet correctly requires knowing which of these three interpretations applies; that is the job of a format definition.

Code Pages and the Problem They Left Behind

Because ASCII defines only 128 code points, the upper half of an eight-bit byte is left empty. Letters such as ğ, ş, and ı in Turkish were placed into this empty half — but since every language community made its own placement, mutually incompatible code pages resulted.

The consequence is that the same byte gives a different letter depending on which code page it is read with. Text opened with the wrong code page turning into garbled symbols is a symptom of this incompatibility. There is also a harder limit: a single byte carries at most 256 characters, so a single multilingual document cannot be written with the code-page approach.

Unicode

Unicode solves this problem by separating the character set from the encoding. It assigns a code point to the characters of the world’s writing systems within a single number space; the code point is written with a U+ prefix, in hexadecimal:

Character	Code point
A	U+0041
ç	U+00E7
ğ	U+011F
漢	U+6F22

The code point space runs from U+0000 to U+10FFFF, that is, more than a million code points. The first 128 code points are identical to ASCII — a deliberate decision made to preserve backward compatibility.

Unicode is not an encoding: it does not say how the number U+011F is to be written into bytes. Encodings say that.

UTF-8

UTF-8 uses between 1 and 4 bytes, depending on the size of the code point. The byte structure is read from the prefix bits of the first byte:

Code point range	Byte count	Pattern
U+0000 – U+007F	1	0xxxxxxx
U+0080 – U+07FF	2	110xxxxx 10xxxxxx
U+0800 – U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx
U+10000 – U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The code point’s bits are placed left to right into the slots marked x. ç (U+00E7 = 0000 0000 1110 0111) requires two bytes; its significant bits split into 00011 100111:

$$\texttt{110}\underline{00011}\ \texttt{10}\underline{100111} = \texttt{0xC3\ 0xA7}$$

The same rule writes 漢 (U+6F22) in three bytes: 0xE6 0xBC 0xA2.

The design has three consequences:

1. ASCII compatibility. Code points up to U+007F are written in a single byte, with values identical to ASCII. A document containing only ASCII is also a valid UTF-8 document.
2. Self-synchronization. Continuation bytes always start with 10; whether a given byte is the start of a character or a continuation of one can be read from its leading bits alone. In a corrupted stream, the boundary of the next character can still be located.
3. Variable length. How many bytes a character occupies depends on its content; this has direct consequences for text processing.

UTF-16 and UTF-32 are alternative encodings. UTF-32 writes every code point in a fixed four bytes; access is direct but space usage is high. UTF-16 writes most characters in two bytes and uses a surrogate pair for the rest — meaning it, too, is variable length, contrary to the common assumption that it is fixed.

Code Points, Bytes, and Graphemes

In a variable-length encoding, “length” is not a single well-defined concept. There are three separate counts, and they can all differ:

• Byte count — the space occupied on disk or on the wire.
• Code point count — the number of Unicode numbers.
• Grapheme count — the units a user perceives as “letters.”

The source of the third distinction is combining marks. The character é can be written in two different ways: as the single code point U+00E9, or as e (U+0065) followed by a combining acute accent (U+0301). The two forms look identical on screen, their code point counts differ, and they do not compare equal directly. The normalization operation removes this difference before comparison by choosing one of the two forms.

It is here that upper- and lowercasing turn out not to be language-independent either. An operation that is a single bit in ASCII requires, in Turkish, the mappings i ↔ İ and ı ↔ I; a conversion performed without knowledge of the language gives the wrong result.

text = "ABCD"
print(text.encode("utf-8"))              # b'ABCD'          — 4 bytes

turkish_word = "çağ"                     # Turkish for "era"
print(turkish_word.encode("utf-8"))      # b'\xc3\xa7a\xc4\x9f'
print(len(turkish_word), len(turkish_word.encode("utf-8")))   # 3 5

print(ord("ç"), hex(ord("ç")))           # 231 0xe7        — code point U+00E7
print(chr(0x6F22))                       # 漢

decomposed = "e\u0301"                   # e + combining accent (U+0301)
composed = "\u00e9"                      # single code point (U+00E9)

print(decomposed == composed)            # False
print(len(decomposed), len(composed))    # 2 1

import unicodedata
print(unicodedata.normalize("NFC", decomposed) == composed)    # True

The fact that len(turkish_word) gives $3$ while its encoded form is $5$ bytes shows why the two counts must not be confused. A string operation that assumes fixed length — “take the first three bytes,” for instance — cuts a multi-byte character in the middle and produces an invalid byte sequence.

Summary

• A character set assigns characters a number (code point); an encoding states how that number is written into bytes. The two layers are separate.
• ASCII defines 128 code points; digits and letters are consecutive, and the upper/lowercase difference is a single bit. The course’s shared example, 0x41424344, reads as ABCD under an ASCII interpretation.
• Single-byte code pages are both mutually incompatible and limited to 256 characters; multilingual text cannot be written with this approach.
• Unicode defines a single code point space and overlaps with ASCII in its first 128 code points.
• UTF-8 writes a code point in 1–4 bytes; it is ASCII-compatible and self-synchronizing because continuation bytes start with 10.
• Byte count, code point count, and grapheme count are different magnitudes; combining marks and language-dependent letter case reveal this difference.

Next Step

A character has been shown to sometimes span multiple bytes. In what order, then, do those bytes sit in memory? The same question applies to numbers: in which order do the four bytes of the value 0x41424344 sit in memory? The next lesson takes up how this order varies by hardware, and why it becomes a source of trouble when data moves between two systems.