--- title: 'Character Encodings' source: 'https://academia.sh/en/courses/how-computers-work/character-encodings' course: 'How Computers Work' language: en updated: '2026-08-17T18:08:09+00:00' license: 'CC BY-SA 4.0' --- # Character Encodings Character sets from ASCII to Unicode, the distinction between code points and byte sequences, and UTF-8 encoding. For numbers, representation rules could be derived from the structure of the hardware: positional value, two's complement, and floating point were all designed around the requirements of arithmetic. Text has no such natural necessity. Which number the letter `A` maps to is a design decision, and that decision has to be **shared** by whoever writes and whoever reads. This lesson covers how that sharing is established. There are two separate questions, and confusing them is a common source of error: which number a given character receives, and how that number is written into bytes. ## Character Set and Encoding A **character set** states which characters exist and which number each one is assigned. This number is called a **code point**. An **encoding** states how a code point is written into bytes. In small sets these two layers coincide: if the character count does not exceed 256, every code point fits into a single byte and the distinction becomes invisible. Once the set grows, the distinction becomes unavoidable — there is more than one way to write a number that will not fit in a single byte, and which one is used must be stated. ## ASCII ASCII is a 7-bit character set: it defines $2^7 = 128$ code points. Its layout is not arbitrary; it is arranged for arithmetic convenience: | Range (hex) | Content | |---|---| | `00`–`1F` | Control characters (newline, tab, end of file, etc.) | | `20` | Space | | `30`–`39` | Digits `0`–`9` | | `41`–`5A` | Uppercase letters `A`–`Z` | | `61`–`7A` | Lowercase letters `a`–`z` | The digits being consecutive reduces going from a digit character to its numeric value to a single subtraction: `'7' - '0' = 7`. The difference between uppercase and lowercase letters is exactly `0x20`, that is, a single bit: `A` = `0x41` = `0100 0001`, `a` = `0x61` = `0110 0001`. Upper- and lowercasing, within ASCII's range, is a single bit operation. The course's shared example reaches its third interpretation here. The four bytes of the `0x41424344` pattern are `0x41`, `0x42`, `0x43`, `0x44`; their ASCII counterparts: $$ \texttt{0x41424344} \longrightarrow \texttt{"ABCD"} $$ The same thirty-two bits read as $1{,}094{,}861{,}636$ as an unsigned integer, $12.141422271728515625$ as binary32, and `ABCD` as text. Reading the contents of a file or a network packet correctly requires knowing which of these three interpretations applies; that is the job of a format definition. ## Code Pages and the Problem They Left Behind Because ASCII defines only 128 code points, the upper half of an eight-bit byte is left empty. Letters such as `ğ`, `ş`, and `ı` in Turkish were placed into this empty half — but since every language community made its own placement, mutually incompatible **code pages** resulted. The consequence is that the same byte gives a different letter depending on which code page it is read with. Text opened with the wrong code page turning into garbled symbols is a symptom of this incompatibility. There is also a harder limit: a single byte carries at most 256 characters, so a single multilingual document cannot be written with the code-page approach. ## Unicode Unicode solves this problem by separating the character set from the encoding. It assigns a code point to the characters of the world's writing systems within a single number space; the code point is written with a `U+` prefix, in hexadecimal: | Character | Code point | |---|---| | `A` | U+0041 | | `ç` | U+00E7 | | `ğ` | U+011F | | `漢` | U+6F22 | The code point space runs from U+0000 to U+10FFFF, that is, more than a million code points. The first 128 code points are identical to ASCII — a deliberate decision made to preserve backward compatibility. Unicode is not an encoding: it does not say how the number `U+011F` is to be written into bytes. Encodings say that. ## UTF-8 UTF-8 uses between 1 and 4 bytes, depending on the size of the code point. The byte structure is read from the prefix bits of the first byte: | Code point range | Byte count | Pattern | |---|---|---| | U+0000 – U+007F | 1 | `0xxxxxxx` | | U+0080 – U+07FF | 2 | `110xxxxx 10xxxxxx` | | U+0800 – U+FFFF | 3 | `1110xxxx 10xxxxxx 10xxxxxx` | | U+10000 – U+10FFFF | 4 | `11110xxx 10xxxxxx 10xxxxxx 10xxxxxx` | The code point's bits are placed left to right into the slots marked `x`. `ç` (U+00E7 = `0000 0000 1110 0111`) requires two bytes; its significant bits split into `00011 100111`: $$ \texttt{110}\underline{00011}\ \texttt{10}\underline{100111} = \texttt{0xC3\ 0xA7} $$ The same rule writes `漢` (U+6F22) in three bytes: `0xE6 0xBC 0xA2`. The design has three consequences: 1. **ASCII compatibility.** Code points up to U+007F are written in a single byte, with values identical to ASCII. A document containing only ASCII is also a valid UTF-8 document. 2. **Self-synchronization.** Continuation bytes always start with `10`; whether a given byte is the start of a character or a continuation of one can be read from its leading bits alone. In a corrupted stream, the boundary of the next character can still be located. 3. **Variable length.** How many bytes a character occupies depends on its content; this has direct consequences for text processing. UTF-16 and UTF-32 are alternative encodings. UTF-32 writes every code point in a fixed four bytes; access is direct but space usage is high. UTF-16 writes most characters in two bytes and uses a **surrogate pair** for the rest — meaning it, too, is variable length, contrary to the common assumption that it is fixed. ## Code Points, Bytes, and Graphemes In a variable-length encoding, "length" is not a single well-defined concept. There are three separate counts, and they can all differ: - **Byte count** — the space occupied on disk or on the wire. - **Code point count** — the number of Unicode numbers. - **Grapheme count** — the units a user perceives as "letters." The source of the third distinction is **combining marks**. The character `é` can be written in two different ways: as the single code point U+00E9, or as `e` (U+0065) followed by a combining acute accent (U+0301). The two forms look identical on screen, their code point counts differ, and they do not compare equal directly. The **normalization** operation removes this difference before comparison by choosing one of the two forms. It is here that upper- and lowercasing turn out not to be language-independent either. An operation that is a single bit in ASCII requires, in Turkish, the mappings `i` ↔ `İ` and `ı` ↔ `I`; a conversion performed without knowledge of the language gives the wrong result. ```python text = "ABCD" print(text.encode("utf-8")) # b'ABCD' — 4 bytes turkish_word = "çağ" # Turkish for "era" print(turkish_word.encode("utf-8")) # b'\xc3\xa7a\xc4\x9f' print(len(turkish_word), len(turkish_word.encode("utf-8"))) # 3 5 print(ord("ç"), hex(ord("ç"))) # 231 0xe7 — code point U+00E7 print(chr(0x6F22)) # 漢 decomposed = "e\u0301" # e + combining accent (U+0301) composed = "\u00e9" # single code point (U+00E9) print(decomposed == composed) # False print(len(decomposed), len(composed)) # 2 1 import unicodedata print(unicodedata.normalize("NFC", decomposed) == composed) # True ``` The fact that `len(turkish_word)` gives $3$ while its encoded form is $5$ bytes shows why the two counts must not be confused. A string operation that assumes fixed length — "take the first three bytes," for instance — cuts a multi-byte character in the middle and produces an invalid byte sequence. ## Summary - A character set assigns characters a number (code point); an encoding states how that number is written into bytes. The two layers are separate. - ASCII defines 128 code points; digits and letters are consecutive, and the upper/lowercase difference is a single bit. The course's shared example, `0x41424344`, reads as `ABCD` under an ASCII interpretation. - Single-byte code pages are both mutually incompatible and limited to 256 characters; multilingual text cannot be written with this approach. - Unicode defines a single code point space and overlaps with ASCII in its first 128 code points. - UTF-8 writes a code point in 1–4 bytes; it is ASCII-compatible and self-synchronizing because continuation bytes start with `10`. - Byte count, code point count, and grapheme count are different magnitudes; combining marks and language-dependent letter case reveal this difference. ## Next Step A character has been shown to sometimes span multiple bytes. In what order, then, do those bytes sit in memory? The same question applies to numbers: in which order do the four bytes of the value `0x41424344` sit in memory? The next lesson takes up how this order varies by hardware, and why it becomes a source of trouble when data moves between two systems.