--- title: 'File Operations' source: 'https://academia.sh/en/courses/python-fundamentals/file-operations' course: 'Python Fundamentals' language: en updated: '2026-08-17T18:10:27+00:00' license: 'CC BY-SA 4.0' --- # File Operations The same 41 bytes give a byte sequence of length 41 in binary mode, and a 34-character string in text mode with utf-8; a wrong encoding breaks at read time, not open time, and latin-1 never breaks at all, silently producing 41 wrong characters. The previous lesson's twelve records sat ready in memory. The `value` field was already a string; the only job was converting it to a number and checking it against the field's bounds. The domain root paid off because the error's source was the domain itself. When records come from a file, one more layer gets inserted. What sits in a file is neither a string nor a number — it is **bytes**. Converting bytes to a string is an **encoding** decision, and if that decision is wrong, the error never reaches the domain layer at all; it shows up while reading. The concepts of file, encoding, and byte were built in the How Computers Work course; the **concept** was built there, what is measured here is what Python's two file modes give on the same bytes. ## Two Modes, Two Types `open` works in two modes, and the difference between them is not an option, it is **the type of the returned object**. **Binary mode** (`"rb"`, `"wb"`) gives a byte sequence. Whatever is in the file is what you get; no translation happens. The length is **the byte count**. **Text mode** (`"r"`, `"w"`) gives a string. A decoder steps in between: bytes are turned into characters according to an **encoding**, and line endings are translated. The length is **the character count**. If `encoding` is not given in text mode, the interpreter picks a default, and that default **depends on the environment it runs in**. Same file, same program, a different environment — a different result. Every measurement in this lesson writes the encoding explicitly; that is the condition for it to be measurable. ## Same Bytes, Different Encodings The measurement starts with a single piece of text: a measurement line carrying Turkish letters, a degree sign, and an em dash. This text is encoded with utf-8, and the resulting bytes are tried for decoding with **five separate encodings**. The measurement's assumptions: - **EF19** — The entire measurement runs on a single byte sequence; the sequence is produced by **encoding** the text, it is not written by hand. - **EF20** — The oracle is the source text itself: whether the decoded string is **identical** to the source can be asked, because we produced the source ourselves. - **EF21** — The "broken byte position" is read from the exception object's own field; the error message is not parsed. - **EF22** — The encoding names come from the language's own decoder catalog; none of them belong to an outside tool. - **EF23** — The chunk-size scan splits the same bytes at different sizes and decodes every chunk **separately**; no state is carried between chunks. ```python """Same bytes, different encodings: does it decode, is it identical, where does it break.""" TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam" RAW = TEXT.encode("utf-8") print(f"characters {len(TEXT)} | utf-8 bytes {len(RAW)} | " f"multi-byte characters {sum(len(c.encode('utf-8')) > 1 for c in TEXT)}") ENCODINGS = ("utf-8", "utf-16", "latin-1", "ascii", "cp1254") print() print(f"{'encoding':<12s}{'decoded':<10s}{'characters':>11s}{'identical':>10s}" f" broken class / byte position") for name in ENCODINGS: try: decoded = RAW.decode(name) except UnicodeDecodeError as e: print(f" {name:<10s}{'no':<10s}{'-':>11s}{'-':>10s}" f" {type(e).__name__} / {e.start}") else: print(f" {name:<10s}{'yes':<10s}{len(decoded):>11d}" f"{('yes' if decoded == TEXT else 'no'):>10s} -") print() print(f"{'error mode':<20s}{'characters':>11s}{'identical':>10s} first seven characters") for mode in ("ignore", "replace", "backslashreplace"): c = RAW.decode("ascii", errors=mode) print(f" {mode:<18s}{len(c):>11d}{('yes' if c == TEXT else 'no'):>10s}" f" {c[:7]!r}") print() print(f"{'chunk size':<12s}{'chunks':>7s}{'broken chunks':>15s}") for size in range(4, 25, 2): chunks = [RAW[i:i + size] for i in range(0, len(RAW), size)] broken = 0 for chunk in chunks: try: chunk.decode("utf-8") except UnicodeDecodeError: broken += 1 print(f" {size:<10d}{len(chunks):>7d}{broken:>15d}") whole = 0 try: RAW.decode("utf-8") except UnicodeDecodeError: whole = 1 print(f"broken when the same bytes are decoded whole {whole}") ``` ``` characters 34 | utf-8 bytes 41 | multi-byte characters 6 encoding decoded characters identical broken class / byte position utf-8 yes 34 yes - utf-16 no - - UnicodeDecodeError / 40 latin-1 yes 41 no - ascii no - - UnicodeDecodeError / 10 cp1254 yes 41 no - error mode characters identical first seven characters ignore 28 no 'kuzey y' replace 41 no 'kuzey y' backslashreplace 80 no 'kuzey y' chunk size chunks broken chunks 4 11 4 6 7 2 8 6 2 10 5 2 12 4 2 14 3 2 16 3 0 18 3 0 20 3 2 22 2 0 24 2 2 broken when the same bytes are decoded whole 0 ``` ## The Silent Wrong Answer The first line gives the lesson's core: **34**-character text holds **41** bytes. The **7**-byte difference comes from the **6** multi-byte characters. Character count and byte count are independent of each other, and which one you are counting depends on which mode you read in. The encoding table shows three separate behaviors. **utf-8** decodes, gives **34** characters, and is **identical** to the source. This is the definition of the correct encoding. **ascii** and **utf-16** cannot decode and raise `UnicodeDecodeError`. The positions differ: ascii breaks at the **10th** byte — the first byte of the first Turkish letter. utf-16, by contrast, breaks at the **40th** byte, nearly at the end; because utf-16 reads bytes in pairs and **41** is an odd number, the last byte is left unpaired. Both break, but **where** they break comes from the encoding's own rule. The real finding is the **latin-1** and **cp1254** rows. Both **decode** — raising no exception at all — and give **41** characters. That is, they turn every byte into a character. But they are **not identical**. A single-byte encoding accepts every byte that comes its way as valid; there is nothing there to break on. This is the measurement's most dangerous result: **a wrong encoding does not always produce an exception.** An encoding that raises one warns you; one that does not silently gives back the wrong text, and the mistake is only noticed by a human reading it. Error modes are the deliberate version of this. `ignore` drops bytes it cannot decode and **28** characters remain — the bytes of **6** characters fell away. `replace` puts one character in place of every faulty byte and gives **41**; the number matches the byte count because the replacement is done byte by byte. `backslashreplace` describes every faulty byte with a four-character notation, and the length grows to **80**. All three have `identical` reading **no**: giving an error mode does not recover the data, it only prevents **breaking**. ## The Chunk Boundary The last table gives the cost of reading in chunks in binary mode. The same **41** bytes are split into different sizes, and each chunk is decoded **separately**. The result is irregular. At **4**-byte chunks, **4** of **11** chunks break; at sizes **16** and **18**, none break; at **20**, **2** chunks break again, **0** at **22**, **2** at **24**. The break count does not depend on the chunk size's magnitude, it depends on whether a chunk boundary **falls in the middle of a multi-byte character**. When the same bytes are decoded **whole**, the break count is **0**. The direct consequence of this: if you are reading in binary mode and decoding the bytes yourself, carrying **state** between chunks is your job. Text mode already does this work — the decoder holds back a character left incomplete at a chunk boundary. Binary mode is fast and impartial; because it does not take on the translation, it does not take on translation's problems either, and leaves them to the caller. ## Mode, Type, and the Break Point The second measurement works with a real file. The file is built in a **temporary directory**, and the directory is deleted when the measurement ends; the measurement leaves no permanent trace. - **EF24** — The file is produced in a temporary directory, and its bytes come from the previous measurement's text; no path is printed, because a path depends on the environment. - **EF25** — "Length" is bytes in binary mode, characters in text mode; the two are measured under the same name but are not the same thing. - **EF26** — The break-point measurement tries opening and reading as **separate steps**; which one produces the exception is read from the run. - **EF27** — In the newline measurement, the file carries **two separate line-ending forms**, one two characters long, one a single character; the translation difference arises from this. ```python """Text mode vs binary mode: same file, different type, different length, different break point.""" import tempfile from pathlib import Path TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam" LINED = b"kuzey\r\nyama\xc3\xa7\nolcum\r\n" with tempfile.TemporaryDirectory() as temp_dir: data_file = Path(temp_dir) / "measurement.dat" data_file.write_bytes(TEXT.encode("utf-8")) lines_file = Path(temp_dir) / "lines.dat" lines_file.write_bytes(LINED) print(f"{'read mode':<22s}{'type':<8s}{'length':>8s}{'identical':>10s}") with open(data_file, "rb") as f: binary = f.read() print(f" {'binary':<20s}{type(binary).__name__:<8s}{len(binary):>8d}{'-':>10s}") for name, encoding, mode in (("text utf-8", "utf-8", "strict"), ("text latin-1", "latin-1", "strict"), ("text ascii replace", "ascii", "replace")): with open(data_file, "r", encoding=encoding, errors=mode) as f: text = f.read() print(f" {name:<20s}{type(text).__name__:<8s}{len(text):>8d}" f"{('yes' if text == TEXT else 'no'):>10s}") print() print("step result") try: f = open(data_file, "r", encoding="ascii") except UnicodeDecodeError as e: print(f" {'open':<18s}{type(e).__name__} / byte {e.start}") else: print(f" {'open':<18s}succeeded, file opened") try: f.read() except UnicodeDecodeError as e: print(f" {'read':<18s}{type(e).__name__} / byte {e.start}") finally: f.close() print() print(f"{'read form':<22s}{'length':>8s}{'CR count':>11s}{'lines':>7s}") with open(lines_file, "rb") as f: raw = f.read() print(f" {'binary':<20s}{len(raw):>8d}{raw.count(13):>11d}" f"{len(raw.splitlines()):>7d}") for name, newline_arg in (("text (default)", None), ('text newline=""', "")): with open(lines_file, "r", encoding="utf-8", newline=newline_arg) as f: text = f.read() with open(lines_file, "r", encoding="utf-8", newline=newline_arg) as f: lines = f.readlines() print(f" {name:<20s}{len(text):>8d}{text.count(chr(13)):>11d}" f"{len(lines):>7d}") print("temp directory deleted:", not Path(temp_dir).exists()) ``` ``` read mode type length identical binary bytes 41 - text utf-8 str 34 yes text latin-1 str 41 no text ascii replace str 41 no step result open succeeded, file opened read UnicodeDecodeError / byte 10 read form length CR count lines binary 21 2 3 text (default) 18 0 3 text newline="" 20 2 3 temp directory deleted: True ``` ## Reading the Numbers The first table gives the two modes' difference by type. Binary mode returns `bytes` and the length is **41**; text mode returns `str`, and with the correct encoding the length is **34**. Same file, same content, two different numbers — and both are correct. `latin-1` again gives **41**, and the `identical` column reads **no**: the silent wrong answer seen in the in-memory measurement repeats exactly the same way when reading from a file. The second table is the lesson's sharpest finding. The `open` call with `ascii` encoding **succeeds** — the file opens, no exception comes out. The exception comes at the `read` call, at byte **10**. `open` only prepares the file and stores the encoding name; the decoding work is done at the first read. An encoding error is **not an open error, it is a read error**, and the practical consequence is this: a `try` block wrapping the `open` call cannot catch this error. The block that catches it has to wrap the read. The third table counts newline translation. The file is **21** bytes and carries **2** carriage-return characters inside it, because two lines end with a two-character line ending and one with a single-character line ending. When text mode reads with the default setting, the length drops to **18** and the carriage-return count becomes **0**: the decoder turned every line-ending form into a single character. When `newline=""` is given, the translation turns off, and the length is **20**, the carriage-return count **2**. The line count is **3** across all three reads. Translation does not change **where** the lines end, it only changes how many characters the line ending is written with. Comparing a file's length when read in text mode to the file's byte size therefore gives the wrong result: both encoding and newline translation get in between. ## The Write Side and Open Modes The read side **assumes** the encoding; the write side **establishes** it. The third measurement tries writing the same text with five encodings and does a round-trip test on each: when the encoded bytes are decoded back with the same encoding, does the source text come back? - **EF28** — The round-trip test is done with **its own** encoding in every case; what is measured is whether the encoding can carry the text. - **EF29** — The open modes are tried on the same file **in sequence**; at every step the file's byte count is read from the file system, not computed. - **EF30** — The file is again built in a temporary directory; no path is printed, and the directory is deleted when the measurement ends. ```python """Write side: unencodable characters and file open modes.""" import tempfile from pathlib import Path TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam" print(f"{'encoding':<12s}{'encoded':<10s}{'bytes':>6s}" f"{'round trip identical':>22s} broken class / position") for name in ("utf-8", "utf-16", "latin-1", "ascii", "cp1254"): try: raw = TEXT.encode(name) except UnicodeEncodeError as e: print(f" {name:<10s}{'no':<10s}{'-':>6s}{'-':>22s}" f" {type(e).__name__} / {e.start}") else: identical = "yes" if raw.decode(name) == TEXT else "no" print(f" {name:<10s}{'yes':<10s}{len(raw):>6d}{identical:>22s} -") print() with tempfile.TemporaryDirectory() as temp_dir: log_file = Path(temp_dir) / "measurement.log" print(f"{'mode':<6s}{'call':<8s}{'file bytes':>12s} result") for step, mode in (("1.", "w"), ("2.", "w"), ("3.", "a"), ("4.", "x")): try: with open(log_file, mode, encoding="utf-8") as f: f.write(TEXT + "\n") except FileExistsError as e: print(f" {mode:<4s}{step:<8s}{log_file.stat().st_size:>12d}" f" {type(e).__name__}") else: print(f" {mode:<4s}{step:<8s}{log_file.stat().st_size:>12d} written") with open(log_file, "r", encoding="utf-8") as f: lines = f.readlines() print(f"final state lines {len(lines)}, " f"characters {sum(len(s) for s in lines)}, " f"bytes {log_file.stat().st_size}") ``` ``` encoding encoded bytes round trip identical broken class / position utf-8 yes 41 yes - utf-16 yes 70 yes - latin-1 no - - UnicodeEncodeError / 21 ascii no - - UnicodeEncodeError / 10 cp1254 yes 34 yes - mode call file bytes result w 1. 42 written w 2. 42 written a 3. 84 written x 4. 84 FileExistsError final state lines 2, characters 70, bytes 84 ``` The `cp1254` row is the counterpart of the read table and the lesson's most instructive pair. There, `cp1254` **silently misread** utf-8 bytes; here, that same encoding writes the text into **34** bytes, and the round trip comes back **identical**. Encoding is not a property of a file; it is **the agreement between the writer and the reader**, and the file itself does not carry that agreement. What was wrong was not `cp1254`, it was mistaking bytes written with utf-8 for `cp1254`. The write side parts ways with the read side at one point: `latin-1` and `ascii` **break** here. `latin-1` breaks at the **21st** character, at the em dash; `ascii` breaks at the **10th** character, at the first Turkish letter. When writing, a character that cannot be encoded cannot silently pass through, because that character has **no counterpart** in the encoding. When reading, a silent wrong answer was possible, because every byte had some counterpart. `utf-16` writes the text into **70** bytes — against utf-8's **41** — and it too comes back identical; choosing an encoding is a **size** decision as much as a correctness one. The open-modes table sets apart three separate behaviors. `"w"` writes **42** bytes on the first call; after the second `"w"` call, the file is still **42** — the mode **truncated** the file and wrote it from scratch, the second record erased the first. The `"a"` call brings it up to **84**: append mode does not truncate. The `"x"` call raises `FileExistsError` without writing anything, and the byte count stays at **84**. The last line says the file holds **2** lines, **70** characters, and **84** bytes; the gap between character count and byte count stands here too. ## Summary - Binary mode gives `bytes` and the length is the byte count; text mode gives `str`, the length is the character count — the same file reads as **41** and **34**. - A wrong encoding does not always break: `ascii` breaks at the **10th** byte, `utf-16` at the **40th**, while `latin-1` and `cp1254` never break and silently produce **41** wrong characters. - Error modes prevent breaking, they do not recover the data: `ignore` gives **28**, `replace` **41**, `backslashreplace` **80** characters, and none is identical to the source. - Decoding bytes chunk by chunk depends on the chunk boundary: the same **41** bytes break in **0**, **2**, or **4** chunks depending on size, and in **0** when decoded whole. - An encoding error shows up not at the `open` call but at the `read` call; newline translation brings the length down from **21** to **18** but keeps the line count at **3**. - Encoding is a property not of the file but of the writer and the reader: `cp1254` gives an identical round trip writing in **34** bytes while silently corrupting utf-8 bytes on read; an unencodable character breaks at the **21st** character in `latin-1` when writing, the **10th** in `ascii`. ## Next Step Every file example in this lesson was opened inside a `with` block, and the file closed when the block ended — the temporary directory could even be deleted at the end of the measurement, because no open handle was left behind. `with` has been used as a habit up to now; but it too is syntax, and there is a protocol underneath it. The next lesson measures that protocol: which special method `with` calls on entry, which on exit, and what changes when **an exception occurs** inside the block. The shared setup's `with n + exception` line's three calls get paid off there.