File Operations

The previous lesson’s twelve records sat ready in memory. The value field was already a string; the only job was converting it to a number and checking it against the field’s bounds. The domain root paid off because the error’s source was the domain itself.

When records come from a file, one more layer gets inserted. What sits in a file is neither a string nor a number — it is bytes. Converting bytes to a string is an encoding decision, and if that decision is wrong, the error never reaches the domain layer at all; it shows up while reading. The concepts of file, encoding, and byte were built in the How Computers Work course; the concept was built there, what is measured here is what Python’s two file modes give on the same bytes.

Two Modes, Two Types

open works in two modes, and the difference between them is not an option, it is the type of the returned object.

Binary mode ("rb", "wb") gives a byte sequence. Whatever is in the file is what you get; no translation happens. The length is the byte count.

Text mode ("r", "w") gives a string. A decoder steps in between: bytes are turned into characters according to an encoding, and line endings are translated. The length is the character count.

If encoding is not given in text mode, the interpreter picks a default, and that default depends on the environment it runs in. Same file, same program, a different environment — a different result. Every measurement in this lesson writes the encoding explicitly; that is the condition for it to be measurable.

Same Bytes, Different Encodings

The measurement starts with a single piece of text: a measurement line carrying Turkish letters, a degree sign, and an em dash. This text is encoded with utf-8, and the resulting bytes are tried for decoding with five separate encodings.

The measurement’s assumptions:

• EF19 — The entire measurement runs on a single byte sequence; the sequence is produced by encoding the text, it is not written by hand.
• EF20 — The oracle is the source text itself: whether the decoded string is identical to the source can be asked, because we produced the source ourselves.
• EF21 — The “broken byte position” is read from the exception object’s own field; the error message is not parsed.
• EF22 — The encoding names come from the language’s own decoder catalog; none of them belong to an outside tool.
• EF23 — The chunk-size scan splits the same bytes at different sizes and decodes every chunk separately; no state is carried between chunks.

"""Same bytes, different encodings: does it decode, is it identical, where does it break."""

TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam"
RAW = TEXT.encode("utf-8")

print(f"characters {len(TEXT)} | utf-8 bytes {len(RAW)} | "
      f"multi-byte characters {sum(len(c.encode('utf-8')) > 1 for c in TEXT)}")

ENCODINGS = ("utf-8", "utf-16", "latin-1", "ascii", "cp1254")

print()
print(f"{'encoding':<12s}{'decoded':<10s}{'characters':>11s}{'identical':>10s}"
      f"   broken class / byte position")
for name in ENCODINGS:
    try:
        decoded = RAW.decode(name)
    except UnicodeDecodeError as e:
        print(f"  {name:<10s}{'no':<10s}{'-':>11s}{'-':>10s}"
              f"   {type(e).__name__} / {e.start}")
    else:
        print(f"  {name:<10s}{'yes':<10s}{len(decoded):>11d}"
              f"{('yes' if decoded == TEXT else 'no'):>10s}   -")

print()
print(f"{'error mode':<20s}{'characters':>11s}{'identical':>10s}   first seven characters")
for mode in ("ignore", "replace", "backslashreplace"):
    c = RAW.decode("ascii", errors=mode)
    print(f"  {mode:<18s}{len(c):>11d}{('yes' if c == TEXT else 'no'):>10s}"
          f"   {c[:7]!r}")

print()
print(f"{'chunk size':<12s}{'chunks':>7s}{'broken chunks':>15s}")
for size in range(4, 25, 2):
    chunks = [RAW[i:i + size] for i in range(0, len(RAW), size)]
    broken = 0
    for chunk in chunks:
        try:
            chunk.decode("utf-8")
        except UnicodeDecodeError:
            broken += 1
    print(f"  {size:<10d}{len(chunks):>7d}{broken:>15d}")
whole = 0
try:
    RAW.decode("utf-8")
except UnicodeDecodeError:
    whole = 1
print(f"broken when the same bytes are decoded whole {whole}")

characters 34 | utf-8 bytes 41 | multi-byte characters 6

encoding    decoded    characters identical   broken class / byte position
  utf-8     yes                34       yes   -
  utf-16    no                  -         -   UnicodeDecodeError / 40
  latin-1   yes                41        no   -
  ascii     no                  -         -   UnicodeDecodeError / 10
  cp1254    yes                41        no   -

error mode           characters identical   first seven characters
  ignore                     28        no   'kuzey y'
  replace                    41        no   'kuzey y'
  backslashreplace           80        no   'kuzey y'

chunk size   chunks  broken chunks
  4              11              4
  6               7              2
  8               6              2
  10              5              2
  12              4              2
  14              3              2
  16              3              0
  18              3              0
  20              3              2
  22              2              0
  24              2              2
broken when the same bytes are decoded whole 0

The Silent Wrong Answer

The first line gives the lesson’s core: 34-character text holds 41 bytes. The 7-byte difference comes from the 6 multi-byte characters. Character count and byte count are independent of each other, and which one you are counting depends on which mode you read in.

The encoding table shows three separate behaviors. utf-8 decodes, gives 34 characters, and is identical to the source. This is the definition of the correct encoding.

ascii and utf-16 cannot decode and raise UnicodeDecodeError. The positions differ: ascii breaks at the 10th byte — the first byte of the first Turkish letter. utf-16, by contrast, breaks at the 40th byte, nearly at the end; because utf-16 reads bytes in pairs and 41 is an odd number, the last byte is left unpaired. Both break, but where they break comes from the encoding’s own rule.

The real finding is the latin-1 and cp1254 rows. Both decode — raising no exception at all — and give 41 characters. That is, they turn every byte into a character. But they are not identical. A single-byte encoding accepts every byte that comes its way as valid; there is nothing there to break on. This is the measurement’s most dangerous result: a wrong encoding does not always produce an exception. An encoding that raises one warns you; one that does not silently gives back the wrong text, and the mistake is only noticed by a human reading it.

Error modes are the deliberate version of this. ignore drops bytes it cannot decode and 28 characters remain — the bytes of 6 characters fell away. replace puts one character in place of every faulty byte and gives 41; the number matches the byte count because the replacement is done byte by byte. backslashreplace describes every faulty byte with a four-character notation, and the length grows to 80. All three have identical reading no: giving an error mode does not recover the data, it only prevents breaking.

The Chunk Boundary

The last table gives the cost of reading in chunks in binary mode. The same 41 bytes are split into different sizes, and each chunk is decoded separately.

The result is irregular. At 4-byte chunks, 4 of 11 chunks break; at sizes 16 and 18, none break; at 20, 2 chunks break again, 0 at 22, 2 at 24. The break count does not depend on the chunk size’s magnitude, it depends on whether a chunk boundary falls in the middle of a multi-byte character. When the same bytes are decoded whole, the break count is 0.

The direct consequence of this: if you are reading in binary mode and decoding the bytes yourself, carrying state between chunks is your job. Text mode already does this work — the decoder holds back a character left incomplete at a chunk boundary. Binary mode is fast and impartial; because it does not take on the translation, it does not take on translation’s problems either, and leaves them to the caller.

Mode, Type, and the Break Point

The second measurement works with a real file. The file is built in a temporary directory, and the directory is deleted when the measurement ends; the measurement leaves no permanent trace.

• EF24 — The file is produced in a temporary directory, and its bytes come from the previous measurement’s text; no path is printed, because a path depends on the environment.
• EF25 — “Length” is bytes in binary mode, characters in text mode; the two are measured under the same name but are not the same thing.
• EF26 — The break-point measurement tries opening and reading as separate steps; which one produces the exception is read from the run.
• EF27 — In the newline measurement, the file carries two separate line-ending forms, one two characters long, one a single character; the translation difference arises from this.

"""Text mode vs binary mode: same file, different type, different length, different break point."""
import tempfile
from pathlib import Path

TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam"
LINED = b"kuzey\r\nyama\xc3\xa7\nolcum\r\n"

with tempfile.TemporaryDirectory() as temp_dir:
    data_file = Path(temp_dir) / "measurement.dat"
    data_file.write_bytes(TEXT.encode("utf-8"))
    lines_file = Path(temp_dir) / "lines.dat"
    lines_file.write_bytes(LINED)

    print(f"{'read mode':<22s}{'type':<8s}{'length':>8s}{'identical':>10s}")
    with open(data_file, "rb") as f:
        binary = f.read()
    print(f"  {'binary':<20s}{type(binary).__name__:<8s}{len(binary):>8d}{'-':>10s}")
    for name, encoding, mode in (("text utf-8", "utf-8", "strict"),
                             ("text latin-1", "latin-1", "strict"),
                             ("text ascii replace", "ascii", "replace")):
        with open(data_file, "r", encoding=encoding, errors=mode) as f:
            text = f.read()
        print(f"  {name:<20s}{type(text).__name__:<8s}{len(text):>8d}"
              f"{('yes' if text == TEXT else 'no'):>10s}")

    print()
    print("step                result")
    try:
        f = open(data_file, "r", encoding="ascii")
    except UnicodeDecodeError as e:
        print(f"  {'open':<18s}{type(e).__name__} / byte {e.start}")
    else:
        print(f"  {'open':<18s}succeeded, file opened")
        try:
            f.read()
        except UnicodeDecodeError as e:
            print(f"  {'read':<18s}{type(e).__name__} / byte {e.start}")
        finally:
            f.close()

    print()
    print(f"{'read form':<22s}{'length':>8s}{'CR count':>11s}{'lines':>7s}")
    with open(lines_file, "rb") as f:
        raw = f.read()
    print(f"  {'binary':<20s}{len(raw):>8d}{raw.count(13):>11d}"
          f"{len(raw.splitlines()):>7d}")
    for name, newline_arg in (("text (default)", None), ('text newline=""', "")):
        with open(lines_file, "r", encoding="utf-8", newline=newline_arg) as f:
            text = f.read()
        with open(lines_file, "r", encoding="utf-8", newline=newline_arg) as f:
            lines = f.readlines()
        print(f"  {name:<20s}{len(text):>8d}{text.count(chr(13)):>11d}"
              f"{len(lines):>7d}")

print("temp directory deleted:", not Path(temp_dir).exists())

read mode             type      length identical
  binary              bytes         41         -
  text utf-8          str           34       yes
  text latin-1        str           41        no
  text ascii replace  str           41        no

step                result
  open              succeeded, file opened
  read              UnicodeDecodeError / byte 10

read form               length   CR count  lines
  binary                    21          2      3
  text (default)            18          0      3
  text newline=""           20          2      3
temp directory deleted: True

Reading the Numbers

The first table gives the two modes’ difference by type. Binary mode returns bytes and the length is 41; text mode returns str, and with the correct encoding the length is 34. Same file, same content, two different numbers — and both are correct. latin-1 again gives 41, and the identical column reads no: the silent wrong answer seen in the in-memory measurement repeats exactly the same way when reading from a file.

The second table is the lesson’s sharpest finding. The open call with ascii encoding succeeds — the file opens, no exception comes out. The exception comes at the read call, at byte 10. open only prepares the file and stores the encoding name; the decoding work is done at the first read. An encoding error is not an open error, it is a read error, and the practical consequence is this: a try block wrapping the open call cannot catch this error. The block that catches it has to wrap the read.

The third table counts newline translation. The file is 21 bytes and carries 2 carriage-return characters inside it, because two lines end with a two-character line ending and one with a single-character line ending. When text mode reads with the default setting, the length drops to 18 and the carriage-return count becomes 0: the decoder turned every line-ending form into a single character. When newline="" is given, the translation turns off, and the length is 20, the carriage-return count 2.

The line count is 3 across all three reads. Translation does not change where the lines end, it only changes how many characters the line ending is written with. Comparing a file’s length when read in text mode to the file’s byte size therefore gives the wrong result: both encoding and newline translation get in between.

The Write Side and Open Modes

The read side assumes the encoding; the write side establishes it. The third measurement tries writing the same text with five encodings and does a round-trip test on each: when the encoded bytes are decoded back with the same encoding, does the source text come back?

• EF28 — The round-trip test is done with its own encoding in every case; what is measured is whether the encoding can carry the text.
• EF29 — The open modes are tried on the same file in sequence; at every step the file’s byte count is read from the file system, not computed.
• EF30 — The file is again built in a temporary directory; no path is printed, and the directory is deleted when the measurement ends.

"""Write side: unencodable characters and file open modes."""
import tempfile
from pathlib import Path

TEXT = "kuzey yamaç: 12,5 °C — ölçüm tamam"

print(f"{'encoding':<12s}{'encoded':<10s}{'bytes':>6s}"
      f"{'round trip identical':>22s}   broken class / position")
for name in ("utf-8", "utf-16", "latin-1", "ascii", "cp1254"):
    try:
        raw = TEXT.encode(name)
    except UnicodeEncodeError as e:
        print(f"  {name:<10s}{'no':<10s}{'-':>6s}{'-':>22s}"
              f"   {type(e).__name__} / {e.start}")
    else:
        identical = "yes" if raw.decode(name) == TEXT else "no"
        print(f"  {name:<10s}{'yes':<10s}{len(raw):>6d}{identical:>22s}   -")

print()
with tempfile.TemporaryDirectory() as temp_dir:
    log_file = Path(temp_dir) / "measurement.log"
    print(f"{'mode':<6s}{'call':<8s}{'file bytes':>12s}   result")
    for step, mode in (("1.", "w"), ("2.", "w"), ("3.", "a"), ("4.", "x")):
        try:
            with open(log_file, mode, encoding="utf-8") as f:
                f.write(TEXT + "\n")
        except FileExistsError as e:
            print(f"  {mode:<4s}{step:<8s}{log_file.stat().st_size:>12d}"
                  f"   {type(e).__name__}")
        else:
            print(f"  {mode:<4s}{step:<8s}{log_file.stat().st_size:>12d}   written")
    with open(log_file, "r", encoding="utf-8") as f:
        lines = f.readlines()
    print(f"final state lines {len(lines)}, "
          f"characters {sum(len(s) for s in lines)}, "
          f"bytes {log_file.stat().st_size}")

encoding    encoded    bytes  round trip identical   broken class / position
  utf-8     yes           41                   yes   -
  utf-16    yes           70                   yes   -
  latin-1   no             -                     -   UnicodeEncodeError / 21
  ascii     no             -                     -   UnicodeEncodeError / 10
  cp1254    yes           34                   yes   -

mode  call      file bytes   result
  w   1.                42   written
  w   2.                42   written
  a   3.                84   written
  x   4.                84   FileExistsError
final state lines 2, characters 70, bytes 84

The cp1254 row is the counterpart of the read table and the lesson’s most instructive pair. There, cp1254 silently misread utf-8 bytes; here, that same encoding writes the text into 34 bytes, and the round trip comes back identical. Encoding is not a property of a file; it is the agreement between the writer and the reader, and the file itself does not carry that agreement. What was wrong was not cp1254, it was mistaking bytes written with utf-8 for cp1254.

The write side parts ways with the read side at one point: latin-1 and ascii break here. latin-1 breaks at the 21st character, at the em dash; ascii breaks at the 10th character, at the first Turkish letter. When writing, a character that cannot be encoded cannot silently pass through, because that character has no counterpart in the encoding. When reading, a silent wrong answer was possible, because every byte had some counterpart. utf-16 writes the text into 70 bytes — against utf-8’s 41 — and it too comes back identical; choosing an encoding is a size decision as much as a correctness one.

The open-modes table sets apart three separate behaviors. "w" writes 42 bytes on the first call; after the second "w" call, the file is still 42 — the mode truncated the file and wrote it from scratch, the second record erased the first. The "a" call brings it up to 84: append mode does not truncate. The "x" call raises FileExistsError without writing anything, and the byte count stays at 84. The last line says the file holds 2 lines, 70 characters, and 84 bytes; the gap between character count and byte count stands here too.

Summary

• Binary mode gives bytes and the length is the byte count; text mode gives str, the length is the character count — the same file reads as 41 and 34.
• A wrong encoding does not always break: ascii breaks at the 10th byte, utf-16 at the 40th, while latin-1 and cp1254 never break and silently produce 41 wrong characters.
• Error modes prevent breaking, they do not recover the data: ignore gives 28, replace 41, backslashreplace 80 characters, and none is identical to the source.
• Decoding bytes chunk by chunk depends on the chunk boundary: the same 41 bytes break in 0, 2, or 4 chunks depending on size, and in 0 when decoded whole.
• An encoding error shows up not at the open call but at the read call; newline translation brings the length down from 21 to 18 but keeps the line count at 3.
• Encoding is a property not of the file but of the writer and the reader: cp1254 gives an identical round trip writing in 34 bytes while silently corrupting utf-8 bytes on read; an unencodable character breaks at the 21st character in latin-1 when writing, the 10th in ascii.

Next Step

Every file example in this lesson was opened inside a with block, and the file closed when the block ended — the temporary directory could even be deleted at the end of the measurement, because no open handle was left behind. with has been used as a habit up to now; but it too is syntax, and there is a protocol underneath it. The next lesson measures that protocol: which special method with calls on entry, which on exit, and what changes when an exception occurs inside the block. The shared setup’s with n + exception line’s three calls get paid off there.