File and I/O Interfaces

The last lesson of the Networking and Data Formats topic closed with a question: a framing rule tells you where a message starts and where it ends, but it does not tell you what the bytes inside that boundary are. This topic asks the same question without carrying it outside the program, to the network. A byte read from a file, a byte accumulated in a buffer, and a byte that just arrived from the network — all three are looking for an answer to the same question: does a byte sequence coming from the other side say, on its own, what type it belongs to?

The answer is no, and this topic’s first lesson shows where that answer comes from: Go’s standard library makes the file, the buffer, the network body and the string speak through a single shared contract — read and write. This contract is so narrow that it says nothing about the source itself; it only streams bytes. The Networking and Data Formats topic already measured that the HTTP body follows this same contract; that is not rebuilt here, and this lesson passes three new sources — a file, a buffer, a string — through the same contract and shows the same result (that type information does not carry across) for each one separately.

Determinism note: the file source sits in a temporary directory built with os.MkdirTemp; the directory’s own path is never printed, and only numbers related to its content are used.

Read and Write Are Two Methods

Go’s standard library has a single condition for a source to count as “readable”: carrying a method with the signature Read(p []byte) (n int, err error). Interfaces themselves — the implicit implementation model, where a type satisfies an interface only by carrying the right methods without a separate declaration saying “I satisfy this interface” — were established in the Interfaces, Errors and Generics course and are not rebuilt here. This lesson measures that model’s most common example in the standard library: the io.Reader interface asks for a single Read method, and the file, network, compression, encryption and text-processing packages are all built on top of this one interface. The write side has a symmetric interface: io.Writer, whose only method is Write(p []byte) (n int, err error).

The smallness of these two interfaces is a design decision, not a shortcoming. Because the only condition for “readable” is this one return signature, types that share no common ancestry at all — os.File, bytes.Buffer, strings.Reader, a network body — can each satisfy the same interface independently. None of them has to imitate the others; each writes its own Read method against its own internal data, and outside code can run the same loop without ever knowing which type it holds. The implicit implementation is especially visible here: the word io.Reader never appears in the definition of the os.File type, but because it carries a Read method with the right signature, it satisfies the interface — the connection is built by signature match, not by declaration.

Four Sources, One Interface

SD1. — The same four-character text ("bolt") is deliberately placed into the three sources below (a file, a buffer, a string); the goal is to compare the tool’s behavior, not the sources’ content. The fourth source — the network body — was already run with the same result (lost) in the Networking and Data Formats topic and is not rebuilt here.

// read.go — a helper that passes three separate sources through the same reader interface
package main

import (
	"bytes"
	"fmt"
	"io"
	"os"
	"strings"
)

func read(r io.Reader) (int, string) {
	b, _ := io.ReadAll(r)
	return len(b), fmt.Sprintf("%T", b)
}

type source struct {
	name string
	r    io.Reader
}

func main() {
	dir, _ := os.MkdirTemp("", "sd1")
	defer os.RemoveAll(dir)
	os.WriteFile(dir+"/content.txt", []byte("bolt"), 0o644)
	file, _ := os.Open(dir + "/content.txt")
	defer file.Close()

	var buffer bytes.Buffer
	buffer.WriteString("bolt")

	str := strings.NewReader("bolt")

	sources := []source{
		{"file", file},
		{"buffer", &buffer},
		{"string", str},
	}
	for _, k := range sources {
		length, kind := read(k.r)
		fmt.Printf("%-10s length=%d type=%s\n", k.name, length, kind)
	}
}

file       length=4 type=[]uint8
buffer     length=4 type=[]uint8
string     length=4 type=[]uint8

The three sources have completely separate implementations: *os.File talks to the operating system through system calls, *bytes.Buffer manages a slice that grows in memory, and *strings.Reader walks over an immutable string. The read function never sees this difference — its parameter is io.Reader, which expects only the Read method. All three produce exactly the same result: four bytes, and a type that is always []uint8 (in Go, byte is an alias for uint8; %T shows it under its own name). The source’s original type — file, buffer, or string — does not survive anywhere past this point; the only thing the read operation produces is a raw byte slice.

This is the first concrete example of this topic’s measurement axis: the value crossing the boundary (the data Read returns) falls into the lost class, because no matter who sent it back on the other side, the result stays in the same raw type. Four bytes read from a file and four bytes read from a string are indistinguishable after the read operation.

The power of implicit implementation does not stop at the standard library’s own types. Any type that writes the Read signature correctly satisfies the io.Reader interface — even a handwritten type that never touches the filesystem or memory.

// handwritten.go — a type that writes its own Read method also satisfies the io.Reader interface
package main

import (
	"fmt"
	"io"
)

// RepeatReader is a handwritten reader that repeats one byte the requested number of times.
type RepeatReader struct {
	Byte      byte
	Remaining int
}

func (t *RepeatReader) Read(p []byte) (int, error) {
	if t.Remaining == 0 {
		return 0, io.EOF
	}
	n := 0
	for n < len(p) && t.Remaining > 0 {
		p[n] = t.Byte
		t.Remaining--
		n++
	}
	return n, nil
}

func main() {
	var source io.Reader = &RepeatReader{Byte: 'x', Remaining: 5}
	data, _ := io.ReadAll(source)
	fmt.Printf("data from our handwritten reader: %q type=%T\n", data, data)
}

data from our handwritten reader: "xxxxx" type=[]uint8

RepeatReader neither opens a file nor wraps a string; it produces bytes according to its own Remaining counter. Yet io.ReadAll sees no difference between it and the other three sources, because the only thing it looks for is a Read method with the right signature. This shows that the four sources’ common ground is actually open to a fifth, a sixth, any number of new sources — the interface stays fixed; the number of implementers is unbounded.

The Write Side: io.Writer

The write direction has the same symmetry. When data is written to a destination that implements io.Writer, the writer does not care whether the destination is a file or a buffer; the only thing that matters is how many bytes were written.

SD2. — The same five-byte data ("screw") is written to both a file and a buffer; both are then read back and checked for equality with each other and with the source.

// write.go — the same data is written to two different writer destinations
package main

import (
	"bytes"
	"fmt"
	"io"
	"os"
)

func write(w io.Writer, data []byte) (int, string) {
	n, _ := w.Write(data)
	return n, fmt.Sprintf("%T", w)
}

func main() {
	dir, _ := os.MkdirTemp("", "sd2")
	defer os.RemoveAll(dir)
	file, _ := os.Create(dir + "/output.txt")

	var buffer bytes.Buffer

	n1, t1 := write(file, []byte("screw"))
	file.Close()
	n2, t2 := write(&buffer, []byte("screw"))

	fmt.Printf("written to file    n=%d type=%s\n", n1, t1)
	fmt.Printf("written to buffer  n=%d type=%s\n", n2, t2)

	readBack, _ := os.ReadFile(dir + "/output.txt")
	fmt.Println("read back from file   :", string(readBack))
	fmt.Println("read back from buffer :", buffer.String())
	fmt.Println("are the two equal     :", string(readBack) == buffer.String())
}

written to file    n=5 type=*os.File
written to buffer  n=5 type=*bytes.Buffer
read back from file   : screw
read back from buffer : screw
are the two equal     : true

The write function’s signature is symmetric to read‘s: its parameter is not a concrete destination but the io.Writer interface. *os.File and *bytes.Buffer are two types that do not resemble each other at all, yet both can pass through the same function because both carry a Write method. The result is symmetric too: the same five bytes are written to both, and the content read back from each is equal to the other. This line adds one more answer to the topic’s question — on the write side too, the value crossing the boundary is carried as the same raw byte sequence regardless of the destination’s own type; whether the destination is *os.File or *bytes.Buffer never affects the content written.

Composition: io.Copy and bufio Are Built on the Same Interface

Narrow interfaces let the standard library write generic tools between a source and a destination. io.Copy(dst io.Writer, src io.Reader) moves data from a source to a destination without ever knowing the concrete type of its parameters — the source can be a file, a buffer, a string, or any other io.Reader; the destination can likewise be any io.Writer. The same idea holds in bufio: bufio.NewScanner takes an io.Reader and turns it into a wrapper that reads it line by line (or word by word), without ever asking whether the source it wraps is a file or a string.

// combine.go — io.Copy and bufio work on any reader/writer without knowing the concrete type
package main

import (
	"bufio"
	"bytes"
	"fmt"
	"io"
	"os"
	"strings"
)

func main() {
	dir, _ := os.MkdirTemp("", "sd4")
	defer os.RemoveAll(dir)
	os.WriteFile(dir+"/source.txt", []byte("nut and bolt"), 0o644)
	file, _ := os.Open(dir + "/source.txt")
	defer file.Close()

	var dest bytes.Buffer
	n, _ := io.Copy(&dest, file)
	fmt.Println("bytes moved from file to buffer :", n)
	fmt.Println("buffer contents                 :", dest.String())

	lineScanner := bufio.NewScanner(strings.NewReader("first line\nsecond line"))
	lineCount := 0
	for lineScanner.Scan() {
		lineCount++
	}
	fmt.Println("line count counted by bufio:", lineCount)
}

bytes moved from file to buffer : 12
buffer contents                 : nut and bolt
line count counted by bufio: 2

The io.Copy call’s signature contains no concrete type at all — no file name, no buffer kind, just two interfaces. This function is written once in the standard library and is reusable in every combination — file to network, network to buffer, buffer to another file; no matter how many concrete types the source and destination have, io.Copy itself never changes. bufio.NewScanner follows the same principle: its input is an io.Reader, and although it wraps another io.Reader, it offers a richer interface (line-by-line scanning) — small interfaces composing on top of each other is the direct result of producing small, pluggable pieces instead of writing one large “file reader” type.

Limiting Measurement: Partial Reads and End of File

The two measurements so far used io.ReadAll — a helper that runs the loop internally until the source is exhausted. But the Read method itself does not guarantee this: a single call is not required to return the full number of requested bytes, and the data ending and the error arriving are not required to happen in the same call. This distinction is written clearly in the standard library’s documentation, but it is hard to see without running it.

SD3. — A five-byte string is deliberately read with a two-byte buffer; how many bytes arrive at each step, and at which step the error appears, are printed separately.

// partial.go — reading with a small buffer returns the data and end of file in separate steps
package main

import (
	"fmt"
	"strings"
)

func main() {
	source := strings.NewReader("board")
	buffer := make([]byte, 2)
	total := 0
	step := 0
	for {
		n, err := source.Read(buffer)
		step++
		fmt.Printf("step=%d read=%d chunk=%q err=%v\n", step, n, buffer[:n], err)
		total += n
		if err != nil {
			break
		}
	}
	fmt.Println("total bytes read:", total)
}

step=1 read=2 chunk="bo" err=<nil>
step=2 read=2 chunk="ar" err=<nil>
step=3 read=1 chunk="d" err=<nil>
step=4 read=0 chunk="" err=EOF
total bytes read: 5

The five-byte string “board” requires four separate Read calls when read with a two-byte buffer — the third step leaves a single byte, and the error is still nil at that step. The end-of-file signal (io.EOF) arrives only on the fourth call, with no data at all. This is a contract of the source’s behavior: Read can return a byte count greater than 0 and a non-nil error at the same time in a single call, or it can split the data and the error across separate calls. Code that relies only on a helper like io.ReadAll never sees this distinction, because the helper completes the loop internally. But code that calls Read directly can skip the last byte at the third step if it assumes the data ending and the error arriving happen in the same step.

This is the boundary specific to this lesson on the topic’s measurement axis: the read interface does not carry a loss — all five bytes arrived in the right order — but it never takes on the job of separating a partial read from the end of file. That distinction stays the responsibility of the code calling Read; the standard library’s io.ReadFull and io.ReadAll helpers take on this responsibility, but they are tools the source itself does not require.

For an in-memory source like strings.NewReader, managing this distinction by hand looks unnecessary — all the data is already ready in memory, and Read not giving everything at once is just a matter of choice. But the interface’s contract is the same for every implementer; for an io.Reader that reads from a network socket or a pipe, a partial read is not a choice, it comes from the nature of the source — if the other side sends the data in pieces, Read returning everything in a single call is not possible. The io.Reader contract does not distinguish between these two cases (an in-memory source and a source arriving in pieces); both carry the same signature, and code that correctly manages partial reads works for both.

Summary

• io.Reader and io.Writer are two of the language’s narrowest interfaces; they ask for a single Read and a single Write method respectively, which is why types that resemble each other not at all — a file, a buffer, a string — can each satisfy the same contract independently.
• The same content read from a file, a buffer and a string becomes indistinguishable after the read operation: all three give a result of the same length and the same []uint8 type — the value crossing the boundary carries no type information.
• The same symmetry holds in the write direction too: *os.File and *bytes.Buffer pass through the same write function, and the content written to each turns out equal to the other, byte for byte.
• Tools like io.Copy and bufio.NewScanner work without ever knowing the concrete type; small interfaces can therefore compose freely across a file, a buffer and a string.
• A partial read and the end of file are separate events: reading with a small buffer can split the data across several steps, and the end-of-file signal arrives independently of the data, in its own step.
• Managing this distinction is the caller’s job; helpers like io.ReadAll take on this responsibility, but the interface itself does not require it.

Next Step

All three sources measured in this lesson sat somewhere reachable while the program was running: on a filesystem path, or in the program’s own memory. But the data a program needs is not always available while it is running — a configuration file, an HTML template, a static asset, can instead travel inside the program’s own binary. The next lesson looks at the mechanism that moves this link to compile time: how does a source file sit directly inside a compiled program without ever being looked up while it runs?