--- title: 'How the Internet Works' source: 'https://academia.sh/en/courses/how-the-internet-works' language: en updated: '2026-08-17T04:39:06+00:00' license: 'CC BY-SA 4.0' lessons: 12 outcomes: - 'Describes the end-to-end journey of a request typed into an address bar' - 'Separates the concepts of domain name, hosting and server' - 'Reads the parts of the HTTP request and response cycle' --- # How the Internet Works The journey of a request typed into an address bar, out across the network to the server and back with the response; the concepts of domain name, hosting and HTTP. ## Foundations of Networks The intuition of endpoints, links, and packet switching; the client-server model, the internet's structure as a network of networks, and IP addressing. ### What Is a Network Source: https://academia.sh/en/courses/how-the-internet-works/what-is-a-network Endpoints, links, and switching; why packet switching was chosen over circuit switching, and the consequences of best-effort delivery. When you type `http://example.test/` into an address bar and send the request, a page appears within a fraction of a second. What happens in between cannot be summarized in a single sentence: a name must be translated into an address, a connection established between two machines, a request formatted and sent, and the returned response rendered. This course unpacks that journey piece by piece. Each lesson builds one link of the chain; the last lesson joins them all, start to finish. At the bottom of the chain stands the most concrete question: how does data move between two machines? #### Endpoints, Links, and Switches A **network** is a structure made of machines that can carry data between each other. It is defined by three components. **Hosts** are the endpoint machines that produce and consume data. The machine where you typed the request is a host; the machine holding the `example.test` page is also a host. The network exists for the hosts; the hosts do not exist for the network. **Links** are the medium that physically joins two points: copper wire, glass fiber, radio waves. Every link has two measurable properties. **Bandwidth** is the amount of data that can be carried per unit of time. **Latency** is the time it takes for data to travel from one end to the other. The two are independent: a link with wide bandwidth can still have high latency. Part of latency is bounded by the laws of physics; a signal cannot travel faster than the medium's propagation speed. **Switching nodes** connect more than two hosts to each other. Attaching every host to every other with a separate link is unworkable: for $n$ hosts the number of links needed is $\frac{n(n-1)}{2}$, which quickly becomes impossible as $n$ grows. Instead, hosts connect to shared nodes, the nodes connect to each other, and data passes from node to node. The node that makes this forwarding decision is called a **router**. #### Two Switching Models There are two different ways to pass data through the nodes, and the choice between them is the most fundamental of the decisions that shape the internet's design. ##### Circuit Switching In the **circuit switching** model, a path is established between two endpoints before communication begins, and the resources along that path are reserved for those two endpoints for the duration of the communication. Once the path is established, data flows through the reserved capacity. This model has one advantage: because the resource is reserved, performance is predictable. If the circuit is established, the reserved capacity is guaranteed. The cost is wasted resources. While reading a page you send no data; while typing text you pause. Data communication is **bursty**: short dense intervals, long silences. The reserved circuit stays reserved through the silence too, and no one else can use that capacity. The second cost is setup time: a single page request requires first establishing a path, then sending the data, then tearing the path down. ##### Packet Switching In the **packet switching** model, no path is established in advance. The data being sent is split into pieces called **packets**. Each packet carries its own destination on itself, and the nodes forward each packet separately, the moment it arrives. Every packet has two parts. The **header** carries the information needed for routing, such as where the packet came from and where it is going. The **payload** is the data being carried itself. The envelope-and-letter analogy holds: the nodes look at the envelope, they do not open the letter. This model's gain is **statistical multiplexing**: a link is shared by everyone who has data to send at that moment. A silent host holds no capacity. Because average total demand is far smaller than the sum of peak demands, the same link serves far more hosts. The second gain is resilience. With no path established in advance, a broken link brings down no "circuit"; the next packets travel a different route instead. Failure of part of the network causes communication to be rerouted rather than stopped entirely. The cost is uncertainty. Because no capacity is reserved, when many packets arrive at the same node at once, queues form; if a queue fills up, packets are dropped. Delivery time varies from packet to packet. ##### The Reason for the Choice The internet is built on packet switching. The reason is the nature of the traffic carried: computer communication is bursty and the number of hosts is enormous. Reserving resources in advance leaves most of the resource idle. Packet switching gives up predictability to buy efficiency and resilience. | Criterion | Circuit switching | Packet switching | |---|---|---| | Resource reservation | Before communication, for its duration | None; shared per packet | | Idle capacity | Stays reserved, unusable | Used by others | | Setup cost | A path is built for each communication | None | | Failure behavior | Communication drops when the path breaks | Packets take another route | | Delivery time | Predictable | Variable with load | | Under load | New circuits are refused | Queues grow, packets may drop | This trade-off also names the promise the network makes: **best-effort** delivery. The network does everything it can to forward the packet, but it makes no promise that it will arrive, that it will arrive in order, or that it will arrive only once. #### Splitting a Message into Packets With such a weak promise, how does a reliable page get displayed? The answer is that the hosts complete what the promise leaves out. The sending endpoint splits the data into numbered packets; the receiving endpoint uses the numbers to restore the order and detect what is missing. The program below shows this splitting and reassembly. Without using the network layer itself, only the operation performed on the data is examined. ```python MESSAGE = "Hello Example Page!" PACKET_SIZE = 5 packets: list[tuple[int, str]] = [ (seq, MESSAGE[i:i + PACKET_SIZE]) for seq, i in enumerate(range(0, len(MESSAGE), PACKET_SIZE)) ] for seq, payload in packets: print(f"seq={seq} payload={payload!r}") print() # Packets can take different routes; arrival order is not send order. arrival_order = [2, 0, 3, 1] received = [packets[i] for i in arrival_order] print("order seen on arrival:", [seq for seq, _ in received]) reassembled = "".join(payload for seq, payload in sorted(received)) print("reassembled:", repr(reassembled)) print("lossless:", reassembled == MESSAGE) ``` Output: ``` seq=0 payload='Hello' seq=1 payload=' Exam' seq=2 payload='ple P' seq=3 payload='age!' order seen on arrival: [2, 0, 3, 1] reassembled: 'Hello Example Page!' lossless: True ``` The packets arrived out of order; thanks to the sequence number, the original message was reassembled without loss. Without the number, the scrambled arrival would have been unrecoverable corruption. This is an example of a principle in network design: **hard guarantees are built at the endpoints.** Asking intermediate nodes to preserve order would complicate and slow them down; the endpoints, meanwhile, already have to see the data as a whole. Reordering, re-requesting a missing piece, and discarding a duplicate are all left to the endpoints. The protocols that carry out this work are the subject of the Network Models and Protocols course. In real packets, the payload's bytes are interpreted according to the rules defined in the How Computers Work course: text is converted to bytes with a specific character encoding, and the numeric fields in the header are written in a specific byte order. The big-endian layout used in network headers was covered there under the name **network byte order**; that is where the name comes from. #### Packet Size and Fragmentation How large a packet can be is not free to choose. Every type of link has a maximum packet size it can carry. Data exceeding that limit must be split. Both directions of that limit carry a cost. As packets get smaller, the proportion of header overhead rises: every packet carries its own header, so more headers are sent to carry the same data. As packets get larger, the cost of a single loss rises, and the time a packet occupies the link grows longer. In the example above, splitting the 19-character message into 5-byte packets produced four packets, and the last one carried only four bytes. It is normal for the last packet to be under-filled; the data size does not have to be a multiple of the packet size. #### Summary - A network is made of hosts, links, and switching nodes that make the forwarding decision; links are characterized by bandwidth and latency. - Circuit switching reserves resources before communication and provides predictability, but leaves capacity idle under bursty traffic. - Packet switching does not reserve resources in advance; it gains efficiency through statistical multiplexing and fault tolerance from not having a fixed path. - The cost of this choice is best-effort delivery: the network makes no promise that a packet will arrive, arrive in order, or arrive only once. - A message is split into numbered packets carrying a header and a payload; reordering and detecting loss happen at the endpoints, not the intermediate nodes. #### Next Step This lesson established how packets are carried, but it did not say who starts the carrying. The machine holding the `example.test` page and the machine where you typed the request are equal hosts as far as the network is concerned; their roles, however, are not symmetric. One waits continuously, the other starts talking whenever it wants. The next lesson defines that role distinction and runs a real server to show how the two machines find each other. ### The Client–Server Model Source: https://academia.sh/en/courses/how-the-internet-works/client-server-model The distinction between the role that initiates a connection and the one that waits, the concept of a port, and a comparison with the peer-to-peer model. The previous lesson established how packets are carried across the network and left open the question of who initiates communication. The network carries both hosts the same way; the difference between them is not the network's property but that of the programs running on them. This lesson defines that difference: why does the program on the machine holding the `example.test` page wait continuously, while the program on your machine starts talking whenever it wants? By the end of the lesson, this course's example server will be up and running. #### The Criterion That Separates the Roles A **server** is a program that waits on a port and responds to incoming requests. A **client** is a program that initiates the connection and sends the request. The only criterion for the distinction is **who initiates the connection**. A common misunderstanding is treating the server as the "powerful machine" and the client as the "weak machine." This is not true: the role belongs to the program's behavior, not the hardware. The same machine can be a server for one program and a client for another; servers are often clients of other servers. The roles produce two asymmetries. **Discoverability.** The client needs to be able to find the server; the server does not need to know the client. This is why the server's address must be stable and known in advance. The client's address can be temporary and can change from connection to connection. **Availability.** The server must be listening at every moment a request might arrive. The client, in contrast, runs only at the moment it wants to speak. The server's requirement to run continuously is the source of the hosting problem covered in later lessons. #### The Port An address specifies a machine. But more than one server program can run on a machine at the same time. Telling which program an incoming packet belongs to requires a second number: the **port**. The address points to the building, the port to the apartment inside it. A connection is defined by four values: source address, source port, destination address, destination port. As long as that four-tuple is unique, countless connections can coexist between the same two machines. Servers listen on well-known numbers; like the client's address, the client's port is usually a temporary number assigned by the operating system at that moment. Unencrypted web traffic is listened for on port 80, encrypted traffic on port 443. These numbers are assigned by a registration authority and stay invisible because they are not written in the address bar: typing `http://example.test/` assumes 80, and `https://example.test/` assumes 443. If a different number is to be used, it is written explicitly in the address. This course's examples use port 8080; choosing an unprivileged number allows the server to run without administrator rights. #### Writing a Server The most certain way to explain what a server does is to run one. The program below listens on a port, accepts a single connection, prints the bytes it receives, and sends back a fixed response. ```python # single_connection_server.py import socket BODY = b"\n

Example Page

\n" RESPONSE = ( b"HTTP/1.1 200 OK\r\n" b"Content-Type: text/html; charset=utf-8\r\n" b"Content-Length: " + str(len(BODY)).encode() + b"\r\n" b"Connection: close\r\n" b"\r\n" ) + BODY def run_server(port: int = 8080) -> None: with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener: listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) listener.bind(("127.0.0.1", port)) listener.listen(1) print("listening on:", port, flush=True) client, address = listener.accept() with client: request = client.recv(4096) print("--- incoming request ---") print(request.decode("utf-8", "replace"), end="") print("--- end of request ---") client.sendall(RESPONSE) run_server() ``` The program executes four steps in order, and these four steps are the same in every server: 1. `bind` — an address and port are claimed. 2. `listen` — the program becomes ready to accept incoming connections. 3. `accept` — the program **waits** until a client connects. Execution stops on this line. 4. `recv` / `sendall` — the request is read, the response is written. The third step is the essence of the role. The server waits without knowing when or who will connect. The **socket** in the `socket` calls is the connection's handle on the program's side; the operating system reads and writes data through this handle. #### Running a Client While the server runs, a client is run from another terminal. `curl` is used here as the client. The `--resolve` option manually declares which address the name `example.test` corresponds to; name resolution is the subject of the next topic, and this option stands in for it for now. The `-A` option fixes the text the client identifies itself with. The `--noproxy '*'` option keeps a proxy setting defined in the environment from affecting this local request. The commands below run as one piece: the server is started in the background and what it prints is redirected to `server_output.txt`, the `until` loop waits for the line that announces the server is listening (a probe request would use up the single connection the server accepts), and `wait` blocks until the server has handled its single connection and exited. If you run the server in a separate terminal, you only need to type the `curl` line. ```sh rm -f server_output.txt python3 single_connection_server.py > server_output.txt & until grep -q listening server_output.txt; do sleep 0.05; done curl -sv --noproxy '*' --resolve example.test:8080:127.0.0.1 -A "example-client" http://example.test:8080/ wait ``` The exchange as seen on the client side (`>` shows outgoing, `<` incoming lines): ``` * Added example.test:8080:127.0.0.1 to DNS cache * Hostname example.test was found in DNS cache * Trying 127.0.0.1:8080... * Connected to example.test (127.0.0.1) port 8080 > GET / HTTP/1.1 > Host: example.test:8080 > User-Agent: example-client > Accept: */* > < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 < Content-Length: 74 < Connection: close < { [74 bytes data] * Closing connection

Example Page

``` Lines starting with `*` are `curl`'s own commentary, not part of the exchange. On the server side, that is in `server_output.txt` (in the server's window if it ran in a separate terminal), the bytes the client sent appeared: ``` listening on: 8080 --- incoming request --- GET / HTTP/1.1 Host: example.test:8080 User-Agent: example-client Accept: */* --- end of request --- ``` The two outputs are two ends of the same exchange. The point to notice is that what the server receives is not a structured object but plain text. Both the request and the response consist of bytes; the only thing that sets the rules is that both sides follow the same format. That format is HTTP, and it is the subject of the third topic. The address `127.0.0.1` refers to the machine itself; packets sent to this address never leave the machine. This makes it possible to examine the whole client–server exchange without needing a real network. #### The Peer-to-Peer Model Client–server is not the only option. In the **peer-to-peer** model, every host acts as both client and server: it both receives resources and offers them to others. There is no privileged machine sitting at the center. The two models are compared along the same criteria. | Criterion | Client–server | Peer-to-peer | |---|---|---| | Location of the resource | On a known host | Spread across participating hosts | | Discovery method | The server's address is known | The list of peers is discovered | | As demand grows | Server load grows | Capacity grows too | | Single point of failure | The server | None | | Control and consistency | Single location, easy | Distributed, hard | | Availability | Depends on the server | Continues if enough peers remain | The peer-to-peer model's clearest advantage is scaling: every new host that wants the resource is, at the same time, new capacity able to offer that resource. As demand grows, so does supply. It has two difficulties in return. The first is discoverability: without a stable address, determining who holds a resource is a separate problem. The second is control: guaranteeing a resource's accuracy, currency, and authorization from a single point is hard in a distributed structure. The web is built on the client–server model. The reason is that these two difficulties directly conflict with the web's basic requirements: a domain name is expected to show specific content, and who has authority over that content must be clear. A stable name and stable responsibility require a stable server. This distinction is not a sharp boundary. Many machines can share the load behind a single server; a direct connection can be established between two browsers. The roles are determined by a specific exchange, not by the machines. #### Summary - The criterion separating client from server is not hardware but who initiates the connection. - The server listens continuously at a stable address; the client's address can be temporary. - The port distinguishes multiple server programs on the same machine; a connection is defined by the four-tuple of source and destination addresses and ports. - A server program follows the `bind`, `listen`, `accept`, and read/write steps; waiting at the `accept` call is the essence of the role. - The request and response are not structured objects but byte sequences that both sides interpret the same way. - The peer-to-peer model gains capacity as demand grows, but the client–server model was chosen for the web, which needs a stable address and central control. #### Next Step It was established that the server has to listen at a stable address; but this server is not on the same local network as your machine. The packets in between pass through many independently operated networks that share no common owner. The next lesson covers how this structure, belonging to no single organization, holds together, and who determines the path to the `example.test` server. ### The Structure of the Internet Source: https://academia.sh/en/courses/how-the-internet-works/structure-of-the-internet The network-of-networks idea, routing between autonomous systems, the absence of central management, and how shared standards hold this structure together. The previous lesson established that the server listens at a stable address. But the `example.test` server is not on the same local network as your machine. The packets between the two endpoints pass through many networks, operated by separate organizations, that share no common owner. This lesson covers how that structure holds together. The question to ask is: in a system that no one owns in full, how is a packet's path to its destination decided? #### The Network of Networks The internet is not a single network. It is a union of thousands of interconnected networks, each operated separately. A university campus network, an organization's internal network, an access provider's network — each chooses its own hardware, sets its own rules, runs with its own staff. What makes these networks the internet is not shared ownership but two shared decisions: using the same addressing scheme, and agreeing to carry each other's traffic. A network is part of the internet the moment it does both; it disconnects the moment it leaves. This design is called the **network of networks**, and it has a direct consequence: the internet has no power switch, no central governing board, no complete map. No organization keeps a list of all the connections, because no such list is produced anywhere. #### The Autonomous System The unit that matters for routing is not individual networks but the **autonomous system**: a network or set of networks operated under a single administrative authority with a shared routing policy. Autonomous systems are numbered. This number is the identity that introduces the system in routing conversations between systems, distributed within a registration scheme; two organizations cannot use the same number. The routing problem thus splits into two levels. **Intra-system routing.** How the path is chosen inside an autonomous system is the operating organization's own decision. No one outside interferes, and no one outside sees this detail. **Inter-system routing.** Autonomous systems announce to each other which address blocks they can reach. A system collects its neighbors' announcements and decides, by its own policy, which neighbor to hand each destination to. Who owns this second-level decision matters: **every autonomous system makes its own decision.** No central authority determines the path; the path emerges from a chain of independent decisions. The route a packet takes is, for this reason, not known in advance at either end. The practical result of this split is scaling. Inter-system routing speaks in address blocks and autonomous systems, not individual hosts. An organization changing its internal network creates no change in the routing tables outside. Without this split, every local change anywhere would have to be announced to every network. #### Types of Relationship Connections between autonomous systems are commercial as much as technical, and it is these relationships that set policy. In a **transit** relationship, one system undertakes to carry another's traffic to the rest of the internet, and charges a fee in return. Traffic flows upward. In a **peering** relationship, two systems open direct exchange to each other, but only for traffic belonging to their own customers. Since neither pays a third party for the carrying, both gain; this is why most peering is free. The result is that the path is not always the shortest one. A system may choose a different neighbor for policy reasons even when a technically shorter path exists. The geometry of the network and the geometry of routing are not the same thing. #### Tracing a Path The intermediate nodes a packet passes through can be observed with path-tracing tools, which send packets whose travel distance grows step by step, making the nodes along the path respond in order. {/* run: skip */} ```sh traceroute example.test ``` This command's output is not given here, and cannot be. It depends on where the machine running it is located, its access provider, and the routing state at that moment; the same command produces different results in different places, and different results in the same place at different times. What does not change is the output's **format**: | Field | Meaning | |---|---| | Line number | Which intermediate node this is | | Name or address | That node's identity | | Duration measurements | Round-trip time; multiple measurements are taken | | `*` mark | The node did not respond | The `*` marks on a line do not mean a failure. Nodes along the path are not obligated to respond to this kind of packet; not responding is a configuration choice and does not block the packets from being carried. Two points matter when reading this output. First, the outbound path and the return path do not have to be the same; the measured duration is the sum of both. Second, the path can change over time; two consecutive measurements can show different nodes. This is a direct consequence of the packet switching established in the previous lesson. #### What the Decentralization Is Decentralized About The internet having no central management does not mean nothing is shared. Some functions have to be singular, because the system breaks when they collide. **What has to be singular.** Addresses and autonomous system numbers must be unique; if two organizations use the same address, packets lose track of which destination to go to. The root of the domain name hierarchy is also singular; the same name must not point to two different places. Distributing these numbers and names is delegated to regional registration authorities. **What does not have to be singular.** Which hardware is used, how internal routing is done, which neighbor to peer with, how much capacity to have — all local decisions. **What has to be shared but cannot be mandated.** Protocols. Two networks talking to each other requires following the same rules, but nothing enforces this. The rules are published as open documents, and implementations interoperate because they conform to those documents. A nonconforming implementation is not banned; no one else can talk to it. This three-way split summarizes the internet's governance model: singularity in names and numbers, autonomy in operation, voluntary compliance in protocols. #### The Cost of This Structure The gains of decentralized design are clear: a new network needs no one's permission to join, one part's failure does not stop the whole, and organizations of very different scales can live in the same structure. The costs are equally clear. No one guarantees end-to-end performance; every system along the path has its own policy. Finding the source of a problem means piecing together a picture spread across multiple organizations' fields of view. Because routing announcements rely on trust, a false one can pull traffic to the wrong place. These costs are not a design flaw but the other side of the chosen trade-off. As with the packet switching decision in the previous lesson, predictability is given up here too, in exchange for flexibility and scalability. #### Summary - The internet is not a single network but a union of independent networks that agreed to use a shared addressing scheme and to carry each other's traffic. - An autonomous system is a set of networks operated under a single administrative authority with a shared policy, and it is the meaningful unit of routing. - Intra-system routing is a local decision; in inter-system routing, each system makes its own decision based on the announcements coming from its neighbors. - Because inter-system relationships are commercial, the chosen path is not always the shortest one. - Path-tracing output depends on place and time; what stays constant is the output's format, and a non-responding node does not mean a failure. - Addresses, autonomous system numbers, and the domain name root have to be singular; operating decisions are local, and protocol compliance is voluntary. #### Next Step This lesson said that packets are routed according to address blocks, but it did not define the address itself. Routing tables being able to carry blocks instead of individual machines depends on addresses being structured rather than arbitrary. The next lesson defines the IP address, shows the reason this structure scales, and explains what the `example.test` server's address actually means. ### The Concept of the IP Address Source: https://academia.sh/en/courses/how-the-internet-works/concept-of-the-ip-address The numeric structure of the address, how splitting it into network and host parts shrinks routing tables, and the distinction between public and private addresses. The previous lesson said that autonomous systems announce **address blocks** to each other, not individual machines. This implies that addresses are not randomly assigned labels; if they can be spoken of in blocks, addresses must have a structure. This lesson defines that structure. The question to ask is: what does an address look like, and what problem was this format chosen to solve? #### An Address Is a Number An **IP address** is a fixed-length number that identifies an interface attached to the network. In the IPv4 scheme this number is 32 bits. The familiar `192.0.2.10` notation is nothing more than splitting those 32 bits into four eight-bit pieces and writing each piece in base ten. The notation is for humans; the machine sees a single number. ```python import ipaddress address = ipaddress.ip_address("192.0.2.10") print(address.version) print(int(address)) print(format(int(address), "032b")) ``` ``` 4 3221225994 11000000000000000000001000001010 ``` The bit string's first octet, `11000000` in binary, is 192; the second, `00000000`, is zero; the third, `00000010`, is two; the last, `00001010`, is ten. The dot separator does not exist in the bit string; it exists only in the written form. Each piece being eight bits limits the values it can hold to the range 0–255. A notation like `192.0.2.300` is invalid because 300 does not fit in eight bits. The order in which the address's bytes are written on the network is separately agreed upon. The byte-order lesson in the How Computers Work course covered this layout under the name **network byte order**: multi-byte numbers in headers are written big-endian. Machines using different byte orders can read the same address the same way because of this agreement. 32 bits total gives $2^{32}$, roughly 4.3 billion distinct values. This number looked plentiful when the scheme was designed; it later fell short. The IPv6 scheme removes this limit by extending the address to 128 bits, written in hexadecimal groups like `2001:db8::1`. The technical details between the two schemes are the subject of the Network Models and Protocols course. #### What the Address Addresses Note that the address identifies not a machine but **a network interface**. A machine with multiple interfaces has multiple addresses; each leg of a router carries a separate address. The address works together with the port defined in the previous lesson. The address says which interface to go to; the port says which program on that machine to deliver to. Reaching the `example.test` server requires both. #### Hierarchy and the Scaling of Routing Suppose addresses could be distributed arbitrarily. In that case every router would have to know, individually, which way to go for every destination address. The table would carry as many rows as there are addresses in use, and every new machine would have to be announced to every network. The solution is to split the address in two: the left-hand bits specify the **network**, the right-hand bits the **host** within that network. Where the split falls is stated by the **prefix length** written alongside the address. ```python import ipaddress network = ipaddress.ip_network("192.0.2.0/24") print(network.network_address) print(network.netmask) print(network.num_addresses) print(ipaddress.ip_address("192.0.2.10") in network) print(ipaddress.ip_address("198.51.100.10") in network) ``` ``` 192.0.2.0 255.255.255.0 256 True False ``` The `/24` notation says the left-hand 24 bits are the network part. That leaves 8 bits for the host, and this block contains $2^8 = 256$ addresses. `192.0.2.10` is inside this block; `198.51.100.10` is not. The value `255.255.255.0` is the same information written as a **netmask**: bits set to one mark the network part, bits set to zero mark the host part. The gain is this: a router now holds a single row for 256 addresses. Every packet headed for the `192.0.2.0/24` block goes the same way; the individual hosts inside it are invisible from outside. The larger the block, the greater the saving in the table. This means that, given an address, which network it belongs to can be read **from the address itself**. An address is not only an identity but also a statement of location. When an organization changes its address block, its machines' addresses change with it; an address is not a portable property of the machine. Blocks can nest, and a router can find more than one matching row. In that case the **longest prefix** wins: the row giving the most specific information is chosen. This way a general block can be summarized in one row while a section inside it is sent a different way by a separate, more specific row. The detail of prefix arithmetic and subnet design belongs to the Network Models and Protocols course. #### Public and Private Addresses Part of the address space is set aside to never be routed on the internet. These **private addresses** carry meaning only within a local network. Three blocks are reserved for this purpose: ```python import ipaddress PRIVATE_BLOCKS = [ ipaddress.ip_network("10.0.0.0/8"), ipaddress.ip_network("172.16.0.0/12"), ipaddress.ip_network("192.168.0.0/16"), ] for text in ["10.0.0.5", "172.16.3.1", "192.168.1.1", "192.0.2.10"]: addr = ipaddress.ip_address(text) private = any(addr in block for block in PRIVATE_BLOCKS) print(f"{text:<14} in_private_block={private}") ``` ``` 10.0.0.5 in_private_block=True 172.16.3.1 in_private_block=True 192.168.1.1 in_private_block=True 192.0.2.10 in_private_block=False ``` The defining property of private addresses is that they are **not required to be unique**. Countless local networks each have a machine at `192.168.1.1` at the same time, and this is not a collision, because the address carries no meaning outside its own network. A router does not forward a packet whose destination is a private address onto the internet. The direct consequence is that a machine with a private address is not directly reachable from outside. Such a machine can still **initiate** an outbound connection: at the network's edge, a mechanism rewrites the packet's source address to its own public address and translates the returning response back to the correct internal machine. This translation is detailed in the address translation lesson of the Linux Network Administration and Troubleshooting course, and in the Network Models and Protocols course. This matches the client–server asymmetry established in the previous lesson: being a client requires no reachable address, being a server does. A server's need for a public address is the technical basis of the hosting topic. One more block is set apart: `127.0.0.0/8` contains the **loopback** addresses that refer to the machine itself. Packets sent to this address never reach the network interface at all. Because the previous lesson's server listened on `127.0.0.1`, it was reachable only from the same machine. This course's examples use the `192.0.2.0/24` and `198.51.100.0/24` blocks. These blocks are reserved for documentation; they do not point to a real host, which is why they are safe to use in examples. #### Where an Address Comes From A machine gets its address one of two ways. **Manual configuration** is a human entering the address; it is used on machines, like servers, whose address must not change. **Automatic assignment** is the machine requesting an address from a local service when it joins the network; this address is leased and renewed when it expires. The consequence of the second method is that client machines' addresses are not permanent. The same machine gets a different address on a different network. The server's address being stable and the client's temporary is the natural split between these two assignment methods. Public address blocks are not handed directly to organizations; they are allocated within the regional registration scheme mentioned in the previous lesson and distributed through access providers. An address block is not owned; it is allocated under specific conditions. #### Summary - An IP address is a fixed-length number identifying a network interface; the dotted notation is only a writing convenience. - An address splits into network and host parts; the split point is stated by the prefix length or an equivalent netmask. - Hierarchy lets routers table blocks instead of individual hosts; on overlapping blocks, the longest prefix is chosen. - An address is a statement of location; when an organization's block changes, its machines' addresses change with it. - Private addresses are not unique and are not routed to the internet; a machine with such an address can initiate a connection but cannot be reached from outside. - Servers use stable, manually configured addresses; clients can work with automatically assigned, temporary ones. #### Next Step It was established that the server has an address, and that the address is kept stable. And yet you did not type a number into the address bar; you typed `example.test`. People do not remember numbers, and an organization's server address changes when it moves, while its name is expected to stay the same. The next topic builds this link between names and addresses: how a name is structured, who it belongs to, and how it is translated into an address. ## Domain Names and Hosting The name hierarchy and registration process, the domain name system's resolution chain and caching, record types and hosting models. ### What Is a Domain Name Source: https://academia.sh/en/courses/how-the-internet-works/what-is-a-domain-name The indirection between name and address, the name hierarchy made of labels, the delegation of authority, and the roles in the registration process. The previous topic established that reaching a server requires an address. Even so, you did not type `192.0.2.10` into the address bar; you typed `example.test`. The two forms point to the same server, but the difference between them is not only readability. This lesson examines the name itself: how it is structured, who owns it and why a separate layer sits on top of the address. Translating the name into an address is the next lesson's subject. #### The Reason for Indirection Separating the name from the address solves two distinct problems. **Memorability** is only the first of these, and the less important one. People struggle to remember number sequences; names carry meaning. **The ability to break the link** is the real reason. As established in the previous lesson, an address is a statement of location: when an organization moves its server to another provider, the address must change. If the address were used directly, the move would affect every user — the value everyone knew would become invalid. Once a name layer sits in between, the move reduces to updating a single record. The name stays fixed; the address it points to changes. The user does nothing. The same indirection works in the other direction too. A single name can point to multiple addresses; requests are distributed among them. A single address can also host multiple names — the subject of this topic's last lesson. The relationship between name and address is not one-to-one. This is a pattern that recurs throughout computer science: placing a stable name over a value that can change. It is the same idea as variable binding from the Programming Fundamentals course; the code that refers to the name does not change even when what the name points to does. #### The Structure of a Name A **domain name** is a sequence of **labels** separated by dots. The hierarchy is read not left to right but **right to left**: the rightmost label denotes the most general level. ```python def split_labels(domain_name: str) -> list[str]: """Splits domain name into labels; root is rightmost.""" return domain_name.rstrip(".").split(".") for domain in ["example.test", "www.example.test", "file.archive.example.test"]: labels = split_labels(domain) print(f"{domain:<27} {len(labels)} labels verification order: {labels[::-1]}") ``` ``` example.test 2 labels verification order: ['test', 'example'] www.example.test 3 labels verification order: ['test', 'example', 'www'] file.archive.example.test 4 labels verification order: ['test', 'example', 'archive', 'file'] ``` Reading this sequence on a tree gives the right intuition. The tree terminology from the Data Structures course applies here exactly as it does there: each label of the name is a node, and the full name is the path from the root to that node. - The **root** is the dot at the very right of the name, and it is usually left unwritten. Writing `example.test` is shorthand for `example.test.`. The form with the trailing dot written out explicitly is called a **fully qualified domain name**. - The **top-level domain** is the label directly under the root: `test` in the example. - The label under that is the registered name: `example`. - Labels further to the left are **subdomains**, created at the name owner's own discretion: `archive`, `file`. The `www` label carries no special meaning. It is an established convention; technically it is no different from any other subdomain. `example.test` and `www.example.test` are two separate names and can point to different places. #### Label Rules Labels are not free text. Their length is limited and their comparison rules are defined. ```python def split_labels(domain_name: str) -> list[str]: return domain_name.rstrip(".").split(".") print(len("a" * 63)) print(all(len(label) <= 63 for label in split_labels("file.archive.example.test"))) ``` ``` 63 True ``` A label can be at most 63 characters, and a full name at most 255 characters. The first limit comes from the form in which the name is carried over the network: six bits of the length byte in front of each label are usable. The second is not a field width at all but a ceiling the specification set to simplify implementations. Names are **case-insensitive**: `EXAMPLE.TEST` and `example.test` denote the same name. This rule, however, applies only to the name; it does not apply to the path portion of the address bar, and that distinction is covered in the Journey of a Web Request topic. The native character set for names is limited to letters, digits and the hyphen; that is the limit registries enforce, in line with host-naming convention, not the limit of what the protocol can carry in a label. Names that contain characters outside this set — including non-ASCII letters — are represented with an encoding that maps back into the limited set. The distinction established in the character encoding lesson of the How Computers Work course holds here as well: the form shown to the user and the form carried over the network need not be the same. #### The Delegation of Authority The real function of the name hierarchy is not to look tidy but to **divide responsibility.** An authority at the root level decides which top-level names will exist. That authority delegates management of the names under each top-level name to that name's operator. When the operator hands the name `example` to an owner, authority over everything under `example.test` passes to that owner. The consequence of delegation is this: creating the subdomain `archive.example.test` requires no one's permission. The name owner has unlimited authority within its own subtree. The root authority is not even aware these names exist. This is the same principle as the autonomous system structure from the previous lesson: singularity at the top level, autonomy at the lower level. The only thing that must be singular is who is authoritative at each level of the hierarchy. #### Registration and Roles Saying a name is "bought" is common but misleading. A name is not bought; a **right of use is leased** for a fixed term. If it is not renewed when the term expires, the name becomes available again. Three roles are separated in the process, and confusing them is a common mistake. | Role | Function | |---|---| | Registry | Holds the single authoritative database of registrations under a top-level name | | Registrar | Mediates between the name owner and the registry, and forwards the registration | | Registrant | The person or organization holding the right of use for the name | There is exactly one registry per top-level name; registrars, in contrast, are numerous, and the name owner chooses among them. A name's registrar can be changed; its registry cannot, because the registry is part of that top-level name's definition. The technically meaningful part of a registration is the declaration of which servers are authoritative for the name. The registry does not hold the address the name itself corresponds to; it holds only the information "ask these servers for this name's answers." The content of the answer is under the name owner's control. This distinction underlies the resolution chain covered in the next lesson. #### Reserved Names Some top-level names are reserved and will never be opened to actual registration. `test`, `example`, `invalid` and `localhost` are among them. This is why this course's examples use the name `example.test`. A reserved name belongs to no one, does not resolve to an actual server, and will never pass into anyone's ownership. Using a real name in the examples would make the example wrong the moment that name's owner changed or its content diverged. The same reasoning applies to the `192.0.2.0/24` address block used in the previous lesson: what is reserved for documentation can be used safely in an example. #### Summary - The name layer lets the value the user knows stay fixed even though the address can change; the real reason is not memorability but the ability to break this link. - A domain name consists of labels and is read right to left, from root to leaves; the trailing dot denotes the root and is usually left unwritten. - The `www` label carries no technical privilege; it sits at the same level as any other subdomain. - Labels are at most 63 characters, full names at most 255; names are case-insensitive. - The function of the hierarchy is to delegate authority: a name owner creates new names within its own subtree without asking permission. - There is exactly one registry per top-level name, and it holds only the identity of the authoritative servers; the content of the answer is under the name owner's control. #### Next Step The structure of a name and who owns it have now been established, but how a machine gets from that name to an address when `example.test` is typed into the address bar has not yet been said. The answer is not stored in a single place: the query follows a chain that starts at the root and advances one step at each level where authority has been delegated. The next lesson covers this chain and the caching scheme that keeps every request from running it from scratch. ### The Domain Name System Source: https://academia.sh/en/courses/how-the-internet-works/domain-name-system The distributed database that translates a name into an address, the resolution chain from root to authoritative server, and time-to-live-based caching. The previous lesson established the structure of a name and how authority over it is delegated. It was said that the registry does not hold the name's address, only the information "ask these servers about this name." This lesson covers the machinery that carries out that asking: the **domain name system**. The question to answer is how the name `example.test` is translated into an address, and why that translation is not redone from scratch on every request. #### Why Not a Single File Keeping a file that maps names to addresses is what was done while the network held few machines. Every machine carried a copy; when a name was added, the file was updated and redistributed to everyone. This scheme has three distinct limits, and all three are about scale. The file grows in proportion to the number of names. Every change requires every copy to be refreshed, and the frequency of change grows with the number of machines. And a single file means a single authority: every name request has to go through the same point. The domain name system overcomes all three limits by using a hierarchy. The database is split, authority over each part is delegated to a separate party, and no one holds the whole. #### The Roles in the Chain Four distinct roles exist in resolution. The **stub resolver** lives inside the machine making the request. The application's call to translate a name into an address goes to this component in the operating system. It does not perform resolution itself; it forwards the question to a resolver and waits for the answer. The **recursive resolver** is the server that takes on the work. It is usually a server run by the access provider or the organization. It carries out every step needed to find the answer and holds the cache itself. The **root servers** sit at the top of the hierarchy. They do not know addresses themselves; they know only which servers each top-level name has been delegated to. The **authoritative servers** are the servers that hold a name's answers as the source of record. For `example.test`, the authoritative server is the one the name owner designated, and it is the same one reported during registration in the previous lesson. #### The Resolution Chain Assuming `example.test` is not present in any cache at all, the query follows these steps: 1. The application asks the stub resolver in the operating system for the address of `example.test`. 2. The stub resolver forwards the question to the recursive resolver and waits for the answer. 3. The recursive resolver asks a root server. The root does not know the address; it says which servers the authority for the top-level name `test` has been delegated to. 4. The recursive resolver asks one of those servers. It does not know the address either; it says which servers the authority for the name `example.test` has been delegated to. 5. The recursive resolver asks the authoritative server and receives the address. 6. The answer returns to the stub resolver, and from there to the application. The division of labor separates two distinct query forms. The question the stub resolver asks the recursive resolver is **recursive**: "find this name's answer and bring it to me." The questions the recursive resolver asks the root and intermediate servers are **iterative**: "tell me if you know this, and if you do not, tell me who to ask." This distinction is not accidental. If the root and top-level servers had to ask someone else and assemble the answer for every incoming question, their load would be unsustainable. Each intermediate server's work is a single, fixed-cost step; assembly is concentrated in one recursive resolver. The fact that the answer at every step takes the form "I do not know the address, ask this one" is a direct reflection of the authority delegation from the previous lesson. The length of the chain is determined not by the number of labels in the name but by how many times authority has been delegated. #### The Shape of a Record Query tools let these answers be seen directly. The command below queries the name that denotes the machine itself. The output depends on the machine's resolver configuration; on a system whose local resolver answers the name `localhost` itself, a sample run looks like this: {/* run: skip */} ```sh dig +noall +answer localhost A ``` ``` localhost. 0 IN A 127.0.0.1 ``` This single line shows a record's five fields: | Field | Value in the example | Meaning | |---|---|---| | Name | `localhost.` | The name the record belongs to; the trailing dot denotes the root | | Time to live | `0` | How many seconds the answer may be kept in cache | | Class | `IN` | The address family; always this value for the internet | | Type | `A` | The record's type; `A` carries an IPv4 address | | Data | `127.0.0.1` | The record's content | When the same command is run for an actual domain name, the format stays the same and the values change. This course gives no such output: the time to live depends on when the query is made, the returned address on where it is made from, and both begin going stale the moment they are written on a page. Run the command yourself and the five fields you see will match the ones above. The details of record types are the next lesson's subject. #### Caching and Time to Live If the chain above were run from scratch on every request, opening a single page would require multiple network round trips, and the root servers would see every query made anywhere in the world. What makes the system operable is the storing of answers. Every record carries a **time to live** with it: how many seconds the answer can be used without being asked again. The recursive resolver keeps the answer in its cache for this period and answers the same question without running the chain when it recurs. The party that sets this duration is the authoritative server issuing the answer — in other words, the name owner. This gives the name owner direct control and requires managing a trade-off. The program below shows this behavior. Because the clock is supplied externally, the output is the same on every run. ```python class NameCache: """TTL-limited name->address cache. Clock is supplied externally.""" def __init__(self) -> None: self.records: dict[str, tuple[str, int]] = {} self.upstream_query_count = 0 def resolve(self, name: str, now: int, ttl: int, address: str) -> str: record = self.records.get(name) if record is not None and record[1] > now: return f"from cache (remaining TTL: {record[1] - now})" self.upstream_query_count += 1 self.records[name] = (address, now + ttl) return f"upstream query (new TTL: {ttl})" cache = NameCache() for second in [0, 100, 250, 300, 400]: result = cache.resolve("example.test", now=second, ttl=300, address="192.0.2.10") print(f"t={second:>3}s {result}") print(f"total upstream queries: {cache.upstream_query_count}") ``` ``` t= 0s upstream query (new TTL: 300) t=100s from cache (remaining TTL: 200) t=250s from cache (remaining TTL: 50) t=300s upstream query (new TTL: 300) t=400s from cache (remaining TTL: 200) total upstream queries: 2 ``` Five resolution requests led to only two upstream queries. On an actual recursive resolver, the ratio is far sharper because many users share the same cached record. The trade-off is this: - A **long time to live** reduces upstream queries and speeds up answers. In exchange, when a change is made, the old answer stays in use until the duration expires. - A **short time to live** lets a change propagate quickly. In exchange, it raises query load and average response time. This has a direct counterpart in operating practice: before moving a server, the time to live is lowered; the move is carried out, and once propagation completes the duration is raised back. Skip the advance lowering, and the old address keeps living in caches for as long as the old duration. This produces a consequence worth keeping in mind throughout: **a domain name change does not appear everywhere at once.** Different users get different answers for a while, depending on the state of their own caches. This is not a fault; it follows directly from the definition of caching. Queries with a negative answer are also stored. When a name that does not exist is queried, the answer "no such name exists" is also kept in cache for a while; this is why a newly created name does not appear immediately. The cache does not live in a single place. The stub resolver, the recursive resolver and the requesting application can each hold a separate cache. Tracing which layer a delay or a stale answer came from therefore requires examining the layers separately. #### Summary - Keeping names in a single file does not scale; the domain name system splits the database by hierarchy and delegates authority over each part. - The resolution chain starts at the root, descends to the top-level name's server and from there to the authoritative server; the answer at each step points not to the next address but to the next authority. - The stub resolver asks the recursive resolver recursively; the recursive resolver asks intermediate servers iteratively. This division of labor keeps the load on root and top-level servers fixed. - A record consists of name, time to live, class, type and data fields. - Time to live is set by the name owner and manages the trade-off between query load and how fast a change propagates. - Caching happens across multiple layers and covers negative answers too; a change does not appear everywhere at the same moment. #### Next Step This lesson showed a record's five fields and passed over the type field with a single example, `A`. Yet a name does not hold only an address underneath it: redirection to another name, mail routing, verification texts and authority declarations are stored in the same layout. The next lesson covers these types, which problem each answers, and the combination rules among them. ### DNS Record Types Source: https://academia.sh/en/courses/how-the-internet-works/dns-record-types Address, alias, mail, name server and text records; which problem each type answers and the combination limits on alias records. The previous lesson showed that a record consists of name, time to live, class, type and data fields, and passed over the type field with a single example, `A`. Yet a name does not hold only an address underneath it. This lesson covers the values the type field can take. Each type answers a distinct problem; seeing why each type exists shows that the domain name system is more than an address book. #### The Zone and the Zone File The full set of records a single authoritative server is responsible for is called a **zone**. The zone written out in text form is called a **zone file**; each line is one record. The zone for the name `example.test` might look like this: ``` example.test. 3600 IN A 192.0.2.10 example.test. 3600 IN AAAA 2001:db8::10 www.example.test. 3600 IN CNAME example.test. example.test. 3600 IN MX 10 mail.example.test. mail.example.test. 3600 IN A 192.0.2.20 example.test. 600 IN TXT "verification=abc123" example.test. 86400 IN NS ns1.example.test. ``` Every line follows the same five-field layout. The program below verifies this: it splits the lines into fields and rewrites them keyed by the type field. ```python ZONE = """\ example.test. 3600 IN A 192.0.2.10 example.test. 3600 IN AAAA 2001:db8::10 www.example.test. 3600 IN CNAME example.test. example.test. 3600 IN MX 10 mail.example.test. mail.example.test. 3600 IN A 192.0.2.20 example.test. 600 IN TXT "verification=abc123" example.test. 86400 IN NS ns1.example.test. """ def parse_record(line: str) -> tuple[str, int, str, str, str]: name, ttl, record_class, record_type, *data = line.split() return name, int(ttl), record_class, record_type, " ".join(data) for line in ZONE.strip().splitlines(): name, ttl, record_class, record_type, data = parse_record(line) print(f"{record_type:<6} name={name:<19} ttl={ttl:<6} data={data}") print() types = sorted({parse_record(l)[3] for l in ZONE.strip().splitlines()}) print("record types in zone:", len(types)) print("types:", types) ``` ``` A name=example.test. ttl=3600 data=192.0.2.10 AAAA name=example.test. ttl=3600 data=2001:db8::10 CNAME name=www.example.test. ttl=3600 data=example.test. MX name=example.test. ttl=3600 data=10 mail.example.test. A name=mail.example.test. ttl=3600 data=192.0.2.20 TXT name=example.test. ttl=600 data="verification=abc123" NS name=example.test. ttl=86400 data=ns1.example.test. record types in zone: 6 types: ['A', 'AAAA', 'CNAME', 'MX', 'NS', 'TXT'] ``` That the parser is identical for every type is a feature of the design: the type field only says how the data field should be interpreted; it does not change the record's frame. Defining a new type does not stop existing servers from carrying the record. The relationship between a record's name and the name being looked up deserves attention: a single zone can hold multiple records for the same name. A query is made with the name **and** type together. #### Address Records An `A` record carries an IPv4 address, an `AAAA` record an IPv6 address. The two are distinct types, and a single name can have both; in the zone above, both are defined for `example.test`. A single name can be given multiple address records. In that case the resolver returns all of them, and the client picks one. This is the simplest way to distribute requests across multiple servers. The limit of this method is clear: the domain name system does not know which of the addresses is actually up. When a server fails, its record stays in the rotation until it is removed by hand, and even after removal it keeps living in caches for as long as the time to live. This is a direct consequence of the time-to-live trade-off established in the previous lesson. #### Alias Records A `CNAME` record does not point to an address but to **another name**. When `www.example.test` is queried, the answer is "this name's counterpart is the name `example.test`," and the resolver continues the lookup through that name. The reason to use it applies the indirection idea from the previous lesson one layer further: the address is defined in one place, and other names attach to it. When the address changes, a single record is updated. This type has two constraints, and both come from the same source: an alias record hands off the **entirety** of the name to another name. **No other record can exist at the same name.** If a name has an alias record, no other type of record can be defined for that name. If it could, a contradiction would arise: the name would say both "my counterpart is this name" and "my address is this." **It cannot be used at a zone's root name.** The name `example.test` itself cannot be an alias, because that name already must hold records that are required to exist — the authority and name server records covered below. Under the first constraint, these cannot coexist with an alias. This second constraint is a common obstacle in practice: the name `www.example.test` can be aliased to a provider's name, but the same cannot be done for `example.test`, which requires an address record written directly at it. #### Mail Records An `MX` record says which server delivers mail sent to that domain name. Its data field has two parts: a priority number and a server name. ``` example.test. 3600 IN MX 10 mail.example.test. ``` A **lower** priority number is preferred. When multiple mail records are defined, the sending side tries the lowest-numbered server first and moves to the next if it cannot be reached. The number itself carries no other meaning; it exists only for ordering. The existence of this record shows an important distinction: a domain name's web traffic and mail traffic can go to different servers. The page for `example.test` may come from `192.0.2.10` while the same name's mail is delivered to `192.0.2.20` via `mail.example.test`. A domain name names an **administrative domain**, not a single machine. The name a mail record points to should not be an alias; it must carry an address record directly. #### Name Server and Authority Records An `NS` record declares which servers a zone's authority belongs to. The "ask this one" answer received at every step of the previous lesson's resolution chain is exactly this record. This record exists in two places at once: within the zone itself and in the zone one level up. The copy at the level above is the record that signals delegation; it is what was reported to the registry during registration in the earlier lesson. A mismatch between the two copies leads to inconsistencies that are hard to diagnose during resolution. Every zone also carries an `SOA` record. This record holds the zone's administrative information: the primary name server, a responsible address, the zone's serial number, and the intervals that determine how often secondary servers refresh their copies. The serial number lets secondary servers tell whether the zone has changed. #### Text Records A `TXT` record carries free text. It has no fixed function of its own, which makes it a common tool for schemes that want to prove ownership of a name. The usage pattern is this: a service asks the name owner to place a specific piece of text in its zone. Once the text appears, control over that name's management is proven. This rests on the authority delegation established in the previous lesson — whoever can write a record into the zone is the authority for that name. Mail verification schemes also operate through this record type, writing text that follows a defined syntax. The details of these schemes are the subject of the Application Layer Protocols course. #### Choosing a Type | Type | Question it answers | |---|---| | `A` | What is this name's IPv4 address | | `AAAA` | What is this name's IPv6 address | | `CNAME` | Which name does this name stand in for | | `MX` | Where is this name's mail delivered | | `NS` | Who holds authority over this zone | | `SOA` | What is this zone's administrative information | | `TXT` | What free text is attached to this name | The order followed when designing a zone follows from this: authority records are required for the zone to exist, address records are needed for the service to be reachable, alias records reduce maintenance, and mail and text records define other services tied to the domain name. #### Summary - A zone is the full set of records a single authoritative server is responsible for; every record follows the same five-field layout, and the type field only determines how the data field is interpreted. - `A` and `AAAA` carry addresses; a name can be given multiple addresses to distribute requests, but the system does not know the servers' health. - `CNAME` hands a name off to another name; because no other record can exist at the same name, it cannot be used at a zone's root name. - `MX` reports the server mail is delivered to, in priority order; a domain name's web and mail traffic can go to different servers. - `NS` declares authority and appears both in the zone and one level up; `SOA` carries the zone's administrative information. - `TXT` carries free text and is used to prove authority over a name. #### Next Step This lesson established at the record level that the name `example.test` points to the address `192.0.2.10`. What remains is the question of what is at that address: a machine must be there, running continuously, waiting for the request. The next lesson covers how that machine is provisioned, how the same address can serve multiple domain names, and the trade-offs among hosting options. ### Hosting Models Source: https://academia.sh/en/courses/how-the-internet-works/hosting-models The requirement that a server keep running continuously, how name-based virtual hosting works, and the trade-offs among shared, virtual, dedicated and managed models. The previous lesson established that the name `example.test` points to the address `192.0.2.10`. What remains is the question: what is at that address? The answer is a relative of the program written in the Client-Server Model lesson: a process waiting on a port, taking a request and answering it. This lesson covers where and how that process is run. #### The Problem Hosting Solves Three conditions must be met for a server program to run; all three were established separately in earlier lessons. **A stable, reachable address.** As seen in the IP Address Concept lesson, a machine with a private address cannot be reached from outside, and automatically assigned addresses change. The server needs a public, unchanging address. **Continuous operation.** As established in the Client-Server Model lesson, a server does not know when a request will arrive; it must be listening at all times. This means the machine must not shut down, the network connection must not drop, and the process must be restarted if it crashes. **Operation.** Hardware failure, software updates, backups and rising load all have to be handled. **Hosting** is providing these three conditions as a service. An organization keeping a machine running continuously in its own building is also hosting; the difference lies in who meets the conditions. #### One Address, Multiple Names The technical basis that makes hosting operable is that a single address can serve multiple domain names. Addresses are a limited resource; needing a separate address for every domain name would make the cost prohibitive. The problem is this: a packet arrives at an address, and the address denotes a single machine. How is the server to know which domain name the request was made for? The answer is carried in the request itself. The client states the name it connected to in the request's `Host` header. This line already appeared in the output of the Client-Server Model lesson: ``` > GET / HTTP/1.1 > Host: example.test:8080 ``` The server decides which site to serve by looking at this header. This scheme is called **name-based virtual hosting**. The server below listens on a single address and port but serves two separate sites: ```python # hosting_server.py from http.server import BaseHTTPRequestHandler, HTTPServer SITES: dict[str, bytes] = { "example.test": b"

Example Site

\n", "demo.test": b"

Demo Site

\n", } class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, format_: str, *arg: object) -> None: pass def do_GET(self) -> None: host = self.headers.get("Host", "").split(":")[0] body = SITES.get(host, b"

Unknown host

\n") status = 200 if host in SITES else 404 self.send_response_only(status) self.send_header("Content-Type", "text/html; charset=utf-8") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) HTTPServer(("127.0.0.1", 8080), Handler).serve_forever() ``` While the server is running, three requests are made. All three go to the same address; only the declared name changes. The commands below run as one piece: the server is started in the background, the `until` loop waits until the server accepts connections, and the server is shut down at the end. If you run the server in a separate terminal, you only need to type the `curl` lines. ```sh python3 hosting_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done curl -s --resolve example.test:8080:127.0.0.1 http://example.test:8080/ curl -s --resolve demo.test:8080:127.0.0.1 http://demo.test:8080/ curl -s --resolve other.test:8080:127.0.0.1 http://other.test:8080/ kill $!; wait $! 2>/dev/null ``` ```

Example Site

Demo Site

Unknown host

``` All three requests went to port 8080 on address `127.0.0.1` and got three different answers. The only thing that made the distinction was the `Host` header. This has two consequences. First, sites sharing an address cannot see each other at the address level; the distinction is made inside the server program. Second, this header is mandatory: a client that fails to declare it makes it impossible for the server to determine which site to serve. #### Hosting Models Who meets the three conditions, and at what level, is what separates the models. ##### Shared Hosting A single machine runs many customers' sites on the same operating system. The virtual hosting scheme above is directly the basis for this model. Because resources are shared, this is the lowest-cost model, and the entire operational burden sits with the provider. The price is weak isolation: the resources one site consumes affect the other sites on the same machine. Configuration options are limited to whatever the provider allows; there is no operating-system-level control. ##### Virtual Private Server A single physical machine is split, through virtualization, into multiple virtual machines isolated from one another. Each customer has its own operating system and its own administrative privileges. Because resources are separated by defined limits, the neighbor effect is markedly reduced; software choice and configuration are under the customer's control. In exchange, operating system maintenance, security updates and backups become the customer's responsibility. How isolation is achieved — namespaces, control groups and virtualization — is covered in the Kernel Interfaces and Isolation course. ##### Dedicated Server The entire physical machine is assigned to a single customer. There is no resource sharing; hardware choice and predictability of performance are at their highest. Cost is paid for the whole machine whether it is used or not. The impact of a hardware failure is direct, and redundancy has to be set up separately. ##### Managed Hosting The distinguishing criterion is not hardware but **the boundary of responsibility.** In a managed model, the provider takes on not just the machine but the operation of the software running on it as well: updates, backups, monitoring and scaling. The customer gives up part of its configuration freedom in exchange for being relieved of the operational burden. This model does not sit on the same axis as the other three; a virtual private server or a dedicated server can equally well be offered in managed form. ##### Comparison | Criterion | Shared | Virtual private | Dedicated | Managed | |---|---|---|---|---| | Isolation | Weak | Defined limits | Full | Depends on model | | Administrative privilege | None | Yes | Yes | Limited | | Operational burden | Provider | Customer | Customer | Provider | | Performance predictability | Low | Medium | High | Depends on model | | Relative cost | Lowest | Medium | Highest | Depends on service | No single option is superior on every criterion. The choice depends on the balance between the site's requirements and the effort that can be devoted to operating it. #### Static and Dynamic Content Another distinction that shapes hosting requirements is how the content is produced. With **static content**, the server sends a file from disk as is. Every request produces the same response. This kind of content requires no computation and is cheap to replicate. With **dynamic content**, the response is computed when the request arrives: a database is queried, sections specific to the user are generated. Every request consumes processor time and memory; scaling it is markedly harder than scaling static content. This distinction leads to a placement decision. Static content can be replicated across servers distributed close to users. In this scheme a user's request goes to the copy nearest to it; because latency, as established in the What Is a Network lesson, is bounded by propagation distance, the gain is direct. Dynamic content, on the other hand, stays tied to where its state is held and cannot be replicated this way. This is why the two kinds of content are separated in practice: a page's unchanging parts come from distributed copies, its user-specific parts from the origin server. #### Summary - Hosting provides a stable address, continuous operation and operational conditions for a server program. - Name-based virtual hosting serves multiple sites at a single address by looking at the request's `Host` header; the distinction is made not at the address level but inside the server program. - Shared hosting gives the lowest cost and weakest isolation, a dedicated server the opposite; a virtual private server sits between the two. - Managed hosting is defined not by hardware but by the boundary of responsibility, and can be layered on top of the other models. - Static content can be replicated to bring it closer to users; dynamic content stays tied to where its state is held. #### Next Step This topic built, start to finish, how a name typed into the address bar reaches a server: the structure of the name, the resolution chain, the records and the machine at that address. Yet what is said once the server is reached has been glossed over with the same two lines throughout. The next topic opens that exchange itself: what parts a request and a response are made of, what status codes report, and why the server remembers nothing between requests. ## The Journey of a Web Request The parts of the HTTP request and response, the stages the browser goes through from request to drawing, the promises HTTPS gives, and the joining of the entire chain end to end. ### HTTP Request and Response Source: https://academia.sh/en/courses/how-the-internet-works/http-request-and-response The parts of a request and response, the safety and idempotency of methods, status code classes, and the statelessness of the protocol. The previous topic established how the name typed into the address bar reaches a server. The Client–Server Model lesson connected to that server and both ends of the exchange were seen. What was not explained is why the text sent takes that particular shape. This lesson defines that shape. **HTTP** is the protocol that sets the rules for the exchange between client and server, and it is the contract the web is built on. #### A Text-Based Protocol The server in the Client–Server Model lesson wrote the bytes it received to the screen exactly as they arrived: ``` GET / HTTP/1.1 Host: example.test:8080 User-Agent: example-client Accept: */* ``` This is not a binary structure, it is readable text. That the design goes in this direction is a choice, and it has a cost: a text representation carries the same information in more bytes than a binary representation would, and parsing it is more expensive. In exchange, the protocol can be inspected without a special tool and can be extended; adding a new header does not break existing implementations. Line endings are written with the two characters `\r\n`. What marks the end of the headers is an empty line — that is the blank space at the end of the output above. #### The Structure of a Request A request has three parts. **The request line** is the first line and carries three fields: method, target, version. ``` GET / HTTP/1.1 ``` `GET` states the operation to perform, `/` the path of the requested resource, `HTTP/1.1` the protocol version the client speaks. Only the path portion of the target is written; the domain name is carried in a separate header. **Headers** carry a `name: value` pair on each line. Header names are case-insensitive. The three headers above each serve a different function: `Host` states which site is being connected to and, as seen in the Hosting Models lesson, is the basis of virtual hosting; `User-Agent` identifies the client; `Accept` states which content formats the client can handle. **The body** is optional data that comes after the empty line. It is usually absent in `GET` requests; it is present in methods that send data. #### The Structure of a Response A response follows the same layout, only its first line differs. ``` HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 Connection: close

Example Page

``` **The status line** carries the version, the numeric status code, and its readable description. The description text is informational; decisions are made on the number. The `Content-Length` header states how many bytes the body is. This header is not a detail — it is information required for the protocol to work: the receiver needs to know where the body ends. A connection is a byte stream, and a stream has no natural boundary. When the length is not known in advance, the body is sent in chunks, with the size of each chunk written before it. `Content-Type` states how the body should be interpreted. The `charset=utf-8` part here answers the question set up in the Character Encodings lesson of the How Computers Work course: which rule converts bytes to characters is not left to the receiver's guess, it is stated explicitly. #### Methods The method states the operation to perform on the resource. Two properties distinguish methods, and these properties determine client behavior. A **safe** method makes no change on the server; it only reads. An **idempotent** method leaves the same result whether it is applied once or many times. Every safe method is idempotent; the reverse is not true. | Method | Function | Safe | Idempotent | |---|---|---|---| | `GET` | Retrieves the resource | Yes | Yes | | `HEAD` | Retrieves only the headers | Yes | Yes | | `POST` | Sends data, may create a new resource | No | No | | `PUT` | Replaces the resource with the given content | No | Yes | | `DELETE` | Deletes the resource | No | Yes | These properties are not theoretical. Whether a request can be retried when no response is received follows directly from them: an idempotent request can be safely repeated, but if a `POST` request is repeated the same operation can happen twice. Even if the `DELETE` method returns a different status code on the second call, the server's state stays the same; idempotency is a property of the state left behind, not of the code returned. These properties **must be preserved** by the server; the protocol cannot enforce them. Writing a `GET` handler that changes state is syntactically possible and violates the protocol. The `HEAD` method returns the same headers as `GET` but sends no body. It is used to learn a resource's size or existence without downloading its content. #### Status Codes A status code has three digits, and its first digit gives the class of the result. | Class | Meaning | |---|---| | `1xx` | Informational; processing continues | | `2xx` | Success | | `3xx` | Redirection; the resource is elsewhere | | `4xx` | Client error; the request is malformed | | `5xx` | Server error; the request was valid but the server could not fulfill it | The distinction between `4xx` and `5xx` is decisive from an operational standpoint: the first requires fixing the request, the second requires examining the server. The following server produces three different status codes and defines a redirect: ```python # redirect_server.py from http.server import BaseHTTPRequestHandler, HTTPServer class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, fmt: str, *args: object) -> None: pass def _send(self, status: int, body: bytes, extra: list[tuple[str, str]] = []) -> None: self.send_response_only(status) self.send_header("Content-Type", "text/plain; charset=utf-8") self.send_header("Content-Length", str(len(body))) for name, value in extra: self.send_header(name, value) self.end_headers() self.wfile.write(body) def do_GET(self) -> None: if self.path == "/old": self._send(301, b"", [("Location", "/new")]) elif self.path == "/new": self._send(200, b"content at the new address\n") else: self._send(404, b"not found\n") def do_POST(self) -> None: length = int(self.headers.get("Content-Length", "0")) body = self.rfile.read(length) self._send(201, b"received body: " + body + b"\n") HTTPServer(("127.0.0.1", 8080), Handler).serve_forever() ``` The codes returned when three paths are queried. The commands below run as one piece: the server is started in the background, the `until` loop waits until the server accepts connections, and the server is shut down at the end. If you run the server in a separate terminal, you only need to type the `curl` lines; the later examples in this lesson start and stop the server the same way. ```sh python3 redirect_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done for path in /new /old /missing; do printf "%-6s " "$path" curl -s -o /dev/null -w "%{http_code}\n" --resolve example.test:8080:127.0.0.1 "http://example.test:8080$path" done kill $!; wait $! 2>/dev/null ``` ``` /new 200 /old 301 /missing 404 ``` Examining the redirect response itself, the new location of the resource is seen to be carried in a header: ```sh python3 redirect_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done curl -sD - -o /dev/null --resolve example.test:8080:127.0.0.1 http://example.test:8080/old kill $!; wait $! 2>/dev/null ``` ``` HTTP/1.1 301 Moved Permanently Content-Type: text/plain; charset=utf-8 Content-Length: 0 Location: /new ``` The body of the response is empty; the information carried is in the `Location` header. Once the client receives this response, it makes a second request to the new target. `curl` does not do this on its own; it does so when asked: ```sh python3 redirect_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done curl -sL --resolve example.test:8080:127.0.0.1 http://example.test:8080/old kill $!; wait $! 2>/dev/null ``` ``` content at the new address ``` Browsers follow redirects on their own. A `301` code states that the move is permanent and lets the client not remember the old address; separate codes exist for temporary moves, and this distinction matters, because permanent redirects can be cached by the client. A request carrying a body follows the same layout: ```sh python3 redirect_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done curl -s -X POST -d "name=test" --resolve example.test:8080:127.0.0.1 http://example.test:8080/register kill $!; wait $! 2>/dev/null ``` ``` received body: name=test ``` #### Statelessness HTTP is a **stateless** protocol: the server remembers nothing about the client between two requests. Each request is interpreted on its own, independent of previous requests. This is an observable property. The following server reads the counter carried by the request and returns an incremented value to the client. The command writes the server to `counter_server.py`: ```sh cat > counter_server.py <<'EOF' from http.server import BaseHTTPRequestHandler, HTTPServer class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, fmt: str, *args: object) -> None: pass def do_GET(self) -> None: incoming = self.headers.get("Cookie", "") counter = int(incoming.removeprefix("counter=")) if incoming.startswith("counter=") else 0 body = f"observed counter: {counter}\n".encode() self.send_response_only(200) self.send_header("Content-Type", "text/plain; charset=utf-8") self.send_header("Content-Length", str(len(body))) self.send_header("Set-Cookie", f"counter={counter + 1}") self.end_headers() self.wfile.write(body) HTTPServer(("127.0.0.1", 8080), Handler).serve_forever() EOF ``` If three requests are made carrying nothing between them: ```sh python3 counter_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done for i in 1 2 3; do curl -s --resolve example.test:8080:127.0.0.1 http://example.test:8080/; done kill $!; wait $! 2>/dev/null ``` ``` observed counter: 0 observed counter: 0 observed counter: 0 ``` The server saw zero all three times. It keeps no record of its own. If the same three requests are made while carrying back the value the server gave: ```sh python3 counter_server.py & until curl -s -o /dev/null 127.0.0.1:8080; do sleep 0.05; done rm -f jar.txt for i in 1 2 3; do curl -s -b jar.txt -c jar.txt --resolve example.test:8080:127.0.0.1 http://example.test:8080/; done kill $!; wait $! 2>/dev/null ``` ``` observed counter: 0 observed counter: 1 observed counter: 2 ``` The server code did not change. What changed is that the client stored the value given by `Set-Cookie` and sent it back in the next request with the `Cookie` header. The state is carried not in the server, but **in the request**. The reason for statelessness is scaling. Because the server holds no client-specific memory, two consecutive requests can be answered by different machines; if the server crashes, there is no session state that gets lost when it restarts. The cost is that every request must carry again the information needed to identify itself. #### Summary - HTTP is a text-based protocol; it trades parsing cost for inspectability and extensibility. - A request consists of a request line, headers, an empty line, and an optional body; a response follows the same layout with a status line. - `Content-Length` is required because it states where the body ends in a byte stream; `Content-Type` states how the bytes should be interpreted. - Methods are distinguished by their safety and idempotency properties; these properties determine whether a request can be retried and must be preserved by the server. - The first digit of a status code gives its class; a `4xx` requires fixing the request, a `5xx` requires fixing the server. - The protocol is stateless; session continuity is built not by the server's memory but by data the client carries back with every request. #### Next Step This lesson examined a single request and a single response. Yet when a page is opened, only one request is not made: the document that arrives refers to other resources, and each of them is requested separately. The next lesson takes up the stages from the text typed into the address bar to the drawing on the screen, and shows through a real server's log how these additional requests arise. ### What the Browser Does Source: https://academia.sh/en/courses/how-the-internet-works/what-the-browser-does The stages from parsing the text in the address bar to the drawing on screen, subresource requests, and the reuse of connections. The previous lesson examined a single request and a single response. Yet when a page is opened, a single request is not made. The document that arrives refers to other resources, and the browser requests those too. This lesson takes up the browser's work in order: from parsing the text in the address bar to the drawing on screen. The browser is not a viewer, it is an orchestrator that carries out these stages. #### Parsing the Address The first job is to split the written text into its parts. An address is not a single string, it is a structured value. ```python from urllib.parse import urlsplit address = "https://example.test:8443/archive/document?topic=network&page=2#section3" parts = urlsplit(address) print("scheme :", parts.scheme) print("host :", parts.hostname) print("port :", parts.port) print("path :", parts.path) print("query :", parts.query) print("fragment :", parts.fragment) ``` ``` scheme : https host : example.test port : 8443 path : /archive/document query : topic=network&page=2 fragment : section3 ``` Each part is used at a different stage: | Part | Where it is used | |---|---| | Scheme | Which protocol will be spoken; also sets the default port | | Host | The name given to the resolution in the Domain Name System lesson | | Port | When the connection is established; the scheme's default if not written | | Path | Placed in the request line, sent to the server | | Query | Placed in the request line together with the path, sent to the server | | Fragment | **Never sent to the server**; used only in the browser | The last line is notable. The `#section3` part never goes out on the network; the browser uses it itself, after receiving the document, to scroll to the section carrying that name. The server never sees this value. It was stated in the What Is a Domain Name lesson that a name is case-insensitive. The same is not true of the path part: `/Archive` and `/archive` can be two different paths, and the server decides this. #### Stages Once the address is parsed, the browser follows these steps: 1. **Name resolution.** The host name is turned into an address. The chain from the Domain Name System lesson runs here, and caches come into play. 2. **Establishing a connection.** A connection is made to the target address and port. 3. **Security handshake.** If the scheme is `https`, an encrypted channel is established; the subject of the next lesson. 4. **Sending the request.** The request is written in the shape defined in the previous lesson. 5. **Receiving and interpreting the response.** What the body is gets determined from the status code and the `Content-Type` header. 6. **Parsing the document.** If the content is a document, its structure is extracted. 7. **Requesting subresources.** Steps 1–5 repeat for every resource the document refers to. 8. **Layout and drawing.** The position and appearance of elements on screen are computed and drawn. Part of these steps overlap. The browser starts parsing before the whole document arrives, and sends the request as soon as it sees the first subresource reference. #### Subresource Requests The seventh step explains why an opened page cannot be reduced to a single request. The following server serves three resources; the root document refers to the other two. The command writes the server to `subresource_server.py`. ```sh cat > subresource_server.py <<'EOF' from http.server import BaseHTTPRequestHandler, HTTPServer PAGES: dict[str, tuple[str, bytes]] = { "/": ("text/html; charset=utf-8", b"\n\n" b"\n" b"

Example Page

\"\"\n" b"\n"), "/style.css": ("text/css; charset=utf-8", b"h1 { color: #333; }\n"), "/logo.svg": ("image/svg+xml", b"\n"), } class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, fmt: str, *args: object) -> None: print(f"{self.command} {self.path} -> {args[1]}", flush=True) def _respond(self, status: int, kind: str, body: bytes, send_body: bool) -> None: self.send_response_only(status) self.send_header("Content-Type", kind) self.send_header("Content-Length", str(len(body))) self.end_headers() if send_body: self.wfile.write(body) def _dispatch(self, send_body: bool) -> None: if self.path in PAGES: kind, body = PAGES[self.path] self.log_request(200) self._respond(200, kind, body, send_body) else: body = b"

404 Not Found

\n" self.log_request(404) self._respond(404, "text/html; charset=utf-8", body, send_body) def do_GET(self) -> None: self._dispatch(send_body=True) def do_HEAD(self) -> None: self._dispatch(send_body=False) HTTPServer(("127.0.0.1", 8080), Handler).serve_forever() EOF ``` The three requests a browser would make, done explicitly with `curl`. The commands below run as one piece: the server is started in the background, the `until` loop connects to the server and closes without sending anything to check that it is listening, and the server is shut down at the end. The loop sends no HTTP request, so it does not appear in the server's log. If you run the server in a separate terminal, you only need to type the `curl` line. The response bodies are not printed (`> /dev/null`); the lines `curl` prints with `*`, `>` and `<` go to stderr, appear mixed with the server's log and are examined in the next section. ```bash python3 subresource_server.py & until (exec 3<>/dev/tcp/127.0.0.1/8080) 2>/dev/null; do sleep 0.05; done curl -sv --resolve example.test:8080:127.0.0.1 -A "example-client" \ http://example.test:8080/ http://example.test:8080/style.css http://example.test:8080/logo.svg > /dev/null kill $!; wait $! 2>/dev/null ``` The server's log shows three requests: ``` GET / -> 200 GET /style.css -> 200 GET /logo.svg -> 200 ``` The user typed a single address; the server saw three requests. In real pages this number is far higher, and subresources can live on other domain names — in that case the name resolution and connection steps are carried out again for each new domain name. #### The Reuse of Connections Establishing a separate connection for every request is expensive. Establishing a connection requires at least one round-trip time; an encrypted connection requires more. As set up in the What Is a Network lesson, part of the latency cannot be physically reduced, so cutting the number of round trips is a direct gain. Examining the connection behavior of the three requests above shows the situation: ``` * Trying 127.0.0.1:8080... * Connected to example.test (127.0.0.1) port 8080 > GET / HTTP/1.1 < HTTP/1.1 200 OK * Connection #0 to host example.test left intact * Re-using existing connection with host example.test > GET /style.css HTTP/1.1 < HTTP/1.1 200 OK * Connection #0 to host example.test left intact * Re-using existing connection with host example.test > GET /logo.svg HTTP/1.1 < HTTP/1.1 200 OK * Connection #0 to host example.test left intact ``` A connection was made once, and three requests passed over the same connection. This is **persistent connection** behavior, and it has become the protocol's default with HTTP/1.1; in the earlier version, a connection was made and closed for every request. A problem persistent connections do not solve remains. Requests on the same connection are processed **in order**: the second cannot be sent before the first response completes. A slow response holds up the ones behind it. This behavior is called **head-of-line blocking**. HTTP/2 answers this problem by letting several requests interleave over the same connection; this is called **multiplexing**. Requests and responses are split into numbered streams, the pieces are sent in mixed order and reassembled on the receiving end. This idea is the same reasoning as the packet numbering in the What Is a Network lesson. | Property | Separate connection | Persistent connection | Multiplexing | |---|---|---|---| | Connection setup per request | Every request | First request | First request | | Requests pending at once | As many as connections | One | Many | | Effect of a slow response | Only itself | Holds up the ones behind it | Does not hold up others | These three behaviors are the technical distinction between the protocol's versions. Which one is used is decided by the client and server agreeing while establishing the connection. #### From Document to Drawing The document received is text; what appears on screen is elements whose layout and appearance have been computed. There are three jobs in between. **Parsing.** The document text is turned into a tree of nested elements. The tree structure from the Data Structures course finds its direct counterpart here: each element is a node, and nesting is the parent–child relationship. **Style computation.** Style rules are applied to elements and each element's appearance properties are determined. **Layout and drawing.** Each element's position and size on screen are computed, then it is drawn. The way subresources affect this order differs, and this difference produces an observable consequence. Drawing before the style file arrives leads to the page appearing unstyled and then changing; for this reason, drawing is done by waiting for the style file. Scripts, since they can modify the document, can halt parsing. Images, however, are not waited for; they are placed once they arrive. This means it is not only the number of subresources but also their **order** that determines how long a page takes to appear. #### The Browser Cache So that fetched resources are not requested again, the browser keeps a cache. The lifetime logic from the Domain Name System lesson applies here too, but control is done with headers. The server states how long a response can be stored. Once the duration expires, instead of downloading the resource again, the browser makes a **conditional request**: it sends the identity of the copy it has and says "send it if it changed." If it has not changed, the server returns a short response with no body, and the body is not carried a second time. This mechanism is the source of the difference between the first opening and later openings, and it rests on the same idea as the `HEAD` method's justification from the previous lesson: getting information about content without carrying the content. #### Summary - An address is a structured value; its scheme, host, port, path, query, and fragment parts are used at different stages, and the fragment part is never sent to the server. - The browser carries out name resolution, connection setup, sending the request, parsing, requesting subresources, and drawing; the steps proceed with overlap. - Writing a single address leads to many requests; if subresources are on different domain names, the resolution and connection steps run again. - A persistent connection spreads the cost of establishing a connection across requests; multiplexing removes the requirement that requests on the same connection wait on each other. - The document is turned into a tree of elements; drawing waits on style, scripts halt parsing, images are processed without being waited on. - The browser cache makes a conditional request for resources whose duration has expired, preventing the body from being carried again. #### Next Step This lesson separated out the scheme part of the address and, for the `https` case, said in passing that a security handshake takes place. That step cannot be skipped over: the exchange must be unreadable and unmodifiable to anyone on the path, and the other party must actually be who it claims to be. The next lesson defines these three promises one by one and shows, with a locally generated certificate, how each of them is tested. ### The Role of HTTPS Source: https://academia.sh/en/courses/how-the-internet-works/role-of-https The promises of confidentiality, integrity, and authentication; how a certificate binds a name to a public key; and the role of trust anchors. The previous lesson said in passing that when the scheme part of the address is `https`, a security handshake takes place. This lesson opens up that step. As seen in the HTTP Request and Response lesson, the request and response are plain text, readable and modifiable by anyone along the way. As established in the What Is a Network lesson, this text passes through numerous intermediate nodes, and, as stated in the Structure of the Internet lesson, these nodes are under the control of different organizations. **HTTPS** is HTTP carried over a security layer called **TLS**. The protocol itself does not change; the channel it is carried over does. #### Three Promises TLS gives three separate promises. Separating them matters, because the common confusion comes from lumping all three under a single "encryption" heading. **Confidentiality.** The content of the exchange cannot be read by anyone but the two ends. Nodes along the path carry encrypted bytes; they cannot see their meaning. **Integrity.** If the content is modified along the way, that modification is detected. What is at stake is not preventing the modification but **noticing** it; corrupted data is not accepted silently. **Authentication.** It is verified that the party being connected to really is that name. Without the third promise, the first two are meaningless. A node in between can cut the connection, present itself as the server, and establish an encrypted channel with you; the content stays confidential and intact, but with the wrong party. Encryption without verified identity is whispering without knowing who you are talking to. This is why the hard part is not encryption but authentication. Encryption is a mathematical problem; authentication is the question of how you decide who a party you have never met really is. #### The Certificate Authentication is done with a **certificate**. A certificate is a document that carries three things together: a name, a public key belonging to that name, and a signature confirming the pairing of the two. The bond a certificate establishes is this: **this public key belongs to this name.** The server proves, during the handshake, that it holds the corresponding private key. This is how "I want to connect to this name" and "I am talking to the owner of this key" become joined. A certificate can be generated locally. The following command produces a key pair and a certificate for the name `example.test`: ```sh openssl req -x509 -newkey rsa:2048 -nodes \ -keyout key.pem -out cert.pem -days 3650 \ -subj "/CN=example.test" -addext "subjectAltName=DNS:example.test" ``` Examining the identity fields of the generated certificate: ```sh openssl x509 -in cert.pem -noout -subject -issuer openssl x509 -in cert.pem -noout -ext subjectAltName ``` ``` subject=CN = example.test issuer=CN = example.test X509v3 Subject Alternative Name: DNS:example.test ``` The two fields are identical. **Subject** states who the certificate belongs to, **issuer** states who confirmed it. The two being the same means the certificate confirmed itself; this is called a **self-signed** certificate. A self-signed certificate has no proof value. This document says, "I am `example.test`, because I say so." Anyone can produce the same document for any name. #### Trust Anchors The problem becomes: who decides that a certificate is correct? The answer is to accept an unfamiliar party based on the endorsement of one you know. A **certificate authority** is an organization that signs a certificate with its own key after verifying that the name really belongs to the applicant. The client has a pre-loaded set of directly trusted certificates; these are called **trust anchors**. Verification starts from the server's certificate and tries to follow its signers to reach one of this set. If it reaches one, the certificate is accepted; if not, it is not. This reduces trust to a chain, whose last link is something the client already had: trust is not produced from scratch, it is transferred. A server can be run with this certificate. The server is the same as the one in the HTTP Request and Response lesson; the only difference is that the listening socket is wrapped with TLS: ```python # tls_server.py import ssl from http.server import BaseHTTPRequestHandler, HTTPServer class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, fmt: str, *args: object) -> None: pass def do_GET(self) -> None: body = b"

Example Page

\n" self.send_response_only(200) self.send_header("Content-Type", "text/html; charset=utf-8") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) context.load_cert_chain("cert.pem", "key.pem") server = HTTPServer(("127.0.0.1", 8443), Handler) server.socket = context.wrap_socket(server.socket, server_side=True) server.serve_forever() ``` Connecting while the server is running, verification fails. The commands below run as one piece: the server is started in the background, the `until` loop waits until the server accepts connections, and the server is shut down at the end. If you run the server in a separate terminal, you only need to type the `curl` line; the later examples in this lesson start and stop the server the same way. ```bash python3 tls_server.py & until (exec 3<>/dev/tcp/127.0.0.1/8443) 2>/dev/null; do sleep 0.05; done curl -sS --resolve example.test:8443:127.0.0.1 https://example.test:8443/ kill $!; wait $! 2>/dev/null ``` ``` curl: (60) SSL certificate problem: self-signed certificate More details here: https://curl.se/docs/sslcerts.html ``` The request came back with nothing; the exit code is 60. The certificate is valid and the name is correct — but its signer is not among the client's trusted parties. The message text varies by tool; what does not change is that verification fails and the connection is not established. When the same certificate is given as a trust anchor, the connection is established: ```bash python3 tls_server.py & until (exec 3<>/dev/tcp/127.0.0.1/8443) 2>/dev/null; do sleep 0.05; done curl -s --cacert cert.pem --resolve example.test:8443:127.0.0.1 https://example.test:8443/ kill $!; wait $! 2>/dev/null ``` ```

Example Page

``` The server, certificate, and key are the same; only what the client trusts changed. This shows that trust is not a property of the certificate, it is **a decision made by the client**. #### Name Verification It is not enough for the certificate to be valid; the name it carries must also match the name being connected to. While the certificate above is considered trustworthy, connecting with a different name: ```bash python3 tls_server.py & until (exec 3<>/dev/tcp/127.0.0.1/8443) 2>/dev/null; do sleep 0.05; done curl -sS --cacert cert.pem --resolve other.test:8443:127.0.0.1 https://other.test:8443/ kill $!; wait $! 2>/dev/null ``` ``` curl: (60) SSL: no alternative certificate subject name matches target host name 'other.test' More details here: https://curl.se/docs/sslcerts.html ``` The certificate carries a trusted signature, but it was issued for the name `example.test`. A client connecting to the name `other.test` does not accept it. Without this check, any party holding a valid certificate for some name could stand in for every other name. A certificate binds a key to a **specific name**; the bond covers the name as well. #### The Handshake The handshake carries out two jobs together. **Agreeing on a key.** The two parties generate a session-specific encryption key, derived from their exchange rather than sent over the network. Even someone who records the entire exchange cannot compute the key. **Proving identity.** The server sends its certificate and shows, through an operation only performable with the corresponding private key, that it holds that key. Copying the certificate is not enough; identity cannot be proven without the private key. Once these are complete, the HTTP exchange begins, in exactly the shape defined in the previous lessons. Encryption does not change HTTP, it wraps it. Regenerating the session key for every connection has a consequence: even if the private key is later compromised, previously recorded exchanges cannot be decrypted. This property depends on the key agreement method chosen. #### What HTTPS Does Not Hide Knowing the boundaries of the promises given matters as much as knowing the promises themselves. **Name resolution is not hidden.** Before the connection is established, the query from the Domain Name System lesson is made, and this query is outside TLS. Which name you want to connect to is known by the parties who see that query. **The destination address is not hidden.** Packet headers cannot be encrypted; if they were, they could not be routed. Who you are talking to is seen by nodes along the path. **The size and timing of traffic are not hidden.** How much data went and when it went is observable. This information does not give the content by itself, but it is useful for drawing inferences. **The correctness of the content is not guaranteed.** TLS verifies that the other party is the name it claims to be; not that what it says is true. A verified site giving false information is not within the protocol's concern. This distinction has a practical consequence: a connection being encrypted does not mean the other party is trustworthy. What is verified is identity, not intent. #### Summary - HTTPS is HTTP carried over TLS; the protocol's shape does not change, the channel does. - TLS gives the promises of confidentiality, integrity, and authentication together; without authentication, the first two can be established with the wrong party. - A certificate binds a public key to a specific name and confirms this bond with a signature. - In a self-signed certificate the subject and issuer are the same and it has no proof value; verification tries to bring the signature chain to one of the trust anchors on the client. - The certificate being valid is not enough; the name it carries is also checked against the name being connected to. - HTTPS does not hide the name resolution query, the destination address, or the size and timing of traffic; nor does it give any promise about the correctness of the content. #### Next Step Every link of the chain is now established separately: packet switching, addressing, name resolution, hosting, the HTTP exchange, and the secure channel. The final lesson will join these together, end to end, over a single request, show what information each stage hands off to the next, and list the symptoms that appear when each link breaks. ### The End-to-End Trace of a Request Source: https://academia.sh/en/courses/how-the-internet-works/end-to-end-trace-of-a-request Joining the entire chain from name resolution to the drawing of the response, the information handed off between stages, and the symptoms of each link's failure. A single example was followed throughout the course: a request made to the address `example.test`. Each lesson opened up one part of this journey — how packets are carried, what addresses mean, how a name is resolved, where the server stands, how the exchange is shaped, and how the channel is secured. This lesson joins the links together. The goal is not to add a new concept but to make the flow of information between them visible. #### The Whole Chain When `https://example.test/` is typed into the address bar and the request is sent, the following steps are carried out. **1. Parsing the address.** The written text is split into parts: scheme `https`, host `example.test`, path `/`. The scheme determines that the port will be 443. *Handed off to the next stage: the name to resolve.* **2. Name resolution.** The provider asks the resolver: a cached answer returns directly, otherwise the chain from the root to the authoritative server runs, and the address is stored for its lifetime. *Handed off to the next stage: the target IP address.* **3. Establishing a connection.** A connection is opened to the target address and port. Packets pass through independently operated autonomous systems along the way; no endpoint determines the route. *Handed off to the next stage: a byte stream capable of carrying data.* **4. Security handshake.** The server presents its certificate; the client brings the signature chain to a trust anchor and checks that the certificate's name matches `example.test`. The two parties generate the session key. *Handed off to the next stage: an encrypted and authenticated channel.* **5. Sending the request.** The request line, headers, and empty line are written. The `Host` header carries the host name — since the address is by now reduced to a number, only this header tells the server which site to serve. *Handed off to the next stage: the method, path, and headers.* **6. The server's processing.** The server selects the site by looking at the `Host` header, finds the resource by looking at the path, and produces a status code with a body. *Handed off to the next stage: the status code, headers, body.* **7. Interpreting the response.** The client reads the status code; if it is a redirect, it returns to step 1 with a new target. `Content-Type` states what the body is, `Content-Length` states where it ends. *Handed off to the next stage: the document.* **8. Parsing and subresources.** The document is turned into a tree of elements. Steps 1–7 repeat for every resource referred to; requests made to the same host use the existing connection. **9. Layout and drawing.** The position of elements is computed and the page is drawn. Two observations about the chain's structure matter. First, **each stage hands off only a single thing to the next** and forgets what came before: after parsing, the name matters; after connecting, the address; after the handshake, the certificate becomes irrelevant. Second, the name comes back at step 5, because the address is no longer distinguishing by then — what virtual hosting, established in the Hosting Models lesson, requires. #### Running the Chain First the certificate and key are generated. If you still have the files from the Role of HTTPS lesson, this step can be skipped: ```sh openssl req -x509 -newkey rsa:2048 -nodes \ -keyout key.pem -out cert.pem -days 3650 \ -subj "/CN=example.test" -addext "subjectAltName=DNS:example.test" ``` All of these steps can be set up locally. The following server is wrapped with TLS and selects a site by the `Host` header, bringing together the pieces built separately throughout the course. ```python # chain_server.py import ssl from http.server import BaseHTTPRequestHandler, HTTPServer SITES: dict[str, bytes] = { "example.test": b"\n

Example Page

\n", } class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" def log_message(self, fmt: str, *args: object) -> None: print(f"[server] {self.command} {self.path} Host={self.headers.get('Host')}", flush=True) def do_GET(self) -> None: name = self.headers.get("Host", "").split(":")[0] body = SITES.get(name) self.log_request(200 if body else 404) if body is None: body = b"

Unknown host

\n" self.send_response_only(404) else: self.send_response_only(200) self.send_header("Content-Type", "text/html; charset=utf-8") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) context.load_cert_chain("cert.pem", "key.pem") server = HTTPServer(("127.0.0.1", 8443), Handler) server.socket = context.wrap_socket(server.socket, server_side=True) server.serve_forever() ``` The certificate is the `cert.pem` file generated above. The request is made with the following command; `--resolve` stands in for name resolution, `--cacert` makes the certificate a trust anchor: {/* run: skip */} ```sh curl -sv --cacert cert.pem --resolve example.test:8443:127.0.0.1 \ -A "example-client" https://example.test:8443/ ``` The trace on the client side (abridged; the removed lines are described below): ``` * Added example.test:8443:127.0.0.1 to DNS cache * Hostname example.test was found in DNS cache * Trying 127.0.0.1:8443... * Connected to example.test (127.0.0.1) port 8443 * CAfile: cert.pem * CApath: none * Server certificate: * subject: CN=example.test * subjectAltName: host "example.test" matched cert's "example.test" * issuer: CN=example.test * SSL certificate verify ok. * using HTTP/1.x > GET / HTTP/1.1 > Host: example.test:8443 > User-Agent: example-client > Accept: */* > < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 < Content-Length: 74 < * Connection #0 to host example.test left intact

Example Page

``` Three line types were removed from this output: certificate validity dates, the agreed cipher suite, and the handshake's intermediate steps. Dates depend on when the certificate was produced and the cipher suite on the parties' capabilities; both vary by machine. The lines above are what stays unchanged in this setup. The server's log is a single line: ``` [server] GET / Host=example.test:8443 ``` The trace reads as follows: the name became an address (1–2), the connection was established (3), the certificate and name match were verified (4), the request was sent (5), the server answered by the `Host` header (6), the response was interpreted (7). No subresources were found, so step 8 stayed at a single document. #### Symptoms of Failure Splitting the chain into steps has a practical value: which step a problem is in can be inferred from its symptom. Every step fails in its own characteristic way. | Symptom | Faulty stage | Cause | |---|---|---| | Name does not resolve | 2 | No record, authoritative server unresponsive, or a negative answer is cached | | Name points to an old address | 2 | The cached record's lifetime has not expired | | Connection refused | 3 | The address is correct, but no process is listening on that port | | Connection timed out | 3 | Packets are not reaching the destination, or no response is coming back | | Certificate could not be verified | 4 | The signature chain does not reach a trust anchor | | Name on the certificate does not match | 4 | The certificate was issued for a different name | | Unexpected site appeared | 6 | No site is defined for the corresponding `Host` header | | `4xx` status code | 6 | The request is malformed: path, method, or authorization | | `5xx` status code | 6 | The request was valid, the server could not fulfill it | | Page arrives but unstyled | 8 | One of the subresource requests failed | Using this table means following the order: once a symptom appears, every stage before it succeeded. A certificate error means the name resolved and the connection was established; a `5xx` means the secure channel and the request both worked. This narrowing method is the same reasoning as the layered diagnostic method from the Linux Network Administration and Troubleshooting course. Two points here are most often confused. **A refused connection and a timeout differ**: a refusal shows the target is reachable and responded explicitly, a timeout means nothing came back at all. **Name resolution problems and server problems sit at different layers**: a name resolving shows only that the record exists, not that the server is running. #### What Stays Constant in the Chain Three principles repeat throughout the structure built during the course, and they are also the ground for the courses that follow. **Indirection.** The name points to an address, the address to a location, the certificate to a key. Each layer stays fixed for the one above it while allowing the layer below it to change. **Layers not knowing about each other.** The network does not know the content of the packet it carries; HTTP does not know whether encryption sits underneath it; TLS does not know that the data it carries is HTTP. This separation lets any one layer change without touching the others. **Hard guarantees are left to the endpoints.** This principle, established in the What Is a Network lesson, repeats throughout the chain: order correction, cache consistency, authentication, and state management are all done at the endpoints. Intermediate nodes are kept simple. #### Summary - A request is carried out in nine stages, and each stage hands off only a single thing to the next, forgetting what came before. - The host name comes back with the `Host` header after the address has been resolved; virtual hosting requires this. - The entire chain can be set up locally and observed in a single trace. - Fields in the trace such as certificate dates and cipher suite depend on the setup; structural lines do not change. - Every stage has its own characteristic failure symptom; once a symptom is observed, the stages before it are known to have succeeded. - Indirection, layers not knowing about each other, and leaving guarantees to the endpoints are three design principles that repeat throughout the chain. #### Course Wrap-Up This course built, end to end, the turning of a name typed into an address bar into a page on screen. Why networks are built on packet switching, why addresses are hierarchical, why names require a separate layer, how the domain name system scales through caching, why HTTP was designed to be stateless, and what three promises TLS gives — all of it was treated as parts of a single example request. One question was deliberately left open. Correcting packet order, re-requesting lost packets, and adapting sending speed to what the network can carry were all passed over by saying "this is done at the endpoints"; the protocol that does this was never defined. Connections were spoken of too, but what a connection is, which layer it stands at, and when connectionless operation is preferred were not said. The Network Models and Protocols course fills this gap. It defines layered models and places the distinctions used intuitively here — which job belongs to which layer — into a systematic framework, takes addressing beyond the prefix arithmetic mentioned here to build subnet design, and justifies the choice between reliable and unreliable transport through the application's requirements. Every point left here by saying "that is the subject of a later course" is met there.