--- title: 'Network Models and Protocols' source: 'https://academia.sh/en/courses/network-models' language: en updated: '2026-08-17T04:39:08+00:00' license: 'CC BY-SA 4.0' lessons: 18 outcomes: - 'Places problems at the correct layer by comparing the OSI and TCP/IP models' - 'Performs subnet calculations and designs an address plan' - "Justifies the choice between TCP and UDP from the application's requirements" --- # Network Models and Protocols Layered network models, IPv4 and IPv6 addressing, subnet design, address translation and transport layer protocols; a course carried out by calculation over a single enterprise network plan. ## Layered Models The seven layers of OSI, the four layers of TCP/IP, the mechanics of encapsulation, and the per-layer data unit terms. ### OSI Model Source: https://academia.sh/en/courses/network-models/osi-model The layer responsibilities of the seven-layer reference model, the service–interface–protocol distinction, and placing a fault at the correct layer. The How the Internet Works course traced the end-to-end journey of a request typed into an address bar: the name was resolved, a connection was established, the request was sent, the response was rendered. In that account the network looked like a single, monolithic service — you make the request, the response arrives. This course opens up that single piece by dividing the work into layers. Getting an application's data from one computer to a computer on another continent involves independent problems: producing an electrical signal, addressing the neighboring device, choosing a path across networks, resending lost pieces, and preserving the meaning of the data. This lesson's question is what criterion separates these problems. #### Why Layering Layered design is the networking counterpart of the top-down decomposition from the Programming Fundamentals course: each layer uses the service of the layer below it and offers a more abstract service above it. **How** a layer works is not the concern of the layer that uses it. The concrete payoff of this separation is that layers can be changed independently of one another. Using fiber instead of copper cable affects no application running above it; changing an application protocol does not touch the routing mechanism beneath it. When a computer switches from a wireless connection to a wired one, most open sessions are unaffected, because the upper layers never see the change. In a layered architecture, three concepts are carefully kept separate: - **Service:** What a layer offers to the layer above it. "A sequential, lossless byte stream" is a service definition. - **Interface:** How the layer above accesses this service. Function calls, parameters, the data transfer format. - **Protocol:** How two instances of the same layer on separate machines talk to each other. It is the internal matter of carrying out the service, and it is closed off from the layer above. As long as the service stays the same, the protocol can change. The durability of the layered model comes from this promise. #### The Seven Layers The **OSI reference model** — Open Systems Interconnection — divides network functions into seven layers. From bottom to top: | No | Layer | Responsibility | Addressing unit | |---|---|---|---| | 7 | Application | Application-specific exchange rules | Resource name | | 6 | Presentation | Data representation, encoding, encryption | — | | 5 | Session | Dialog setup, synchronization, recovery | — | | 4 | Transport | End-to-end reliability and multiplexing | Port | | 3 | Network | Inter-network addressing and path selection | IP address | | 2 | Data Link | Frame delivery between neighboring nodes | Hardware address | | 1 | Physical | Bits turning into signals | — | The layers' responsibilities are separated by a single criterion: **the scope of a problem**. The **Physical layer** is concerned only with how a single bit is transmitted. Voltage levels, light pulses, radio carriers, connector shapes, and timing are this layer's subject matter. There is no "address" at this layer; the signal placed on the medium carries no known meaning. The **Data Link layer** carries traffic between devices sharing the same physical medium. It marks the start and end boundaries of a bit sequence, identifies the destination device by its hardware address, adds a check field for error detection, and arbitrates access when multiple devices share the medium. Its scope is a single link; it does not see beyond the next device. The **Network layer** carries traffic between networks that are not directly adjacent. It takes on two new tasks: giving each endpoint an address independent of the network, and choosing the path to that address. Routers operate at this layer. It gives no guarantee of end-to-end delivery; it offers only a "best-effort" level of service. The **Transport layer** turns communication between two endpoints into something applications can use. It distinguishes multiple applications running on the same machine by port number and — depending on the protocol — takes on the problems of loss, reordering, and duplication. The Network layer can lose packets; the Transport layer can hide this from the layer above. The **Session layer** structures the dialog between two endpoints: which side speaks when, and where to resume after an interruption. The **Presentation layer** governs the representation of data: character encoding, byte order, compression, and encryption. In practice, the functions of these two layers mostly live inside the Application layer or in libraries. The **Application layer** hosts the protocols the user's program uses directly. How a resource is requested, and in what format a piece of mail is delivered, are this layer's subject matter. #### The Network Followed Throughout the Course All of this course's calculations will be carried out over a single example: the single-campus network of a research institution. - The institution has received **a single public address** from its provider: `203.0.113.10`. - The internal network uses **a private block**: `192.168.10.0/24`. - Dividing this block into segments will be designed step by step in the lessons following this topic. - The client followed is the machine at `192.168.10.196` in the administration department; its hardware address is `00:00:5e:00:53:01`. - This machine's default gateway is the institution's router at `192.168.10.193`. - The client's destination in the outside world is a server at `198.51.100.20`. The blocks `203.0.113.0/24`, `198.51.100.0/24`, and `192.0.2.0/24` used in the example are reserved for documentation; they do not appear on real networks. The hardware addresses are likewise chosen from the `00:00:5e:00:53:xx` range reserved for documentation purposes. This choice is deliberate: an example address stops being an example the moment it points at a real system. When the client requests a resource on `198.51.100.20`, each layer does its own part: the Application layer formats the request, the Transport layer fits it into a connection, the Network layer selects the path based on the destination address, the Data Link layer delivers the frame to the gateway, and the Physical layer puts the bits on the cable. Each of these steps will be opened up one by one in the lessons that follow. #### Placing a Fault at a Layer The everyday value of the layered model is that it fixes a diagnostic order. When it is not known which layer a problem is at, narrowing proceeds from the bottom up: if a lower layer is not working, none of the layers above it work either. | Symptom | Responsible layer | Reason | |---|---|---| | Interface link light is off | Physical | No signal is being carried | | The neighboring device's hardware address does not resolve | Data Link | Address mapping fails on the local network | | The local network works, the external address is unreachable | Network | Path selection or gateway configuration is wrong | | The address is reachable, the port is unresponsive | Transport | A listening process or filtering problem at the destination | | A connection is established, the response is meaningless | Presentation / Application | Encoding or protocol mismatch | Reversing this order wastes a great deal of effort: hours can be spent on application configuration only to discover that the cable was never plugged in. The layered diagnostic method offers a systematic narrowing to prevent this waste. #### Limits of the Model OSI is a **reference model**; it is not an implementation. The protocol stack that runs on the Internet was not designed according to this model, and it does not fit it exactly. The model's known limits must be stated plainly: - The Session and Presentation layers do not exist as separate layers in most stacks; their functions are distributed into the Application layer or into libraries. - Some protocols concern more than one layer at once. Secure transport protocols sit between the Transport and Application layers; address resolution protocols draw on both the Network and Data Link layers together. - When tunneling is used, the layer sequence repeats: a Network-layer packet can become the payload of another Network-layer packet. Despite these limits, the model survives for two reasons. First, it offers a shared vocabulary: the phrase "a Layer 3 problem" carries a common meaning. Second, it gives a diagnostic order: it defines a systematic way of narrowing a problem's scope. #### Summary - Layered design separates network work by scope; each layer uses the service of the one below it and offers a more abstract service to the one above it. - Service (what is offered), interface (how it is accessed), and protocol (how peers talk to each other) are separate concepts; the protocol can change while the service stays fixed. - The OSI model defines seven layers: Physical, Data Link, Network, Transport, Session, Presentation, Application. - The fundamental distinction between layers is scope: a bit, a link, a path across networks, an end-to-end session. - Diagnosis narrows from the bottom up; if a lower layer is not working, none of the layers above it work either. - OSI is a reference model; implemented stacks do not fit it exactly, but it provides a shared vocabulary and a diagnostic order. #### Next Step The seven-layer model gives a design vocabulary, but the stack that runs on the Internet was not built according to it. The next lesson takes up the four-layer TCP/IP model that is actually implemented, maps the layers of the two models onto each other, and shows why the single common layer at the center of TCP/IP is so decisive. ### TCP/IP Model Source: https://academia.sh/en/courses/network-models/tcp-ip-model The four-layer practical model, its mapping onto OSI, the hourglass architecture, and the end-to-end principle's effect on design. The previous lesson introduced a seven-layer reference model and noted that it does not fit implemented stacks exactly. The stack that runs on the Internet comes from a different lineage: the protocols were written first, and the model was described afterward to describe them. This lesson's question is how that stack is actually divided, and what design decisions the division forces. The resulting structure is distinguished not by its number of layers, but by the singularity of the one layer in the middle. #### The Four Layers The TCP/IP model defines four layers: | No | Layer | Responsibility | Example protocol class | |---|---|---|---| | 4 | Application | Application-specific exchange rules | Resource transfer, name resolution, mail | | 3 | Transport | End-to-end multiplexing and optional reliability | TCP, UDP | | 2 | Internet | Inter-network addressing and path selection | IP, ICMP | | 1 | Link | Frame delivery on a single network | Ethernet, wireless LAN | The model's name comes from its two central protocols: **IP (Internet Protocol)** and **TCP (Transmission Control Protocol)**. The name can be misleading: the Transport layer holds other protocols besides TCP, and the model does not exclude them. Two points where the model departs from OSI stand out. First, OSI's Physical and Data Link layers merge here into a single **Link layer**. This merger is deliberate: IP promises not to know the details of the network technology beneath it, so how many layers those details are divided into is not its concern. Second, OSI's Session and Presentation layers do not exist as separate layers; their functions are left to the Application layer. #### Mapping the Two Models | OSI | TCP/IP | Note | |---|---|---| | 7 Application | Application | Direct correspondence | | 6 Presentation | Application | Encoding and encryption fall to the application or a library | | 5 Session | Application | Dialog management is not a separate layer | | 4 Transport | Transport | Direct correspondence | | 3 Network | Internet | Direct correspondence | | 2 Data Link | Link | Merger | | 1 Physical | Link | Merger | The places where the mapping breaks down need to be known. Secure transport protocols use the Transport layer's stream and encrypt the application data; they fall to the Presentation layer in the OSI vocabulary and to the Application layer in the TCP/IP vocabulary. An address resolution protocol translates a Network-layer address into a Link-layer address and belongs to both layers at once. When tunneling is used, a Network-layer packet becomes the payload of another Network-layer packet; the layer sequence repeats. For this reason, the practical value of layer numbers is not precise classification but a shared vocabulary. Saying a problem is "a Layer 3 problem" is a shorthand for saying the problem has to do with path selection. #### The Hourglass Architecture The distinguishing feature of the TCP/IP stack is not its number of layers, but **the distribution of protocol diversity per layer**. The Link layer holds a large number of technologies: wired LANs, wireless LANs, cellular connections, satellite links, point-to-point serial lines. The Application layer likewise holds countless protocols. In the middle, however, stands **a single protocol**: IP. This structure is called the **hourglass**: width at both ends, a narrow neck in the middle. The narrowness of the neck is not a limitation but the architecture itself. Because there is a single protocol in the middle: - The moment a new link technology learns to carry IP packets, it becomes available to every application. None of the applications need to change. - The moment a new application protocol runs over IP, it becomes reachable over every link technology. No change is needed in the network infrastructure. A product turns into a sum here: for $n$ link technologies and $m$ application protocols, instead of writing $n \times m$ adaptations, $n + m$ adaptations suffice. The cost of the neck is the difficulty of changing IP itself: touching the narrow neck concerns everything at both ends. This structure is why widening the address space is a project that has spanned decades — a subject taken up in the IPv6 lesson. #### The End-to-End Principle TCP/IP's second decisive design decision concerns where functions are carried out. The **end-to-end principle** states that a function should not be placed inside the network unless it can only be carried out by the network itself — a function the endpoints can perform on their own should stay at the endpoints. The reasoning is a correctness argument. An application that wants to be sure a file was transferred without error must verify this itself: even if every intermediate node performs error checking on its own link, memory corruption or a software bug inside a node falls outside those checks. Since verification at the endpoint is required regardless, verification in between is not necessary for correctness. Checking at intermediate nodes is only a performance optimization: discarding a corrupted frame early is cheaper than carrying it all the way to the endpoint and discarding it there. The concrete consequence of this principle is that the service IP offers is **best-effort**. IP does not prevent packet loss, reordering, or duplication. Whoever wants these corrected corrects them at the Transport layer or in the application. The Network layer is kept simple; complexity is pushed to the endpoints. There is a cost as well. Because the network does not know what traffic it is carrying, it cannot give traffic-specific guarantees. Prioritizing a latency-sensitive stream requires extra mechanisms. Address translation and stateful middleboxes directly strain the principle; this tension is taken up in the NAT and PAT lesson. #### The Layers' Counterpart on the Example Network Over the enterprise network introduced in the previous lesson, the four layers' counterpart is as follows. When the client at `192.168.10.196` requests a resource from the server at `198.51.100.20`: | Layer | What happens at this step | Address used | |---|---|---| | Application | The request is formatted | The resource's name | | Transport | The request is fitted into a connection, given a sequence number | Port 49152 → 443 | | Internet | The path is chosen based on the destination address | 192.168.10.196 → 198.51.100.20 | | Link | The frame is delivered to the gateway | 00:00:5e:00:53:01 → the gateway's hardware address | The distinction to notice is this: **Link-layer addresses change at every step, Network-layer addresses do not.** In the frame from the client to the gateway, the destination hardware address is the gateway's; in the frame from the gateway to the provider, the destination hardware address is the provider's router's. By contrast, the source and destination addresses in the IP header stay the same along the entire path — the exception to this rule is address translation. This distinction is the direct consequence of the two layers' difference in scope: the Link layer's scope is a single link, the Network layer's scope is the end-to-end path. #### Which Model, When The two models are not competitors; they serve different jobs. The OSI model is well suited for **teaching and diagnosis**. Seven layers give someone trying to narrow down a problem's scope a finer grid; separating the Physical layer from the Data Link layer makes it easier to distinguish a cable fault from an addressing fault. The TCP/IP model is the correct description for **implementation**. An operating system's network stack is structured around these four layers: the socket interface sits above the Transport layer, the routing table lives at the Internet layer, the interface drivers are at the Link layer. This course uses both vocabularies: OSI numbering when referring to layer numbers, TCP/IP's division when describing where protocols sit. #### Summary - The TCP/IP model consists of four layers: Link, Internet, Transport, Application. - OSI's Physical and Data Link layers merge into the Link layer; the Session and Presentation layers do not appear as separate layers. - In the hourglass architecture, a single protocol (IP) stands in the middle; this narrowness reduces the $n \times m$ adaptation problem to $n + m$, at the cost of making IP itself hard to change. - Under the end-to-end principle, a function is not placed inside the network if it can be carried out at the endpoints; IP offering a best-effort service is a consequence of this principle. - On a packet's journey, Link-layer addresses change at every step, Network- layer addresses do not. - OSI is used for diagnosis and teaching, TCP/IP for describing implementation. #### Next Step Both models say that data is wrapped as it moves through the layers, but the wrapping itself has a cost: each layer adds its own header, and these headers take up space that could carry data. The next lesson takes up the mechanics of encapsulation at the bit level, builds an IP header field by field, and calculates what percentage of the payload goes to headers. ### Encapsulation and Headers Source: https://academia.sh/en/courses/network-models/encapsulation-and-headers How data is wrapped as it moves through the layers, the fields of the IPv4 header, the header checksum calculation, and the effect of header overhead on throughput. The previous two lessons separated the layers' responsibilities. The concrete form of the exchange between layers has not yet been opened up: what exactly does a layer do with the data it receives from the layer above it? The answer can be given in one word: **it wraps it**. Each layer adds its own control information to the data coming from above and hands it to the layer below. This lesson's question is what this wrapping looks like field by field, and what it costs. #### Encapsulation **Encapsulation** is a layer wrapping the data it receives from above with its own header and handing it to the layer below. Data coming from the upper layer is a meaningless byte sequence to the lower layer — it is called the **payload**. The lower layer does not interpret the payload's contents; it only carries it. Suppose the client at `192.168.10.196` sends a 100-byte application request. Wrapping proceeds downward as follows: | Layer | Header added | Resulting size | Unit name | |---|---|---|---| | Application | — | 100 bytes | Data | | Transport (TCP) | 20 bytes | 120 bytes | Segment | | Internet (IPv4) | 20 bytes | 140 bytes | Datagram | | Link (Ethernet) | 14-byte header + 4-byte trailer | 158 bytes | Frame | At the receiver, the process runs in reverse. **Decapsulation** reads and strips off its own header at each layer and hands the remaining payload to the layer above. This symmetry is the layered model's counterpart in implementation: a stack on the sending side, the other face of the mirror on the receiving side. The 4-byte piece the Link layer adds is called not a header but a **trailer**: the frame check sequence must sit at the end of the data it was computed over. #### How a Layer Knows Where to Deliver While decapsulating, each layer must decide **which** upper-layer protocol to hand the payload to. Every header carries a **multiplexing key** for this decision: | Layer | Field | Example value | Meaning | |---|---|---|---| | Ethernet | Type (EtherType) | `0x0800` | The payload is an IPv4 datagram | | Ethernet | Type | `0x86DD` | The payload is an IPv6 datagram | | Ethernet | Type | `0x0806` | The payload is an address resolution message | | IPv4 | Protocol | `6` | The payload is a TCP segment | | IPv4 | Protocol | `17` | The payload is a UDP datagram | | IPv4 | Protocol | `1` | The payload is an ICMP message | | TCP / UDP | Destination port | `443` | The local application to receive the payload | Without this chain, decapsulation would not be possible: the receiver knows what a byte sequence in hand is only as far as the previous header told it. #### The IPv4 Header The IPv4 header, without options, is 20 bytes and consists of five 32-bit words: | Bit range | Field | Width | Function | |---|---|---|---| | 0–3 | Version | 4 bits | 4 for IPv4 | | 4–7 | Header length (IHL) | 4 bits | Number of 32-bit words, minimum 5 | | 8–15 | Type of service | 8 bits | Priority and congestion notification | | 16–31 | Total length | 16 bits | Datagram size including the header | | 32–47 | Identification | 16 bits | Which datagram the fragments belong to | | 48–50 | Flags | 3 bits | Don't fragment / more fragments | | 51–63 | Fragment offset | 13 bits | The payload's starting position (in 8-byte units) | | 64–71 | Time to live (TTL) | 8 bits | Remaining hop count | | 72–79 | Protocol | 8 bits | The payload's type | | 80–95 | Header checksum | 16 bits | Error detection over the header only | | 96–127 | Source address | 32 bits | Sender | | 128–159 | Destination address | 32 bits | Receiver | Three fields deserve particular attention. **Header length** is a word count: if its value is 5, the header is $5 \times 4 = 20$ bytes. **Fragment offset** counts 8-byte units, not bytes; with 13 bits, positions up to $8 \times 8191 = 65\,528$ bytes can be represented. **Header checksum** protects only the header, not the payload — the payload's integrity is the upper layer's job. #### Building the Header at the Bit Level The fields' bit alignment is built with the shifting and masking operations introduced in the Bit-Level Operations lesson of the How Computers Work course. Version and header length are packed into the same byte: version is shifted left four bits and combined with the length. The program below builds and reads back the IPv4 header of an empty TCP segment going from the client on the example network to `198.51.100.20`. ```python import ipaddress import struct def header_checksum(data: bytes) -> int: """The one's complement of the one's complement sum of 16-bit words.""" total = 0 for i in range(0, len(data), 2): total += (data[i] << 8) + data[i + 1] while total >> 16: # add the overflow bit back in total = (total & 0xFFFF) + (total >> 16) return ~total & 0xFFFF def ipv4_header(source: str, dest: str, total_length: int, identifier: int, ttl: int, protocol: int) -> bytes: version_ihl = (4 << 4) | 5 # version 4, header 5 words = 20 bytes flags_offset = (0b010 << 13) | 0 # don't fragment (DF), offset 0 raw = struct.pack( "!BBHHHBBH4s4s", version_ihl, 0, total_length, identifier, flags_offset, ttl, protocol, 0, ipaddress.ip_address(source).packed, ipaddress.ip_address(dest).packed, ) c = header_checksum(raw) return raw[:10] + struct.pack("!H", c) + raw[12:] header = ipv4_header("192.168.10.196", "198.51.100.20", total_length=40, identifier=7238, ttl=64, protocol=6) print(header.hex()) print("length:", len(header), "bytes") print("checksum check:", header_checksum(header)) fields = struct.unpack("!BBHHHBBH4s4s", header) print("version :", fields[0] >> 4) print("header size :", (fields[0] & 0x0F) * 4, "bytes") print("total size :", fields[2], "bytes") print("identifier :", fields[3]) print("flags :", format(fields[4] >> 13, "03b")) print("frag offset :", fields[4] & 0x1FFF) print("TTL :", fields[5]) print("protocol :", fields[6]) print("checksum :", hex(fields[7])) print("source :", ipaddress.ip_address(fields[8])) print("dest :", ipaddress.ip_address(fields[9])) ``` Output: ``` 450000281c464000400628d6c0a80ac4c6336414 length: 20 bytes checksum check: 0 version : 4 header size : 20 bytes total size : 40 bytes identifier : 7238 flags : 010 frag offset : 0 TTL : 64 protocol : 6 checksum : 0x28d6 source : 192.168.10.196 dest : 198.51.100.20 ``` The `!` prefix in the `struct` format string selects **network byte order**, meaning big-endian. This ordering, introduced in the Byte Order lesson of the How Computers Work course, is mandatory in protocol headers: the sending and receiving machines' internal byte orders can differ, but the order on the wire is single. The first byte of the hex output is `45`: the high nibble `4` gives the version, the low nibble `5` gives the header length. The next two bytes are `0028`, that is 40 — the total length. The first three bits of the value `4000` are `010`, the "don't fragment" flag. #### The Header Checksum The checksum is the one's complement of the one's complement sum of 16-bit words. During the calculation, the checksum field itself is taken as zero. The `while total >> 16` loop adds the part that overflows 16 bits back into the sum; this step is a requirement of one's complement arithmetic. The elegance of verification is this: when the receiver performs the same calculation on the header with the checksum field **filled in**, it must get zero. The `checksum check: 0` line in the output above shows this. As a result, the receiver does not need to make a separate comparison. The method's power is limited: if word order changes, the sum does not change, so this error cannot be detected. It provides no protection against deliberate tampering. Still, it is cheap, and because TTL changes at every hop, the checksum must be recomputed at every router — keeping the cost low is a design requirement. #### The Header Overhead Ratio Every header takes space away from the data that could be carried. **Overhead** gives the size of this share. Ethernet's fixed 18-byte cost (14 header + 4 trailer), IPv4's 20 bytes, and TCP's 20 bytes add up to 58 bytes. | Application data | Frame size | Throughput ratio | |---|---|---| | 100 bytes | 158 bytes | 63.29% | | 500 bytes | 558 bytes | 89.61% | | 1460 bytes | 1518 bytes | 96.18% | The ratio improves rapidly as the data size grows. This is the numerical justification for a fundamental rule in network programming: batching small writes and sending them at once is markedly more efficient than sending the same data piece by piece. The real cost on the wire is a bit higher. Ethernet requires an 8-byte preamble before every frame and a 12-byte gap between frames. For 1460 bytes of data, the total on the wire is $1460 + 58 + 20 = 1538$ bytes; the throughput ratio drops to 94.93%. #### Summary - Encapsulation is each layer wrapping the data it receives from above with its own header and handing it to the layer below; at the receiver, the process runs in reverse, symmetrically. - Every header carries a multiplexing key: the type field in Ethernet, the protocol field in IP, the port number at the Transport layer. - The IPv4 header without options is 20 bytes; header length counts 32-bit words, and fragment offset counts 8-byte units. - The header checksum is the one's complement of the one's complement sum of 16-bit words; recomputed over the filled-in header, it must yield zero. - The checksum protects only the header, not the payload, and cannot detect word-order changes. - Header overhead is 58 bytes; throughput is 63.29% for 100 bytes of data and 96.18% for 1460 bytes. #### Next Step This lesson used the names "segment," "datagram," and "frame," each denoting a specific layer's data unit. These terms are used interchangeably in everyday speech, yet the difference between them determines size limits. The next lesson will pin down the per-layer data unit terms, define what the maximum transmission unit is, and calculate piece by piece how a 4000-byte payload passes through a 1500-byte medium. ### Frame, Packet, Segment Source: https://academia.sh/en/courses/network-models/frame-packet-segment Per-layer data unit terms, the maximum transmission unit, the maximum segment size, and the IPv4 fragmentation calculation. The previous lesson traced 100 bytes of application data turning into a 158-byte frame, and used three separate names along the way: segment, datagram, frame. These names are not arbitrary; each denotes a specific layer's data unit. Pinning down these names is not merely terminological rigor. Each layer has its own size limit, and these limits determine one another. This lesson's question is how large a piece of data can be at most, and what happens when it exceeds the limit. #### Data Unit Per Layer The unit a layer forms together with its own header is called a **protocol data unit (PDU)**. The naming per layer is as follows: | Layer | Unit name | Contents | |---|---|---| | Application | Data / message | The byte sequence the application produces | | Transport (TCP) | Segment | TCP header + application data | | Transport (UDP) | Datagram | UDP header + application data | | Internet | Packet / datagram | IP header + transport unit | | Link | Frame | Link header + IP packet + trailer | | Physical | Bit / symbol | The signal placed on the medium | Two distinctions are especially prone to confusion. The first is the **segment versus datagram** distinction. TCP's unit is the segment; UDP's unit is the datagram. The difference in name is not an accident: a segment is a slice cut from a byte stream, and its place within the stream is known by its sequence number. A datagram, by contrast, is a self-contained message whose boundaries are preserved. This difference will be taken up in detail in the Transport Layer topic. The second is the two meanings of the word **packet**. In the narrow sense, packet is the Network layer's unit. In the broad sense, it denotes any piece of data traveling over a network — this is the usage in the term "packet switching." This course uses the terms "packet" or "datagram" for the Network-layer unit; to avoid loose usage, the other layers' units are referred to by their own names. #### Maximum Transmission Unit The **maximum transmission unit (MTU)** is the upper limit on the payload a link-layer technology can carry in a single frame. The measure **does not include** the link header: MTU is the size of the Network-layer packet the frame can carry. In the common Ethernet implementation, this limit is 1500 bytes. The reason the limit is an upper bound is that the medium is shared: the longer a single frame takes, the longer the devices waiting their turn have to wait. The reason there is also a lower bound is that header overhead becomes intolerable in small frames. The MTU value can change along a path. A tunnel, since it must reserve room for its own header, lowers the MTU of the connection inside it. The smallest value that holds across the entire path is called the **path MTU**. #### Maximum Segment Size The **maximum segment size (MSS)** is the upper limit on the application data a TCP segment can carry. It is derived from the MTU, with the headers subtracted: $$ \text{MSS} = \text{MTU} - \text{IP header} - \text{TCP header} $$ | Medium | MTU | IP header | TCP header | MSS | |---|---|---|---|---| | Ethernet, IPv4 | 1500 | 20 | 20 | 1460 | | Ethernet, IPv6 | 1500 | 40 | 20 | 1440 | | Tunneled connection | 1492 | 20 | 20 | 1452 | The corresponding limit for UDP is $1500 - 20 - 8 = 1472$ bytes; the UDP header is 8 bytes. MSS is announced between the two ends when a TCP connection is established: each end tells the other the largest segment size it can accept, and the smaller of the two governs. This announcement lets the sending side produce segments without falling into fragmentation. When options are used, the TCP header exceeds 20 bytes and MSS shrinks accordingly. With options such as timestamps and selective acknowledgment active, the usable data space narrows. #### Fragmentation An IPv4 packet is **fragmented** if it is larger than the MTU of the link it must cross. Fragmentation is carried out with three fields in the IP header: - **Identification:** The same across all fragments coming from the same original datagram. - **More Fragments flag (MF):** 1 in every fragment except the last. - **Fragment offset:** Gives which byte, counted from the start of the original payload, this fragment's payload begins at. Since the field is 13 bits, the offset is counted **in 8-byte units**; this is why the payload of every fragment except the last must be a multiple of 8. The program below takes a given payload size and MTU value and computes the fragments. ```python def fragment(payload_size: int, mtu: int, header: int = 20) -> None: """Splits an IPv4 datagram's payload into fragments according to the MTU limit.""" fragment_payload = (mtu - header) // 8 * 8 # offset must be a multiple of 8 bytes offset = 0 remaining = payload_size index = 1 while remaining > 0: this_fragment = min(fragment_payload, remaining) remaining -= this_fragment print(f"fragment {index}: payload={this_fragment:4d} bytes " f"offset field={offset // 8:3d} MF={1 if remaining else 0} " f"datagram={this_fragment + header:4d} bytes") offset += this_fragment index += 1 fragment(4000, mtu=1500) ``` Output: ``` fragment 1: payload=1480 bytes offset field= 0 MF=1 datagram=1500 bytes fragment 2: payload=1480 bytes offset field=185 MF=1 datagram=1500 bytes fragment 3: payload=1040 bytes offset field=370 MF=0 datagram=1060 bytes ``` The steps of the calculation can be followed. Since MTU is 1500 and the IP header is 20 bytes, at most $1500 - 20 = 1480$ bytes of payload can be carried per fragment; 1480 is already a multiple of 8. The first fragment starts at byte 0, so its offset field is 0. The second fragment starts at byte 1480: $1480 / 8 = 185$. The third fragment starts at byte 2960: $2960 / 8 = 370$, and it carries the remaining $4000 - 2960 = 1040$ bytes. The total payload carried is $1480 + 1480 + 1040 = 4000$ bytes, but the total on the wire comes to $1500 + 1500 + 1060 = 4060$ bytes: two extra IP headers have been added. A smaller MTU value quickly increases the number of fragments: ``` fragment(4000, mtu=576) ``` ``` fragment 1: payload= 552 bytes offset field= 0 MF=1 datagram= 572 bytes fragment 2: payload= 552 bytes offset field= 69 MF=1 datagram= 572 bytes fragment 3: payload= 552 bytes offset field=138 MF=1 datagram= 572 bytes fragment 4: payload= 552 bytes offset field=207 MF=1 datagram= 572 bytes fragment 5: payload= 552 bytes offset field=276 MF=1 datagram= 572 bytes fragment 6: payload= 552 bytes offset field=345 MF=1 datagram= 572 bytes fragment 7: payload= 552 bytes offset field=414 MF=1 datagram= 572 bytes fragment 8: payload= 136 bytes offset field=483 MF=0 datagram= 156 bytes ``` Here, since $576 - 20 = 556$ is not a multiple of 8, the fragment payload has been rounded down to 552. #### The Cost of Fragmentation Fragmentation is undesirable for three reasons. **The cost of loss is multiplied.** Fragments can only be reassembled once all of them arrive. If one of eight fragments is lost, the other seven are discarded as well and the original datagram must be resent. Under the assumption of independent loss, if the per-fragment loss probability is $p$, the probability that an $n$-fragment datagram arrives is $(1-p)^n$; for $p = 0.01$ and $n = 8$, this ratio is about $0.923$, meaning datagram loss rises to 7.7%. **The reassembly burden falls on the receiver.** Fragments are not reassembled along the path; they are reassembled only at the final destination. The receiver must hold memory and run a timeout while waiting for missing fragments. **The job of intermediate devices gets harder.** Only the first fragment carries the Transport-layer header; later fragments have no port number. Mechanisms that decide based on port cannot classify the later fragments. For this reason, common practice is to avoid fragmentation. The sender sets the **don't fragment (DF)** flag in the IP header; when a packet arrives at a link larger than the MTU, the router drops it and returns an error message to the sender. The sender shrinks the segment size based on the MTU value in this message. This cycle is called **path MTU discovery**, and it will be taken up in detail in the ICMP lesson. IPv6 has removed fragmentation from routers along the path entirely: if a packet is larger than the MTU, the router does not fragment it, it drops it and reports this. If fragmentation is needed, only the sending endpoint performs it. #### Limits on the Example Network What happens when the client on the enterprise network wants to send a 3000-byte body to `198.51.100.20` is determined by the calculation built in this lesson. If TCP is used, fragmentation never occurs: the Transport layer splits the body itself and produces three segments of $1460 + 1460 + 80$ bytes. Each segment goes out as a separate IP packet, and none of them exceeds the MTU. If UDP is used, the situation is different: UDP does not split the body, it produces a single 3000-byte datagram, and the job of splitting falls to the IP layer. This is a concrete example of the choice of transport layer directly determining network-layer behavior. #### Summary - Each layer's data unit is called by a separate name: segment (TCP), datagram (UDP and IP), packet (IP), frame (Link layer). - MTU is the upper limit on the Network-layer packet a link can carry in a single frame, and it does not include the link header. - MSS is found by subtracting the IP and TCP headers from the MTU: 1460 bytes for Ethernet and IPv4. - Fragmentation is carried out with the identification, more-fragments flag, and fragment offset fields; since offset is counted in 8-byte units, the payloads of every fragment except the last must be a multiple of 8. - A 4000-byte payload splits into three fragments of 1480 + 1480 + 1040 bytes at a 1500-byte MTU; the total on the wire comes to 4060 bytes. - Fragmentation multiplies the cost of loss; common practice is to avoid it with the don't-fragment flag and path MTU discovery. #### Next Step This topic defined the shape data takes across the layers. The next topic descends into how these shapes are actually carried. The first stop is the lowest addressing level: how devices sharing the same cable or the same wireless cell find each other, and how a switch decides where to send a given frame. The next lesson takes up the Ethernet frame and the structure of hardware addresses. ## Link and Network Layer The mechanics of local networking and routing, from the Ethernet frame to IPv4 and IPv6 addressing, from subnet design to address translation. ### Ethernet and MAC Addresses Source: https://academia.sh/en/courses/network-models/ethernet-and-mac-addresses The fields of the Ethernet frame, the bit structure of the 48-bit hardware address, broadcast addresses, and a switch's address-learning mechanism. The previous topic defined how data is wrapped across the layers and named the outermost wrapping the frame. The frame's contents, though, have not been opened yet: what fields does a frame consist of, and what does the address on it mean? This lesson's question is how devices sharing the same physical medium find each other. The answer lies in an addressing scheme that is entirely local, independent of network-layer addresses. #### The Ethernet Frame An Ethernet frame consists of the following fields: | Field | Size | Function | |---|---|---| | Destination address | 6 bytes | Hardware address of the device receiving the frame | | Source address | 6 bytes | Hardware address of the sending device | | Type / length | 2 bytes | The payload's protocol (`0x0800` IPv4, `0x86DD` IPv6, `0x0806` ARP) | | Payload | 46–1500 bytes | The network-layer packet being carried | | Frame check sequence | 4 bytes | Cyclic redundancy check (CRC-32) | Before the frame, a 7-byte preamble and a 1-byte start-of-frame delimiter are placed on the wire; the medium is left idle for 12 bytes between frames. These 20 bytes do not belong to the frame, but they consume time on the wire. The payload's **lower** bound is 46 bytes. If the payload is shorter, padding is added to fill it out to 46. The reason for this bound is collision detection: a frame must take long enough to reach the farthest device on the wire and return, or the sender finishes sending without noticing the collision. Together with the address and type fields, the smallest frame is $6 + 6 + 2 + 46 + 4 = 64$ bytes. The payload's **upper** bound is 1500 bytes — this is the MTU value defined in the previous lesson. The frame's own headers are outside this bound; the largest frame is 1518 bytes. The frame check sequence is a cyclic redundancy check computed over the entire frame. Unlike the checksum in the IP header, it also covers the payload and detects bit errors noticeably more strongly. If the check fails, the frame is **silently dropped**; Ethernet does not retransmit. Noticing and recovering from the loss is the job of the upper layers. #### The Structure of the MAC Address A **MAC address (media access control address)** is a 48-bit value that addresses a network interface at the physical level. It is written as six hexadecimal bytes separated by colons or hyphens: `00:00:5e:00:53:01`. The address consists of two parts: - **First 3 bytes:** the organizationally unique identifier. Assigned to the organization that manufactures the interface. - **Last 3 bytes:** the interface number the organization assigns itself. The first byte's lowest two bits carry a separate meaning, and are the first thing to check when reading an address: | Bit | Name | If 0 | If 1 | |---|---|---|---| | 0 (lowest) | I/G | Unicast address (single interface) | Group address (broadcast or multicast) | | 1 | U/L | Universally administered (assigned by manufacturer) | Locally administered (assigned locally) | The program below decodes these two bits. ```python def decode_mac(mac: str) -> None: bytes_ = [int(x, 16) for x in mac.split(":")] first = bytes_[0] kind = "group" if first & 0b1 else "unicast" source = "local" if first & 0b10 else "universal" print(f"{mac} first byte={first:08b} {kind:7s} {source}") for address in ("00:00:5e:00:53:01", # client on the example network "00:00:5e:00:53:00", # gateway's internal interface "ff:ff:ff:ff:ff:ff", # broadcast "01:00:5e:00:00:01", # IPv4 multicast "33:33:00:00:00:01", # IPv6 multicast "02:00:5e:00:53:2f"): # locally administered decode_mac(address) ``` Output: ``` 00:00:5e:00:53:01 first byte=00000000 unicast universal 00:00:5e:00:53:00 first byte=00000000 unicast universal ff:ff:ff:ff:ff:ff first byte=11111111 group local 01:00:5e:00:00:01 first byte=00000001 group universal 33:33:00:00:00:01 first byte=00110011 group local 02:00:5e:00:53:2f first byte=00000010 unicast local ``` The masking operations are a direct application of the method introduced in the Bit-Level Operations lesson of the How Computers Work course: `first & 0b1` isolates the lowest bit, `first & 0b10` isolates the next one. Three address classes are distinguished. `ff:ff:ff:ff:ff:ff` is the **broadcast** address; every interface on the local network processes this frame. Addresses starting with `01:00:5e:...` are for IPv4 multicast, those starting with `33:33:...` for IPv6 multicast; only interfaces that joined the relevant group process them. The rest are unicast addresses. Locally administered addresses are addresses that do not come from a manufacturer: virtual interfaces, container networks, and wireless clients that rotate their address for privacy use this block. In `02:00:5e:00:53:2f`, the second bit is set, so the address is local. #### The Scope of Addresses A MAC address is a **local** address. When a frame passes through a router, the frame is entirely rebuilt: the source address becomes the router's outgoing interface, and the destination address becomes the next device. The distinction emphasized in the previous topic becomes concrete here — link-layer addresses change at every hop, IP addresses do not. One consequence of this: a MAC address can be seen only by devices on the same local network. A remote server does not know the hardware address of the client connecting to it. #### How a Switch Learns Addresses A **switch** is a device that operates at the link layer and forwards frames only to the port they need to reach. It does not know in advance which address is on which port; it learns **by observing source addresses**. The rule has three steps: 1. An incoming frame's **source** address is written to the table together with the port it arrived on. 2. If the frame's **destination** address is in the table, the frame is forwarded only to that port. 3. If the destination address is not in the table, or is a group address, the frame is sent to every port except the one it arrived on. This is called **flooding**. On the switch in the example network's administration section, after the client sends its first frame to the gateway, the table becomes this: | MAC address | Port | How it was learned | |---|---|---| | `00:00:5e:00:53:01` | 4 | Source address of the frame the client sent | The gateway's address is not yet in the table; the client's frame goes to every port by flooding. When the gateway responds, its address is learned too: | MAC address | Port | How it was learned | |---|---|---| | `00:00:5e:00:53:01` | 4 | The client's frame | | `00:00:5e:00:53:00` | 1 | The gateway's response frame | Table entries are removed after an aging period; this lets the table correct itself when a device moves to a different port. One consequence of this learning method is that two addresses are not naturally learned together: if a device has never sent a frame, the switch does not know where it is, and every frame headed to it is flooded. #### Collision Domain and Broadcast Domain Two scope concepts must be distinguished. A **collision domain** is the set of devices where a collision can occur if they send at the same time. Every port of a switch is its own collision domain; on a link running full duplex, no collision occurs at all. A **broadcast domain** is the set of devices a broadcast frame reaches. Because a switch floods broadcast frames, it does not split the broadcast domain. The device that splits a broadcast domain is the router: a router does not pass broadcast frames to the other side. This distinction is one of the reasons the example network is split into sections. As the number of devices in a single broadcast domain grows, so does the work each broadcast frame creates: in a hundred-device broadcast domain, one broadcast frame causes ninety-nine unnecessary processing events. Putting the organization's sections into separate networks limits this load. The **virtual LAN (VLAN)** mechanism makes it possible to define multiple broadcast domains on a single physical switch: every port is tagged with an identifier, and frames do not pass between ports with different identifiers. The sections in the example network could just as well be separated this way, on one switch instead of separate switches; the address plan is the same either way. #### Summary - An Ethernet frame consists of a destination address, source address, type field, 46–1500-byte payload, and 4-byte check sequence; the smallest frame is 64 bytes, the largest 1518. - If the check sequence fails, the frame is silently dropped; recovering the loss is the job of the upper layers. - A MAC address is 48 bits; the first byte's lowest bit gives the unicast/group distinction, the next bit the universal/local distinction. - `ff:ff:ff:ff:ff:ff` is the broadcast address; the `01:00:5e:...` and `33:33:...` prefixes are for multicast. - A switch learns its address table by observing incoming frames' source addresses; it floods frames with an unknown destination. - A switch splits collision domains but not the broadcast domain; the device that splits the broadcast domain is the router. #### Next Step To send a frame, the destination's hardware address must be known. Yet applications specify the destination by IP address; a client knows it will send to `192.168.10.193`, but does not know which hardware address that corresponds to. The next lesson covers the address resolution protocol that bridges these two address spaces, and the security gap it leaves behind. ### ARP Source: https://academia.sh/en/courses/network-models/arp Resolving an IP address to a hardware address, the request–reply cycle, cache behavior, proxy and gratuitous usage, and address poisoning. The previous lesson showed that a hardware address is written into a frame's destination field. Yet applications specify the destination by IP address: the client on the example network wants to reach `198.51.100.20`, knows its gateway is `192.168.10.193`, but knows the hardware equivalent of neither address. A bridge is needed between the two address spaces. This lesson's question is how that bridge is built, and what assumptions are made while building it. #### Two Address Spaces There is **no** computable relationship between an IP address and a MAC address. One is a logical name assigned by a network administrator or a leasing service; the other is a physical address tied to the interface's hardware. When the same machine moves from network to network, its IP address changes; its MAC address does not. Since the mapping cannot be computed, it **must be asked**. **ARP (Address Resolution Protocol)** is the protocol that asks this question. The method is direct: a broadcast is sent to the entire local network asking "who has this IP address?" and the owner of the address replies. ARP is not carried inside IP; it is carried directly in the Ethernet frame with type value `0x0806`. This is a concrete example of the observation from the previous topic that "some protocols concern two layers at once": an ARP message carries a network-layer address, but does not use a network-layer header. #### The Fields of the Message An ARP message is 28 bytes and consists of nine fields. | Field | Size | Example value | |---|---|---| | Hardware type | 2 bytes | `1` (Ethernet) | | Protocol type | 2 bytes | `0x0800` (IPv4) | | Hardware address length | 1 byte | `6` | | Protocol address length | 1 byte | `4` | | Operation | 2 bytes | `1` request, `2` reply | | Sender hardware address | 6 bytes | MAC address of the one asking | | Sender protocol address | 4 bytes | IP address of the one asking | | Target hardware address | 6 bytes | Zero in the request, filled in the reply | | Target protocol address | 4 bytes | The IP address being asked about | The presence of the length fields shows the protocol is not limited to Ethernet and IPv4: the format is designed for technologies with different address widths. The program below builds the request the client on the example network sends asking for its gateway, and the reply that comes back. ```python import ipaddress import struct FORMAT = "!HHBBH6s4s6s4s" def pack_mac(mac: str) -> bytes: return bytes(int(x, 16) for x in mac.split(":")) def format_mac(raw: bytes) -> str: return ":".join(f"{b:02x}" for b in raw) def arp_message(operation: int, sender_mac: str, sender_ip: str, target_mac: str, target_ip: str) -> bytes: return struct.pack( FORMAT, 1, # hardware type: Ethernet 0x0800, # protocol type: IPv4 6, # hardware address length 4, # protocol address length operation, # 1 = request, 2 = reply pack_mac(sender_mac), ipaddress.ip_address(sender_ip).packed, pack_mac(target_mac), ipaddress.ip_address(target_ip).packed, ) def print_arp(raw: bytes) -> None: fields = struct.unpack(FORMAT, raw) operation = {1: "request", 2: "reply"}[fields[4]] print(f"op={operation:7s} sender={format_mac(fields[5])} / {ipaddress.ip_address(fields[6])}" f" target={format_mac(fields[7])} / {ipaddress.ip_address(fields[8])}") request = arp_message(1, "00:00:5e:00:53:01", "192.168.10.196", "00:00:00:00:00:00", "192.168.10.193") reply = arp_message(2, "00:00:5e:00:53:00", "192.168.10.193", "00:00:5e:00:53:01", "192.168.10.196") print("message size:", len(request), "bytes") print(request.hex()) print_arp(request) print(reply.hex()) print_arp(reply) ``` Output: ``` message size: 28 bytes 000108000604000100005e005301c0a80ac4000000000000c0a80ac1 op=request sender=00:00:5e:00:53:01 / 192.168.10.196 target=00:00:00:00:00:00 / 192.168.10.193 000108000604000200005e005300c0a80ac100005e005301c0a80ac4 op=reply sender=00:00:5e:00:53:00 / 192.168.10.193 target=00:00:5e:00:53:01 / 192.168.10.196 ``` The target hardware address field in the request message is zero — that is the question itself. In the reply this field is filled in, and the sender and target roles swap. #### How the Request and Reply Are Carried As much as the message itself, the addresses of the frame carrying it matter: | | ARP request | ARP reply | |---|---|---| | Frame source address | `00:00:5e:00:53:01` | `00:00:5e:00:53:00` | | Frame destination address | `ff:ff:ff:ff:ff:ff` | `00:00:5e:00:53:01` | | Devices it reaches | The entire broadcast domain | Only the one who asked | The request goes by broadcast because it is not known which device holds the address being asked about. The reply, by contrast, goes to a unicast address: the responder has learned both of the asker's addresses from the sender fields inside the request. This asymmetry has a side effect: every device in the broadcast domain sees the request and, if it wants, writes the sender's mapping into its own cache. This behavior is the basis of the security problem covered in a later section. #### Local or Remote The first decision made before a packet is sent is whether the destination is on the same network. The decision is made by comparing the destination address against the local netmask; netmask arithmetic is built up in the Subnetting lesson. The result of this decision determines **which address gets resolved**: | Destination | Same network? | Address asked via ARP | Frame's destination MAC | |---|---|---|---| | `192.168.10.200` | Yes | `192.168.10.200` | The destination's own MAC address | | `198.51.100.20` | No | `192.168.10.193` (gateway) | The gateway's MAC address | The second row is the most frequently misunderstood point in network mechanics. In a packet headed to a remote destination, **the IP destination address is the remote server, the MAC destination address is the gateway.** The two addresses point to different devices, and this is not an inconsistency: the IP address specifies the end-to-end destination, the MAC address specifies the next hop. The remote server's MAC address is never asked for. It cannot be, either: an ARP request travels by broadcast, and a broadcast does not cross a router. #### The ARP Cache Because broadcasting for every resolution would be expensive, results are kept in a **cache**. The client on the example network's cache, after talking to its gateway and reaching a printer in the same section, holds this content: | IP address | MAC address | State | |---|---|---| | `192.168.10.193` | `00:00:5e:00:53:00` | Reachable | | `192.168.10.210` | `00:00:5e:00:53:04` | Reachable | | `192.168.10.205` | — | Incomplete (awaiting reply) | Entries are not permanent. An entry unused for a while is removed; removal lets the mapping correct itself when a device's network card changes or an address moves to another device. The first send after an entry is removed triggers a fresh resolution, and that send is delayed slightly. The commands used to read cache state and their output formats vary by operating system. What stays constant is the three pieces of information shown in the table above: protocol address, hardware address, and the entry's state. #### Gratuitous and Proxy Usage **Gratuitous ARP** is a device announcing its own address without anyone asking: a request is broadcast whose target protocol address is the device's own address. It serves three purposes. It detects an address conflict — if a reply comes back, the same address is on another device. A standby device announces it has taken over an address during failover and refreshes switches' tables. It updates neighbors' stale mappings when an interface comes up. **Proxy ARP** is a router replying for an address that is not its own. If the address asked about is remote, the router gives its own MAC address and takes on the traffic. It keeps clients with a misconfigured netmask working; but because it hides the problem itself and produces many identical-looking entries in caches, it makes diagnosis harder. #### ARP Poisoning ARP's design has no authentication. Who sent a reply is not checked; even an unsolicited reply gets written to the cache in many implementations. The result is the attack called **ARP poisoning**. If a device in the same broadcast domain sends replies claiming that `192.168.10.193` is at its own MAC address, its neighbors' caches are corrupted and traffic meant for the gateway gets redirected to the attacker. The attacker can keep the connection alive by forwarding the traffic on to the gateway; in that case the user sees no interruption at all. The source of the vulnerability is not an implementation bug but the protocol's design assumption: devices on the same local network are assumed trustworthy. Countermeasures are therefore outside ARP itself: - Shrinking broadcast domains and putting untrusted devices on separate networks. - ARP inspection on the switch: allowing a port to announce only the addresses assigned to that device. - Pinning critical mappings. - Using end-to-end encryption and authentication: an intermediary listening to the traffic cannot read or modify the content. The last item is an application of the end-to-end principle: when a layer inside the network cannot make a security promise, the promise has to be made at the endpoints. #### Its Counterpart in IPv6 IPv6 does not use ARP. The equivalent function has moved into ICMPv6 under the name **neighbor discovery**, and uses multicast instead of broadcast: the question is delivered only to the multicast group associated with the address being asked about. This keeps every device in the broadcast domain from having to process every question. Neighbor discovery also takes on functions like duplicate address detection and router advertisement; these are covered in the IPv6 lesson. #### Summary - There is no computable relationship between an IP address and a MAC address; the mapping is asked for with ARP. - An ARP request travels by broadcast, the reply returns to a unicast address; the message is 28 bytes and uses no IP header. - If the destination is on the same network, its own address is resolved; if not, the gateway's address is; in a packet headed to a remote destination, the IP target is the server, the MAC target is the gateway. - Resolution results are kept in a time-limited cache; when the time expires, the mapping corrects itself. - Gratuitous ARP is used for address announcement and conflict detection, proxy ARP for taking on another network's address. - ARP has no authentication; countermeasures against poisoning are shrinking the broadcast domain, switch-level inspection, and end-to-end encryption. #### Next Step Up to this point, IP addresses were used like labels: `192.168.10.196` was written, its meaning was not questioned. Yet there is a structure inside this number — one part points to the network, one part to the host within that network. The next lesson covers the bit structure of the IPv4 address, the split between network and host parts, and the blocks set aside for special purposes. ### IPv4 Addressing Source: https://academia.sh/en/courses/network-models/ipv4-addressing The structure of the 32-bit address, the split between network and host portions, the legacy of classful addressing, and blocks reserved for special purposes. The previous two lessons used IP addresses like labels: `192.168.10.196` was written, it was said which device it pointed to, but the number itself was not decoded. Yet inside this number is a structure that routing depends on. This lesson's question is what that structure is: which part of the 32 bits points to the network, which part to the host within that network, and who determines this split? #### The Structure of the Address An IPv4 address is a **32-bit unsigned integer**. For readability it is split into four bytes, each byte written in decimal, with dots placed between them — this is called **dotted decimal notation**. The notation is a convenience; the address itself is a bit string. ```python import ipaddress address = ipaddress.ip_address("192.168.10.196") print("integer :", int(address)) print("binary :", format(int(address), "032b")) print("byte-wise :", " ".join(f"{b:08b}" for b in address.packed)) print("hex :", address.packed.hex()) ``` ``` integer : 3232238276 binary : 11000000101010000000101011000100 byte-wise : 11000000 10101000 00001010 11000100 hex : c0a80ac4 ``` The four bytes' values are read directly: `11000000` = 192, `10101000` = 168, `00001010` = 10, `11000100` = 196. The base conversions are a direct application of the method built up in the Binary Number System and Hexadecimal and Octal Bases lessons of the How Computers Work course. The total number of addresses is $2^{32} = 4\,294\,967\,296$. This number falling short against the number of endpoint devices in the world is the reason many mechanisms covered in this course — address translation, variable-length subnets, IPv6 — exist at all. #### Network and Host Portion An address is not a single-piece identity; it splits into **two portions**: - **Network portion:** the left-hand bits. Says which network the address belongs to. The same across every address on that network. - **Host portion:** the right-hand bits. Distinguishes the address within that network. The split point is stated by the **prefix length**, written after the address with a slash: in the notation `192.168.10.196/24`, the left-hand 24 bits specify the network, the remaining 8 bits the host. | Notation | Network portion | Host portion | |---|---|---| | `192.168.10.196/24` | `11000000 10101000 00001010` | `11000100` | | `192.168.10.196/27` | `11000000 10101000 00001010 110` | `00100` | The same address belongs to different networks with different prefix lengths. This produces a very important consequence: **an address by itself is incomplete information.** Configuring an interface requires giving the address and the prefix length together; if either is missing, which destinations are local cannot be determined. The split's counterpart in routing is direct. Routers know **networks**, not individual hosts. A row in a routing table maps a prefix to a next hop; the number of rows in the table is proportional to the number of networks, not the number of hosts. This is why addresses are distributed hierarchically. #### The Legacy of Classful Addressing IPv4 addresses were originally split into **classes**: the boundary between network and host was determined by looking at the address's first bits. | Class | First bits | Address range | Network portion | Hosts per network | |---|---|---|---|---| | A | `0` | 0.0.0.0 – 127.255.255.255 | 8 bits | 16,777,214 | | B | `10` | 128.0.0.0 – 191.255.255.255 | 16 bits | 65,534 | | C | `110` | 192.0.0.0 – 223.255.255.255 | 24 bits | 254 | | D | `1110` | 224.0.0.0 – 239.255.255.255 | Multicast | — | | E | `1111` | 240.0.0.0 – 255.255.255.255 | Reserved | — | This scheme had two flaws. First, the boundary being able to take only three values led to address waste: a class C is not enough for a 300-host organization, and giving it a class B wastes 65,234 addresses. Second, the size of routing tables could not be controlled: every network needed its own row, and rows could not be merged. The solution was to make the boundary independent of the address. With **classless inter-domain routing (CIDR)**, the prefix length is carried explicitly alongside the address and can take any value from 0 to 32. Classes are no longer used in address allocation; but the traces they left remain — some tools' default mask suggestions and the phrase "class C network" in some documents come from this legacy. This course takes classless notation as the basis. Class names are mentioned only for historical context. #### Special-Purpose Blocks Part of the address space is set aside to never be routed on the public internet. | Block | Purpose | Routed on the public internet? | |---|---|---| | `10.0.0.0/8` | Private use | No | | `172.16.0.0/12` | Private use | No | | `192.168.0.0/16` | Private use | No | | `127.0.0.0/8` | Loopback (own machine) | No | | `169.254.0.0/16` | Link-local (when no address can be obtained) | No | | `100.64.0.0/10` | Carrier-grade NAT | No | | `224.0.0.0/4` | Multicast | Conditional | | `255.255.255.255` | Limited broadcast | No | | `192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24` | Documentation | No | The three **private use** blocks can be used freely on internal organizational networks. Different organizations can use the same private addresses at the same time; there is no collision, because these addresses never leave the organization. Going outward is provided by address translation, the subject of the NAT and PAT lesson. The **loopback** block refers to the machine itself; a packet sent to this block never reaches the network interface. The **link-local** block is for the temporary address an interface chooses for itself when it cannot obtain an address; this address can only reach other devices on the same network. The documentation blocks are used throughout this course wherever an example needs a public address. The organization's external address, `203.0.113.10`, and the destination server, `198.51.100.20`, come from these blocks. ```python import ipaddress for text in ("10.0.0.1", "127.0.0.1", "169.254.7.3", "224.0.0.251", "192.168.10.196", "198.51.100.20"): addr = ipaddress.ip_address(text) print(f"{text:16s} private={str(addr.is_private):5s} loopback={str(addr.is_loopback):5s} " f"link_local={str(addr.is_link_local):5s} multicast={addr.is_multicast}") ``` ``` 10.0.0.1 private=True loopback=False link_local=False multicast=False 127.0.0.1 private=True loopback=True link_local=False multicast=False 169.254.7.3 private=True loopback=False link_local=True multicast=False 224.0.0.251 private=False loopback=False link_local=False multicast=True 192.168.10.196 private=True loopback=False link_local=False multicast=False 198.51.100.20 private=True loopback=False link_local=False multicast=False ``` The last row deserves attention: `198.51.100.20` is a documentation address, not in the private use block; yet `is_private` still returns true. This property does not answer "is it in the private use block," but "is it routable on the public internet." Differences like this between a library property's name and its meaning make reading the documented meaning mandatory. #### Who an Address Belongs To An IP address belongs not to a machine but to an **interface**. A machine with two interfaces has two addresses; more than one address can also be assigned to the same interface. Routers are the clearest example of this rule: on the example network, the organization's router carries `192.168.10.193` on its internal interface and `203.0.113.10` on its external interface, and these two addresses belong to different networks. This distinction helps in diagnosis. "I cannot reach the server" is incomplete; which of the server's interfaces cannot be reached must be asked. A machine reachable through one interface can be unreachable through another. #### The Example Network's Block The organization's internal network uses the `192.168.10.0/24` block. The reasons for the choice are these: - The block is in the private use range; it requires no allocation from a provider. - A 24-bit prefix leaves an 8-bit host portion: $2^8 = 256$ addresses. - Two of these 256 addresses are reserved — the lowest denotes the network itself, the highest the broadcast address — leaving 254 usable addresses. The organization's sections have 182 hosts in total: 100 on the guest wireless network, 50 in the lab, 20 in administration, 10 in the server room, and 2 on the point-to-point link between routers. Since that number is less than 254, the block is sufficient — but the sections cannot be gathered into a single broadcast domain. Why they cannot be gathered was seen in the previous lesson: as a broadcast domain grows, the cost of every broadcast frame rises, and no access control can be established between sections. The block needs to be split into sections. #### Summary - An IPv4 address is a 32-bit integer; dotted decimal notation is only for readability. - The address splits into network and host portions; the boundary is stated by the prefix length and cannot be read from the address alone. - Without a prefix length, an address is incomplete information; which destinations are local cannot be determined. - Classful addressing confined the boundary to three values and led to waste; CIDR freed the prefix length to any value from 0 to 32. - Private use, loopback, link-local, multicast, and documentation blocks are not routed on the public internet. - An address belongs to an interface, not a machine; a machine can have more than one address. #### Next Step The `192.168.10.0/24` block cannot be handed to the organization's sections as a single piece. The block has to be cut so each section gets a separate network, and the cutting is done by borrowing bits from the host portion. The next lesson builds up mask arithmetic, shows how network and broadcast addresses are computed, and produces the first plan that splits the block into equal parts. ### Subnetting Source: https://academia.sh/en/courses/network-models/subnetting Netmask arithmetic, computing network and broadcast addresses, borrowing bits, and splitting the example block into equal subnets. The previous lesson showed that the `192.168.10.0/24` block is numerically sufficient for the organization's 182 hosts, but that the sections cannot be gathered into a single broadcast domain. The block has to be cut. The cutting operation is called **subnetting**, and it rests on a single mechanism: borrowing bits from the host portion and adding them to the network portion. This lesson's question is the arithmetic of that borrowing. #### The Netmask A **netmask** is a 32-bit pattern showing which bits of an address belong to the network portion: network bits are 1, host bits are 0. In a mask, the 1s are always on the left and unbroken; a pattern like `255.255.0.255` is not a valid mask. The prefix length and the mask carry the same information in two separate forms: | Prefix | Mask | Host bits | Addresses per block | |---|---|---|---| | /24 | 255.255.255.0 | 8 | 256 | | /25 | 255.255.255.128 | 7 | 128 | | /26 | 255.255.255.192 | 6 | 64 | | /27 | 255.255.255.224 | 5 | 32 | | /28 | 255.255.255.240 | 4 | 16 | | /29 | 255.255.255.248 | 3 | 8 | | /30 | 255.255.255.252 | 2 | 4 | The last column is computed with $2^{32 - \text{prefix}}$. When the prefix grows by one bit, the block halves — this binary relationship makes computing each case individually unnecessary when planning. #### Network and Broadcast Addresses The network an address belongs to is found by putting the address and the mask through a **bitwise AND** operation. This is a direct application of the masking pattern introduced in the Bit-Level Operations lesson of the How Computers Work course: bits where the mask is 0 are cleared, bits where it is 1 are kept. The broadcast address is found by setting all the host bits to 1. This uses the **wildcard mask**, the inverse of the netmask, put through an OR operation. $$ \text{network} = \text{address} \mathbin{\&} \text{mask}, \qquad \text{broadcast} = \text{network} \mathbin{|} \lnot\text{mask} $$ For address `192.168.10.196` and mask `255.255.255.192`: ```python import ipaddress address = ipaddress.ip_address("192.168.10.196") mask = ipaddress.ip_address("255.255.255.192") network = ipaddress.ip_address(int(address) & int(mask)) wildcard = ipaddress.ip_address(int(mask) ^ 0xFFFFFFFF) broadcast = ipaddress.ip_address(int(network) | int(wildcard)) print("address :", format(int(address), "032b"), address) print("mask :", format(int(mask), "032b"), mask) print("network :", format(int(network), "032b"), network) print("wildcard :", format(int(wildcard), "032b"), wildcard) print("broadcast :", format(int(broadcast), "032b"), broadcast) ``` ``` address : 11000000101010000000101011000100 192.168.10.196 mask : 11111111111111111111111111000000 255.255.255.192 network : 11000000101010000000101011000000 192.168.10.192 wildcard : 00000000000000000000000000111111 0.0.0.63 broadcast : 11000000101010000000101011111111 192.168.10.255 ``` Except for the last two bits, the network portion is preserved: the `11000100` byte's upper six bits are `110001`, and its host bits are `00`. The network address is `192.168.10.192`, the broadcast address is `192.168.10.255`, and the 62 addresses in between are usable host addresses. #### The Number of Usable Hosts Every network has two addresses reserved: the address with all host bits 0 denotes **the network itself**, the one with all host bits 1 denotes **the broadcast address**. Neither can be assigned to an interface. The number of usable hosts: $$ N = 2^{32 - p} - 2 $$ | Prefix | Total addresses | Usable hosts | |---|---|---| | /24 | 256 | 254 | | /25 | 128 | 126 | | /26 | 64 | 62 | | /27 | 32 | 30 | | /28 | 16 | 14 | | /29 | 8 | 6 | | /30 | 4 | 2 | On point-to-point links, the `/31` prefix can also be used: since a link where two devices connect directly has no need for a broadcast address, both addresses can be assigned. This course uses the more common `/30` choice in its example plan. The reverse calculation is the one used when planning: the prefix length needed for $N$ hosts is the largest $p$ that satisfies the inequality $2^{32-p} - 2 \ge N$. | Hosts needed | Smallest sufficient block | Usable | Wasted | |---|---|---|---| | 2 | /30 | 2 | 0 | | 10 | /28 | 14 | 4 | | 20 | /27 | 30 | 10 | | 50 | /26 | 62 | 12 | | 100 | /25 | 126 | 26 | #### Borrowing Bits Splitting a block means borrowing bits from the host portion and adding them to the network portion. Every bit borrowed doubles the number of subnets and halves the block size: | Bits borrowed | New prefix | Number of subnets | Hosts per subnet | |---|---|---|---| | 0 | /24 | 1 | 254 | | 1 | /25 | 2 | 126 | | 2 | /26 | 4 | 62 | | 3 | /27 | 8 | 30 | | 4 | /28 | 16 | 14 | The number of subnets is $2^k$, hosts per subnet is $2^{8-k} - 2$; their product never reaches 254, because every subnet spends its own network and broadcast address. Splitting trades address loss for separation. Subnets' **starting addresses** are multiples of the block size. In a `/26` split, the block size is 64, so the starting points are 0, 64, 128, 192. This increment is called the **block size (increment)** and is read from the mask's last byte that is not zero: for mask `255.255.255.192`, $256 - 192 = 64$. #### Splitting the Example Block in Four The organization has four sections and one point-to-point link. The first attempt is to split the block into four equal parts. ```python import ipaddress block = ipaddress.ip_network("192.168.10.0/24") for subnet in block.subnets(new_prefix=26): hosts = list(subnet.hosts()) print(f"{str(subnet):18s} mask={subnet.netmask:15s} network={str(subnet.network_address):15s} " f"first={str(hosts[0]):15s} last={str(hosts[-1]):15s} " f"broadcast={str(subnet.broadcast_address):15s} n={len(hosts)}") ``` ``` 192.168.10.0/26 mask=255.255.255.192 network=192.168.10.0 first=192.168.10.1 last=192.168.10.62 broadcast=192.168.10.63 n=62 192.168.10.64/26 mask=255.255.255.192 network=192.168.10.64 first=192.168.10.65 last=192.168.10.126 broadcast=192.168.10.127 n=62 192.168.10.128/26 mask=255.255.255.192 network=192.168.10.128 first=192.168.10.129 last=192.168.10.190 broadcast=192.168.10.191 n=62 192.168.10.192/26 mask=255.255.255.192 network=192.168.10.192 first=192.168.10.193 last=192.168.10.254 broadcast=192.168.10.255 n=62 ``` This table is a check on the hand calculation. The same result can be derived on paper: block increment 64, starting points 0/64/128/192, every block's broadcast address is one less than the next starting point. The plan compares against the requirements as follows: | Section | Hosts needed | Block assigned | Usable | Result | |---|---|---|---|---| | Guest wireless | 100 | `192.168.10.0/26` | 62 | Insufficient | | Lab | 50 | `192.168.10.64/26` | 62 | Sufficient, 12 spare | | Administration | 20 | `192.168.10.128/26` | 62 | Sufficient, 42 spare | | Server room | 10 | `192.168.10.192/26` | 62 | Sufficient, 52 spare | | Point-to-point link | 2 | — | — | No block left | The plan fails at two points. The guest network needs 100 hosts, gets 62 addresses. No block is left to give the point-to-point link. Meanwhile, 52 addresses sit unused in the server room network. #### The Inherent Flaw of Equal Splitting The reason for the failure is not a calculation error but the method itself. In equal splitting, **every subnet has to be sized for the largest requirement**. If the largest section needs 100 hosts, every subnet must be at least `/25`; and only two `/25` blocks fit inside a `/24`. The total addresses needed under equal splitting is $5 \times 128 = 640$. There are 256 addresses on hand. Yet the sections' real total requirement is 182. The gap comes entirely from the method. The source of the problem is a single assumption: **that every subnet must carry the same mask.** This assumption disappeared with CIDR, and once it is removed, the plan fits. #### Which Subnet an Address Falls Into After a subnet plan is set up, a frequently asked question is which subnet a given address belongs to. The answer is found by ANDing the address with each mask in the plan and comparing the result against the network address. The same address falls into different networks with different masks: | Address | Mask | Network address | Broadcast address | |---|---|---|---| | `192.168.10.196` | /24 | `192.168.10.0` | `192.168.10.255` | | `192.168.10.196` | /25 | `192.168.10.128` | `192.168.10.255` | | `192.168.10.196` | /26 | `192.168.10.192` | `192.168.10.255` | | `192.168.10.196` | /27 | `192.168.10.192` | `192.168.10.223` | | `192.168.10.196` | /28 | `192.168.10.192` | `192.168.10.207` | The table is the numeric counterpart of the observation made in the previous lesson: an address alone is incomplete information. If a client's mask is misconfigured, it mistakes local destinations for remote ones, or remote destinations for local ones. In the first case it sends traffic to the gateway needlessly; in the second it broadcasts an ARP request for an address it can never reach and gets no reply. #### Summary - A netmask is a 32-bit pattern marking an address's network bits with 1; the 1s are on the left and unbroken. - The network address is found by ANDing the address with the mask, the broadcast address by ORing the network with the wildcard mask. - Every subnet has two addresses reserved; the number of usable hosts is $2^{32-p} - 2$. - Every bit borrowed doubles the number of subnets and halves the subnet size; the block increment is the difference between 256 and the mask's last significant byte. - Splitting the `192.168.10.0/24` block into four equal `/26` parts does not cover the guest network and leaves no room for the link network; equal splitting needs 640 addresses in total. - The flaw of equal splitting is that every subnet has to be sized for the largest requirement. #### Next Step Every subnet carrying the same mask is not a requirement. If each section is given a mask sized to its own requirement, the same `192.168.10.0/24` block covers all five networks and still leaves addresses to spare. The next lesson builds this plan step by step, computes each section's network and broadcast addresses, and tests that the plan is free of overlap. ### Variable-Length Subnetting Source: https://academia.sh/en/courses/network-models/variable-length-subnetting The method of splitting with different masks, the alignment rule, building the example organization's plan with an overlap test, and route summarization. The previous lesson showed that equal splitting could not stretch the `192.168.10.0/24` block to cover the organization's requirements: the method needed 640 addresses, and only 256 were on hand. The source of the flaw was the requirement that every subnet carry the same mask. CIDR removes this requirement. In the **variable length subnet mask (VLSM)** approach, each subnet is sized to its own requirement. This lesson's question is what rule governs this split and how the result is tested. #### The Alignment Rule Variable splitting carries exactly one constraint: **a subnet's starting address must be a multiple of its own size.** The rule is not a formatting preference; it is the forced consequence of mask arithmetic. A `/27` block's last five bits are host bits, and all of them must be zero in the network address; this only holds when the address is a multiple of 32. The notation `192.168.10.200/27` does not define a valid network, because 200 is not a multiple of 32 — that address is a host inside the `192.168.10.192/27` network. | Prefix | Block size | Valid starting points (last byte) | |---|---|---| | /25 | 128 | 0, 128 | | /26 | 64 | 0, 64, 128, 192 | | /27 | 32 | 0, 32, 64, …, 224 | | /28 | 16 | 0, 16, 32, …, 240 | | /30 | 4 | 0, 4, 8, …, 252 | The rule's consequence for planning is this: **larger blocks must be placed first.** If smaller blocks are placed first, the next larger block is forced to jump ahead to its own alignment point, leaving an unusable gap in between. #### Building the Plan The method is three steps: 1. Sort the sections by host requirement in **descending** order. 2. For each section, pick the longest prefix satisfying $2^{32-p} - 2 \ge N$. 3. Place the blocks in order, respecting alignment. The organization's requirements, already sorted: guest wireless 100, lab 50, administration 20, server room 10, point-to-point link 2 hosts. **Guest wireless, 100 hosts.** Since $2^7 - 2 = 126 \ge 100$, `/25` is sufficient; `/26` gives only 62. Block `192.168.10.0/25`, range `.1` – `.126`, broadcast `.127`. **Lab, 50 hosts.** `/26` gives 62 hosts. The next aligned address is 128. Block `192.168.10.128/26`, range `.129` – `.190`, broadcast `.191`. **Administration, 20 hosts.** `/27` gives 30 hosts. The next aligned address is 192. Block `192.168.10.192/27`, range `.193` – `.222`, broadcast `.223`. **Server room, 10 hosts.** `/28` gives 14 hosts. The next aligned address is 224. Block `192.168.10.224/28`, range `.225` – `.238`, broadcast `.239`. **Point-to-point link, 2 hosts.** `/30` gives exactly 2 hosts. The next aligned address is 240. Block `192.168.10.240/30`, range `.241` – `.242`, broadcast `.243`. #### Verifying the Plan A hand computation must be tested with a program. The program below encodes the method and produces the plan. ```python import ipaddress import math BLOCK = ipaddress.ip_network("192.168.10.0/24") REQUIREMENTS = [ ("Guest wireless", 100), ("Lab", 50), ("Administration", 20), ("Server room", 10), ("Link", 2), ] def required_prefix(hosts: int) -> int: """Longest sufficient prefix for `hosts` usable addresses.""" return 32 - math.ceil(math.log2(hosts + 2)) def vlsm_plan(block: ipaddress.IPv4Network, requirements: list[tuple[str, int]]) -> list[tuple]: sorted_reqs = sorted(requirements, key=lambda k: k[1], reverse=True) cursor = int(block.network_address) plan = [] for name, hosts in sorted_reqs: prefix = required_prefix(hosts) size = 2 ** (32 - prefix) if cursor % size: # alignment: the block must land on its own size cursor += size - (cursor % size) subnet = ipaddress.ip_network((cursor, prefix)) if not subnet.subnet_of(block): raise ValueError(f"no room left for {name}") plan.append((name, hosts, subnet)) cursor += size return plan plan = vlsm_plan(BLOCK, REQUIREMENTS) print(f"{'Section':18s} {'Hosts':>7s} {'Block':<19s} {'Mask':<16s} " f"{'Broadcast':<15s} {'Range':<31s} {'Spare':>4s}") for name, hosts, subnet in plan: d = list(subnet.hosts()) rng = f"{d[0]} - {d[-1]}" print(f"{name:18s} {hosts:7d} {str(subnet):<19s} {str(subnet.netmask):<16s} " f"{str(subnet.broadcast_address):<15s} {rng:<31s} {len(d) - hosts:4d}") used = sum(subnet.num_addresses for _, _, subnet in plan) print(f"\naddresses used: {used} / {BLOCK.num_addresses} " f"spare: {BLOCK.num_addresses - used}") networks = [subnet for _, _, subnet in plan] for i in range(len(networks)): for j in range(i + 1, len(networks)): assert not networks[i].overlaps(networks[j]), (networks[i], networks[j]) print("overlap test: passed") ``` ``` Section Hosts Block Mask Broadcast Range Spare Guest wireless 100 192.168.10.0/25 255.255.255.128 192.168.10.127 192.168.10.1 - 192.168.10.126 26 Lab 50 192.168.10.128/26 255.255.255.192 192.168.10.191 192.168.10.129 - 192.168.10.190 12 Administration 20 192.168.10.192/27 255.255.255.224 192.168.10.223 192.168.10.193 - 192.168.10.222 10 Server room 10 192.168.10.224/28 255.255.255.240 192.168.10.239 192.168.10.225 - 192.168.10.238 4 Link 2 192.168.10.240/30 255.255.255.252 192.168.10.243 192.168.10.241 - 192.168.10.242 0 addresses used: 244 / 256 spare: 12 overlap test: passed ``` The hand computation is fully confirmed. Where equal splitting demanded 640 addresses, variable splitting made do with 244, leaving 12 addresses spare. The `hosts + 2` term inside `required_prefix` encodes the fact that every network spends its own network and broadcast addresses. The call to `math.ceil(math.log2(...))` rounds the required host count up: for 102, $\lceil \log_2 102 \rceil = 7$, giving prefix $32 - 7 = 25$. The overlap test is not a formality. The most common mistake in hand-built plans is one block falling inside the previous one; the `overlaps` call catches this mistake instead of letting it pass silently. #### The Finalized Plan This is the plan the course uses from here on: | Section | Block | Mask | Gateway | Usable range | Broadcast | |---|---|---|---|---|---| | Guest wireless | `192.168.10.0/25` | 255.255.255.128 | `.1` | `.1` – `.126` | `.127` | | Lab | `192.168.10.128/26` | 255.255.255.192 | `.129` | `.129` – `.190` | `.191` | | Administration | `192.168.10.192/27` | 255.255.255.224 | `.193` | `.193` – `.222` | `.223` | | Server room | `192.168.10.224/28` | 255.255.255.240 | `.225` | `.225` – `.238` | `.239` | | Link | `192.168.10.240/30` | 255.255.255.252 | — | `.241` – `.242` | `.243` | In each section, the first usable address is reserved for the router interface. The client introduced in the course's first lesson, `192.168.10.196`, is in the administration network, and its gateway is `192.168.10.193` — the plan confirms both values. The 12 spare addresses lie between `192.168.10.244` and `192.168.10.255` and correspond to three `/30` blocks: `.244/30`, `.248/30`, `.252/30`. These are available for point-to-point links added later. #### The Effect of Ordering What would happen if the sections were placed in ascending order? The smallest block goes first, and each subsequent block jumps ahead to align: | Order | Section | Block | Gap skipped | |---|---|---|---| | Ascending | Link | `192.168.10.0/30` | — | | Ascending | Server room | `192.168.10.16/28` | `.4` – `.15` (12 addresses) | | Ascending | Administration | `192.168.10.32/27` | — | | Ascending | Lab | `192.168.10.64/26` | — | | Ascending | Guest | `192.168.10.128/25` | — | The plan still fits, but this time the spare 12 addresses sit in the **middle** of the block, between the Link and Server networks. Alignment creates the gap: a `/28` can only start at a multiple of 16, so Server starts at `.16` rather than `.4`. Under descending order this skip is never needed; each block starts right after the larger blocks and is therefore already aligned, and the spare 12 addresses collect at the **end** of the plan. In neither order is the gap a single CIDR block (`.4/30` + `.8/29` versus `.244/30` + `.248/29`); the difference is where the gap sits. The gap in the middle is an alignment leftover squeezed between two networks; the gap at the end sits where the plan ends without interruption, where later subnets are added in turn. This is the address-space counterpart of the fragmentation problem introduced in the context of allocators in the Memory Layout — Stack and Heap lesson of the How Computers Work course. #### Route Summarization Variable splitting's second benefit is in the routing table. Contiguous, aligned subnets can be **summarized** into a single prefix — a technique called **route summarization**. The organization's five subnets are all inside the `192.168.10.0/24` block. The provider's router does not need to know these five networks individually; a single line is enough: ``` 192.168.10.0/24 -> 203.0.113.10 ``` Summarization is a direct consequence of the alignment rule. Four contiguous `/24` blocks — `192.168.8.0`, `192.168.9.0`, `192.168.10.0`, `192.168.11.0` — can be referred to with a single `192.168.8.0/22` prefix, because all four form an aligned group of four starting at a third octet that is a multiple of four — here 8. The same group of four could not be summarized if it started at `192.168.9.0`: 9 is not a multiple of four. The gain from summarization is direct: the number of rows in the routing table drops, lookups get cheaper, and a change inside the internal network does not affect external tables. This is the rationale for distributing addresses hierarchically. #### Documenting the Plan An address plan is used not only at the moment it is computed but at every subsequent change. The fields that must be documented are: section name, block, mask, gateway address, the range reserved for static assignments, the range left for dynamic distribution, and growth headroom. Growth headroom is easy to overlook. In the plan above, the guest network has 26 spare addresses, the lab has 12, and administration has 10; the server room network has only 4 spare addresses left. If the number of servers rises to 15, the `/28` is not enough and the block must be resized. Whoever builds the plan should note upfront which section overflows at which threshold. #### Summary - In variable-length splitting, each subnet is sized to its own requirement; the only constraint is that the starting address be a multiple of the block size. - The method is three steps: sort requirements in descending order, pick the longest sufficient prefix for each, and place them respecting alignment. - The organization's five networks fit inside `192.168.10.0/24` using 244 addresses; equal splitting would have needed 640 addresses for the same job. - Descending order leaves the spare addresses at the end of the block, contiguous; ascending order fragments the gap in the middle. - Contiguous, aligned subnets can be summarized into a single prefix; the five subnets are advertised outward as `192.168.10.0/24`. - Growth headroom must also be recorded when documenting the plan; in the example plan, the tightest headroom is 4 addresses, in the server room network. #### Next Step This plan is one way of coping with the scarcity of the 32-bit address space: using private addresses and splitting them carefully. There is another way that removes the scarcity itself — expanding the address space. The next lesson covers 128-bit addresses, their notation rules, how interfaces configure their own addresses, and the methods by which the two versions interoperate. ### IPv6 Source: https://academia.sh/en/courses/network-models/ipv6 The notation rules of the 128-bit address, address types, autoconfiguration, header differences, and how the two versions interoperate. The previous two lessons built methods for coping with the scarcity of the 32-bit address space: splitting private addresses carefully, giving each section exactly what it needs. These methods manage the scarcity; they do not remove it. The solution aimed at the scarcity itself is expanding the address space. This lesson's question is what decisions come bundled with quadrupling the address width — because address width sits at the neck of the hourglass introduced in the previous topic, and touching it affects everything. #### 128 Bits An IPv6 address is 128 bits. The total number of addresses is $2^{128}$, roughly $3.4 \times 10^{38}$. For comparison: IPv4's $2^{32}$ addresses are one $2^{96}$th of IPv6's space. Choosing such an extravagant width was not meant to maximize the address count. The goal is to **ease hierarchical distribution**: when addresses are abundant, every level can be given generous blocks, blocks can be kept contiguous, and prefixes can be summarized. In IPv4, the effort to conserve addresses had led to fragmented routing tables. The visible consequence of this is that IPv6 subnets that carry hosts are **`/64`**. Regardless of how many hosts a subnet has — even if it is only two devices — it is given a 64-bit host portion. The one place outside the rule is router links, which need no autoconfiguration: for a point-to-point link between two routers, narrower prefixes such as `/127` are separately defined. Viewed with IPv4 habits, this is wasteful; in IPv6's design, it is the precondition for autoconfiguration. #### Notation Rules The address is written as eight hexadecimal groups separated by colons. There are three shortening rules: 1. Leading zeros in each group can be dropped: `0db8` → `db8`, `0000` → `0`. 2. Consecutive zero groups can be dropped once with `::`. 3. `::` can be used **at most once** in an address; using it twice would leave it ambiguous how many groups each gap stands for. ```python import ipaddress for text in ("2001:0db8:0010:0003:0000:0000:0000:0001", "fe80:0000:0000:0000:0200:5eff:fe00:5301", "ff02:0000:0000:0000:0000:0000:0000:0001", "0000:0000:0000:0000:0000:0000:0000:0001", "2001:0db8:0000:0000:0001:0000:0000:0001"): a = ipaddress.ip_address(text) print(f"{text} -> {a.compressed}") ``` ``` 2001:0db8:0010:0003:0000:0000:0000:0001 -> 2001:db8:10:3::1 fe80:0000:0000:0000:0200:5eff:fe00:5301 -> fe80::200:5eff:fe00:5301 ff02:0000:0000:0000:0000:0000:0000:0001 -> ff02::1 0000:0000:0000:0000:0000:0000:0000:0001 -> ::1 2001:0db8:0000:0000:0001:0000:0000:0001 -> 2001:db8::1:0:0:1 ``` The last line shows the third rule: the address has two separate runs of zeros, and only the **longer one** is compressed. The shorter one must be written as `0:0`. If the two runs are equal in length, the left one is compressed. When written together with a port number, the address is enclosed in square brackets: `[2001:db8:10:3::1]:443`. Otherwise it cannot be told whether the final colon belongs to the address or the port. #### Address Types IPv6 has **no broadcast address**. Broadcast's function is met by multicast instead; this gives an "ask those who care" behavior in place of "ask everyone," and it reduces unnecessary processing. | Prefix | Type | Scope | |---|---|---| | `2000::/3` | Global unicast address | Global Internet | | `fe80::/10` | Link-local | Same link only | | `fc00::/7` | Unique local address | Within the organization | | `ff00::/8` | Multicast | Embedded in prefix | | `::1/128` | Loopback | The machine itself | | `::/128` | Unspecified address | "I have no address" in the source field | | `2001:db8::/32` | Documentation | Not routed | A **link-local address** is present on every IPv6 interface and is mandatory; it exists even without a global address. Router advertisements and neighbor discovery run over these addresses. Because the address is only meaningful within a single link, the same link-local address can repeat on different interfaces; this is why the interface must be specified when it is used. A **unique local address** is the counterpart of IPv4's private-use blocks: it is used within the organization and is not routed on the public Internet. In multicast addresses, the prefix also carries the scope. `ff02::1` denotes all nodes within link scope, `ff02::2` all routers within link scope. The **solicited-node multicast address** is the mechanism that solves ARP's broadcast problem: the target address's last 24 bits are appended to the `ff02::1:ff00:0/104` prefix. The neighbor discovery query is sent to this address, and only the small number of interfaces whose last 24 bits match process the message. This prevents every device in the broadcast domain from having to process every query. #### Autoconfiguration IPv6 interfaces can configure an address without a server. **Stateless address autoconfiguration (SLAAC)** is four steps: 1. The interface generates itself a **link-local address** (`fe80::` prefix + interface identifier). 2. It performs **duplicate address detection**: it sends a neighbor request to the solicited-node group of the address it generated. If a reply arrives, the address is in use. 3. It sends a **router solicitation** to `ff02::2`. The router replies with a **router advertisement** containing the `/64` prefix used on the link. 4. The interface combines the advertised prefix with its own interface identifier to build its global address. One way to generate the interface identifier is the **EUI-64** conversion: `ff:fe` is inserted into the middle of the 48-bit MAC address, and the universal/local bit is flipped. ```python import ipaddress def eui64(mac: str) -> str: b = [int(x, 16) for x in mac.split(":")] b[0] ^= 0x02 # flip the universal/local bit raw = bytes(b[:3] + [0xFF, 0xFE] + b[3:]) return ":".join(raw[i:i + 2].hex() for i in range(0, 8, 2)) interface_id = eui64("00:00:5e:00:53:01") print("interface identifier:", interface_id) print("link-local address :", ipaddress.ip_address("fe80::" + interface_id)) print("global address :", ipaddress.ip_address("2001:db8:10:3:" + interface_id)) ``` ``` interface identifier: 0200:5eff:fe00:5301 link-local address : fe80::200:5eff:fe00:5301 global address : 2001:db8:10:3:200:5eff:fe00:5301 ``` The example network's client had MAC address `00:00:5e:00:53:01`; the first byte's second bit was flipped to make it `02`, and `ff:fe` was inserted in the middle. The reason for the `/64` subnet requirement is visible here: the interface identifier is 64 bits. The EUI-64 conversion has a privacy problem: because the address embeds the hardware address, the machine can be tracked even as it moves from network to network. For this reason, end hosts mostly generate the interface identifier randomly and rotate it periodically. Servers, by contrast, use fixed and readable identifiers — manually assigned addresses like `2001:db8:10:4::20` are common. SLAAC does not carry name server information. That information is delivered either through an option in the router advertisement or through the stateless leasing protocol defined for IPv6. #### Header Differences The IPv6 header is a fixed 40 bytes; it is larger than IPv4's 20-byte header, but it has fewer fields and no variable length. | Change | Rationale | |---|---| | Header checksum removed | The link layer and transport layer already check; recomputing at every hop is redundant | | Fragmentation at routers removed | The cost of fragmentation is pushed to the endpoints; an intermediate node drops the packet and reports it | | Header length fixed | The processing pipeline is simplified | | Options moved to extension headers | They are appended as a chain, not bloating the main header | | Minimum MTU raised to 1280 bytes | Reduces the need for path MTU discovery | Removing the checksum is a direct application of the end-to-end principle: the check performed at the intermediate node was not necessary for correctness — it only added cost. #### The Example Network's IPv6 Plan The organization has received a `2001:db8:10::/48` allocation from its provider. This allocation contains 65,536 `/64` subnets — far more than enough for the organization's five networks. ```python import ipaddress allocation = ipaddress.ip_network("2001:db8:10::/48") plan = { "Guest wireless": "2001:db8:10:1::/64", "Lab": "2001:db8:10:2::/64", "Administration": "2001:db8:10:3::/64", "Server room": "2001:db8:10:4::/64", "Link": "2001:db8:10:ffff::/64", } print("allocation:", allocation, "— /64 subnets inside:", 2 ** (64 - allocation.prefixlen)) for name, cidr in plan.items(): net = ipaddress.ip_network(cidr) print(f"{name:18s} {str(net):24s} inside allocation={net.subnet_of(allocation)}") ``` ``` allocation: 2001:db8:10::/48 — /64 subnets inside: 65536 Guest wireless 2001:db8:10:1::/64 inside allocation=True Lab 2001:db8:10:2::/64 inside allocation=True Administration 2001:db8:10:3::/64 inside allocation=True Server room 2001:db8:10:4::/64 inside allocation=True Link 2001:db8:10:ffff::/64 inside allocation=True ``` The contrast between the two plans is instructive. In the IPv4 plan, every bit was counted, spare addresses were noted one by one, and growth headroom was a constraint. In the IPv6 plan, host count never entered the calculation; the only decision was which subnet number to assign to which section. The subnet numbers were chosen for readability — giving the point-to-point link `ffff` is a stylistic choice, starting from the end of the allocation. #### Interoperability of the Two Versions IPv4 and IPv6 **do not talk to each other directly**: different address widths, different headers, different neighbor discovery. There are three approaches to interoperability. **Dual stack:** Endpoints and routers run both protocols. The result of name resolution decides which version to use for a given destination. This is the simplest method; its cost is that everything is configured and monitored twice. **Tunneling:** An IPv6 packet is carried as the payload of an IPv4 packet. It is used when the network in between does not know IPv6. The tunnel header lowers the MTU; this is why the fragmentation and path MTU problems covered in the previous topic are common in tunnels. **Translation:** An intermediate node converts IPv6 addresses to IPv4 addresses. It breaks end-to-end transparency and inherits the address-translation problems covered in the next lesson. If an endpoint has both versions, which one it picks depends on a preference rule; the general tendency is to try IPv6 first and fall back to IPv4 on failure. This behavior can increase latency when there is a silent failure along the IPv6 path. #### Summary - An IPv6 address is 128 bits; the point of the width is less about raising the address count and more about easing hierarchical distribution and prefix summarization. - Subnets that carry hosts are `/64`; the 64-bit interface identifier is the precondition for autoconfiguration. - In notation, leading zeros are dropped, and the longest run of zeros is compressed once with `::`. - There is no broadcast address; its function is met by multicast and the solicited-node group. - Stateless autoconfiguration consists of generating a link-local address, duplicate address detection, router advertisement, and prefix combination. - The header checksum and router fragmentation have been removed; the two versions interoperate through dual stack, tunneling, or translation. #### Next Step This lesson used the phrase "the router drops the packet and reports it" several times. The reporting itself is also a protocol, and it does more than announce errors: reachability testing, path discovery, and MTU notification all run over it too. The next lesson covers these control messages, how the time-to-live field turns into traceroute, and why blocking these messages leads to silent failures. ### ICMP Source: https://academia.sh/en/courses/network-models/icmp The format of control messages, the echo test, how the time-to-live field turns into traceroute, path MTU discovery, and the consequences of blocking these messages. The previous lessons said "the router drops the packet and reports it" several times: when the time to live runs out, when no route to the destination network can be found, when the packet is larger than the MTU. How these reports are made has not yet been explained. IP itself gives no feedback; it is a best-effort delivery service. Feedback is a separate protocol's job. This lesson's question is what that protocol says, and what happens when it cannot. #### The Place of Control Messages **ICMP (Internet Control Message Protocol)** is the protocol that reports conditions arising while IP packets are carried. It is carried with value `1` in the IP header's protocol field — meaning it travels as IP's payload, but it is not a transport-layer protocol. Its position looks like an oddity: IP's control protocol is carried inside IP. The rationale is practical; an ICMP message also needs to be routed, and the mechanism that routes it is already IP. ICMP's limits must be known explicitly: - There is **no guarantee** that an ICMP message will arrive; it can be lost itself. - When an ICMP message is lost, **no second message is generated**; there is no error report for an error report. - ICMP messages do not **fix** errors; they only report them. The endpoint decides what to do. #### Message Format Every ICMP message begins with an 8-byte header: | Field | Size | Function | |---|---|---| | Type | 1 byte | The message's class | | Code | 1 byte | Detail within the class | | Checksum | 2 bytes | Over the entire message | | Variable field | 4 bytes | Gains meaning based on the type | In error-reporting messages, the body carries **the IP header of the packet that caused the error, plus the first bytes of its payload**. This detail matters: the endpoint can only match an incoming error message to its own connection because of this copy. Because the copy also contains the first bytes of the transport-layer header, the port numbers can be read too. In echo messages, the variable field is used as an identifier and a sequence number: ```python import struct def checksum(data: bytes) -> int: if len(data) % 2: data += b"\x00" total = 0 for i in range(0, len(data), 2): total += (data[i] << 8) + data[i + 1] while total >> 16: total = (total & 0xFFFF) + (total >> 16) return ~total & 0xFFFF def echo_request(ident: int, seq: int, payload: bytes) -> bytes: body = struct.pack("!BBHHH", 8, 0, 0, ident, seq) + payload value = checksum(body) return body[:2] + struct.pack("!H", value) + body[4:] message = echo_request(ident=0x1234, seq=1, payload=bytes(range(32))) print("message size:", len(message), "bytes") print("header :", message[:8].hex()) print("checksum :", hex(struct.unpack("!H", message[2:4])[0])) print("verify :", checksum(message)) icmp_type, code, _, ident, seq = struct.unpack("!BBHHH", message[:8]) print(f"type={icmp_type} code={code} id={hex(ident)} seq={seq} payload={len(message) - 8} bytes") ``` ``` message size: 40 bytes header : 0800f4c912340001 checksum : 0xf4c9 verify : 0 type=8 code=0 id=0x1234 seq=1 payload=32 bytes ``` The checksum method is the same one built for the IPv4 header in the Encapsulation and Headers lesson; the difference is that in ICMP it covers the payload as well as the header. Verification should again come out to zero. #### Message Types | Type | Code | Meaning | When it is generated | |---|---|---|---| | 0 | 0 | Echo reply | In response to an echo request | | 3 | 0 | Network unreachable | No route to the destination network | | 3 | 1 | Host unreachable | Target unresponsive on the local network | | 3 | 3 | Port unreachable | No process listening at the target | | 3 | 4 | Fragmentation needed, blocked | Packet larger than the MTU and DF set | | 8 | 0 | Echo request | Reachability test | | 11 | 0 | Time to live exceeded | TTL reached zero | | 11 | 1 | Reassembly time exceeded | Fragments not collected in time | | 12 | 0 | Header error | One of the fields is inconsistent | Type 3, code 4 carries a distinct significance: the message also carries the acceptable MTU value, and it is the basis for path MTU discovery. #### The Echo Test The **echo request** and **echo reply** pair is used to test an address's reachability. The sender puts in an identifier and an increasing sequence number; the reply returns the same values. This matches which reply belongs to which request and measures the round-trip time. What the test proves and what it does not prove must be kept separate. **If a reply arrives**, the following are true: a path exists in both directions, the target interface is active, and the IP layer is working. **If no reply arrives**, none of this is certain. The target might be down; the target might be up but not answering echo requests; a filter along the path might have dropped the request or the reply. No response does not mean the target is not working. Whether a port is serving cannot be determined with an echo test. Echo is a test at the IP level; it says nothing about the transport layer or the application layer. A server that answers echo requests may still have a stalled application. #### Time to Live and Traceroute The **time to live (TTL)** field in the IP header limits the number of routers a packet can pass through. Every router decrements the field by one; when it reaches zero, the router drops the packet and sends a type 11 message to the sender. The field's original purpose is preventing packets from circulating forever in routing loops. But the field also serves an entirely different purpose: **traceroute**. The method is this. A packet with TTL 1 is sent to the destination first; the first router drops it and reports itself. Then a packet with TTL 2 is sent; the second router reports. As the value is raised one by one, each step of the path is revealed in turn. When the destination is reached, the incoming message is not type 11 but either an echo reply or a port-unreachable report; this marks the end of the sequence. Since the topology of the path from the example network's client to `198.51.100.20` is known, which steps the trace will produce can be derived directly: | TTL | Responder | Role | Reply type | |---|---|---|---| | 1 | `192.168.10.193` | Organization router, administration interface | Type 11 | | 2 | `203.0.113.1` | Provider edge router | Type 11 | | 3 | `198.51.100.1` | Destination network's edge router | Type 11 | | 4 | `198.51.100.20` | Destination endpoint | Type 0 or type 3 code 3 | Three limits must be known when reading traceroute results. Not every step of the path is required to reply; a step that does not reply appears empty, and this does not mean it was not traversed. The outbound path is seen, the return path is not; the two directions can differ. The measured times depend on the intermediate router's priority for generating error messages and may not reflect the actual delay at that point. #### Path MTU Discovery The Frame, Packet, Segment lesson defined a way to avoid fragmentation: setting the don't-fragment flag and shrinking the segment size based on the incoming report. The report is the ICMP type 3 code 4 message. The loop works as follows. The sender generates packets sized to its own link's MTU. If a link along the path has a smaller MTU, that router drops the packet and reports the MTU value it can accept. The sender lowers the segment size according to that value and resends. The process continues until packets start getting through. The method's weak point is its dependence on the ICMP message itself. If the message is blocked along the path, the sender learns nothing: its packets keep getting dropped, but it never learns why. This condition is called a **black hole**, and its typical symptom is this — small requests work, large transfers stall. The echo test succeeds, the connection is established, the first few bytes go through, and then the stream stops. It is diagnosed by manually lowering the MTU: if the transfer works once the segment size is capped, the problem is in path MTU discovery. #### The Cost of Blocking ICMP ICMP messages are often blocked outright because they reveal information about the network. This produces several concrete failures: - **If type 3 code 4 is blocked**, path MTU discovery does not work; a black hole forms. - **If type 3 code 3 is blocked**, a connection attempt to a closed port times out instead of being rejected instantly; wait times grow longer. - **If type 11 is blocked**, traceroute does not work, and routing loops become invisible. - **If types 8 and 0 are blocked**, reachability testing cannot be performed; diagnosis proceeds with blind spots. The measured approach is to select by type rather than block outright: error reports are let through, and if the response to inbound echo requests needs restricting, it is limited rather than blocked entirely. Whatever the selection, type 3 code 4 must be let through; without it, transfers break silently. #### ICMPv6's Expanded Role In IPv6, ICMP's scope expands markedly. As noted in the ARP lesson, neighbor discovery is part of ICMPv6; router solicitation and advertisement, duplicate address detection, and multicast group management also run over this protocol. This has a direct consequence: **if ICMPv6 is blocked outright, IPv6 does not work.** Shutting off ICMP in IPv4 makes diagnosis harder; in IPv6 it halts basic functions. Addresses cannot be configured, neighbors cannot be resolved, routers cannot be found. #### Summary - ICMP is the control protocol carried as IP's payload; it does not fix errors, only reports them, and there is no guarantee of its own delivery. - Error messages carry the header of the packet that caused the error; the endpoint matches the message to its own connection through this copy. - The echo test shows reachability at the IP level; no response does not mean the target is down, and it does not prove a service exists. - The time-to-live field reveals the path's steps in order through packets sent with rising values; the outbound path is seen, the return path is not. - Path MTU discovery depends on the type 3 code 4 message; if the message is blocked, a black hole forms in which small requests work but large transfers stall. - ICMPv6 also takes on neighbor discovery and router advertisement; blocking it outright renders IPv6 unable to work. #### Next Step Up to this point, every address on the example network was assumed to be assigned by hand. That assumption cannot hold on a guest network with a hundred devices: entering address, mask, gateway, and name server information into every device one by one is not feasible. The next lesson covers the process of leasing addresses, the four steps of leasing, and how the subnet plan translates into that process. ### DHCP Source: https://academia.sh/en/courses/network-models/dhcp The four steps of dynamic address leasing, lease duration and renewal thresholds, deriving pools from the example plan, and the relay agent's role. The previous lessons assumed every address on the example network was assigned by hand. That assumption holds for the ten machines in the server room; it does not hold for the hundred devices on the guest wireless network. Entering address, mask, gateway, and name server information into every device one by one is not feasible. This lesson's question is where a device gets these four pieces of information the moment it joins the network. The answer starts with a chicken-and-egg problem: for the device to request the information, it needs an address, but it is requesting because it does not have one. #### Talking Without an Address The problem is solved at the link layer. The device uses the **unspecified address** `0.0.0.0` instead of its own address and writes the **limited broadcast address** `255.255.255.255` as the destination. At the frame level, the destination becomes the broadcast address `ff:ff:ff:ff:ff:ff`. This broadcast reaches every device in the broadcast domain; whichever one distributes configuration replies. **DHCP (Dynamic Host Configuration Protocol)** is the protocol that carries out this exchange. It runs over UDP; the client uses port 68, the server port 67. The transport-layer choice is forced: a protocol that requires connection setup cannot be used when there is no address yet. #### The Four Steps Leasing is completed with four messages. | Step | Message | Direction | Addressing | Information carried | |---|---|---|---|---| | 1 | Discover | Client → Broadcast | `0.0.0.0` → `255.255.255.255` | Client's hardware address, transaction ID | | 2 | Offer | Server → Client | Server address → Broadcast or unicast | Proposed address, mask, gateway, lease duration | | 3 | Request | Client → Broadcast | `0.0.0.0` → `255.255.255.255` | Accepted offer and the server's identifier | | 4 | Acknowledge | Server → Client | Server address → Broadcast or unicast | Lease finalization and options | Why the third step is sent **as a broadcast** is not obvious at first glance: the client already knows which server made the offer. The reason is that multiple servers may be present. When the request goes out as a broadcast, the servers whose offers were not accepted see it too and release the address they had reserved. Without the broadcast, those addresses would sit reserved uselessly for an entire lease period. In the second step, because the client does not yet have an address, the server's reply contains an oddity: the reply must be sent to an address that has not been assigned yet. Implementations resolve this two ways — either sending the reply as a broadcast, or building the frame directly to the client's hardware address. The client indicates which it prefers with a flag in the discover message. #### Options The real value of the lease is the configuration information that comes along with the address. These are carried in **option** fields: | Option | Information carried | Value on the example network (administration section) | |---|---|---| | Subnet mask | Network/host boundary | `255.255.255.224` | | Router | Default gateway | `192.168.10.193` | | Name server | Resolver addresses | `192.168.10.226` | | Domain name | Search suffix | The organization's internal domain | | Lease duration | Validity | In seconds | | Server identifier | Who issued the lease | The server's address | | Renewal thresholds | T1 and T2 | Fractions of the lease duration | The mask's presence in this list recalls the observation from the Subnetting lesson: an address alone is incomplete information. The lease delivers the address together with its mask. #### Lease Duration and Renewal An address is not granted permanently; it is **leased**. At the end of the period, the lease expires and the address returns to the pool. The reason it is time-limited is to prevent devices that connect once and leave from holding their addresses indefinitely. The client does not wait out the period; it attempts renewal at two thresholds: | Threshold | Timing | Behavior | |---|---|---| | T1 | 50% of the lease duration | Direct renewal request to the server that issued the lease | | T2 | 87.5% of the lease duration | Broadcast rebinding request to any server | | Expiration | 100% | Address is released, process starts over | For an eight-hour lease, T1 arrives at four hours, T2 at seven. If renewal succeeds, the period restarts from zero and the address does not change. This is why a device mostly keeps the same address even when the lease duration is short. Choosing the lease duration is a trade-off. A short duration returns released addresses to the pool quickly and prevents pool exhaustion on a busy guest network; in exchange, renewal traffic increases. A long duration reduces traffic; in exchange, released addresses stay blocked for a long time and configuration changes propagate late. #### The Example Network's Pools The plan finalized in the Variable-Length Subnetting lesson must be translated into a leasing configuration. In each subnet, addresses split into three: the first address reserved for the router, a range reserved for manual assignment, and the remainder as the dynamic pool. ```python import ipaddress # (section, block, addresses reserved for manual assignment, expected dynamic clients) SECTIONS = [ ("Guest wireless", "192.168.10.0/25", 9, 100), ("Lab", "192.168.10.128/26", 5, 45), ("Administration", "192.168.10.192/27", 9, 11), ("Server room", "192.168.10.224/28", 13, 0), ("Link", "192.168.10.240/30", 1, 0), ] for name, cidr, reserved, expected in SECTIONS: net = ipaddress.ip_network(cidr) addrs = list(net.hosts()) router = addrs[0] manual = addrs[1:1 + reserved] pool = addrs[1 + reserved:] manual_text = (f"{manual[0]}-{manual[-1]}" if manual else "none") pool_text = (f"{pool[0]}-{pool[-1]} ({len(pool)})" if pool else "none") print(f"{name:18s} {cidr:20s} router={str(router):15s} " f"manual={manual_text:31s} pool={pool_text:35s} " f"spare={len(pool) - expected if pool else 0}") ``` ``` Guest wireless 192.168.10.0/25 router=192.168.10.1 manual=192.168.10.2-192.168.10.10 pool=192.168.10.11-192.168.10.126 (116) spare=16 Lab 192.168.10.128/26 router=192.168.10.129 manual=192.168.10.130-192.168.10.134 pool=192.168.10.135-192.168.10.190 (56) spare=11 Administration 192.168.10.192/27 router=192.168.10.193 manual=192.168.10.194-192.168.10.202 pool=192.168.10.203-192.168.10.222 (20) spare=9 Server room 192.168.10.224/28 router=192.168.10.225 manual=192.168.10.226-192.168.10.238 pool=none spare=0 Link 192.168.10.240/30 router=192.168.10.241 manual=192.168.10.242-192.168.10.242 pool=none spare=0 ``` The server room and link networks have no dynamic pool; server addresses must not change, and both ends of the point-to-point link are routers. The pool margins are meaningful. The guest network has 116 addresses for an expected 100 devices; the 16-address margin covers more devices connecting at once than expected. The lease duration should be kept short on this network: the addresses of devices that come and go should return to the pool quickly. #### Reserved Addresses and Fixed Leasing The manual assignment range is the set of addresses the leasing server never touches. But there is a second way to give a device a fixed address: **reservation**. The server always hands out the same IP address to a particular hardware address. The client the course has been tracking is configured this way: | Hardware address | Assigned address | Method | |---|---|---| | `00:00:5e:00:53:01` | `192.168.10.196` | Reservation | The difference between the two methods is administrative. With manual assignment, the configuration lives on the device; when the mask or gateway changes, every device must be visited individually. With reservation, the configuration is centralized; the device still gets its address through leasing, but always the same one. When a network-wide option changes, it changes in one place. The hardware address that reservation relies on may not stay fixed on devices that rotate their address for privacy. In that case, the reservation does not hold, and the device gets an address from the pool. #### The Relay Agent The discover message goes out as a broadcast, and a broadcast does not cross a router. The example network has five subnets; placing a separate leasing server in each is an unnecessary administrative burden. The solution is the **relay agent**. The router captures the broadcast arriving on its interface, forwards it to the central server as a unicast packet, and adds a field to the message indicating which interface it came from. The server decides which pool to serve from by looking at that field. If `192.168.10.226` in the server room provides both name resolution and leasing service on the example network, relaying is configured on every internal interface of the router. A client in the administration section's discover message reaches the server carrying `192.168.10.193` in the interface field, and the server serves an address from the administration pool. #### Failure Patterns **Rogue leasing server.** If a device added to the broadcast domain sends out leasing offers, clients accept whichever offer arrives first. A wrong gateway or wrong name server can get distributed. The countermeasure operates at the switch level: leasing replies are only allowed to arrive from designated ports. **Pool exhaustion.** If every address in the pool is leased, new clients cannot get an address. The symptom is that some devices work and others do not. The measure is to monitor pool occupancy and adjust the lease duration to demand. **Falling back to a link-local address.** An interface that cannot get an address picks one for itself from the `169.254.0.0/16` block. A device carrying an address in this block is a direct sign that leasing failed: the server is unreachable, relaying is not configured, or the pool is exhausted. **Address conflict.** If a manually assigned address falls inside the pool range, that same address can also get leased out. This is why the manual assignment range is kept outside the pool when building the plan. #### Summary - The client talks without an address: source `0.0.0.0`, destination `255.255.255.255`; the protocol uses ports 67 and 68 over UDP. - Leasing is completed with four messages; the third step goes out as a broadcast so that servers whose offers were not accepted release their addresses. - The mask, gateway, name server, and lease duration are delivered along with the address. - Renewal is attempted at the 50% and 87.5% thresholds of the lease duration; the address does not change on a successful renewal. - In the example plan, the guest pool is 116 addresses, the lab 56, administration 20; the server room and link networks have no dynamic pool. - Because broadcasts do not cross a router, networks with many subnets use a relay agent; an address from the `169.254.0.0/16` block indicates that leasing failed. #### Next Step Every device on the example network now has an address; but all of these addresses come from the private-use block and are not routed on the public Internet. The organization has exactly one public address: `203.0.113.10`. The next lesson covers how hundreds of private addresses go outbound through a single public address, how the translation table is kept, and what this mechanism does to end-to-end connectivity. ### NAT and PAT Source: https://academia.sh/en/courses/network-models/nat-and-pat Types of address translation, the translation table's five-component key, capacity calculations, and the breakdown of end-to-end connectivity. Every device on the example network now has an address. But all of these addresses come from the `192.168.10.0/24` block, and that block is not routed on the public Internet: even if a packet with source address `192.168.10.196` reaches the provider's router, there is no path by which a return packet could be brought back. The organization has exactly one public address: `203.0.113.10`. This lesson's question is how 182 internal addresses can go outbound through a single public address. #### Three Kinds of Translation **Network address translation (NAT)** is changing the addresses in a packet's header while it passes through. Three forms are distinguished. | Type | Mapping | How many public addresses | Inbound connection | |---|---|---|---| | Static NAT | One-to-one, fixed | As many as internal hosts | Can be established | | Dynamic NAT | One-to-one, from a pool | As many as concurrent hosts | Cannot be established | | Port address translation (PAT) | Many-to-one | One address is enough | Only if configured | **Static NAT** permanently binds one internal address to one public address. It is used for a server that must be reachable from outside. **Dynamic NAT** establishes a temporary mapping from a pool of public addresses. When the pool is exhausted, new hosts cannot go outbound. Since the organization has only one public address, this type does not suit the example network. **Port address translation (PAT)** fits a large number of internal hosts into a single public address by changing the source port number along with the address. This is the method the example network uses. In common usage, when people say "NAT" they usually mean this type. #### The Translation Table The device performing translation must know which return packet belongs to which internal host. For this it keeps a **translation table**. The table's key has five components: source address, source port, destination address, destination port, and protocol. ```python EXTERNAL_ADDRESS = "203.0.113.10" class Pat: """A translation table that performs port translation.""" def __init__(self, external_address: str, initial_port: int = 61000) -> None: self.external_address = external_address self.next_port = initial_port self.forward: dict[tuple, tuple] = {} # internal key -> external key self.reverse: dict[tuple, tuple] = {} # external key -> internal key def outbound(self, internal_ip: str, internal_port: int, dest_ip: str, dest_port: int) -> tuple[str, int]: key = (internal_ip, internal_port, dest_ip, dest_port) if key not in self.forward: self.next_port += 1 external = (self.external_address, self.next_port) self.forward[key] = external self.reverse[(external[0], external[1], dest_ip, dest_port)] = (internal_ip, internal_port) return self.forward[key] def inbound(self, external_ip: str, external_port: int, source_ip: str, source_port: int) -> tuple[str, int] | None: return self.reverse.get((external_ip, external_port, source_ip, source_port)) pat = Pat(EXTERNAL_ADDRESS) flows = [ ("192.168.10.196", 49152, "198.51.100.20", 443), ("192.168.10.197", 49152, "198.51.100.20", 443), ("192.168.10.196", 49153, "198.51.100.20", 443), ("192.168.10.60", 51000, "198.51.100.20", 80), ] print(f"{'internal source':22s} {'external source':22s} {'destination'}") for internal_ip, internal_port, dest_ip, dest_port in flows: ext_ip, ext_port = pat.outbound(internal_ip, internal_port, dest_ip, dest_port) print(f"{internal_ip + ':' + str(internal_port):22s} {ext_ip + ':' + str(ext_port):22s} " f"{dest_ip + ':' + str(dest_port)}") print() print("return packet 203.0.113.10:61001 <- 198.51.100.20:443 ->", pat.inbound("203.0.113.10", 61001, "198.51.100.20", 443)) print("return packet 203.0.113.10:61003 <- 198.51.100.20:443 ->", pat.inbound("203.0.113.10", 61003, "198.51.100.20", 443)) print("unsolicited packet 203.0.113.10:61999 <- 198.51.100.20:443 ->", pat.inbound("203.0.113.10", 61999, "198.51.100.20", 443)) ``` ``` internal source external source destination 192.168.10.196:49152 203.0.113.10:61001 198.51.100.20:443 192.168.10.197:49152 203.0.113.10:61002 198.51.100.20:443 192.168.10.196:49153 203.0.113.10:61003 198.51.100.20:443 192.168.10.60:51000 203.0.113.10:61004 198.51.100.20:80 return packet 203.0.113.10:61001 <- 198.51.100.20:443 -> ('192.168.10.196', 49152) return packet 203.0.113.10:61003 <- 198.51.100.20:443 -> ('192.168.10.196', 49153) unsolicited packet 203.0.113.10:61999 <- 198.51.100.20:443 -> None ``` The second row shows why translation is necessary: two different internal hosts used the same source port number (49152). If the numbers were not changed, the return packets could not be told apart. The last line is translation's most visible side effect: a packet with no counterpart in the table is dropped. No connection arriving from outside that was not initiated from inside can get in. #### Fields That Change When a packet is translated, it is not only the address fields that change. Since the source address in the IP header and the source port in the transport header change, the checksums that cover them must be recomputed too. | Field | On the outbound packet | On the return packet | |---|---|---| | IP source address | Internal address → public address | Unchanged | | IP destination address | Unchanged | Public address → internal address | | Transport source port | Internal number → translated number | Unchanged | | Transport destination port | Unchanged | Translated number → internal number | | IP header checksum | Recomputed | Recomputed | | Transport checksum | Recomputed | Recomputed | The reason the transport checksum also changes is that TCP and UDP checksums are computed over a pseudo-header that includes the IP addresses. This detail makes concrete that address translation is a layer violation: a device operating at the network layer is forced to modify the transport-layer header. For the same reason, a translator that cannot read the transport header cannot work. A transport layer encrypted end to end makes translation impossible. #### Capacity Limit The number of simultaneous translations that can be done through a single public address is limited by the available port numbers. ```python available = 65535 - 1024 + 1 print("available ports:", available) for n in (182, 1000, 4000): print(f"{n:5d} clients -> concurrent flows per client: {available // n}") ``` ``` available ports: 64512 182 clients -> concurrent flows per client: 354 1000 clients -> concurrent flows per client: 64 4000 clients -> concurrent flows per client: 16 ``` For the example network's 182 hosts, that works out to 354 concurrent flows per client — a generous margin. But the number is inversely proportional to the host count and drops to 64 at a thousand hosts. Applications that open dozens of connections per page push against this limit. The calculation is pessimistic: because the translation key also includes the destination, flows going to different destinations can share the same translated number. Still, the limit is real, and this is why large networks use multiple public addresses. A second limit is memory: a table entry is kept for every flow. Entries are held for a period and time out — this produces one of translation's most visible side effects. #### The Breakdown of End-to-End Connectivity The end-to-end principle introduced in the TCP/IP Model lesson says that state should not be placed inside the network. Address translation directly violates this principle: the translation table is state kept in the middle of the network. Its consequences can be listed. **A connection cannot be established from outside.** A flow that was not initiated from inside has no counterpart in the table. For a device inside to be reachable from outside, a rule must be written by hand — **port forwarding**. If the internal server `192.168.10.226` on the example network is to be exposed, a rule is defined that forwards traffic arriving at `203.0.113.10:443` to it. **Idle flows drop.** When table entries are deleted by timeout, the return path is lost for a connection that has not carried data in a long time. Both ends think the connection is still open, but packets do not get through. Applications prevent this with regular **keep-alive** packets; the TCP lesson details this mechanism. **Peers cannot meet directly.** If both ends are behind translation, neither can connect to the other. The workaround is for both ends to connect to a shared external server, learn their own external address and port, and then send packets to each other simultaneously. This method only works if the translator's mapping behavior is predictable; when it does not work, traffic is carried through a relay server and latency increases. **An internal client may not be able to reach its own external address.** A packet from a device inside trying to connect to `203.0.113.10:443` arrives at the translator from its own internal interface and needs to turn back around. This behavior is called **hairpinning**, and not every implementation supports it. The symptom is typical: a service works from outside, but not from inside the same network. #### Translation Is Not a Security Mechanism Not being able to establish a connection from outside looks like security, but translation was not designed as a security mechanism and should not be treated as one. The access barrier is a side effect, not a policy. Every flow initiated from inside is opened outward; malicious software gets past the barrier by initiating the connection from inside. Translation also does not look at the content of the traffic passing through; it does not inspect what is carried inside a permitted flow. Access policy is established through explicitly written filtering rules. Address translation solves address scarcity; access control is the name of a separate job. #### Carrier-Grade Translation When address scarcity is felt at the provider level too, translation is applied one more layer: subscribers are given addresses from the `100.64.0.0/10` block instead of a public address, and outbound traffic goes through the provider's public addresses. This is called **carrier-grade NAT**. Under two layers of translation, port forwarding cannot be defined by the subscriber, peer-to-peer rendezvous fails more often, and address-based diagnosis loses its meaning because a large number of subscribers share one external address. IPv6 removes the need for any of these layers: once every endpoint can be given a public address, the translation table becomes unnecessary too. This is one of the rationales for the transition methods covered in the previous lesson. #### Summary - Private-use addresses are not routed on the public Internet; outbound access is provided through address translation. - Static translation is one-to-one and permanent, dynamic translation is temporary from a pool, and port address translation is many-to-one. - The translation table's key has five components; return packets are told apart even when different internal hosts use the same source port. - Because the address and port change, both the IP and transport checksums are recomputed; this is evidence that translation is a layer violation. - A single public address has 64,512 ports; that works out to 354 concurrent flows per host for 182 hosts and 64 for a thousand hosts. - Translation breaks end-to-end connectivity: connections cannot be established from outside, idle flows drop, and peers cannot meet directly. It is not a security mechanism. #### Next Step Throughout this topic, the term "port" was used without being defined: in the translation table's key, in the leasing protocol's numbers 67 and 68, in the destination server's 443. What these numbers are, and how an operating system decides which application an incoming segment goes to, is the transport layer's subject. The next lesson covers ports, the concept of a socket, and the four-tuple that makes a connection unique. ## Transport Layer Ports and sockets, TCP's reliability mechanisms, congestion control, UDP, and the delay-throughput calculation. ### Ports and Sockets Source: https://academia.sh/en/courses/network-models/ports-and-sockets Addressing application endpoints, port number ranges, the four-tuple that makes a connection unique, and the distinction between a listening socket and a connection socket. Throughout the previous topic, the term "port" was used without being defined: in the translation table's key, in the leasing protocol's numbers 67 and 68, in the destination server's 443. The place where the term is defined is the transport layer. The network layer delivers a packet to the correct **machine** and stops there; yet a single machine has dozens of programs using the network at the same time. This lesson's question is how it is decided which program an incoming segment is handed to. #### Multiplexing A single machine has several network flows at once: a browser talks to several servers, a mail client runs in the background, a backup tool sends data. All of them share the same IP address. The transport layer's first job is to separate these flows. Two fields are placed in the header for the separation: **source port** and **destination port**. Each is 16 bits; the value range is between $0$ and $65\,535$. Handing an incoming segment to the correct flow is called **demultiplexing**; loading outgoing segments onto a single network-layer flow is called **multiplexing**. This is the last link in the chain of multiplexing keys introduced in the Encapsulation and Headers lesson: the Ethernet type field leads to IP, the IP protocol field leads to TCP, the TCP port number leads to the application. #### Number Ranges Numbers are divided into three ranges: | Range | Name | Use | |---|---|---| | 0–1023 | Well-known | Standardized services; assignment requires authorization | | 1024–49151 | Registered | Numbers registered to specific applications | | 49152–65535 | Dynamic / ephemeral | The range clients use as their source number | Numbers in the first range cannot be bound without elevated privilege on most operating systems. This restriction is a security measure: it prevents an unprivileged program from seizing a standard service's number and impersonating it. Some well-known numbers: | Number | Protocol | Service | |---|---|---| | 22 | TCP | Secure shell | | 53 | UDP and TCP | Name resolution | | 67, 68 | UDP | Address leasing (server, client) | | 80 | TCP | Resource transfer | | 443 | TCP and UDP | Secure resource transfer | It should be noted that numbers are independent of protocol: TCP 443 and UDP 443 are **separate endpoints**. On the same machine, one can belong to one application and the other to a different one. There are two separate namespaces. Numbers are a convention, not a requirement. A service can also run on a different number; in that case the client needs to know the number explicitly. #### The Four-Tuple That Makes a Connection Unique A port number alone is not enough to identify a flow. Thousands of clients connect to a server's port 443 at the same time, and all of them use the same destination number. What makes a TCP connection unique is the **four-tuple**: $$ (\text{source address},\ \text{source port},\ \text{destination address},\ \text{destination port}) $$ When the protocol is also counted, it becomes a five-tuple; the translation table's key in the NAT and PAT lesson was this five-tuple. **Socket**, at one end, is the address-and-port pair; at the operating-system level it is the object representing that endpoint. Two sockets form a connection. The following program makes two connections to the same listening endpoint and prints the four-tuples from both sides. ```python import socket import threading import time def server(ready: threading.Event, state: dict) -> None: listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM) listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) listener.bind(("127.0.0.1", 9101)) listener.listen(5) state["listener"] = listener.getsockname() ready.set() for _ in range(2): connection, peer = listener.accept() state.setdefault("connections", []).append((connection.getsockname(), peer)) connection.close() listener.close() ready = threading.Event() state: dict = {} threading.Thread(target=server, args=(ready, state), daemon=True).start() ready.wait() print("listening endpoint:", state["listener"]) for i in range(2): client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) client.connect(("127.0.0.1", 9101)) print(f"client {i + 1}: local={client.getsockname()} remote={client.getpeername()}") client.close() time.sleep(0.2) for local, peer in state["connections"]: print(f"server side: local={local} peer={peer}") ``` Output: ``` listening endpoint: ('127.0.0.1', 9101) client 1: local=('127.0.0.1', 53044) remote=('127.0.0.1', 9101) client 2: local=('127.0.0.1', 53045) remote=('127.0.0.1', 9101) server side: local=('127.0.0.1', 9101) peer=('127.0.0.1', 53044) server side: local=('127.0.0.1', 9101) peer=('127.0.0.1', 53045) ``` Ephemeral port numbers (`53044`, `53045`) **change on every run**; the operating system picks them from the ephemeral range. What stays constant is that the two connections receive different numbers. The four-tuples: | Connection | Source | Destination | |---|---|---| | 1 | `127.0.0.1:53044` | `127.0.0.1:9101` | | 2 | `127.0.0.1:53045` | `127.0.0.1:9101` | Three components are shared; only the source port differs — that is what provides the distinction. #### Listening Socket and Connection Socket One detail in the output stands out: on the server side, the number `9101` appears in both the listening socket and the two connection sockets. This is not a conflict, because the two socket types are looked up by separate keys. | Socket type | Key | Role | |---|---|---| | Listening socket | (local address, local port) | Accepts new connection requests | | Connection socket | The full four-tuple | Carries data for an established connection | For an incoming segment, the operating system first looks for a full four-tuple match; if it does not find one and the segment is a new connection request, it routes the segment to the listening socket. This two-step lookup is what lets a single listening socket serve an unlimited number of connections. The same distinction explains why a server's concurrent capacity is not limited by the number of ports. The server's number is fixed at a single value; what varies is the client side's address and number. On the client side, however, the limit is real: the number of concurrent connections a single client machine can establish to a single destination is limited by the count of numbers in the ephemeral range. This is the same capacity limit computed in the NAT and PAT lesson. #### Endpoints in the Example Network The flow the course follows is written as a four-tuple as follows: | Location | Source | Destination | |---|---|---| | At the client | `192.168.10.196:49152` | `198.51.100.20:443` | | After translation | `203.0.113.10:61001` | `198.51.100.20:443` | | As seen at the server | `203.0.113.10:61001` | `198.51.100.20:443` | The server never sees the client's real address and number. The translation has changed two components of the four-tuple, and this change is invisible to both ends. A flow that stays within the internal network, however, is not subject to translation. A request from the client in the administration section to the server room's `192.168.10.226` address travels with the following four-tuple, unchanged along the way: ``` 192.168.10.196:49160 -> 192.168.10.226:443 ``` #### Binding Failures An application's attempt to bind to a number fails for two reasons. **Number in use.** Another listening socket is already bound to the same address-and-number pair. This error is common when a server program restarts; the reason is that the previous connections have not finished their closing process. The detail of this behavior will be covered in the TCP lesson. The `SO_REUSEADDR` option in the program above exists precisely to get past this situation. **Insufficient privilege.** An unprivileged process cannot bind to a number below 1024. The address given when binding also matters. A socket bound to `127.0.0.1` is reachable only from the same machine; a socket bound to `0.0.0.0`, which represents all interfaces, is also reachable from the network. A service running during local development being unreachable from the network is most often caused by this distinction. #### Summary - The transport layer separates flows on the same machine using 16-bit port numbers. - Numbers are divided into three ranges: well-known (0–1023), registered (1024–49151), ephemeral (49152–65535). - TCP and UDP numbers are separate namespaces; the same number means two separate endpoints across the two protocols. - What makes a connection unique is the four-tuple: source address, source number, destination address, destination number. - A listening socket is looked up only by the local pair, a connection socket by the full four-tuple; this is why a single listening socket can serve unlimited connections. - The address given when binding a socket determines the scope of access: `127.0.0.1` is local only, `0.0.0.0` is all interfaces. #### Next Step The four-tuple names a flow; it says nothing about whether the flow will be reliable. The network layer can lose a packet, reorder it, or deliver the same packet twice. The next lesson covers the protocol that solves these three problems at once — its handshake, sequence numbers, retransmission, and flow control. ### TCP Source: https://academia.sh/en/courses/network-models/tcp Three-way handshake, sequence and acknowledgement numbers, computing the retransmission timer, flow control, and the connection state machine. The previous lesson established the four-tuple that names a flow. Naming says nothing about whether the flow will be reliable: the network layer can lose a packet, reorder it, deliver the same packet twice, and feed the receiver faster than it can process. **TCP (Transmission Control Protocol)** takes on all four of these problems at once and offers the application a single abstraction: an ordered, loss-free, duplicate-free byte stream. This lesson's question is which mechanisms keep that promise. #### Promises Made | Promise | How it is kept | |---|---| | Delivery | Unacknowledged data is retransmitted | | Order | Every byte gets a sequence number, the receiver reorders | | No duplication | A second copy at the same sequence number is discarded | | Flow control | The receiver announces how many bytes it can accept | | Integrity | The checksum is computed together with the pseudo header | What is not promised must also be known. TCP does **not** preserve message boundaries: an application's three separate write calls can arrive on the other side merged into a single read. It gives no upper bound on delay; in the event of loss, retransmission noticeably increases delay. Nor does it guarantee that the other side has processed the data; an acknowledgement shows only that the data has reached the transport layer's buffer. #### Header The TCP header is 20 bytes without options. | Field | Size | Function | |---|---|---| | Source port | 2 bytes | Sending endpoint | | Destination port | 2 bytes | Receiving endpoint | | Sequence number | 4 bytes | The position of this segment's first byte in the stream | | Acknowledgement number | 4 bytes | The number of the next byte expected | | Data offset | 4 bits | Header length (in 32-bit words) | | Flags | 8 bits | SYN, ACK, FIN, RST, PSH, URG, and congestion bits | | Window | 2 bytes | The number of bytes the receiver can accept | | Checksum | 2 bytes | Computed over the header, payload, and pseudo header | | Urgent pointer | 2 bytes | Meaningful only together with the URG flag | | Options | 0–40 bytes | MSS, window scale, timestamp, selective acknowledgement | Computing the checksum over the **pseudo header** means the source and destination IP addresses are also factored in. The purpose is detecting a segment delivered to the wrong machine. A side effect of this, as seen in the NAT and PAT lesson, is that an address translator must recompute the checksum. #### Three-Way Handshake The connection is established before any data is sent. Setup takes three segments: | Step | Direction | Flags | Carries | |---|---|---|---| | 1 | Client → Server | SYN | The client's initial sequence number, MSS, and options | | 2 | Server → Client | SYN + ACK | The server's initial sequence number, acknowledgement of the client's | | 3 | Client → Server | ACK | Acknowledgement of the server's sequence number | The necessity of three steps is symmetric: **each direction is established separately.** TCP is bidirectional; each direction has its own sequence-number series. The client announces its own number and has it acknowledged, and the server does the same. Because one of the two symmetric exchanges (the server's acknowledgement together with its own announcement) is merged into a single segment, the total comes to three segments. The **initial sequence number** is not zero; it is chosen at random. The reason is twofold. Delayed segments from an old connection must not be mixed into a new connection established with the same four-tuple. The second is security: if the number were predictable, an outside party could inject segments into the flow that appear valid. The handshake carries a round-trip-time cost. At least one full round trip is waited out before any data is sent; in short exchanges this cost can make up the largest part of the total time. #### Sequence and Acknowledgement Numbers TCP assigns a sequence number not to the segment but to the **byte**. A segment's sequence number is the position, within the stream, of the first byte it carries. A flow with initial number 1000 and MSS 1460: | Segment | Sequence number | Range covered | |---|---|---| | 1 | 1000 | 1000 – 2459 | | 2 | 2460 | 2460 – 3919 | | 3 | 3920 | 3920 – 5379 | | 4 | 5380 | 5380 – 6839 | The acknowledgement number is the number of the next byte expected, and it is **cumulative**: an acknowledgement of 6840 means everything up to 6839 has been received. Cumulative acknowledgement has a consequence. If the second segment is lost and the third and fourth arrive, the receiver is forced to hold the acknowledgement number at 2460 — because there is a gap starting at 2460. It cannot report that it received the third and fourth segments. This shortcoming is fixed by the **selective acknowledgement** option: the receiver additionally lists the contiguous ranges it has received. The sender can then retransmit only the segment that is actually missing; without the option, it may be forced to retransmit everything after the gap. The SYN flag in the handshake and the FIN flag at closing each consume one sequence number. Even though they carry no data, they need to be acknowledged; consuming a sequence number is what makes that possible. #### Retransmission Every segment sent is kept in a buffer until it is acknowledged. If no acknowledgement arrives, the segment is retransmitted. How long to wait is decided by the **retransmission timeout (RTO)**. A fixed duration cannot be chosen: the round-trip time on the same connection varies along the way. If the duration is chosen too short, unnecessary retransmissions occur and load an already congested network further; if it is chosen too long, loss is noticed late and throughput drops. The solution is to produce a running estimate from measured round-trip times: $$ \text{SRTT} \leftarrow (1-\alpha)\,\text{SRTT} + \alpha R $$ $$ \text{RTTVAR} \leftarrow (1-\beta)\,\text{RTTVAR} + \beta \lvert \text{SRTT} - R \rvert $$ $$ \text{RTO} = \text{SRTT} + 4 \cdot \text{RTTVAR} $$ Here $R$ is the latest measurement, $\alpha = 1/8$, and $\beta = 1/4$. The second term accounts for the **variability** of the duration: on a stable path RTO approaches the measurement, on a fluctuating path it moves away from it. ```python ALPHA, BETA, K = 1 / 8, 1 / 4, 4 measurements = [100.0, 104.0, 96.0, 180.0, 102.0, 98.0] # ms srtt = measurements[0] rttvar = measurements[0] / 2 rto = srtt + K * rttvar print(f"{'measurement':>11s} {'SRTT':>8s} {'RTTVAR':>8s} {'RTO':>8s}") print(f"{measurements[0]:11.2f} {srtt:8.2f} {rttvar:8.2f} {rto:8.2f}") for r in measurements[1:]: rttvar = (1 - BETA) * rttvar + BETA * abs(srtt - r) srtt = (1 - ALPHA) * srtt + ALPHA * r rto = srtt + K * rttvar print(f"{r:11.2f} {srtt:8.2f} {rttvar:8.2f} {rto:8.2f}") ``` ``` measurement SRTT RTTVAR RTO 100.00 100.00 50.00 300.00 104.00 100.50 38.50 254.50 96.00 99.94 30.00 219.94 180.00 109.95 42.52 280.01 102.00 108.95 33.87 244.44 98.00 107.58 28.14 220.15 ``` The 180 ms jump in the fourth row alone has pushed the average to 110 ms, and pushed the variability from 30 to 42.5; RTO rises from 220 ms to 280 ms. With the following stable measurements, the value comes back down. When a timeout occurs, RTO **doubles** and the segment is retransmitted. On consecutive timeouts the duration grows exponentially. This behavior prevents retransmission traffic from making the problem worse when the network is not responding. There is a second mechanism for cases where waiting out the timeout is expensive. When the receiver gets a segment that is out of order, it repeats the same acknowledgement number; if the sender receives the same acknowledgement three times, it retransmits the segment without waiting for the timeout. This is called **fast retransmit** and works together with the congestion behavior in the next lesson. #### Flow Control If the sender transmits faster than the receiver can process, the receiver's buffer overflows and data is dropped. **Flow control** prevents this: the receiver announces, with every acknowledgement, how many bytes it can accept. This value is called the **receive window**. The amount of data the sender can transmit without acknowledgement cannot exceed the window. This has a direct throughput consequence: $$ \text{max throughput} = \frac{\text{window}}{\text{RTT}} $$ ```python for window, rtt in ((65535, 0.020), (65535, 0.100), (262144, 0.100)): print(f"window {window:7d} bytes, RTT {rtt*1000:5.0f} ms -> max {window*8/rtt/1e6:8.2f} Mb/s") ``` ``` window 65535 bytes, RTT 20 ms -> max 26.21 Mb/s window 65535 bytes, RTT 100 ms -> max 5.24 Mb/s window 262144 bytes, RTT 100 ms -> max 20.97 Mb/s ``` The window field is 16 bits; its largest value is 65,535 bytes. On a high-latency path this value caps throughput at 5 Mb/s — regardless of the connection's capacity. The problem is solved with the **window scale** option: a shift amount is announced during the handshake, and the window value is multiplied by $2^k$. The largest shift is 14, which takes the window to just under a gibibyte. When the receiver's buffer fills, a window of zero is announced and the sender stops. When the receiver frees up space, it sends a window update; if this update is lost, the connection locks up. To prevent this, the sender regularly sends one-byte **probe** segments. Flow control watches only the **receiver's** capacity. The path's capacity is a separate problem and is the subject of the next lesson. #### State Machine A TCP endpoint transitions among defined states. | State | Meaning | |---|---| | CLOSED | No connection | | LISTEN | The server is waiting for incoming requests | | SYN-SENT | The client has sent SYN and is waiting for a reply | | SYN-RECEIVED | The server received SYN and sent SYN+ACK | | ESTABLISHED | Data exchange can take place | | FIN-WAIT-1 / FIN-WAIT-2 | The wait of the side that initiated closing | | CLOSE-WAIT | The other side closed, the local application has not closed yet | | LAST-ACK | The final acknowledgement is awaited | | TIME-WAIT | Closing is complete, delayed segments are awaited | Closing, unlike setup, takes **four** segments, and the two directions are closed separately. When one side sends FIN, it announces only that the data it has to send is finished; the other side can continue sending data. This state is called **half-closed**. Connections that stay in `CLOSE-WAIT` for a long time are the signature of an application bug: the other side has closed, and the local application has forgotten to close the socket. The `TIME-WAIT` state is held for a period on the side that initiated closing. It has two reasons. If the final acknowledgement is lost, the other side repeats the FIN, and the state needs to remain in place so it can be answered. The second is preventing delayed segments belonging to this connection from mixing into a new connection established with the same four-tuple. A server getting an "address in use" error when it restarts stems from this state; the `SO_REUSEADDR` option from the previous lesson loosens this restriction. The **RST** flag follows a separate path: it terminates the connection instantly, without a handshake. This is the response given to a segment arriving at a closed port and to connections whose state has become corrupted. #### Summary - TCP offers an ordered, loss-free, duplicate-free byte stream; it does not preserve message boundaries and gives no upper bound on delay. - The handshake is three segments because the two directions are established separately; the initial sequence number is chosen at random to prevent old segments from mixing in and to prevent injection. - The sequence number is assigned to the byte; the acknowledgement is cumulative and cannot advance once a gap forms — selective acknowledgement fixes this shortcoming. - RTO is computed from a running average and its variability; it doubles on timeout, and three duplicate acknowledgements trigger fast retransmit. - The receive window limits throughput by $\text{window}/\text{RTT}$; at 65,535 bytes and 100 ms this limit is 5.24 Mb/s and is overcome with the window scale option. - Closing is four segments and the directions are closed separately; `TIME-WAIT` is held for a lost final acknowledgement and for delayed segments. #### Next Step Flow control protects the receiver's capacity; it says nothing about the path's capacity. If the receiver announces a one-megabyte window, the sender will fill it regardless of the network's condition. The next lesson covers, by calculation, how the sender estimates the network's capacity, how it reads loss as a signal, and the window's effect on throughput. ### Congestion Control Source: https://academia.sh/en/courses/network-models/congestion-control Congestion window, slow start and additive increase-multiplicative decrease, reading loss as a signal, and the relationship between loss rate and throughput. The previous lesson's flow control protects the receiver's capacity. If the receiver announces a one-megabyte window, the sender earns the right to send that amount without acknowledgement — without accounting for what happens along the way. The path may not be able to carry that much data. Router queues fill up, packets are dropped, dropped packets are retransmitted, and retransmissions fill the queues even further. This lesson's question is how the sender can estimate a capacity it cannot see. #### Congestion **Congestion** is traffic arriving at a link exceeding that link's capacity. The router queues the excess packets; once the queue fills, it starts dropping them. Left uncontrolled, a feedback loop forms. Dropped packets are retransmitted, retransmissions increase the load, and as the load increases, more packets are dropped. As a result, the amount of **useful** data crossing the link falls even as load rises. This is called **congestion collapse**. The difficulty of the problem is lack of information. The sender knows neither the path's capacity nor who else shares that path. The only information it has is what happens to the segments it sends itself. For this reason, congestion control is a **measure-and-adapt** loop: the sender increases its rate, observes the network's response, and adjusts based on the response. #### Congestion Window In addition to the receive window, the sender keeps a second limit: the **congestion window (cwnd)**. This value does not come from the other side; it is the sender's own estimate. The amount of data that can be sent without acknowledgement is the smaller of the two: $$ \text{effective window} = \min(\text{cwnd},\ \text{rwnd}) $$ The distinction matters. The receive window protects the **receiver**, the congestion window protects the **network**. If a transfer is slow, the question to ask is which one is limiting it: if the receiver's buffer is small, the receive window is the bottleneck; if the path is congested, the congestion window is. #### Slow Start When a connection is established, there is no information about the path's capacity. The initial value is chosen small and is increased with **every acknowledgement**. Since a window's worth of segments gets acknowledged over one round, the window doubles every round. ```python MSS = 1460 cwnd = 1 for rnd in range(1, 12): print(f"round {rnd:2d}: cwnd={cwnd:5d} MSS = {cwnd * MSS / 1024:8.1f} KiB") cwnd *= 2 ``` ``` round 1: cwnd= 1 MSS = 1.4 KiB round 2: cwnd= 2 MSS = 2.9 KiB round 3: cwnd= 4 MSS = 5.7 KiB round 4: cwnd= 8 MSS = 11.4 KiB round 5: cwnd= 16 MSS = 22.8 KiB round 6: cwnd= 32 MSS = 45.6 KiB round 7: cwnd= 64 MSS = 91.2 KiB round 8: cwnd= 128 MSS = 182.5 KiB round 9: cwnd= 256 MSS = 365.0 KiB round 10: cwnd= 512 MSS = 730.0 KiB round 11: cwnd= 1024 MSS = 1460.0 KiB ``` The phase's name is misleading: the growth is **exponential** and fast. The word "slow" describes starting from a small value rather than jumping directly to capacity. Exponential growth continues up to a threshold. The phase changes when the **slow start threshold** is exceeded or a loss is detected. #### Additive Increase, Multiplicative Decrease The phase after the threshold is **congestion avoidance**, and the growth is linear: one MSS is added to the window every round. When loss is detected, the window is reduced **multiplicatively** — the common behavior is halving it. The two rules together are called **additive increase, multiplicative decrease (AIMD)**. The asymmetry is deliberate. Because the increase is slow, capacity overshoot stays bounded; because the decrease is fast, congestion eases quickly. Flows sharing the same path balance their shares under this rule: a flow taking more than its share is affected more by loss and its share drops. The cost of this asymmetry shows up on high-capacity, high-latency paths. ```python import math MSS = 1460 bdp = 1000e6 * 0.100 / 8 target = bdp / MSS print(f"bandwidth-delay product: {bdp/1e6:.1f} MB = {target:.0f} MSS") rounds_exp = math.ceil(math.log2(target)) print(f"with exponential growth: {rounds_exp} rounds = {rounds_exp * 100} ms") rounds_lin = round(target / 2) print(f"with linear growth (from half): {rounds_lin} rounds = {rounds_lin * 100 / 1000:.1f} s") ``` ``` bandwidth-delay product: 12.5 MB = 8562 MSS with exponential growth: 14 rounds = 1400 ms with linear growth (from half): 4281 rounds = 428.1 s ``` On a path with 1 Gb/s capacity and 100 ms latency, the window reaching capacity in the exponential phase takes 1.4 seconds when it starts from a single MSS, as the calculation above assumes; the ten-MSS initial window common today shortens this by four rounds, to one second. If a single loss halves the window, climbing back with linear growth takes more than seven minutes. On paths at this scale, classic AIMD behavior cannot use the capacity; this is the rationale for the approaches covered in the next section. #### Reading Loss as a Signal Classic congestion control treats packet loss as an indicator of congestion. The assumption is this: on a wired path, bit errors are rare, so the cause of loss is most likely a full queue. There are cases where the assumption does not hold. On wireless links, packet loss can stem from interference or a weak signal; in that case halving the window does not solve the problem, it only lowers throughput. Likewise, reordering caused by a temporary path change produces duplicate acknowledgements and can trigger an unnecessary reduction. There is a way to signal congestion without waiting for loss to occur: **explicit congestion notification**. A router whose queue starts filling marks two bits in the IP header instead of dropping the packet; the receiver reports this back in its acknowledgement, and the sender lowers its rate without losing a packet. For the method to work, the nodes along the path and both ends must support it. A third approach is to read the **increase in delay** as a signal without waiting for loss at all: round-trip time rises as queues start filling. Delay-based methods keep queues empty and keep delay low; but when they share a path with loss-based flows, they lose share, because the others do not back off until the queue fills. #### Fast Retransmit and Fast Recovery Loss is detected in two ways, and the response differs. | Detection path | Meaning | Response | |---|---|---| | Timeout | No word from the network at all | Window drops to its smallest value, returns to slow start | | Three duplicate acknowledgements | Later segments are arriving | Window halves, congestion avoidance continues | The reason for the distinction is the amount of information available. If duplicate acknowledgements are arriving, the path has not broken entirely: later segments are reaching the receiver. In that case starting from zero is unnecessary; the window is halved and the flow continues. This is called **fast recovery**. On a timeout, however, no acknowledgement has arrived at all; the state of the path is unknown and it must be measured from scratch. #### Loss Rate and Throughput The long-term throughput of a loss-based flow depends on the loss rate and the round-trip time. Averaging the sawtooth-shaped window behavior yields the following approximation: $$ \text{throughput} \approx \frac{\text{MSS}}{\text{RTT}} \cdot \sqrt{\frac{3}{2p}} $$ ```python import math MSS = 1460 for p in (1e-2, 1e-3, 1e-4, 1e-5, 1e-6): v = MSS * math.sqrt(1.5) / (0.100 * math.sqrt(p)) print(f"p={p:.0e} -> {v * 8 / 1e6:8.2f} Mb/s") ``` ``` p=1e-02 -> 1.43 Mb/s p=1e-03 -> 4.52 Mb/s p=1e-04 -> 14.31 Mb/s p=1e-05 -> 45.24 Mb/s p=1e-06 -> 143.05 Mb/s ``` Two results can be read off. First, throughput is inversely proportional to the **square root** of the loss rate: reducing loss a hundredfold increases throughput only tenfold. Second, throughput is inversely proportional to round-trip time — a distant server yields lower throughput at the same loss rate. This relationship also explains why, of two flows sharing the same path, the one with lower latency takes a larger share. A short round-trip time gives more frequent opportunities to grow its window. The formula is an approximation and has limits: it is valid only for loss-based behavior, in long-lived and stable flows. Short transfers most often do not even finish the slow start phase. #### Bufferbloat Keeping router buffers large reduces packet loss but increases queuing delay. If the control algorithm is waiting for loss, it keeps increasing its rate until the buffer fills; the result is reduced loss and increased delay. This is called **bufferbloat**. The symptom is familiar: while a large transfer is underway, the response time of interactive use on the same link noticeably lengthens. The cause is not insufficient capacity but queued packets waiting behind a full queue. The countermeasure is not shrinking the buffer but managing the queue **actively**: dropping or marking packets early once queue wait exceeds a threshold. This way the sender gets the signal before the queue is completely full. #### Fairness How flows sharing the same bottleneck divide up capacity is a separate metric. AIMD converges toward balance among flows with equal round-trip times. If the times differ, the balance breaks down; the flow with the shorter time gets a larger share. An application opening many concurrent connections also increases its share: each connection runs its own AIMD cycle. This is the simplest way an application can get around congestion control, and it is one of the design rationales for application protocols that multiplex over a single connection. #### Summary - Congestion is traffic arriving at a link exceeding its capacity; left uncontrolled, retransmissions raise the load and lead to collapse. - The sender keeps two windows; the amount that can be sent without acknowledgement is the smaller of the congestion and receive windows. - In slow start the window doubles every round; after the threshold it grows by one MSS per round and shrinks multiplicatively on loss. - On a 1 Gb/s, 100 ms path, capacity is reached in 1.4 seconds in the exponential phase — starting from a single MSS; climbing back from half in the linear phase takes 428 seconds. - Loss-based throughput is approximated by $\text{MSS}/\text{RTT} \cdot \sqrt{3/2p}$; throughput is inversely proportional to the square root of the loss rate and to round-trip time. - A timeout returns the flow to slow start; three duplicate acknowledgements only halve the window; large buffers reduce loss while increasing delay. #### Next Step The guarantees TCP offers are not free: the handshake costs one round, retransmission on loss costs extra rounds, and ordered delivery brings waiting. Not every application needs to pay this cost. The next lesson covers the transport protocol that offers no guarantees at all, and which applications this simplicity is the right choice for. ### UDP Source: https://academia.sh/en/courses/network-models/udp Eight-byte header, message boundary preservation, applications suited to unreliable delivery, and building reliability at the application layer. The previous two lessons established TCP's guarantees and their cost: the handshake costs one round, retransmission on loss costs extra rounds, and ordered delivery brings waiting. This cost is not necessary for every application; for some it is directly harmful. This lesson's question is what a transport protocol that makes no guarantees at all is good for, and in which requirements it is the right choice. #### Eight-Byte Header The **UDP (User Datagram Protocol)** header is 8 bytes: | Field | Size | Function | |---|---|---| | Source port | 2 bytes | Sending endpoint; zero if unused | | Destination port | 2 bytes | Receiving endpoint | | Length | 2 bytes | Datagram size including the header | | Checksum | 2 bytes | Computed over the header, payload, and pseudo header | Most of the fields in TCP's 20-byte header are missing here: sequence number, acknowledgement number, window, flags. The reason for their absence is that the corresponding mechanisms are absent too. The checksum field is optional in IPv4; if left at zero, no check is performed. It is mandatory in IPv6, because the header checksum has been removed at the network layer, leaving the only check at the transport layer. #### Promises Not Made | TCP's promise | UDP | |---|---| | Delivery guarantee | None — a lost datagram is not resent | | Order | None — datagrams can arrive in a different order than they were sent | | No duplication | None — the same datagram can be delivered twice | | Flow control | None — the receiver's capacity is not taken into account | | Congestion control | None — the network's condition is not taken into account | | Connection state | None — there is no handshake or closing | This list should not be read as a list of shortcomings. Each row is also the removal of a cost: no state is kept, no round is waited out, and delayed data does not hold up the order. #### Message Boundary The only structural promise UDP makes is **preserving the message boundary**. Every datagram sent is delivered to the receiver in a single read operation, at the same size. TCP makes no such promise: the stream is an uninterrupted sequence of bytes, and the boundaries of the send calls are lost. The following program sends the same three messages with both protocols and prints what is received. ```python import socket import threading import time MESSAGES = [b"ALFA", b"BRAVO", b"CHARLIE"] def tcp_server(received: list, ready: threading.Event) -> None: listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM) listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) listener.bind(("127.0.0.1", 9201)) listener.listen(1) ready.set() connection, _ = listener.accept() time.sleep(0.3) # all three sends pile up in the buffer while True: chunk = connection.recv(4096) if not chunk: break received.append(chunk) connection.close() listener.close() def udp_server(received: list, ready: threading.Event) -> None: sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.bind(("127.0.0.1", 9202)) ready.set() for _ in range(len(MESSAGES)): data, _ = sock.recvfrom(4096) received.append(data) sock.close() tcp_received: list = [] ready = threading.Event() threading.Thread(target=tcp_server, args=(tcp_received, ready), daemon=True).start() ready.wait() client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) client.connect(("127.0.0.1", 9201)) for m in MESSAGES: client.send(m) time.sleep(0.5) client.close() time.sleep(0.3) print("TCP sent :", MESSAGES) print("TCP received:", tcp_received) udp_received: list = [] ready2 = threading.Event() threading.Thread(target=udp_server, args=(udp_received, ready2), daemon=True).start() ready2.wait() udp_client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) for m in MESSAGES: udp_client.sendto(m, ("127.0.0.1", 9202)) time.sleep(0.5) udp_client.close() print("UDP sent :", MESSAGES) print("UDP received:", udp_received) ``` Output: ``` TCP sent : [b'ALFA', b'BRAVO', b'CHARLIE'] TCP received: [b'ALFABRAVOCHARLIE'] UDP sent : [b'ALFA', b'BRAVO', b'CHARLIE'] UDP received: [b'ALFA', b'BRAVO', b'CHARLIE'] ``` On the TCP side, the three sends have merged into a single read. This merging is **not guaranteed**; depending on timing, it could just as well have arrived in two or three separate reads. What is guaranteed is that boundaries are not preserved: an application carrying messages over TCP must encode the message boundary itself — using a fixed length, a length prefix, or a delimiter. On the UDP side, the three datagrams arrived in three separate reads, and this is **guaranteed**. If the receiver's read buffer is smaller than the datagram, the datagram is truncated; the remaining part is lost. #### When to Choose The selection criterion is what the application **does with delayed data**. | Application class | Choice | Rationale | |---|---|---| | File transfer | TCP | Every byte is required; order and integrity are mandatory | | Name resolution | UDP | One request, one response; the handshake round would be most of the cost | | Address leasing | UDP | A connection cannot be established without an address | | Real-time audio and video | UDP | A delayed frame is worthless; retransmission does harm | | Interactive game state | UDP | Stale position data is invalidated by newer data | | Continuous measurement reporting | UDP | The loss of a single sample is negligible | | Broadcast and multicast | UDP | TCP does not support multiple recipients | The real-time streaming row shows the essence of the choice. If an audio frame arrives 200 ms late, the moment it was meant to be played has already passed; requesting it again only burdens the network further. Putting silence or an estimate in place of a lost frame is better than waiting for the delayed one. #### Head-of-Line Blocking TCP's ordered-delivery promise carries a second cost. The receiver cannot hand later data to the application until it fills a gap in the stream — even if it already has that data. When a segment is lost, the segments that arrive after it are held in the buffer. This is called **head-of-line blocking**. On a connection carrying a single stream, this is the correct behavior. But if a single TCP connection carries multiple independent logical streams, loss in one stream holds up the others too — even though there is no dependency between them. UDP has no such blocking, because it makes no ordering promise either. This is one of the reasons protocols that multiplex independent streams are built on top of UDP: order and reliability are managed per stream, and streams do not hold each other up. #### Building Reliability at the Application Layer Choosing UDP does not mean giving up on reliability; it means building **only as much of it as needed**. The pieces commonly built at the application layer are: - **Retry and timeout.** The typical pattern in name resolution: if no response arrives, the same query is repeated, and after a few attempts another server is tried. - **Sequence number.** Numbering datagrams makes loss and reordering detectable; the application decides which gap to make up for. - **Selective retransmission.** Only data that is still valuable is requested again. - **Rate control.** Since there is no congestion control, an application sending continuous data must impose its own rate limit. The last item must not be neglected. An uncontrolled UDP flow forces the TCP flows sharing the same path to back off and unfairly grows its own share; it also creates a congestion risk for the network as a whole. If all of these pieces are built, it looks like rewriting TCP. The difference is **which pieces** get built: the application knows which part of its own data becomes worthless when delayed; TCP cannot know that. #### Size and Fragmentation UDP does not split the body. If the application sends a 3000-byte datagram, the splitting work falls to the IP layer, and the fragmentation cost from the Frame, Packet, Segment lesson arises: if one of the fragments is lost, the entire datagram is lost. For this reason, protocols that use UDP keep the datagram size under the path MTU. For Ethernet and IPv4 the limit is $1500 - 20 - 8 = 1472$ bytes; choosing a safer threshold against the possibility of a tunnel being present is common practice. #### Source Address Spoofing Since UDP has no handshake, the sender's address is never verified at any stage. A datagram whose source address is made to appear as belonging to someone else causes the response to be sent to that address. The scale problem this creates appears in protocols where the response is larger than the request: a small request directs a large response at the target, and the traffic is reflected with amplification. The countermeasures lie outside the protocol: - **Source address validation.** The network operator checks that the source addresses of packets leaving its network belong to that network. A packet with a spoofed source cannot leave the network. - **Response rate limiting.** The number of responses given to the same source is limited. - **Query validation.** Before giving a large response, the server tests with a small exchange whether the request really came from that address. The same spoofing is harder in TCP: seeing the sequence number in the second step of the handshake is required, and the number is chosen at random. This is one of the reasons for the randomness requirement noted in the TCP lesson. #### Summary - The UDP header is 8 bytes; it has no sequence, acknowledgement, window, or flag fields, because the corresponding mechanisms are absent too. - Delivery, order, no-duplication, flow-control, and congestion-control guarantees are not made; in exchange, no state is kept and no round is waited out. - UDP preserves the message boundary; TCP does not — an application carrying messages over TCP must encode the boundary itself. - The selection criterion is whether delayed data is still valuable to the application. - Head-of-line blocking causes independent streams to hold each other up on a single TCP connection; UDP has no such blocking. - Retry, sequence numbering, and rate control can be built at the application layer; neglecting rate control disrupts flow sharing. #### Next Step The throughput of both protocols runs into the same physical realities: putting a bit on the wire takes time, light travels at a finite speed, queues hold packets up. The next lesson separates these components to show, by calculation, where delay comes from, why the product of bandwidth and delay is a measure of capacity, and why what determines a transfer's duration is most often not bandwidth. ### Delay and Throughput Source: https://academia.sh/en/courses/network-models/delay-and-throughput The four components of delay, the bandwidth-delay product, window sizing, and breaking down a transfer's duration into its components. The previous two lessons showed how transport protocols limit throughput: window, loss rate, congestion behavior. Beneath these limits lies a more fundamental layer — putting a bit on the wire takes time, and the signal travels at a finite speed. This lesson's question is what the sentence "my connection is 100 Mb/s" says, and does not say, about a transfer's duration. The answer, in most cases, differs from what is expected: what determines the duration is most often not bandwidth. #### The Four Components of Delay A packet crossing a link consists of four components. **Transmission delay** is the time it takes to put all of a packet's bits on the wire. It is the ratio of packet size to link rate: $$ d_{\text{transmission}} = \frac{L}{R} $$ ```python for rate in (10e6, 100e6, 1e9, 10e9): print(f"{rate/1e6:8.0f} Mb/s -> {1500*8/rate*1e6:8.2f} us") ``` ``` 10 Mb/s -> 1200.00 us 100 Mb/s -> 120.00 us 1000 Mb/s -> 12.00 us 10000 Mb/s -> 1.20 us ``` **Propagation delay** is the time it takes for the signal to cross the distance. It is the ratio of distance to the medium's propagation speed, and it is **independent** of packet size. Inside fiber, the speed is approximately $2 \times 10^8$ m/s. ```python for km in (1, 100, 1000, 10000): print(f"{km:6d} km -> {km*1000/2e8*1000:8.3f} ms") ``` ``` 1 km -> 0.005 ms 100 km -> 0.500 ms 1000 km -> 5.000 ms 10000 km -> 50.000 ms ``` **Queuing delay** is the packet waiting its turn at the router. It is the only variable component: it grows as load increases and rises sharply as capacity is approached. The effect of bufferbloat on delay shows up in this component. **Processing delay** is the time it takes to read the header and make the routing decision; it stays small next to the others. The scale difference between two components is decisive. On a 1 Gb/s link, transmitting a 1500-byte packet takes 12 microseconds, while the same packet traveling a thousand kilometers takes 5 milliseconds — four hundred times as long. **Over long distances, what determines the duration is not bandwidth but the speed of light.** Increasing bandwidth shortens transmission delay; it cannot touch propagation delay. #### Round-Trip Time **Round-trip time (RTT)** is the total time it takes for a packet to reach its destination and for the response to return. It is the sum of all components in both directions. The measured value cannot fall below the geographic distance between the two ends. A thousand-kilometer path requires at least 10 ms of propagation delay round trip; intermediate routers and queues add to that. A measured value falling below this lower bound shows that the destination is not where it is assumed to be. RTT is the unit of measure for every mechanism in the transport layer: the handshake takes one RTT, a window update takes one RTT, and every increase of the congestion window takes one RTT. #### Bandwidth-Delay Product The **bandwidth-delay product (BDP)** is the amount of data that can be "in flight" on the path at the same time: $$ \text{BDP} = R \times \text{RTT} $$ The measure comes from thinking of the path as a pipe: bandwidth is the pipe's cross-section, round-trip time is its length; the product is the pipe's volume. ```python for mbps, rtt in ((100, 0.040), (100, 0.100), (1000, 0.040), (1000, 0.100)): bdp = mbps*1e6*rtt/8 print(f"{mbps:5d} Mb/s x {rtt*1000:5.0f} ms = {bdp:12.0f} bytes = " f"{bdp/1024:9.2f} KiB = {bdp/1460:8.1f} MSS") ``` ``` 100 Mb/s x 40 ms = 500000 bytes = 488.28 KiB = 342.5 MSS 100 Mb/s x 100 ms = 1250000 bytes = 1220.70 KiB = 856.2 MSS 1000 Mb/s x 40 ms = 5000000 bytes = 4882.81 KiB = 3424.7 MSS 1000 Mb/s x 100 ms = 12500000 bytes = 12207.03 KiB = 8561.6 MSS ``` BDP has a direct consequence: **if the window is smaller than the BDP, the connection sits idle.** The sender fills the window, waits for acknowledgement, and sends nothing during that time. Throughput is limited by the relationship established in the TCP lesson. The required window size and window scale follow directly: | Target throughput | RTT | Required window | Required scale | |---|---|---|---| | 100 Mb/s | 40 ms | 488.3 KiB | 3 | | 500 Mb/s | 100 ms | 6103.5 KiB | 7 | The largest window without scaling is 65,535 bytes; both rows exceed it. On high-capacity paths, the window scale option is not an optimization — it is a necessity. #### Components of a Transfer Suppose the client on the example network fetches a 2 MiB resource from `198.51.100.20`. The external link is 100 Mb/s, the round-trip time to the server is 40 ms, MSS is 1460 bytes, and the initial congestion window is 10 MSS. ```python MSS = 1460 SIZE = 2 * 1024 * 1024 RATE = 100e6 RTT = 0.040 bdp_mss = RATE * RTT / 8 / MSS cwnd = 10 sent = 0 round_ = 0 while sent < SIZE and cwnd < bdp_mss: round_ += 1 this_round = cwnd * MSS sent += this_round print(f"round {round_}: cwnd={cwnd:4d} MSS, {this_round:7d} bytes this round, " f"total {sent:8d} bytes") cwnd *= 2 remaining = SIZE - sent total = RTT + round_ * RTT + remaining * 8 / RATE print(f"handshake {RTT*1000:.0f} ms + slow start {round_*RTT*1000:.0f} ms + " f"remaining {remaining*8/RATE*1000:.1f} ms = {total*1000:.0f} ms") print(f"pure transmission time would be: {SIZE*8/RATE*1000:.0f} ms") print(f"achieved average throughput: {SIZE*8/total/1e6:.1f} Mb/s") ``` ``` round 1: cwnd= 10 MSS, 14600 bytes this round, total 14600 bytes round 2: cwnd= 20 MSS, 29200 bytes this round, total 43800 bytes round 3: cwnd= 40 MSS, 58400 bytes this round, total 102200 bytes round 4: cwnd= 80 MSS, 116800 bytes this round, total 219000 bytes round 5: cwnd= 160 MSS, 233600 bytes this round, total 452600 bytes round 6: cwnd= 320 MSS, 467200 bytes this round, total 919800 bytes handshake 40 ms + slow start 240 ms + remaining 94.2 ms = 374 ms pure transmission time would be: 168 ms achieved average throughput: 44.8 Mb/s ``` The result is worth reading closely. Pure transmission would take 168 ms; the actual duration is 374 ms. The entire difference comes from delay: one round for the handshake, six rounds of window growth. Average throughput ends up below half the connection's capacity. This is a model calculation; loss, name resolution, and the secure transport handshake are not accounted for. Each of them adds at least one more round. What the model shows is not the absolute duration but **the proportion between the components**. #### What Determines Small Transfers As a transfer gets smaller, delay's share grows. ```python RATE = 100e6 # same connection as the previous example RTT = 0.040 small = 20 * 1024 print(f"pure transmission: {small*8/RATE*1000:.2f} ms; " f"handshake+first response: {2*RTT*1000:.0f} ms") print(f"delay's share: {2*RTT/(2*RTT + small*8/RATE)*100:.1f} %") ``` ``` pure transmission: 1.64 ms; handshake+first response: 80 ms delay's share: 98.0 % ``` For a 20 KiB resource, 98% of the duration passes in waiting. Doubling bandwidth brings 1.64 ms down to 0.82 ms and makes no noticeable difference to the total; halving round-trip time, on the other hand, cuts the total duration almost in half. This asymmetry is the rationale behind many design decisions: keeping content geographically close to the user, reducing the number of rounds, reusing connections. All of them target delay, not bandwidth. #### Throughput and Bandwidth Are Not the Same Thing Three terms need to be kept separate. | Term | Definition | Measured? | |---|---|---| | Bandwidth | The connection's theoretical upper bound | Usually stated | | Throughput | The rate actually achieved on a given flow | Measured | | Goodput | The rate of useful data reaching the application | Header overhead and retransmission subtracted | The difference among the three draws on every mechanism built throughout this course: header overhead, retransmission, window limits, congestion behavior, fragmentation. When a transfer is said to be slow, the questions to ask, in order, are these. Is the bottleneck the receive window or the congestion window? What is the loss rate? Is the round-trip time above what is expected? Is path MTU discovery working? The answer to every question was established in one of this course's lessons. #### Summary - Delay consists of four components: transmission, propagation, queuing, and processing. Only transmission delay depends on bandwidth. - Over long distances, propagation determines the duration: at 1 Gb/s, transmitting a 1500-byte packet takes 12 µs, while its propagation over a thousand kilometers takes 5 ms. - The bandwidth-delay product is the amount of data that can be in flight on the path at the same time; if the window is smaller than this, the connection sits idle. - For 100 Mb/s and 40 ms, the BDP is 488 KiB; since the largest window without scaling is 64 KiB, the window scale option is mandatory. - For a 2 MiB transfer, pure transmission takes 168 ms; with the handshake and slow start included, it takes 374 ms; average throughput ends up below half of capacity. - In small transfers, almost the entire duration is waiting; increasing bandwidth makes no difference, shortening round-trip time does. #### Course Wrap-Up This course ran on a single example: an organization's network holding the `192.168.10.0/24` block, and a packet sent to the outside world by the client at `192.168.10.196` on that network. Layered Models defined which problems the packet solves, and in what order, and gave a way to place a fault at the correct layer. Encapsulation showed the layer-by-layer wrapping of data and the cost of that wrapping. The Link and Network Layer topic built the address plan. Where equal-size subnetting required 640 addresses, variable-length subnetting did the same job with 244; the plan turned into leasing pools and a translation table for external egress. When the same network was readdressed with IPv6, address counting was not done at all — the disappearance of scarcity had changed the planning work itself. The Transport Layer topic showed that this packet is part of a flow. The four-tuple named the flow; TCP kept its reliability promise with sequence numbers, retransmission, and windows; congestion control estimated an invisible capacity through measurement and adaptation; UDP showed in which cases these promises are unnecessary. The final lesson separated out the physical limits underlying all of these mechanisms. The course's outcomes converge: placing a problem at the correct layer, designing and testing an address plan, and choosing a transport protocol based on an application's requirements. All three rest on the same habit — tracing a behavior back to the mechanism that produces it. The next course, Application Layer Protocols, covers the protocols that run on top of the transport abstraction built here: the design differences among HTTP versions and their relationship to head-of-line blocking, the extra rounds of the secure transport handshake and the verification of the certificate chain, and protocol selection for real-time communication. The delay budget computed in this course will be the yardstick for every design decision made there.