Syntax and Indentation

The previous lesson showed that the source is compiled as a whole before anything runs, and said that the first step of compilation is splitting the text into tokens. That step left a question open: what happens to the whitespace at the start of a line?

In most languages that build a block with braces, nothing — the whitespace is discarded, braces say where the body starts and ends, and indentation is there only for the reader. In Python, the whitespace is not discarded. Indentation adds its own tokens to the token sequence, and it is exactly those tokens that build the block structure. The observable consequence is this: the same token sequence turns into a different tree and a different result just by changing its indentation.

Whitespace Builds the Block, Not Braces

The tokenizing step measures the whitespace at the start of every line and keeps it on a stack. If a line sits deeper than the one before it, an INDENT token is placed in the stream; if it sits shallower, a DEDENT token is placed for every level that closes. If several levels close at once, that many DEDENT tokens are produced, so the number of DEDENT tokens in the sequence gives the number of blocks that closed. These tokens have no character in the source text that corresponds to them — they are produced from the whitespace itself.

The two sources below are indistinguishable if you only look at the tokens’ names. Only one line’s indentation differs: the print call sits outside the loop body in one, inside it in the other.

The measurement’s assumptions:

• LF8 — The two sources differ only in the last line’s indentation; no character was added, removed, or changed.
• LF9 — The comparison is made through the tokens’ names; INDENT, DEDENT, NEWLINE, NL, and the end-of-file token are counted as “layout tokens” and set aside, the rest count as “content tokens.”
• LF10 — The tree is written with the class names of the statements at the parsed structure’s top level; the loop statement’s body is given in square brackets. What is read is not the tree’s full dump but the shell that is enough for this measurement.
• LF11 — Execution output is captured into a separate buffer and written as a list of printed lines; no environment-dependent value is read.

"""Same token sequence, two indentations: three views."""
import ast
import contextlib
import io
import tokenize

OUTSIDE = "total = 0\nfor x in (1, 2, 3):\n    total += x\nprint(total)\n"
INSIDE = "total = 0\nfor x in (1, 2, 3):\n    total += x\n    print(total)\n"
SOURCE = {"outside the body": OUTSIDE, "inside the body": INSIDE}
LAYOUT = {"INDENT", "DEDENT", "NEWLINE", "NL", "ENDMARKER"}


def tokens(source):
    reader = io.StringIO(source).readline
    return [tokenize.tok_name[t.type] for t in tokenize.generate_tokens(reader)]


def tree(source):
    parts = []
    for d in ast.parse(source).body:
        name = type(d).__name__
        if isinstance(d, ast.For):
            name += "[" + ",".join(type(g).__name__ for g in d.body) + "]"
        parts.append(name)
    return " ".join(parts)


def run(source):
    buffer = io.StringIO()
    with contextlib.redirect_stdout(buffer):
        exec(compile(source, "<lesson>", "exec"), {})
    return buffer.getvalue().split()


for name, source in SOURCE.items():
    all_tokens = tokens(source)
    content = [t for t in all_tokens if t not in LAYOUT]
    print(f"{name:17s} tokens {len(all_tokens)}  non-layout {len(content)}  "
          f"INDENT/DEDENT {[i for i, t in enumerate(all_tokens) if t in ('INDENT', 'DEDENT')]}")

a, b = (tokens(k) for k in SOURCE.values())
print(f"equal once layout tokens are dropped: "
      f"{[x for x in a if x not in LAYOUT] == [x for x in b if x not in LAYOUT]}")
print()
for name, source in SOURCE.items():
    print(f"{name:17s} tree: {tree(source)}")
    print(f"{'':17s} printed: {run(source)}")

outside the body  tokens 28  non-layout 21  INDENT/DEDENT [16, 21]
inside the body   tokens 28  non-layout 21  INDENT/DEDENT [16, 26]
equal once layout tokens are dropped: True

outside the body  tree: Assign For[AugAssign] Expr
                  printed: ['6']
inside the body   tree: Assign For[AugAssign,Expr]
                  printed: ['1', '3', '6']

Both sources produce 28 tokens, and 21 of them are content tokens. Once layout tokens are dropped, the two remaining sequences are exactly equal — same names, same order. The difference sits at a single token’s position: DEDENT stands at position 21 in one, position 26 in the other. The only thing that changed in the source is four spaces, and those four spaces moved one token five positions over.

That single difference changes the tree. In the first source, the call is the loop’s sibling: in the tree, For[AugAssign] finishes and a separate Expr follows. In the second, the call is the loop’s child: For[AugAssign,Expr]. The result splits accordingly — one prints a single line, the other prints three.

The rule that follows from this closes one of the most expensive misunderstandings when approaching the language: breaking indentation is not a style defect, it is changing the program. In a brace-based language, these same two sources would be formatted versions of each other and both would do the same thing. Here indentation is the sole determinant; no other marker states the body’s boundary.

Width Is Free, Consistency Is Mandatory

Indentation being grammatical does not mean a particular width is grammatical. The language does not impose a number; what it imposes is that a block be internally consistent. The boundary is observable: the same structure can be written with four different widths and their trees compared, then four inconsistent sources can be tried for compilation.

• LF12 — In valid sources, width changes but stays consistent within a block; in broken sources, consistency breaks in four different ways. The sources are compiled, not executed — what is measured is whether the compile phase accepts them.

"""The rule of indentation: width is free, consistency is mandatory."""
import ast

WIDTH = {"two spaces": "if 1:\n  a = 1\n  b = 2\n",
         "four spaces": "if 1:\n    a = 1\n    b = 2\n",
         "eight spaces": "if 1:\n        a = 1\n        b = 2\n",
         "tab": "if 1:\n\ta = 1\n\tb = 2\n"}

BROKEN = {"no indentation at all": "for x in (1, 2):\nb = 1\n",
          "unexpected indent": "a = 0\n    b = 1\n",
          "misaligned dedent": "if 1:\n        a = 1\n    b = 2\n",
          "tab mixed with spaces": "if 1:\n\ta = 1\n        b = 2\n"}


def tree(source):
    d = ast.parse(source).body[0]
    return f"{type(d).__name__}[{','.join(type(g).__name__ for g in d.body)}]"


print("same structure, different width:")
for name, source in WIDTH.items():
    print(f"  {name:12s} tree: {tree(source)}")
print(f"  are all four trees the same: "
      f"{len({tree(k) for k in WIDTH.values()}) == 1}")

print("\ninconsistent indentation:")
for name, source in BROKEN.items():
    try:
        compile(source, "<lesson>", "exec")
        print(f"  {name:24s} compiled")
    except SyntaxError as e:
        print(f"  {name:24s} {type(e).__name__}")
print(f"  is IndentationError a SyntaxError: "
      f"{issubclass(IndentationError, SyntaxError)}")
print(f"  is TabError an IndentationError: "
      f"{issubclass(TabError, IndentationError)}")

same structure, different width:
  two spaces   tree: If[Assign,Assign]
  four spaces  tree: If[Assign,Assign]
  eight spaces tree: If[Assign,Assign]
  tab          tree: If[Assign,Assign]
  are all four trees the same: True

inconsistent indentation:
  no indentation at all    IndentationError
  unexpected indent        IndentationError
  misaligned dedent        IndentationError
  tab mixed with spaces    TabError
  is IndentationError a SyntaxError: True
  is TabError an IndentationError: True

Four widths give a single tree. What is grammatical is not the amount of indentation but its structure: a level opens and continues at the same alignment, and when it closes it lands on an alignment that was already opened. A program written with two spaces and one written with eight spaces are the same program; the choice between them is a matter of team convention, not the language.

When consistency breaks, compilation stops in all four cases. Three give IndentationError: a block whose body is never opened, a line that slides inward without opening anything, and a dedent that lands on no previously opened alignment. The fourth stands apart — a source mixing tabs and spaces gives TabError, because the width of these two characters cannot be resolved just by looking at the source. The output’s last two lines show all three come from a single lineage: TabError is an IndentationError, which is in turn a SyntaxError.

This lineage also settles the previous lesson’s boundary. Indentation defects are found at compile time; none of them make it to execution. A broken indentation on the file’s last line keeps even the assignment on the first line from running.

Logical Line vs. Physical Line

Indentation being grammatical stops somewhere, and that limit is worth knowing. What builds the block structure is the whitespace at the start of a logical line; not every line start on screen begins a logical line. While an expression sits inside an open bracket, a line ending does not finish the statement — an NL is placed in the stream instead of a NEWLINE, and the indentation of those lines produces no token at all. The same continuation can also be built with a backslash.

• LF13 — Four forms write the same sum and differ only in how they split across lines; what is measured is the physical line count, the logical line ending (NEWLINE), the continuation line ending (NL), and the INDENT count. The value column is the sum the source itself produces.

"""Logical line vs physical line: indentation is not grammatical inside parens."""
import ast
import io
import tokenize

FORM = {
    "single line": "total = 1 + 2 + 3\n",
    "aligned inside parens": "total = (1 +\n          2 +\n          3)\n",
    "ragged inside parens": "total = (1 +\n2 +\n              3)\n",
    "with backslash": "total = 1 + \\\n    2 + \\\n    3\n",
}


def count(source):
    reader = io.StringIO(source).readline
    names = [tokenize.tok_name[t.type] for t in tokenize.generate_tokens(reader)]
    return names.count("NEWLINE"), names.count("NL"), names.count("INDENT")


print(f"{'form':<22s} {'physical':>8s} {'NEWLINE':>7s} {'NL':>3s} {'INDENT':>6s}"
      f"  tree     value")
for name, source in FORM.items():
    logical, nl, indent = count(source)
    env = {}
    exec(compile(source, "<lesson>", "exec"), env)
    tree = " ".join(type(d).__name__ for d in ast.parse(source).body)
    print(f"{name:<22s} {len(source.splitlines()):8d} {logical:7d} {nl:3d} {indent:6d}"
          f"  {tree:8s} {env['total']}")

form                   physical NEWLINE  NL INDENT  tree     value
single line                   1       1   0      0  Assign   6
aligned inside parens         3       1   2      0  Assign   6
ragged inside parens          3       1   2      0  Assign   6
with backslash                3       1   0      0  Assign   6

All four forms give a single Assign node and a single value. Even though three of them spread across 3 physical lines, the NEWLINE count is 1 in every one: there is exactly one logical line. In the two forms inside parentheses, the line endings in between pass through as NL, meaning “no statement ended here.” In the backslash form, the line ending does not turn into a token at all; the continuation is resolved before the text is even split into tokens.

The most telling column is INDENT: 0 in all four. The “ragged” form’s second line starts at column zero, its third starts deep in, and neither changes anything. So indentation stops being grammatical inside brackets; the alignment done there is entirely for the reader, and inconsistency there produces no defect. The rule can be narrowed like this: indentation only carries meaning at the start of a logical line.

Statement and Expression

Indentation builds a block; a block, in turn, is made of statements. This requires the language’s second fundamental distinction. An expression produces a value: it is computed and leaves an object behind. A statement does a job: it binds a name, enters a block, directs the flow; it leaves no value behind.

The distinction does not have to be left to intuition, because the compile mode states it directly. eval mode accepts only an expression; exec mode accepts a sequence of statements.

• LF14 — Every part is attempted in both modes; what is measured is whether it is accepted. Parts whose value is printed are evaluated in an environment where a single name is bound, so the value column depends on that binding, not on the environment.

"""Statement or expression: 'eval' mode only compiles an expression, 'exec' compiles both."""

PARTS = (
    "2 + 3",
    "total",
    "len('value')",
    "[k for k in (1, 2, 3)]",
    "total if total else 0",
    "total = 5",
    "total += 1",
    "if total: pass",
    "for k in (1, 2): pass",
    "pass",
)

print(f"{'part':<26s} {'expr':>5s} {'stmt':>5s}  value")
expr_count = stmt_count = 0
for p in PARTS:
    try:
        code_obj = compile(p, "<lesson>", "eval")
        is_expr = True
    except SyntaxError:
        is_expr = False
    try:
        compile(p, "<lesson>", "exec")
        is_stmt = True
    except SyntaxError:
        is_stmt = False
    expr_count += is_expr
    stmt_count += is_stmt
    value = repr(eval(code_obj, {"total": 3})) if is_expr else "—"
    print(f"{p:<26s} {str(is_expr):>5s} {str(is_stmt):>5s}  {value}")

print(f"\n{len(PARTS)} parts: expression {expr_count}, statement {stmt_count}; "
      f"statement-only {stmt_count - expr_count}")

part                        expr  stmt  value
2 + 3                       True  True  5
total                       True  True  3
len('value')                True  True  5
[k for k in (1, 2, 3)]      True  True  [1, 2, 3]
total if total else 0       True  True  3
total = 5                  False  True  —
total += 1                 False  True  —
if total: pass             False  True  —
for k in (1, 2): pass      False  True  —
pass                       False  True  —

10 parts: expression 5, statement 10; statement-only 5

5 of ten parts are expressions, 10 are statements. The statement column being entirely filled gives the direction of the distinction: every expression is also a statement — placed alone on a line, its value is computed and discarded. The reverse does not hold; the 5 statement-only parts cannot be placed anywhere an expression could stand.

The distinction has a direct consequence for the course. All eleven forms the previous lesson measured consumed an object, and what they consumed was an expression’s value; special methods operate on expressions. This is also why it matters that the row total += 1 does not come out as an expression: the Programming Fundamentals course said that a compound assignment is a statement; what is measured here is which method that statement calls and what it does with the returned value. The result of the __iadd__ call does not sit around as a value, it is bound straight back to the name — the detail is paid off in the operators lesson.

One last observation hides in the table: if total: pass, even though written on a single line, is a valid statement and uses no indentation at all. Indentation is not the only way to write a block; what makes a block mandatory is the statement itself, and indentation is the way to write a body with more than one statement.

Summary

• Indentation is not discarded; the whitespace at the start of a line is turned into INDENT and DEDENT tokens, and it is these tokens that build the block structure.
• Two sources differing only in indentation carry the same 21 content tokens in the same order; the position of a single DEDENT token changes the tree and the result.
• Indentation’s width is free — four different widths give a single tree — but it must be consistent within a block.
• Inconsistent indentation stops at compile time; TabError is an IndentationError, which is in turn a SyntaxError.
• Indentation is only grammatical at the start of a logical line: 3 physical lines inside an open bracket produce a single NEWLINE, and the INDENT count stays 0.
• 5 of ten parts are expressions, 10 are statements: every expression can be written as a statement, no statement can stand in for an expression.

Next Step

In this lesson, assignment — the most commonly written statement — showed up only as “not an expression” in the table; what it does was never examined. The Programming Fundamentals course established that assignment is a binding — a name is not a box, it is a label attached to an object. The next lesson measures this bond from the object’s side: what does it mean when two names are bound to the same object, under what condition do two equal values turn out to be the same object, and how does a type being “immutable” affect binding?