Regular Expressions

The previous lesson left behind the last syntax form this course measures. Every measurement up to now was a statement or expression calling the language’s own protocols: for, if, +, with.

This lesson’s subject is not the language’s syntax. A regular expression is a separate small language the standard library offers: a pattern is written in a string, interpreted by a separate rule set, and the result comes back as an object. The pattern-matching algorithms themselves — brute force search, hash-based search, prefix table search — were built in the Algorithms course’s text algorithms topic and are not repeated here. What gets measured is something else: what the match object carries and in what order the parentheses in the pattern get numbered.

A Parenthesis’s Two Jobs in a Pattern

A parenthesis in a pattern has two separate jobs, and both appear with the same notation. The first is grouping: it turns the piece inside it into a single unit so a repetition or alternation operator can apply to it. The second is capturing: it stores the matched text separately and numbers it.

An ordinary parenthesis does both jobs at once. A parenthesis opened with (?: only groups, it does not capture — it gets no number. One opened with (?P<name> both captures and puts a name next to its number.

The numbering rule is one sentence: the number is given by the opening parenthesis’s left-to-right order. Even in nested parentheses, the outer one is numbered first, because its opening sits further left. Closing order is never looked at.

Numbering the Groups

The measurement builds a single pattern that parses a measurement-record line: station name, year, optional month, value, and unit. The pattern contains both a named and a non-capturing parenthesis.

• EF43 — Parenthesis counts are counted from the pattern string itself; the number of numbered groups is read from the compiled pattern’s own field. The two are separate sources and confirm each other.
• EF44 — The oracle is the setup that wrote the pattern: which piece should fall into which group is known in advance.
• EF45 — Spans are read from the match object, not counted by hand; it is a character span, not a byte one.
• EF46 — An optional group that does not match returns None; this is not an error, it is the declaration that the group did not take part.
• EF47 — The nested-pattern measurement is kept separate purely to test numbering; it measures nothing else.

"""Capture groups: numbering the parentheses and what the match object carries."""
import re

PATTERN = r"(?P<station>[a-z]+)-(\d{4})(?:-(\d{2}))?:\s*(-?\d+(?:[.,]\d+)?)\s*(°C|%)"
COMPILED = re.compile(PATTERN)

print(f"opening parens in pattern {PATTERN.count('(')} | "
      f"non-capturing parens {PATTERN.count('(?:')} | "
      f"named group {PATTERN.count('(?P<')} | numbered groups {COMPILED.groups}")
print("name -> number:", COMPILED.groupindex)

LINES = ("north-2031: 12,5 °C", "slope-2031-07: -3.5 °C",
            "summary-2031: 88 %", "broken record")

print()
print(f"{'line':<26s}{'matched':<9s}{'span':<10s}{'last group':>10s}   groups")
for s in LINES:
    e = COMPILED.search(s)
    if e is None:
        print(f"  {s:<24s}{'no':<9s}{'-':<10s}{'-':>10s}   -")
    else:
        print(f"  {s:<24s}{'yes':<9s}{str(e.span()):<10s}"
              f"{e.lastindex:>10d}   {e.groups()}")

print()
e = COMPILED.search(LINES[1])
print("second line's match object:")
print(f"  group(0) (the whole match): {e.group(0)!r}")
for i in range(1, COMPILED.groups + 1):
    print(f"  group({i}): {e.group(i)!r}  span {e.span(i)}")
print(f"  access by name: {e.group('station')!r} | "
      f"named group dict: {e.groupdict()}")

print()
NESTED = re.compile(r"((\d{4})-(\d{2}))")
i = NESTED.search("slope-2031-07: -3.5 °C")
print(f"numbered groups in nested pattern {NESTED.groups}; "
      f"number is given by opening-paren order:")
for n in range(1, NESTED.groups + 1):
    print(f"  group({n}) = {i.group(n)!r}")

opening parens in pattern 7 | non-capturing parens 2 | named group 1 | numbered groups 5
name -> number: {'station': 1}

line                      matched  span      last group   groups
  north-2031: 12,5 °C     yes      (0, 19)            5   ('north', '2031', None, '12,5', '°C')
  slope-2031-07: -3.5 °C  yes      (0, 22)            5   ('slope', '2031', '07', '-3.5', '°C')
  summary-2031: 88 %      yes      (0, 18)            5   ('summary', '2031', None, '88', '%')
  broken record           no       -                  -   -

second line's match object:
  group(0) (the whole match): 'slope-2031-07: -3.5 °C'
  group(1): 'slope'  span (0, 5)
  group(2): '2031'  span (6, 10)
  group(3): '07'  span (11, 13)
  group(4): '-3.5'  span (15, 19)
  group(5): '°C'  span (20, 22)
  access by name: 'slope' | named group dict: {'station': 'slope'}

numbered groups in nested pattern 3; number is given by opening-paren order:
  group(1) = '2031-07'
  group(2) = '2031'
  group(3) = '07'

What the Match Object Carries

The first line gives the arithmetic: the pattern has 7 opening parentheses, 2 of them non-capturing, leaving 5 numbered groups. One number came from the pattern string, the other from the compiled pattern, and they agree. The named group’s number is not lost: the name station corresponds to number 1. A name is an alias, it does not replace the number.

The second table resolves four lines. Three match, one does not. The matching ones’ groups column gives a five-item tuple, and the optional month group returns None on lines where no month is written. None is not an error here: it means the group did not take part, and it is distinct from an empty string — an empty string would mean “it matched, but its content was empty.”

The third block opens up everything the match object carries. group(0) is the whole match and is not a group; numbers start after it. Every group gives both its text and its span: 2031 sits between characters 6 and 10 of the record. This span lets you point at the matched text inside the source without pulling it out of place.

The last block settles the numbering. The nested pattern has 3 groups, and group(1) gives the outer whole, group(2) the year, group(3) the month. Even though the outer parenthesis closes later, it is numbered first, because it opens first.

Forms of Searching

The second measurement runs the same pattern with different search forms and tests what a non-match produces.

• EF48 — Three search forms are tried with the same compiled pattern and the same lines; only the method called changes.
• EF49 — Multi-match counts are read from the length of the list; the line count is counted from the text’s own lines.
• EF50 — In the non-matching case, the returned value is printed directly; that the result is not an exception is shown by the run.
• EF51 — The position measurement searches the same content once as a string and once as a byte sequence; the difference comes purely from the content’s type.

"""Forms of matching search, the non-matching case, and position on a string vs bytes."""
import re

PATTERN = r"(?P<station>[a-z]+)-(\d{4})(?:-(\d{2}))?:\s*(-?\d+(?:[.,]\d+)?)\s*(°C|%)"
COMPILED = re.compile(PATTERN)

ATTEMPTS = ("north-2031: 12,5 °C", "  north-2031: 12,5 °C",
             "north-2031: 12,5 °C extra", "broken record")

print(f"{'line':<30s}{'search':<9s}{'match':<8s}fullmatch")
for s in ATTEMPTS:
    results = [("yes" if method(s) else "no")
         for method in (COMPILED.search, COMPILED.match, COMPILED.fullmatch)]
    print(f"  {repr(s):<28s}{results[0]:<9s}{results[1]:<8s}{results[2]}")

RECORDS = ("north-2031: 12,5 °C\n"
            "slope-2031-07: -3.5 °C\n"
            "summary-2031: 88 %\n"
            "broken record\n"
            "south-2032: 41 °C")

print()
found = COMPILED.findall(RECORDS)
iterated = list(COMPILED.finditer(RECORDS))
print(f"text lines {len(RECORDS.splitlines())} | findall {len(found)} | "
      f"finditer {len(iterated)} | length of a findall item "
      f"{len(found[0])} | group count {COMPILED.groups}")
print("finditer spans:", [e.span() for e in iterated])

print()
none_result = COMPILED.search("broken record")
print("result of a non-matching search:", none_result, "| did it raise an exception: no")
try:
    none_result.group(0)
except AttributeError as e:
    print(f"calling group on the result: {type(e).__name__} — "
          f"a program defect in the first lesson's classification")

print()
print("substitution reuses the groups:")
print(" ", COMPILED.sub(r"\g<station>=\4\5", RECORDS).replace("\n", " | "))

print()
TEXT = "kuzey yamaç: 12,5 °C"
RAW = TEXT.encode("utf-8")
print(f"characters {len(TEXT)} | bytes {len(RAW)}")
print(f"position of 'C' in the string {re.search(r'C', TEXT).start()} | "
      f"in the byte sequence {re.search(rb'C', RAW).start()}")

line                          search   match   fullmatch
  'north-2031: 12,5 °C'       yes      yes     yes
  '  north-2031: 12,5 °C'     yes      no      no
  'north-2031: 12,5 °C extra' yes      yes     no
  'broken record'             no       no      no

text lines 5 | findall 4 | finditer 4 | length of a findall item 5 | group count 5
finditer spans: [(0, 19), (20, 42), (43, 61), (76, 93)]

result of a non-matching search: None | did it raise an exception: no
calling group on the result: AttributeError — a program defect in the first lesson's classification

substitution reuses the groups:
  north=12,5°C | slope=-3.5°C | summary=88% | broken record | south=41°C

characters 20 | bytes 22
position of 'C' in the string 19 | in the byte sequence 21

Reading the Numbers

The first table sets apart three search forms. search looks for the pattern anywhere and finds it in three of four lines. match only tries from the start: it fails on the line with leading whitespace. fullmatch requires the whole line to match: it fails on the line with extra text at the end. Same pattern, same line, three different answers — depending on which question you ask.

The second line counts multi-matching: a 5-line text has 4 matches, and findall and finditer give the same count. The difference is in what they carry. findall gives a 5-length tuple per match — only the groups’ text; span and name information are lost. finditer gives the match objects themselves, and the spans can be read. When groups are present, findall is a simplification, and what it simplifies away is exactly the information measured in the previous section.

The third block says what a non-match is: the result is None and there is no exception. A regular expression does not count non-matching as an error, because non-matching is an expected outcome. An exception is only born if group is called on the None, and the class that comes then is AttributeError — a program defect in the first lesson’s classification, meaning something to be prevented by testing, not handled with except. A search result is tested before it is used.

The substitution line shows the group’s second job: captured pieces can be reused by number or by name in the replacement text. A non-matching line is left as it is.

The last two lines tie back to this course’s third lesson. The same content is 20 characters as a string, 22 bytes as a byte sequence; the position of the letter searched for is 19 in one, 21 in the other. A regular expression counts characters on a string, bytes on a byte sequence. The pattern and the thing it is searched against must be the same type, and the spans it returns are in that type’s unit.

Summary

• A parenthesis has two jobs: grouping and capturing; (?: only groups, (?P<name> both captures and names — of the pattern’s 7 opening parentheses, 2 do not capture, leaving 5 groups.
• The number is given by the opening parenthesis’s order; in a nested pattern the outer group is numbered first, and group(0) is not a group, it is the whole match.
• The match object carries every group’s text, span, and name if it has one; an optional group that did not take part returns None, distinct from an empty string.
• search, match, and fullmatch give different answers on the same line; findall returns 5-item tuples and drops span information, finditer keeps the match object.
• Non-matching is not an exception, it is None; an exception is only born if group is called without testing first, and it comes as AttributeError.

Course Wrap-Up

This course measured a single claim across sixteen lessons: in Python, syntax is a shorthand. Every written form calls either a specific special method by name or a specific rule by name; a lesson’s number is which protocol the syntax it covers calls, and how many times.

lesson	syntax covered	protocol called
Python’s Execution Model	eleven syntax forms	12 special methods, 18 calls total
Syntax and Indentation	indentation, block, statement, and expression	no special method: indentation tokens and the tree
Variables and Data Types	name binding, x += extra	the __add__/__iadd__ split, object identity
Operators	n + m, n += m, x in n	__add__, __iadd__, __contains__ or __iter__ if absent
Working with Strings	string methods and formatting notations	__format__, __str__, __repr__; immutability forces a new object
Type Conversion
Conditionals	if n	__bool__, or __len__ if absent, or true if neither
Loops	for x in n	__iter__ and __next__; StopIteration ends it
Function Definition	def, default value, argument forms	the default is evaluated once, at definition time
Scope Rules	name reading and assignment	the four-level local, enclosing, global, built-in lookup
Built-in Functions
Exceptions	try, except, raise, finally	exception hierarchy; catching looks at the class lineage
Custom Exceptions	class ...(Exception)	the class’s place in the hierarchy decides catching behavior
File Operations	open’s text mode and binary mode	the encoding decoder and newline translation
Context Managers	with n	__enter__, __exit__, and the return value’s swallowing decision
Regular Expressions	pattern, group, substitution	the match object and capture-group numbering

The table’s blank rows are lessons not present on disk at the time this lesson was written; no row was invented for them.

The distance between the second and third columns is the course’s entire finding. Two notations that look alike call different protocols — n + m and n += m, for instance. The same notation calls a different protocol depending on the object — if n, for instance. And a protocol failing is part of the protocol too: StopIteration ends a loop, __exit__ sees the exception inside a with block and decides, through its return value, whether to swallow it.

The reading that follows from this is: what an object can do is decided by the special methods it defines, not by its type’s name. The object measured throughout the course descended from no built-in collection; the only reason it could enter a for, be tested with in, or be used in a with block was that it defined the relevant methods.

Throughout this course, protocols were always used. The next course, Data Structures and Functional Tools, takes the other half of the work: writing the protocol. The two are different things. Someone writing for is calling the iteration protocol; someone writing their own iterator, their own generator, their own sort key is building the other side of the protocol. There, lists, tuples, dictionaries, and sets are treated not as data structures but as protocol implementations — and the condition for your own object to stand in their place is defining, yourself, the methods counted in this course.