--- title: 'Regular Expressions' source: 'https://academia.sh/en/courses/python-fundamentals/regular-expressions' course: 'Python Fundamentals' language: en updated: '2026-08-17T18:10:27+00:00' license: 'CC BY-SA 4.0' --- # Regular Expressions Two of the pattern's seven opening parentheses do not capture, leaving five numbered groups; the number is given by the opening parenthesis's order, a non-match is None rather than an exception, and the match object carries every group's text, span, and name. The previous lesson left behind the last syntax form this course measures. Every measurement up to now was a statement or expression calling the language's own protocols: `for`, `if`, `+`, `with`. This lesson's subject is not the language's syntax. A **regular expression** is a separate small language the standard library offers: a pattern is written in a string, interpreted by a separate rule set, and the result comes back as an **object**. The pattern-matching algorithms themselves — brute force search, hash-based search, prefix table search — were built in the Algorithms course's text algorithms topic and are not repeated here. What gets measured is something else: **what the match object carries** and **in what order the parentheses in the pattern get numbered**. ## A Parenthesis's Two Jobs in a Pattern A parenthesis in a pattern has two separate jobs, and both appear with the same notation. The first is **grouping**: it turns the piece inside it into a single unit so a repetition or alternation operator can apply to it. The second is **capturing**: it stores the matched text separately and numbers it. An ordinary parenthesis does **both jobs at once**. A parenthesis opened with `(?:` only groups, it does not capture — it gets no number. One opened with `(?P` both captures and puts a name next to its number. The numbering rule is one sentence: **the number is given by the opening parenthesis's left-to-right order.** Even in nested parentheses, the outer one is numbered first, because its opening sits further left. Closing order is never looked at. ## Numbering the Groups The measurement builds a single pattern that parses a measurement-record line: station name, year, optional month, value, and unit. The pattern contains both a named and a non-capturing parenthesis. - **EF43** — Parenthesis counts are counted from the pattern string itself; the number of numbered groups is read from the compiled pattern's own field. The two are separate sources and confirm each other. - **EF44** — The oracle is the setup that wrote the pattern: which piece should fall into which group is known in advance. - **EF45** — Spans are read from the match object, not counted by hand; it is a **character** span, not a byte one. - **EF46** — An optional group that does not match returns `None`; this is not an error, it is the declaration that the group did not take part. - **EF47** — The nested-pattern measurement is kept separate purely to test numbering; it measures nothing else. ```python """Capture groups: numbering the parentheses and what the match object carries.""" import re PATTERN = r"(?P[a-z]+)-(\d{4})(?:-(\d{2}))?:\s*(-?\d+(?:[.,]\d+)?)\s*(°C|%)" COMPILED = re.compile(PATTERN) print(f"opening parens in pattern {PATTERN.count('(')} | " f"non-capturing parens {PATTERN.count('(?:')} | " f"named group {PATTERN.count('(?P<')} | numbered groups {COMPILED.groups}") print("name -> number:", COMPILED.groupindex) LINES = ("north-2031: 12,5 °C", "slope-2031-07: -3.5 °C", "summary-2031: 88 %", "broken record") print() print(f"{'line':<26s}{'matched':<9s}{'span':<10s}{'last group':>10s} groups") for s in LINES: e = COMPILED.search(s) if e is None: print(f" {s:<24s}{'no':<9s}{'-':<10s}{'-':>10s} -") else: print(f" {s:<24s}{'yes':<9s}{str(e.span()):<10s}" f"{e.lastindex:>10d} {e.groups()}") print() e = COMPILED.search(LINES[1]) print("second line's match object:") print(f" group(0) (the whole match): {e.group(0)!r}") for i in range(1, COMPILED.groups + 1): print(f" group({i}): {e.group(i)!r} span {e.span(i)}") print(f" access by name: {e.group('station')!r} | " f"named group dict: {e.groupdict()}") print() NESTED = re.compile(r"((\d{4})-(\d{2}))") i = NESTED.search("slope-2031-07: -3.5 °C") print(f"numbered groups in nested pattern {NESTED.groups}; " f"number is given by opening-paren order:") for n in range(1, NESTED.groups + 1): print(f" group({n}) = {i.group(n)!r}") ``` ``` opening parens in pattern 7 | non-capturing parens 2 | named group 1 | numbered groups 5 name -> number: {'station': 1} line matched span last group groups north-2031: 12,5 °C yes (0, 19) 5 ('north', '2031', None, '12,5', '°C') slope-2031-07: -3.5 °C yes (0, 22) 5 ('slope', '2031', '07', '-3.5', '°C') summary-2031: 88 % yes (0, 18) 5 ('summary', '2031', None, '88', '%') broken record no - - - second line's match object: group(0) (the whole match): 'slope-2031-07: -3.5 °C' group(1): 'slope' span (0, 5) group(2): '2031' span (6, 10) group(3): '07' span (11, 13) group(4): '-3.5' span (15, 19) group(5): '°C' span (20, 22) access by name: 'slope' | named group dict: {'station': 'slope'} numbered groups in nested pattern 3; number is given by opening-paren order: group(1) = '2031-07' group(2) = '2031' group(3) = '07' ``` ## What the Match Object Carries The first line gives the arithmetic: the pattern has **7** opening parentheses, **2** of them non-capturing, leaving **5** numbered groups. One number came from the pattern string, the other from the compiled pattern, and they agree. The named group's number is not lost: the name `station` corresponds to number **1**. A name is an alias, it does not replace the number. The second table resolves four lines. Three match, one does not. The matching ones' `groups` column gives a five-item tuple, and the optional month group returns `None` on lines where no month is written. `None` is not an error here: it means the group **did not take part**, and it is distinct from an empty string — an empty string would mean "it matched, but its content was empty." The third block opens up everything the match object carries. `group(0)` is **the whole match** and is not a group; numbers start after it. Every group gives both its text and its **span**: `2031` sits between characters **6** and **10** of the record. This span lets you point at the matched text inside the source without pulling it out of place. The last block settles the numbering. The nested pattern has **3** groups, and `group(1)` gives the outer whole, `group(2)` the year, `group(3)` the month. Even though the outer parenthesis closes later, it is numbered **first**, because it **opens first**. ## Forms of Searching The second measurement runs the same pattern with different search forms and tests what a non-match produces. - **EF48** — Three search forms are tried with the **same** compiled pattern and the same lines; only the method called changes. - **EF49** — Multi-match counts are read from the length of the list; the line count is counted from the text's own lines. - **EF50** — In the non-matching case, the returned value is printed directly; that the result is not an exception is shown by the run. - **EF51** — The position measurement searches the same content once as a string and once as a byte sequence; the difference comes purely from the content's type. ```python """Forms of matching search, the non-matching case, and position on a string vs bytes.""" import re PATTERN = r"(?P[a-z]+)-(\d{4})(?:-(\d{2}))?:\s*(-?\d+(?:[.,]\d+)?)\s*(°C|%)" COMPILED = re.compile(PATTERN) ATTEMPTS = ("north-2031: 12,5 °C", " north-2031: 12,5 °C", "north-2031: 12,5 °C extra", "broken record") print(f"{'line':<30s}{'search':<9s}{'match':<8s}fullmatch") for s in ATTEMPTS: results = [("yes" if method(s) else "no") for method in (COMPILED.search, COMPILED.match, COMPILED.fullmatch)] print(f" {repr(s):<28s}{results[0]:<9s}{results[1]:<8s}{results[2]}") RECORDS = ("north-2031: 12,5 °C\n" "slope-2031-07: -3.5 °C\n" "summary-2031: 88 %\n" "broken record\n" "south-2032: 41 °C") print() found = COMPILED.findall(RECORDS) iterated = list(COMPILED.finditer(RECORDS)) print(f"text lines {len(RECORDS.splitlines())} | findall {len(found)} | " f"finditer {len(iterated)} | length of a findall item " f"{len(found[0])} | group count {COMPILED.groups}") print("finditer spans:", [e.span() for e in iterated]) print() none_result = COMPILED.search("broken record") print("result of a non-matching search:", none_result, "| did it raise an exception: no") try: none_result.group(0) except AttributeError as e: print(f"calling group on the result: {type(e).__name__} — " f"a program defect in the first lesson's classification") print() print("substitution reuses the groups:") print(" ", COMPILED.sub(r"\g=\4\5", RECORDS).replace("\n", " | ")) print() TEXT = "kuzey yamaç: 12,5 °C" RAW = TEXT.encode("utf-8") print(f"characters {len(TEXT)} | bytes {len(RAW)}") print(f"position of 'C' in the string {re.search(r'C', TEXT).start()} | " f"in the byte sequence {re.search(rb'C', RAW).start()}") ``` ``` line search match fullmatch 'north-2031: 12,5 °C' yes yes yes ' north-2031: 12,5 °C' yes no no 'north-2031: 12,5 °C extra' yes yes no 'broken record' no no no text lines 5 | findall 4 | finditer 4 | length of a findall item 5 | group count 5 finditer spans: [(0, 19), (20, 42), (43, 61), (76, 93)] result of a non-matching search: None | did it raise an exception: no calling group on the result: AttributeError — a program defect in the first lesson's classification substitution reuses the groups: north=12,5°C | slope=-3.5°C | summary=88% | broken record | south=41°C characters 20 | bytes 22 position of 'C' in the string 19 | in the byte sequence 21 ``` ## Reading the Numbers The first table sets apart three search forms. `search` looks for the pattern **anywhere** and finds it in three of four lines. `match` only tries **from the start**: it fails on the line with leading whitespace. `fullmatch` requires **the whole** line to match: it fails on the line with extra text at the end. Same pattern, same line, three different answers — depending on which question you ask. The second line counts multi-matching: a **5**-line text has **4** matches, and `findall` and `finditer` give the same count. The difference is in what they carry. `findall` gives a **5**-length tuple per match — only the groups' text; span and name information are lost. `finditer` gives the match objects themselves, and the spans can be read. When groups are present, `findall` is a simplification, and what it simplifies away is exactly the information measured in the previous section. The third block says what a non-match is: the result is **`None`** and **there is no exception**. A regular expression does not count non-matching as an error, because non-matching is an expected outcome. An exception is only born if `group` is called on the `None`, and the class that comes then is `AttributeError` — a **program defect** in the first lesson's classification, meaning something to be prevented by testing, not handled with `except`. A search result is tested before it is used. The substitution line shows the group's second job: captured pieces can be reused **by number or by name** in the replacement text. A non-matching line is left as it is. The last two lines tie back to this course's third lesson. The same content is **20** characters as a string, **22** bytes as a byte sequence; the position of the letter searched for is **19** in one, **21** in the other. A regular expression counts **characters** on a string, **bytes** on a byte sequence. The pattern and the thing it is searched against must be the same type, and the spans it returns are in that type's unit. ## Summary - A parenthesis has two jobs: grouping and capturing; `(?:` only groups, `(?P` both captures and names — of the pattern's **7** opening parentheses, **2** do not capture, leaving **5** groups. - The number is given by **the opening parenthesis's order**; in a nested pattern the outer group is numbered first, and `group(0)` is not a group, it is the whole match. - The match object carries every group's text, **span**, and name if it has one; an optional group that did not take part returns `None`, distinct from an empty string. - `search`, `match`, and `fullmatch` give different answers on the same line; `findall` returns **5**-item tuples and drops span information, `finditer` keeps the match object. - Non-matching is not an exception, it is `None`; an exception is only born if `group` is called without testing first, and it comes as `AttributeError`. ## Course Wrap-Up This course measured a single claim across sixteen lessons: **in Python, syntax is a shorthand.** Every written form calls either a specific **special method** by name or a specific rule by name; a lesson's number is which protocol the syntax it covers calls, and how many times. | lesson | syntax covered | protocol called | |---|---|---| | Python's Execution Model | eleven syntax forms | 12 special methods, 18 calls total | | Syntax and Indentation | indentation, block, statement, and expression | no special method: indentation tokens and the tree | | Variables and Data Types | name binding, `x += extra` | the `__add__`/`__iadd__` split, object identity | | Operators | `n + m`, `n += m`, `x in n` | `__add__`, `__iadd__`, `__contains__` or `__iter__` if absent | | Working with Strings | string methods and formatting notations | `__format__`, `__str__`, `__repr__`; immutability forces a new object | | Type Conversion | | | | Conditionals | `if n` | `__bool__`, or `__len__` if absent, or true if neither | | Loops | `for x in n` | `__iter__` and `__next__`; `StopIteration` ends it | | Function Definition | `def`, default value, argument forms | the default is evaluated once, at definition time | | Scope Rules | name reading and assignment | the four-level local, enclosing, global, built-in lookup | | Built-in Functions | | | | Exceptions | `try`, `except`, `raise`, `finally` | exception hierarchy; catching looks at the class lineage | | Custom Exceptions | `class ...(Exception)` | the class's place in the hierarchy decides catching behavior | | File Operations | `open`'s text mode and binary mode | the encoding decoder and newline translation | | Context Managers | `with n` | `__enter__`, `__exit__`, and the return value's swallowing decision | | Regular Expressions | pattern, group, substitution | the match object and capture-group numbering | The table's blank rows are lessons not present on disk at the time this lesson was written; no row was invented for them. The distance between the second and third columns is the course's entire finding. Two notations that look alike call different protocols — `n + m` and `n += m`, for instance. The same notation calls a different protocol depending on the object — `if n`, for instance. And a protocol **failing** is part of the protocol too: `StopIteration` ends a loop, `__exit__` sees the exception inside a `with` block and decides, through its return value, whether to swallow it. The reading that follows from this is: **what an object can do is decided by the special methods it defines, not by its type's name.** The object measured throughout the course descended from no built-in collection; the only reason it could enter a `for`, be tested with `in`, or be used in a `with` block was that it **defined** the relevant methods. Throughout this course, protocols were always **used**. The next course, Data Structures and Functional Tools, takes the other half of the work: **writing** the protocol. The two are different things. Someone writing `for` is calling the iteration protocol; someone writing their own iterator, their own generator, their own sort key is building **the other side** of the protocol. There, lists, tuples, dictionaries, and sets are treated not as data structures but as **protocol implementations** — and the condition for your own object to stand in their place is defining, yourself, the methods counted in this course.