Working with Strings

The previous lesson measured that the + operator calls __add__, the += notation calls __iadd__, and said that immutable types cannot define __iadd__. A string is exactly such a type; every operation on it derives from this single constraint.

This lesson’s question is not the list of string methods. The question is: how does a type being immutable force the methods that can be defined on it, and how much does that forcing cost, counted?

There Is No Such Thing as Writing in Place

There is no way to change a single character of a string. The syntax exists — writing with square brackets — but no method backing it is defined.

• LF31 — A single starting string is used in the measurement, and four operations are run on it in sequence; at every step, whether the result is the same object as the previous one is asked with is. No identity number is printed.
• LF32 — The four chosen operations really change the content. Whether an operation whose result is byte-for-byte identical to its input (strip on a string with no whitespace, for instance) returns the same object or not is implementation-dependent and is not brought into the measurement.
• LF33 — The methods in the second table do not transform the string, they ask it a question; what is measured is the type of the returned value.

"""A string is immutable: producing methods return a new object, query methods return another type."""

RAW = "  12; north; 18  "

try:
    RAW[0] = "x"
except TypeError as e:
    print("writing in place to a string:", type(e).__name__)

STEPS = (("start", lambda s: s),
           ("strip()", lambda s: s.strip()),
           ("upper()", lambda s: s.upper()),
           ("replace(';', ',')", lambda s: s.replace(";", ",")),
           ("split(',')", lambda s: s.split(",")))

print(f"\n{'step':<20s} {'type':<5s} {'same as previous':>17s}  value")
previous = RAW
for name, operation in STEPS:
    current = operation(previous)
    print(f"{name:<20s} {type(current).__name__:<5s} "
          f"{str(current is previous):>17s}  {current!r}")
    previous = current
print("starting string after the measurement:", repr(RAW))

TEXT = "12; north; 18"
QUERIES = (("startswith('12')", lambda s: s.startswith("12")),
            ("find('north')", lambda s: s.find("north")),
            ("count(';')", lambda s: s.count(";")),
            ("isdigit()", lambda s: s.isdigit()),
            ("len(s)", lambda s: len(s)))

print(f"\n{'query method':<20s} {'return type':<5s} value")
for name, function in QUERIES:
    value = function(TEXT)
    print(f"{name:<20s} {type(value).__name__:<5s} {value!r}")
print("string after the queries:", repr(TEXT))

writing in place to a string: TypeError

step                 type   same as previous  value
start                str                True  '  12; north; 18  '
strip()              str               False  '12; north; 18'
upper()              str               False  '12; NORTH; 18'
replace(';', ',')    str               False  '12, NORTH, 18'
split(',')           list              False  ['12', ' NORTH', ' 18']
starting string after the measurement: '  12; north; 18  '

query method         return type value
startswith('12')     bool  True
find('north')        int   4
count(';')           int   2
isdigit()            bool  False
len(s)               int   13
string after the queries: '12; north; 18'

The first line states the constraint by name: writing with square brackets gives TypeError, because the string type has no method backing that operation. It is the exact same result taken for the division operator in the previous lesson — the defect is not a prohibition, it is an absence.

In all four steps of the chain, the “same as previous” column reads False. Every method built a new object and did not touch its input; the last line confirms this — after four operations, the starting string still stands as it was. It matters to see that this is not a choice. A string method could not mutate its input even if it wanted to; the only thing it can do is build and return a new object. Immutability writes the method’s contract on its own, which is why the string type carries no method that works in place, like a list’s append or sort.

A direct rule follows from this: if a string method’s return value is not used, nothing has happened. The line text.strip() written alone builds a new object, does not bind it to a name, and discards it; text stays as it was. This is the single most common silent defect in code written by someone new to the language.

The lower table sets apart the second class of method. startswith, find, count, isdigit, and len do not produce a string; they give a boolean, a position, a count, in turn. These do not transform the string, they ask it a question — and so they sit outside the new-object discussion. The find method returns -1 when it cannot find the piece sought; it does not raise an exception. The index method, which does the same job, raises ValueError when it cannot find it. The choice between the two methods is not a matter of style: when not finding something is an expected situation, the notation that works with a return value is chosen; when it is a situation that should not happen, the notation that raises an exception is chosen. The counterpart of this same distinction on the conversion side is the next lesson’s subject.

A String Is a Sequence

In the third lesson’s built-in type table, str filled all five columns: its length could be asked, a loop could be built over it, membership could be tested, it could be sliced, and it could be added. This means a string is not only text but also a sequence — and being a sequence has two observable consequences.

• LF34 — In the slicing measurement, every result is compared to the source string with is; a slice that asks for the whole source is not used, because whether it returns the same object is implementation-dependent.
• LF35 — In the escaping measurement, the character count of the three written notations is read; the number shows how many characters the notation produces, not how many characters were written in the source.

"""A string is a sequence: slicing, iteration, and membership rest on the same protocols."""

TEXT = "12; north; 18"

SLICES = (("[0]", lambda s: s[0]),
            ("[-1]", lambda s: s[-1]),
            ("[4:9]", lambda s: s[4:9]),
            ("[:2]", lambda s: s[:2]),
            ("[::-1]", lambda s: s[::-1]),
            ("[::4]", lambda s: s[::4]))

print(f"{'slice':<8s} {'same object as source':>22s}  value")
for name, function in SLICES:
    piece = function(TEXT)
    print(f"{name:<8s} {str(piece is TEXT):>22s}  {piece!r}")

print(f"\nlength {len(TEXT)}, first four items {list(TEXT)[:4]}")
print("membership:", "'nor' in text ->", "nor" in TEXT,
      "| 'rth' in text ->", "rth" in TEXT,
      "| 'thr' in text ->", "thr" in TEXT)
print("same question on a list:", "[1, 2] in [1, 2, 3] ->", [1, 2] in [1, 2, 3],
      "| 2 in [1, 2, 3] ->", 2 in [1, 2, 3])

ESCAPES = (("'a\\tb'", "a\tb"), ("r'a\\tb'", r"a\tb"),
         ("'''two\\nlines'''", "two\nlines"))
print(f"\n{'notation':<18s} {'characters':>10s}  actual content")
for name, value in ESCAPES:
    print(f"{name:<18s} {len(value):10d}  {value!r}")

slice     same object as source  value
[0]                       False  '1'
[-1]                      False  '8'
[4:9]                     False  'north'
[:2]                      False  '12'
[::-1]                    False  '81 ;htron ;21'
[::4]                     False  '1nh8'

length 13, first four items ['1', '2', ';', ' ']
membership: 'nor' in text -> True | 'rth' in text -> True | 'thr' in text -> False
same question on a list: [1, 2] in [1, 2, 3] -> False | 2 in [1, 2, 3] -> True

notation           characters  actual content
'a\tb'                      3  'a\tb'
r'a\tb'                     4  'a\\tb'
'''two\nlines'''            9  'two\nlines'

All six slices give an object separate from the source. Slicing is the __getitem__ call measured in the first lesson, and it carries the same rule on a string: a new object is built, the source stays as it is. A negative position counts from the end, the third number sets the step; the [::-1] notation reversing the text is not a separate method’s job, it is the consequence of the step being negative.

The iteration line shows that a string’s items are characters: a thirteen-character text gives thirteen items, and every item is itself a string. Python has no separate type representing a single character; a character is a string whose length is one.

The membership line, in turn, sets a string apart from other sequences. "nor" in text gives True — even though what is being asked for is not one item but a sequence of items. When the same question is asked of a list, [1, 2] in [1, 2, 3] gives False, because in on a list only looks for an item. A string’s __contains__ method is deliberately written differently: it searches for a substring. The syntax is the same across the two types, the contract is not.

The last table sets apart notation forms. Backslash escape sequences are written with two characters in the source but produce a single character: the notation 'a\tb' is a three-character string. In the raw form, prefixed with r, the escape is not resolved, and the same source gives four characters. The triple-quoted form carries a line ending as-is. In all three, the character count written in the source and the length of the string produced are different; this is the reason a raw notation is preferred when writing a file path or a pattern string.

The Cost of Concatenation

Immutability’s measurable cost shows up when building text piece by piece. Because every + operation builds a new object, a string grown inside a loop leaves behind intermediate objects built and discarded along the way.

• LF36 — Two methods build the same text from the same seven parts; what is measured is the number of intermediate objects built along the way, not the time elapsed. In the incremental method, every step’s result is compared to the previous one with is and counted; because joining builds a single object, its count is written directly.

"""The cost of concatenation: how many intermediate objects get built?"""

PARTS = ("north", "; ", "12", "; ", "slope", "; ", "18")


def incremental(parts):
    text = ""
    intermediate = 0
    for p in parts:
        previous = text
        text = text + p          # a new string on every step
        intermediate += text is not previous
    return text, intermediate


def joining(parts):
    text = "".join(parts)      # one object in one step
    return text, 1


print(f"{'method':<18s} {'parts':>5s} {'intermediate objects':>21s}  result")
for name, function in (("incrementing with +", incremental), ("with join", joining)):
    result, intermediate = function(PARTS)
    print(f"{name:<18s} {len(PARTS):5d} {intermediate:21d}  {result!r}")

print("are the two results equal:", incremental(PARTS)[0] == joining(PARTS)[0])

method             parts  intermediate objects  result
incrementing with +     7                     7  'north; 12; slope; 18'
with join              7                     1  'north; 12; slope; 18'
are the two results equal: True

Seven parts build 7 intermediate objects with the incremental method; join builds 1. The results are byte-for-byte equal. The pattern grows linearly with the part count: when the part count doubles, the number of objects the incremental method builds doubles too, while join’s stays at 1.

The reason for the difference is not that one method is written better, it is that it already has the information. join sees all the parts up front, can compute the total length, and builds the result in one pass. The + operator, by contrast, only sees two parts on each call; not knowing what comes at the next step, it has to fully build that step’s result.

The same reasoning also explains why collecting into a list and joining at the end is a common pattern: a list is a mutable type, appending to it builds no new object. The text is produced only once, at the very end.

The join method has one condition that is easy to overlook: all the parts have to be strings. If a number slips in among them, the method does not convert it to text on its own — it raises TypeError. The string the method is called on is not one of the parts either, it is the separator: in the notation "; ".join(...), the text placed in between is the string the call is made on.

Which Protocol Formatting Calls

There is more than one notation for producing text, and they look alike. The course’s measure applies here too: each notation calls a different method, and the text each produces can therefore diverge.

• LF37 — The measured object takes part in all three text protocols: __str__, __repr__, and __format__. All three produce different text, so which one was called can also be read from the output.
• LF38 — The __format__ method itself calls no other protocol; if the spec is empty it builds its own text, if it is filled it applies the spec to the number it carries. This way, the log shows only what the notation called.

"""Which special method do formatting notations call?"""

LOG = []


def record(name):
    LOG.append(name)


class Measurement:
    """Takes part in all three text protocols and logs each participation."""

    def __init__(self, value=12):
        self.value = value

    def __str__(self):
        record("__str__")
        return f"measurement {self.value}"

    def __repr__(self):
        record("__repr__")
        return f"Measurement({self.value})"

    def __format__(self, spec):
        record("__format__")
        return format(self.value, spec) if spec else f"measurement {self.value}"


FORMS = (
    ('f"{n}"', lambda n: f"{n}"),
    ('f"{n:>6}"', lambda n: f"{n:>6}"),
    ('f"{n!s}"', lambda n: f"{n!s}"),
    ('f"{n!r}"', lambda n: f"{n!r}"),
    ('"{}".format(n)', lambda n: "{}".format(n)),
    ('"%s" % n', lambda n: "%s" % n),
    ('str(n)', lambda n: str(n)),
    ('repr(n)', lambda n: repr(n)),
)

unique = set()
print(f"{'notation':<16s} {'protocol':<12s} text produced")
for name, function in FORMS:
    LOG.clear()
    text = function(Measurement())
    unique.update(LOG)
    print(f"{name:<16s} {' '.join(LOG):<12s} {text!r}")

print(f"\n{len(FORMS)} notations, distinct protocols {len(unique)}: "
      f"{', '.join(sorted(unique))}")

notation         protocol     text produced
f"{n}"           __format__   'measurement 12'
f"{n:>6}"        __format__   '    12'
f"{n!s}"         __str__      'measurement 12'
f"{n!r}"         __repr__     'Measurement(12)'
"{}".format(n)   __format__   'measurement 12'
"%s" % n         __str__      'measurement 12'
str(n)           __str__      'measurement 12'
repr(n)          __repr__     'Measurement(12)'

8 notations, distinct protocols 3: __format__, __repr__, __str__

Eight notations spread across 3 distinct protocols. The curly-brace notation and the format method call __format__; the percent-sign notation and the str built-in call __str__; the repr built-in and the !r marker call __repr__. The same object gives three different pieces of text across three notations, and none stands in for another.

Two points deserve a closer read. First, __format__ also receives the spec: on the second line, the spec >6 was passed to the method, and the method applied it to the number it carries. Alignment, digit count, and width rules are handed to the object itself this way. Second, the markers are not a shortcut, they are a protocol selector: the !r notation switches the method called from __format__ to __repr__.

The curly-brace notation has one more distinct feature: what goes inside the braces does not have to be a value, it can be an expression, and the expression is evaluated right there. This is why the second lesson’s statement/expression distinction applies here too; an assignment cannot be placed inside the braces, because an assignment is not an expression. The format method, by contrast, takes values as arguments and keeps the spec separate.

The three methods’ contracts differ too. __str__ gives readable text; __repr__ gives text that identifies the object, usually written in a form that could rebuild it — the measurement’s 'Measurement(12)' is an example of this. If an object defines only __repr__, a str call falls to it too; this is the source of the behavior seen on the bare object in the first lesson.

Summary

• Writing to a single character of a string gives TypeError; no method backs that operation.
• In all four producing methods, the result is a new object and the input stays as it is; if the return value is not used, nothing has happened.
• Query methods do not produce a string, they give a response of another type; find returns -1 when it cannot find something, it does not raise an exception.
• A string is a sequence: slicing calls __getitem__ and gives a new object, its items are characters, and unlike a list, the in test looks for a substring.
• Concatenating seven parts incrementally builds 7 intermediate objects, join builds 1; the difference comes from join seeing all the parts up front.
• Eight formatting notations spread across 3 distinct protocols: __format__, __str__, and __repr__; markers like !r change which method gets called.

Next Step

This lesson measured that the notation str(n) calls __str__ — meaning turning an object into text is itself a protocol call. The same question can be asked in the reverse direction: what does int("12") call, what does float(n) ask of an object, and what happens when a conversion with no counterpart, like int("north"), is requested? The next lesson counts which special methods conversion rests on, and shows that a failed conversion produces not a return value but an exception.