Procedural Macros

The previous lesson showed that a macro definition and the macro’s call can stand in the same file: the compiler read the definition, expanded it, and checked the source it produced, all in a single pass. This lesson measures the same “still holds” class from an entirely different setup. A procedural macro is an ordinary Rust function that processes source text by hand and returns new source text — but this function has to be compiled on its own, in a separate crate, and the source that uses it can only work once the compiler finds that crate as an already-compiled binary. Generation no longer happens in the same pass; it happens across two separate compilation steps.

This two-step nature shows up in the language’s own vocabulary too. Declarative macros worked with the language’s own pattern-matching mechanism — the way a match arm meets a value, the incoming tokens were matched against a pattern and a fixed template was put in its place. A procedural macro is not tied to such a pattern-matching mechanism: it can process the token stream it receives as input like any Rust code it likes — it can build a loop, write a condition, accumulate into a helper data structure — and in the end produce and return a new token stream. This is a different kind of power from a declarative macro’s syntactic templating: generation is now programmable. The cost of this power starts right here — programmable generation requires an independent program the compiler can run, and an independent program cannot exist in the same pass as the rest of the source.

This difference can be summed up in a single question: a declarative macro only matches a pattern and substitutes, a procedural macro runs. This gap lines up exactly with this course’s measurement axis, which looks not at which tool produced the core but at when and where generation actually happens.

Why Generation Moves to a Separate Step

A procedural macro function takes a token stream (TokenStream) as input and returns another token stream as output. This transformation does not run at runtime; it runs at compile time — but for it to run, the compiler itself has to be able to call this function like a subroutine while it is still processing the source it is compiling. For a function to be callable in the middle of the compilation process, that function has to already exist as a linkable binary before compilation starts — it cannot be compiled in the same pass, at the same time, as the rest of the source. This is why the language itself imposes a restriction: a crate containing a procedural macro must be compiled with a separate crate type named proc-macro, and every crate that uses it must take that crate as a dependency in its own build, already compiled.

This is a fundamental split from the declarative macro in the previous lesson. There, the macro definition and its call could stand in the same file, because expansion was a textual substitution and the compiler could do it in a single pass. Here, generation is not a text substitution but the output of a program that runs; that program first has to be brought into existence on its own, through a complete compilation step.

Rust has three separate forms of procedural macro: the form that takes a call-style invocation as input, the attribute macro that takes the item an attribute line affects, and the derive macro that expands the derive attribute itself. All three carry the same restriction — all three must be compiled in their own crate — and all three share the same kind of function signature: a token stream in, a token stream out. An attribute macro is triggered by a line written above a function or struct declaration and takes the item beneath it as a token stream; a derive macro, meanwhile, expands the #[derive(...)] line itself, as seen in the Derived Traits lesson, and can only produce a new implementation — it cannot change the item it is written on. This lesson measures the plainest form, the call-style one, because this form can produce a fixed output without even parsing its input, and that keeps the measurement’s focus from drifting away from the generation mechanics.

Measurement: Two-Stage Compilation

The measurement is set up from inside the program. A Rust program calls the compiler twice with std::process::Command: first it compiles the procedural macro crate with --crate-type proc-macro, then it compiles the source that uses it, linking it to that crate with --extern, and finally it runs the resulting binary and checks its output against what is expected. The pattern matches the one the Rust’s Design Goals lesson adopted from the Go Fundamentals course: the compiler is called as a subprocess, and only its exit status and output are read; error text, path, duration, and machine name are never read.

The measurement itself provides two things at once. First, the measurement is deterministic: both sources sit inside the program as fixed string constants, neither is read from outside, so the same program compiles the same two sources on every run and prints the same three lines. Second, the measurement erases its own trace: the working directory is opened inside a temporary folder, and the program’s last line removes that folder entirely — it leaves no persistent file behind and does not let one run’s leftovers affect the next.

MM4. The two crates’ source text is kept inside the program as constant strings. The crate source begins with the line extern crate proc_macro; — this brings in the one non-third-party crate that sits ready in the compiler’s sysroot. The generated token stream is built by parsing a string constant (.parse().unwrap()) rather than by hand-building a token tree; this is the most direct route that requires no dependency outside the crate.

Real procedural macros mostly process the input token stream too — they look at which name, which fields appear, and shape the output accordingly. Parsing a token stream by hand, by building a token tree, is possible without a third-party crate as well, but this lesson never reads the input at all: the _input parameter is left unused, because what is being measured is not token-handling skill but which compilation step generation happens in. A macro that reads its input and branches on it would be subject to the same two-stage compilation; reading the input does not change where generation happens, only the content of the generated text.

// generation.rs — a procedural macro crate is compiled from inside the program, then
// the source that uses it is compiled and run with a second call.
use std::env;
use std::fs;
use std::path::PathBuf;
use std::process::Command;

const CRATE_SOURCE: &str = r#"
extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro]
pub fn generate_body(_input: TokenStream) -> TokenStream {
    "fn generated_value() -> i32 { 42 }".parse().unwrap()
}
"#;

const CONSUMER_SOURCE: &str = r#"
macro_crate::generate_body!();

fn apply<F: Fn() -> i32>(func: F) -> i32 {
    func()
}

fn main() {
    let direct: i32 = generated_value();
    let inside_generic_bound: i32 = apply(generated_value);
    println!("generated_value={} inside_generic_bound={}", direct, inside_generic_bound);
}
"#;

const EXPECTED_OUTPUT: &str = "generated_value=42 inside_generic_bound=42\n";

fn succeeds(command: &mut Command) -> bool {
    command.output().map(|c| c.status.success()).unwrap_or(false)
}

fn classify(successful: bool) -> &'static str {
    if successful { "compiled" } else { "not compiled" }
}

fn main() {
    let workdir = env::temp_dir().join(format!("procedural-{}", std::process::id()));
    fs::create_dir_all(&workdir).unwrap();

    let crate_path = workdir.join("crate.rs");
    fs::write(&crate_path, CRATE_SOURCE).unwrap();

    let crate_binary: PathBuf = workdir.join(format!("libmacro_crate.{}", env::consts::DLL_EXTENSION));
    let crate_compiled = succeeds(
        Command::new("rustc")
            .args(["--edition", "2021", "--crate-type", "proc-macro", "--crate-name", "macro_crate"])
            .arg("-o").arg(&crate_binary)
            .arg(&crate_path),
    );
    println!("crate: {}", classify(crate_compiled));

    let consumer_path = workdir.join("consumer.rs");
    fs::write(&consumer_path, CONSUMER_SOURCE).unwrap();
    let consumer_binary = workdir.join("consumer");

    let consumer_compiled = crate_compiled
        && succeeds(
            Command::new("rustc")
                .args(["--edition", "2021"])
                .arg("--extern").arg(format!("macro_crate={}", crate_binary.display()))
                .arg("-o").arg(&consumer_binary)
                .arg(&consumer_path),
        );
    println!("consumer: {}", classify(consumer_compiled));

    let matched = consumer_compiled
        && Command::new(&consumer_binary)
            .output()
            .map(|c| c.stdout == EXPECTED_OUTPUT.as_bytes())
            .unwrap_or(false);
    println!("output matched: {}", matched);

    let _ = fs::remove_dir_all(&workdir);
}

crate: compiled
consumer: compiled
output matched: true

MM5. The binary file’s extension is not hard-coded anywhere; env::consts::DLL_EXTENSION gives the platform’s own shared-library extension, and the program reads it and appends it to the file name. This is a way to build a portable example without printing a real machine name or directory path: the extension is never printed on screen, it is only used, invisibly, inside the file name. The --extern macro_crate=<path> binding introduces the binary the first compilation produced to the second compilation under a crate name — the consumer source writes its call to the generating macro exactly as it would write a call into any standard-library crate, prefixing it with the crate name.

This binding step requires no package registry and no dependency resolver; --extern only maps a name to a file path, and the compiler reads the binary at that path directly. The measurement is therefore set up without opening a network connection or an outside registry, using only two rustc calls and a file path passed between them — the course’s rule against using a third-party crate is not broken here either: the only crate linked is macro_crate, which the program produces itself.

The result is three lines: the crate compiled, the consumer source compiled, the output matched what was expected. All three are classifying results — none of them carries error text, duration, or a path. The consumer source uses the generated function in two separate ways: once through a direct call, and once from inside the F: Fn() -> i32 bound of the apply function. Both give the same result (42) and both compile — the generated body, exactly as seen in the previous lesson, is checked here too as if it were an ordinary function inside a generic bound. The only difference is where generation happens: in the previous lesson, generation was inside the same pass; here, generation is already finished and enters the second compilation as a compiled binary.

MM6. The second line of the consumer source, which tests the generic bound, is not an accidental flourish: this lesson deliberately repeats the same test to show that the “still holds” claim from the previous lesson is not specific to that single setup and remains true once generation moves to a separate compilation step. The two lessons use different generation mechanisms — one a textual substitution, the other the running of a separate program — but the source each one produces has the same status in the compiler’s eyes: ordinary Rust source.

This similarity has a limit. The compiles function built in the Go’s Design Goals lesson tested a single source with a single compiler call; here, two sources and two sequential calls are needed, because the second call depends on the first one’s output. The consumer_compiled variable starting with crate_compiled && makes this dependency visible right in the code: if the crate did not compile, there is no point trying to compile the consumer source, because there is no binary to link against. The same logic is repeated once more in the matched variable — if the consumer source did not compile, there is no binary to run either. This chained short-circuit also documents the order of generation in the code itself: once one step fails, the following steps are never even attempted, and the false value on the last line just says “the chain did not complete” without distinguishing which step it stopped at.

The Derived Traits lesson showed that an attribute line can give rise to an implementation with no body at all in the source, and that how many times this body ran could not be counted. This lesson does not repeat that counting claim: what is measured here is not how many times something ran, but which compilation step generation happens in, and whether that step actually passed. The two lessons ask different questions but share the same observation: a body that is not hand-written in the source is, from the compiler’s point of view, no different from an ordinary body.

The Limiting Measurement

A procedural macro has to live in its own crate, and this is a restriction of the language, not a preference. A piece of code that tries to compile the crate source (CRATE_SOURCE) and the consumer source (CONSUMER_SOURCE) in a single file with a single compilation call is rejected by the compiler: the proc-macro crate type can only export macro definitions, and a crate cannot call the procedural macro it defines itself from within itself. This rejection is described in prose rather than placed in a code block, because under the course’s general rule, source that does not compile never goes into a code block.

This restriction directly shaped the measurement’s design. In the previous lesson, a single rustc call was enough, because the macro definition and its call were part of the same pass. Here, two separate rustc calls are needed, because the crate first has to exist — the second call takes the binary the first one produced as an input. The procedural macro falling into the “still holds” class therefore also exposes an architectural decision of the language: the code the compiler runs during expansion has to exist before, and separately from, the source being compiled itself. Checking the generated body does not change, but when and in how many stages that checking happens does.

The second face of the restriction is in what the crate is allowed to export. A crate compiled with the proc-macro crate type can only export procedural macro definitions; an ordinary pub fn or a pub struct cannot be exported from the same crate. This is an extension of the rejection in the previous paragraph: the crate type determines the crate’s role, and that role is limited to “macro generation” alone. If a project wants to offer both an ordinary function and a procedural macro, it has to split the two into two separate crates — one of the proc-macro type, the other an ordinary library crate. This split is followed in the measurement’s setup as well: crate.rs exports nothing but the generating macro.

This restriction carries one more cost: a two-crate structure always requires more compilation steps than a single-crate one. In the previous lesson, when a change was made, the compiler read a single file and produced a single binary; here, even the smallest change to the crate source requires the crate to be recompiled first, then the consumer source to be recompiled against that new binary. This is the price of generation being programmable: every generation form that goes beyond a textual substitution brings its own separate compilation timeline along with it.

Summary

• A procedural macro is an ordinary Rust function that processes an input token stream and returns an output token stream; this function must be compiled in its own proc-macro crate, and only macro definitions can be exported from that crate.
• The measurement was set up from inside the program: the compiler was called twice, in sequence, and only the compiled/not-compiled classification and the output match were printed — error text, path, duration, and machine name were never read.
• The generated body falls into the still holds class in this setup too: it is checked like ordinary source both when it is called directly and when it is used inside a generic bound; the difference is that generation happens not in the same pass but in a separate compilation step.
• A procedural macro having to live in its own crate is not a preference but a restriction of the language: this is why the measurement required two separate, sequential compilation calls instead of one.
• This restriction carries a cost: even the smallest change to the crate source requires both the crate and the source that uses it to be recompiled.

Next Step

These two lessons showed how generated source is checked in two different setups: once through a textual substitution in the same pass, once through a program run in a separate compilation step. Neither has yet asked a question the two share — why generation is preferred over writing by hand. The next lesson, which is also this topic’s final lesson, takes up that question: the same structure is built separately with a macro-written version and a hand-written version, shown to produce the same result, and the macro version is measured for how many lines it shortens the same job by. It is then shown, with an example, that this shortening carries no guarantee of correctness, only a gain in readability.