Item 28: Use macros judiciously

"In some cases it's easy to decide to write a macro instead of a function, because only a macro can do what's needed." – Paul Graham, "On Lisp (Prentice Hall)"

Rust's macro systems allow you to perform metaprogramming: to write code that emits code into your project. This is most valuable when there are chunks of "boilerplate" code that are deterministic and repetitive and that would otherwise need to be kept in sync manually.

Programmers coming to Rust may have previously encountered the macros provided by C/C++'s preprocessor, which perform textual substitution on the tokens of the input text. Rust's macros are a different beast, because they work on either the parsed tokens of the program or on the abstract syntax tree (AST) of the program, rather than just its textual content.

This means Rust macros can be aware of code structure and can consequently avoid entire classes of macro-related footguns. In particular, we see in the following section that Rust's declarative macros are hygienic—they cannot accidentally refer to ("capture") local variables in the surrounding code.

One way to think about macros is to see them as a different level of abstraction in the code. A simple form of abstraction is a function: it abstracts away the differences between different values of the same type, with implementation code that can use any of the features and methods of that type, regardless of the current value being operated on. A generic is a different level of abstraction: it abstracts away the difference between different types that satisfy a trait bound, with implementation code that can use any of the methods provided by the trait bounds, regardless of the current type being operated on.

A macro abstracts away the difference between different fragments of the program that play the same role (type, identifier, expression, etc.); the implementation can then include any code that makes use of those fragments in the same role.

Rust provides two ways to define macros:

  • Declarative macros, also known as "macros by example", allow the insertion of arbitrary Rust code into the program, based on the input parameters to the macro (which are categorized according to their role in the AST).
  • Procedural macros allow the insertion of arbitrary Rust code into the program, based on the parsed tokens of the source code. This is most commonly used for derive macros, which can generate code based on the contents of data structure definitions.

Declarative Macros

Although this Item isn't the place to reproduce the documentation for declarative macros, a few reminders of details to watch out for are in order.

First, be aware that the scoping rules for using a declarative macro are different than for other Rust items. If a declarative macro is defined in a source code file, only the code after the macro definition can make use of it:

fn before() {
    println!("[before] square {} is {}", 2, square!(2));
}

/// Macro that squares its argument.
macro_rules! square {
    { $e:expr } => { $e * $e }
}

fn after() {
    println!("[after] square {} is {}", 2, square!(2));
}
error: cannot find macro `square` in this scope
 --> src/main.rs:4:45
  |
4 |     println!("[before] square {} is {}", 2, square!(2));
  |                                             ^^^^^^
  |
  = help: have you added the `#[macro_use]` on the module/import?

The #[macro_export] attribute makes a macro more widely visible, but this also has an oddity: a macro appears at the top level of a crate, even if it's defined in a module:

#![allow(unused)]
fn main() {
mod submod {
    #[macro_export]
    macro_rules! cube {
        { $e:expr } => { $e * $e * $e }
    }
}

mod user {
    pub fn use_macro() {
        // Note: *not* `crate::submod::cube!`
        let cubed = crate::cube!(3);
        println!("cube {} is {}", 3, cubed);
    }
}
}

Rust's declarative macros are what's known as hygienic: the expanded code in the body of the macro is not allowed to make use of local variable bindings. For example, a macro that assumes that some variable x exists:

#![allow(unused)]
fn main() {
// Create a macro that assumes the existence of a local `x`.
macro_rules! increment_x {
    {} => { x += 1; };
}
}

will trigger a compilation failure when it is used:

let mut x = 2;
increment_x!();
println!("x = {}", x);
error[E0425]: cannot find value `x` in this scope
   --> src/main.rs:55:13
    |
55  |     {} => { x += 1; };
    |             ^ not found in this scope
...
314 |     increment_x!();
    |     -------------- in this macro invocation
    |
    = note: this error originates in the macro `increment_x`

This hygienic property means that Rust's macros are safer than C preprocessor macros. However, there are still a couple of minor gotchas to be aware of when using them.

The first is to realize that even if a macro invocation looks like a function invocation, it's not. A macro generates code at the point of invocation, and that generated code can perform manipulations of its arguments:

#![allow(unused)]
fn main() {
macro_rules! inc_item {
    { $x:ident } => { $x.contents += 1; }
}
}

This means that the normal intuition about whether parameters are moved or &-referred-to doesn't apply:

let mut x = Item { contents: 42 }; // type is not `Copy`

// Item is *not* moved, despite the (x) syntax,
// but the body of the macro *can* modify `x`.
inc_item!(x);

println!("x is {x:?}");
x is Item { contents: 43 }

This becomes clear if we remember that the macro inserts code at the point of invocation—in this case, adding a line of code that increments x.contents. The cargo-expand tool shows the code that the compiler sees, after macro expansion:

let mut x = Item { contents: 42 };
x.contents += 1;
{
    ::std::io::_print(format_args!("x is {0:?}\n", x));
};

The expanded code includes the modification in place, via the owner of the item, not a reference. (It's also interesting to see the expanded version of println!, which relies on the format_args! macro, to be discussed shortly.)1

So the exclamation mark serves as a warning: the expanded code for the macro may do arbitrary things to or with its arguments.

The expanded code can also include control flow operations that aren't visible in the calling code, whether they be loops, conditionals, return statements, or use of the ? operator. Obviously, this is likely to violate the principle of least astonishment, so prefer macros whose behavior aligns with normal Rust where possible and appropriate. (On the other hand, if the purpose of the macro is to allow weird control flow, go for it! But help out your users by making sure the control flow behavior is clearly documented.)

For example, consider a macro (for checking HTTP status codes) that silently includes a return in its body:

#![allow(unused)]
fn main() {
/// Check that an HTTP status is successful; exit function if not.
macro_rules! check_successful {
    { $e:expr } => {
        if $e.group() != Group::Successful {
            return Err(MyError("HTTP operation failed"));
        }
    }
}
}

Code that uses this macro to check the result of some kind of HTTP operation can end up with control flow that's somewhat obscure:

let rc = perform_http_operation();
check_successful!(rc); // may silently exit the function

// ...

An alternative version of the macro that generates code that emits a Result:

#![allow(unused)]
fn main() {
/// Convert an HTTP status into a `Result<(), MyError>` indicating success.
macro_rules! check_success {
    { $e:expr } => {
        match $e.group() {
            Group::Successful => Ok(()),
            _ => Err(MyError("HTTP operation failed")),
        }
    }
}
}

gives code that's easier to follow:

let rc = perform_http_operation();
check_success!(rc)?; // error flow is visible via `?`

// ...

The second thing to watch out for with declarative macros is a problem shared with the C preprocessor: if the argument to a macro is an expression with side effects, beware of repeated use of the argument in the macro. The square! macro defined earlier takes an arbitrary expression as an argument and then uses that argument twice, which can lead to surprises:

let mut x = 1;
let y = square!({
    x += 1;
    x
});
println!("x = {x}, y = {y}");
// output: x = 3, y = 6

Assuming that this behavior isn't intended, one way to fix it is simply to evaluate the expression once and assign the result to a local variable:

#![allow(unused)]
fn main() {
macro_rules! square_once {
    { $e:expr } => {
        {
            let x = $e;
            x*x // Note: there's a detail here to be explained later...
        }
    }
}
// output now: x = 2, y = 4
}

The other alternative is not to allow an arbitrary expression as input to the macro. If the expr syntax fragment specifier is replaced with an ident fragment specifier, then the macro will only accept identifiers as inputs, and the attempt to feed it an arbitrary expression will no longer compile.

Formatting Values

One common style of declarative macro involves assembling a message that includes various values from the current state of the code. For example, the standard library includes format! for assembling a String, println! for printing to standard output, eprintln! for printing to standard error, and so on. The documentation describes the syntax of the formatting directives, which are roughly equivalent to C's printf statement. However, the format arguments are type safe and checked at compile time, and the implementations of the macro use the Display and Debug traits described in Item 10 to format individual values.2

You can (and should) use the same formatting syntax for any macros of your own that perform a similar function. For example, the logging macros provided by the log crate use the same syntax as format!. To do this, use format_args! for macros that perform argument formatting rather than attempting to reinvent the wheel:

#![allow(unused)]
fn main() {
/// Log an error including code location, with `format!`-like arguments.
/// Real code would probably use the `log` crate.
macro_rules! my_log {
    { $($arg:tt)+ } => {
        eprintln!("{}:{}: {}", file!(), line!(), format_args!($($arg)+));
    }
}
}
let x = 10u8;
// Format specifiers:
// - `x` says print as hex
// - `#` says prefix with '0x'
// - `04` says add leading zeroes so width is at least 4
//   (this includes the '0x' prefix).
my_log!("x = {:#04x}", x);
src/main.rs:331: x = 0x0a

Procedural Macros

Rust also supports procedural macros, often known as proc macros. Like a declarative macro, a procedural macro has the ability to insert arbitrary Rust code into the program's source code. However, the inputs to the macro are no longer just the specific arguments passed to it; instead, a procedural macro has access to the parsed tokens corresponding to some chunk of the original source code. This gives a level of expressive power that approaches the flexibility of dynamic languages such as Lisp—but still with compile-time guarantees. It also helps mitigate the limitations of reflection in Rust, as discussed in Item 19.

Procedural macros must be defined in a separate crate (of crate type proc-macro) from where they are used, and that crate will almost certainly need to depend on either proc-macro (provided by the standard toolchain) or proc-macro2 (provided by David Tolnay) as a support library, to make it possible to work with the input tokens.

There are three distinct types of procedural macro:

  • Function-like macros: Invoked with an argument
  • Attribute macros: Attached to some chunk of syntax in the program
  • Derive macros: Attached to the definition of a data structure

Function-like macros

Function-like procedural macros are invoked with an argument, and the macro definition has access to the parsed tokens that make up the argument, and emits arbitrary tokens as a result. Note that the previous sentence says "argument", singular—even if a function-like macro is invoked with what looks like multiple arguments:

my_func_macro!(15, x + y, f32::consts::PI);

the macro itself receives a single argument, which is a stream of parsed tokens. A macro implementation that just prints (at compile time) the contents of the stream:

use proc_macro::TokenStream;

// Function-like macro that just prints (at compile time) its input stream.
#[proc_macro]
pub fn my_func_macro(args: TokenStream) -> TokenStream {
    println!("Input TokenStream is:");
    for tt in args {
        println!("  {tt:?}");
    }
    // Return an empty token stream to replace the macro invocation with.
    TokenStream::new()
}

shows the stream corresponding to the input:

Input TokenStream is:
  Literal { kind: Integer, symbol: "15", suffix: None,
            span: #0 bytes(10976..10978) }
  Punct { ch: ',', spacing: Alone, span: #0 bytes(10978..10979) }
  Ident { ident: "x", span: #0 bytes(10980..10981) }
  Punct { ch: '+', spacing: Alone, span: #0 bytes(10982..10983) }
  Ident { ident: "y", span: #0 bytes(10984..10985) }
  Punct { ch: ',', spacing: Alone, span: #0 bytes(10985..10986) }
  Ident { ident: "f32", span: #0 bytes(10987..10990) }
  Punct { ch: ':', spacing: Joint, span: #0 bytes(10990..10991) }
  Punct { ch: ':', spacing: Alone, span: #0 bytes(10991..10992) }
  Ident { ident: "consts", span: #0 bytes(10992..10998) }
  Punct { ch: ':', spacing: Joint, span: #0 bytes(10998..10999) }
  Punct { ch: ':', spacing: Alone, span: #0 bytes(10999..11000) }
  Ident { ident: "PI", span: #0 bytes(11000..11002) }

The low-level nature of this input stream means that the macro implementation has to do its own parsing. For example, separating out what appear to be separate arguments to the macro involves looking for TokenTree::Punct tokens that hold the commas dividing the arguments. The syn crate (from David Tolnay) provides a parsing library that can help with this, as described in a section that follows.

Because of this, it's usually easier to use a declarative macro than a function-like procedural macro, because the expected structure of the macro's inputs can be expressed in the matching pattern.

The flip side of this need for manual processing is that function-like proc macros have the flexibility to accept inputs that don't parse as normal Rust code. That's not often needed (or sensible), so function-like macros are comparatively rare as a result.

Attribute macros

Attribute macros are invoked by placing them before some item in the program, and the parsed tokens for that item are the input to the macro. The macro can again emit arbitrary tokens as output, but the output is typically some transformation of the input.

For example, an attribute macro can be used to wrap the body of a function:

#[log_invocation]
fn add_three(x: u32) -> u32 {
    x + 3
}

so that invocations of the function are logged:

let x = 2;
let y = add_three(x);
println!("add_three({x}) = {y}");
log: calling function 'add_three'
log: called function 'add_three' => 5
add_three(2) = 5

The implementation of this macro is too large to include here, because the code needs to check the structure of the input tokens and to build up the new output tokens, but the syn crate can again help with this processing.

Derive macros

The final type of procedural macro is the derive macro, which allows generated code to be automatically attached to a data structure definition (a struct, enum, or union). This is similar to an attribute macro but there are a few derive-specific aspects to be aware of.

The first is that derive macros add to the input tokens, instead of replacing them altogether. This means that the data structure definition is left intact but the macro has the opportunity to append related code.

The second is that a derive macro can declare associated helper attributes, which can then be used to mark parts of the data structure that need special processing. For example, serde's Deserialize derive macro has a serde helper attribute that can provide metadata to guide the deserialization process:

fn generate_value() -> String {
    "unknown".to_string()
}

#[derive(Debug, Deserialize)]
struct MyData {
    // If `value` is missing when deserializing, invoke
    // `generate_value()` to populate the field instead.
    #[serde(default = "generate_value")]
    value: String,
}

The final aspect of derive macros to be aware of is that the syn crate can take care of much of the heavy lifting involved in parsing the input tokens into the equivalent nodes in the AST. The syn::parse_macro_input! macro converts the tokens into a syn::DeriveInput data structure that describes the content of the item, and DeriveInput is much easier to deal with than a raw stream of tokens.

In practice, derive macros are the most commonly encountered type of procedural macro—the ability to generate field-by-field (for structs) or variant-by-variant (for enums) implementations allows for a lot of functionality to be provided with little effort from the programmer—for example, by adding a single line like #[derive(Debug, Clone, PartialEq, Eq)].

Because the derived implementations are auto-generated, it also means that the implementations automatically stay in sync with the data structure definition. For example, if you were to add a new field to a struct, a manual implementation of Debug would need to be manually updated, whereas an automatically derived version would display the new field with no additional effort (or would fail to compile if that wasn't possible).

When to Use Macros

The primary reason to use macros is to avoid repetitive code—especially repetitive code that would otherwise have to be manually kept in sync with other parts of the code. In this respect, writing a macro is just an extension of the same kind of generalization process that normally forms part of programming:

  • If you repeat exactly the same code for multiple values of a specific type, encapsulate that code into a common function and call the function from all of the repeated places.
  • If you repeat exactly the same code for multiple types, encapsulate that code into a generic with a trait bound and use the generic from all of the repeated places.
  • If you repeat the same structure of code in multiple places, encapsulate that code into a macro and use the macro from all of the repeated places.

For example, avoiding repetition for code that works on different enum variants can be done only by a macro:

enum Multi {
    Byte(u8),
    Int(i32),
    Str(String),
}

/// Extract copies of all the values of a specific enum variant.
#[macro_export]
macro_rules! values_of_type {
    { $values:expr, $variant:ident } => {
        {
            let mut result = Vec::new();
            for val in $values {
                if let Multi::$variant(v) = val {
                    result.push(v.clone());
                }
            }
            result
        }
    }
}

fn main() {
    let values = vec![
        Multi::Byte(1),
        Multi::Int(1000),
        Multi::Str("a string".to_string()),
        Multi::Byte(2),
    ];

    let ints = values_of_type!(&values, Int);
    println!("Integer values: {ints:?}");

    let bytes = values_of_type!(&values, Byte);
    println!("Byte values: {bytes:?}");

    // Output:
    //   Integer values: [1000]
    //   Byte values: [1, 2]
}

Another scenario where macros help avoid manual repetition is when information about a collection of data values would otherwise be spread out across different areas of the code.

For example, consider a data structure that encodes information about HTTP status codes; a macro can help keep all of the related information together:

// http.rs module

#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum Group {
    Informational, // 1xx
    Successful,    // 2xx
    Redirection,   // 3xx
    ClientError,   // 4xx
    ServerError,   // 5xx
}

// Information about HTTP response codes.
http_codes! {
    Continue           => (100, Informational, "Continue"),
    SwitchingProtocols => (101, Informational, "Switching Protocols"),
    // ...
    Ok                 => (200, Successful, "Ok"),
    Created            => (201, Successful, "Created"),
    // ...
}

The macro invocation holds all the related information—numeric value, group, description—for each HTTP status code, acting as a kind of domain-specific language (DSL) holding the source of truth for the data.

The macro definition then describes the generated code; each line of the form $( ... )+ expands to multiple lines in the generated code, one per argument to the macro:

#![allow(unused)]
fn main() {
macro_rules! http_codes {
    { $( $name:ident => ($val:literal, $group:ident, $text:literal), )+ } => {
        #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
        #[repr(i32)]
        enum Status {
            $( $name = $val, )+
        }
        impl Status {
            fn group(&self) -> Group {
                match self {
                    $( Self::$name => Group::$group, )+
                }
            }
            fn text(&self) -> &'static str {
                match self {
                    $( Self::$name => $text, )+
                }
            }
        }
        impl core::convert::TryFrom<i32> for Status {
            type Error = ();
            fn try_from(v: i32) -> Result<Self, Self::Error> {
                match v {
                    $( $val => Ok(Self::$name), )+
                    _ => Err(())
                }
            }
        }
    }
}
}

As a result, the overall output from the macro takes care of generating all of the code that derives from the source-of-truth values:

  • The definition of an enum holding all the variants
  • The definition of a group() method, which indicates which group an HTTP status belongs to
  • The definition of a text() method, which maps a status to a text description
  • An implementation of TryFrom<i32> to convert numbers to status enum values

If an extra value needs to be added later, all that's needed is a single additional line:

ImATeapot => (418, ClientError, "I'm a teapot"),

Without the macro, four different places would have to be manually updated. The compiler would point out some of them (because match expressions need to cover all cases) but not all—TryFrom<i32> could easily be forgotten.

Because macros are expanded in place in the invoking code, they can also be used to automatically emit additional diagnostic information—in particular, by using the standard library's file!() and line!() macros, which emit source code location information:

#![allow(unused)]
fn main() {
macro_rules! log_failure {
    { $e:expr } => {
        {
            let result = $e;
            if let Err(err) = &result {
                eprintln!("{}:{}: operation '{}' failed: {:?}",
                          file!(),
                          line!(),
                          stringify!($e),
                          err);
            }
            result
        }
    }
}
}

When failures occur, the log file then automatically includes details of what failed and where:

use std::convert::TryInto;

let x: Result<u8, _> = log_failure!(512.try_into()); // too big for `u8`
let y = log_failure!(std::str::from_utf8(b"\xc3\x28")); // invalid UTF-8
src/main.rs:340: operation '512.try_into()' failed: TryFromIntError(())
src/main.rs:341: operation 'std::str::from_utf8(b"\xc3\x28")' failed:
                 Utf8Error { valid_up_to: 0, error_len: Some(1) }

Disadvantages of Macros

The primary disadvantage of using a macro is the impact that it has on code readability and maintainability. The preceding "Declarative Macros" section explained that macros allow you to create a DSL to concisely express key features of your code and data. However, this means that anyone reading or maintaining the code now has to understand this DSL—and its implementation in macro definitions—in addition to understanding Rust. For example, the http_codes! example in the previous section creates a Rust enum named Status, but it's not visible in the DSL used for the macro invocation.

This potential impenetrability of macro-based code extends beyond other engineers: various tools that analyze and interact with Rust code may treat the code as opaque, because it no longer follows the syntactical conventions of Rust code. The square_once! macro shown earlier provided one trivial example of this: the body of the macro has not been formatted according to the normal rustfmt rules:

{
    let x = $e;
    // The `rustfmt` tool doesn't really cope with code in
    // macros, so this has not been reformatted to `x * x`.
    x*x
}

Another example is the earlier http_codes! macro, where the DSL uses Group enum variant names like Informational with neither a Group:: prefix nor a use statement, which may confuse some code navigation tools.

Even the compiler itself is less helpful: its error messages don't always follow the chain of macro use and definition. (However, there are parts of the tooling ecosystem [see Item 31] that can help with this, such as David Tolnay's cargo-expand, used earlier.)

Another possible downside for macro use is the possibility of code bloat—a single line of macro invocation can result in hundreds of lines of generated code, which will be invisible to a cursory survey of the code. This is less likely to be a problem when the code is first written, because at that point the code is needed and saves the humans involved from having to write it themselves. However, if the code subsequently stops being necessary, it's not so obvious that there are large amounts of code that could be deleted.

Advice

Although the previous section listed some downsides of macros, they are still fundamentally the right tool for the job when there are different chunks of code that need to be kept consistent but that cannot be coalesced any other way: use a macro whenever it's the only way to ensure that disparate code stays in sync.

Macros are also the tool to reach for when there's boilerplate code to be squashed: use a macro for repeated boilerplate code that can't be coalesced into a function or a generic.

To reduce the impact on readability, try to avoid syntax in your macros that clashes with Rust's normal syntax rules; either make the macro invocation look like normal code or make it look sufficiently different so that no one could confuse the two. In particular, follow these guidelines:

  • Avoid macro expansions that insert references where possible—a macro invocation like my_macro!(&list) aligns better with normal Rust code than my_macro!(list) would.
  • Prefer to avoid nonlocal control flow operations in macros so that anyone reading the code is able to follow the flow without needing to know the details of the macro.

This preference for Rust-like readability sometimes affects the choice between declarative macros and procedural macros. If you need to emit code for each field of a structure, or each variant of an enum, prefer a derive macro to a procedural macro that emits a type (despite the example shown in an earlier section)—it's more idiomatic and makes the code easier to read.

However, if you're adding a derive macro with functionality that's not specific to your project, check whether an external crate already provides what you need (see Item 25). For example, the problem of converting integer values into the appropriate variant of a C-like enum is well-covered: all of enumn::N, num_enum::TryFromPrimitive, num_derive::FromPrimitive, and strum::FromRepr cover some aspect of this problem.

1

An eagle-eyed reader might notice that format_args! still looks like a macro invocation, even after macros have been expanded. That's because it's a special macro that's built into the compiler.

2

The std::fmt module also includes various other traits that are used when displaying data in particular formats. For example, LowerHex is used when an x format specifier indicates that lower-case hexadecimal output is required.