Lexical analysis

Any sequence of identifier characters followed by an exclamation point (exclamation mark, UK English) will be tokenized as a MACRO_NAME.

Statement form

macro_stmt = MACRO_NAME testlist [ "import" NAME ] [ "as"  NAME ] [ ":" NEWLINE suite ]

Expression form

macro_expr = MACRO_NAME "(" testlist ")"

Resolving ambiguity

The statement form of a macro takes precedence, so that the code macro_name!(x) will be parsed as a macro statement, not as an expression statement containing a macro expression.

Compilation

Upon encountering a macro during translation to bytecode, the code generator will look up the macro processor registered for the macro, and pass the AST, rooted at the macro to the processor function. The returned AST will then be substituted for the original tree.

For macros with multiple names, several trees will be passed to the macro processor, but only one will be returned and substituted, shorting the enclosing block of statements.

This process can be repeated, to enable macros to return AST nodes including other macros.

The compiler will not look up a macro processor until that macro is reached, so that inner macros do not need to have processors registered. For example, in a switch macro, the case and default macros wouldn’t need processors registered as they would be eliminated by the switch processor.

To enable definition of macros to be imported, the macros import! and from! are predefined. They support the following syntax:

"import!" dotted_name "as" name

"from!" dotted_name "import" name [ "as" name ]

The import! macro performs a compile-time import of dotted_name to find the macro processor, then registers it under name for the scope currently being compiled.

The from! macro performs a compile-time import of dotted_name.name to find the macro processor, then registers it under name (using the name following “as”, if present) for the scope currently being compiled.

Note that, since import! and from! only define the macro for the scope in which the import is present, all uses of a macro must be preceded by an explicit import! or from! to improve clarity.

For example, to import the macro “compile” from “my.compiler”:

from! my.compiler import compile

Defining macro processors

A macro processor is defined by a four-tuple, consisting of (func, kind, version, additional_names)

• func must be a callable that takes len(additional_names)+1 arguments, all of which are abstract syntax trees, and returns a single abstract syntax tree.
• kind must be one of the following:
  • macros.STMT_MACRO A statement macro where the body of the macro is indented. This is the only form allowed to have additional names.
  • macros.SIBLING_MACRO A statement macro where the body of the macro is the next statement is the same block. The following statement is moved into the macro as its body.
  • macros.EXPR_MACRO An expression macro.
• version is used to track versions of macros, so that generated bytecodes can be correctly cached. It must be an integer.
• additional_names are the names of the additional parts of the macro, and must be a tuple of strings.

# (func, _ast.STMT_MACRO, VERSION, ())
stmt_macro!:
    multi_statement_body

# (func, _ast.SIBLING_MACRO, VERSION, ())
sibling_macro!
single_statement_body

# (func, _ast.EXPR_MACRO, VERSION, ())
x = expr_macro!(...)

# (func, _ast.STMT_MACRO, VERSION, ("subsequent_macro_part",))
multi_part_macro!:
    multi_statement_body
subsequent_macro_part!:
    multi_statement_body

The compiler will check that the syntax used matches the declared kind.

For convenience, the decorator macro_processor is provided in the macros module to mark a function as a macro processor:

def macro_processor(kind, version, *additional_names):
    def deco(func):
        return func, kind, version, additional_names
    return deco

Which can be used to help declare macro processors, for example:

@macros.macro_processor(macros.STMT_MACRO, 1_08)
def switch(astnode):
    ...

AST extensions

Two new AST nodes will be needed to express macros, macro_stmt and macro_expr.

class macro_stmt(_ast.stmt):

    _fields = "name", "args", "importname", "asname", "body"

class macro_expr(_ast.expr):

    _fields = "name", "args"

In addition, macro processors will needs a means to express control flow or side-effecting code, that produces a value. To support this, a new ast node, called stmt_expr, that combines a statement and expression will be added. This new ast node will be a subtype of expr, but include a statement to allow side effects. It will be compiled to bytecode by compiling the statement, then compiling the value.

class stmt_expr(_ast.expr):

    _fields = "stmt", "value"

Hygiene and debugging

Macro processors will often need to create new variables. Those variables need to named in such as way as to avoid contaminating the original code and other macros. No rules for naming will be enforced, but to ensure hygiene and help debugging, the following naming scheme is recommended:

• All generated variable names should start with a $
• Purely artificial variable names should start $$mname where mname is the name of the macro.
• Variables derived from real variables should start $vname where vname is the name of the variable.
• All variable names should include the line number and the column offset, separated by an underscore.

Examples:

• Purely generated name: $$macro_17_0
• Name derived from a variable for an expression macro: $var_12_5

Compile-time-checked data structures

It is common to encode tables of data in Python as large dictionaries. However, these can be hard to maintain and error prone. Macros allow such data to be written in a more readable format. Then, at compile time, the data can be verified and converted to an efficient format.

For example, suppose we have a two dictionary literals mapping codes to names, and vice versa. This is error prone, as the dictionaries may have duplicate keys, or one table may not be the inverse of the other. A macro could generate the two mappings from a single table and, at the same time, verify that no duplicates are present.

color_to_code = {
    "red": 1,
    "blue": 2,
    "green": 3,
}

code_to_color = {
    1: "red",
    2: "blue",
    3: "yellow", # error
}

would become:

bijection! color_to_code, code_to_color:
    "red" = 1
    "blue" = 2
    "green" = 3

Domain-specific extensions

Where I see macros having real value is in specific domains, not in general-purpose language features.

For example, parsers. Here’s part of a parser definition for Python, using macros:

choice! single_input:
    NEWLINE
    simple_stmt
    sequence!:
        compound_stmt
        NEWLINE

Compilers

Runtime compilers, such as numba have to reconstitute the Python source, or attempt to analyze the bytecode. It would be simpler and more reliable for them to get the AST directly:

from! my.jit.library import jit

jit!
def func():
    ...

Matching symbolic expressions

When matching something representing syntax, such a Python ast node, or a sympy expression, it is convenient to match against the actual syntax, not the data structure representing it. For example, a calculator could be implemented using a domain-specific macro for matching syntax:

from! ast_matcher import match

def calculate(node):
    if isinstance(node, Num):
        return node.n
    match! node:
        case! a + b:
            return calculate(a) + calculate(b)
        case! a - b:
            return calculate(a) - calculate(b)
        case! a * b:
            return calculate(a) * calculate(b)
        case! a / b:
            return calculate(a) / calculate(b)

Which could be converted to:

def calculate(node):
    if isinstance(node, Num):
        return node.n
    $$match_4_0 = node
    if isinstance($$match_4_0, _ast.Add):
        a, b = $$match_4_0.left, $$match_4_0.right
        return calculate(a) + calculate(b)
    elif isinstance($$match_4_0, _ast.Sub):
        a, b = $$match_4_0.left, $$match_4_0.right
        return calculate(a) - calculate(b)
    elif isinstance($$match_4_0, _ast.Mul):
        a, b = $$match_4_0.left, $$match_4_0.right
        return calculate(a) * calculate(b)
    elif isinstance($$match_4_0, _ast.Div):
        a, b = $$match_4_0.left, $$match_4_0.right
        return calculate(a) / calculate(b)

Zero-cost markers and annotations

Annotations, either decorators or PEP 3107 function annotations, have a runtime cost even if they serve only as markers for checkers or as documentation.

@do_nothing_marker
def foo(...):
    ...

can be replaced with the zero-cost macro:

do_nothing_marker!:
def foo(...):
    ...

Protyping language extensions

Although macros would be most valuable for domain-specific extensions, it is possible to demonstrate possible language extensions using macros.

try_!:
    body
finally!:
    closing

try:
    body
except:
    closing
else:
    closing

with! open(filename) as fd:
    return fd.read()

with! open!(filename) as fd:
    return fd.read()

Macro definition macros

Languages that have syntactic macros usually provide a macro for defining macros. This PEP intentionally does not do that, as it is not yet clear what a good design would be, and we want to allow the community to define their own macros.

One possible form could be:

macro_def! name:
    input:
        ... # input pattern, defining meta-variables
    output:
        ... # output pattern, using meta-variables