Pseudo-code Overview

Here is an overview of how the modified importers will operate. Details such as logging or handling of errors and invalid states are left out, and C code is presented with a concise Python-like syntax.

The framework that calls the importers is explained in PEP 451 8.

importlib/_bootstrap.py:

class BuiltinImporter:
    def create_module(self, spec):
        module = _imp.create_builtin(spec)

    def exec_module(self, module):
        _imp.exec_dynamic(module)

    def load_module(self, name):
        # use a backwards compatibility shim
        _load_module_shim(self, name)

importlib/_bootstrap_external.py:

class ExtensionFileLoader:
    def create_module(self, spec):
        module = _imp.create_dynamic(spec)

    def exec_module(self, module):
        _imp.exec_dynamic(module)

    def load_module(self, name):
        # use a backwards compatibility shim
        _load_module_shim(self, name)

Python/import.c (the _imp module):

def create_dynamic(spec):
    name = spec.name
    path = spec.origin

    # Find an already loaded module that used single-phase init.
    # For multi-phase initialization, mod is NULL, so a new module
    # is always created.
    mod = _PyImport_FindExtensionObject(name, name)
    if mod:
        return mod

    return _PyImport_LoadDynamicModuleWithSpec(spec)

def exec_dynamic(module):
    if not isinstance(module, types.ModuleType):
        # non-modules are skipped -- PyModule_GetDef fails on them
        return

    def = PyModule_GetDef(module)
    state = PyModule_GetState(module)
    if state is NULL:
        PyModule_ExecDef(module, def)

def create_builtin(spec):
    name = spec.name

    # Find an already loaded module that used single-phase init.
    # For multi-phase initialization, mod is NULL, so a new module
    # is always created.
    mod = _PyImport_FindExtensionObject(name, name)
    if mod:
        return mod

    for initname, initfunc in PyImport_Inittab:
        if name == initname:
            m = initfunc()
            if isinstance(m, PyModuleDef):
                def = m
                return PyModule_FromDefAndSpec(def, spec)
            else:
                # fall back to single-phase initialization
                module = m
                _PyImport_FixupExtensionObject(module, name, name)
                return module

Python/importdl.c:

def _PyImport_LoadDynamicModuleWithSpec(spec):
    path = spec.origin
    package, dot, name = spec.name.rpartition('.')

    # see the "Non-ASCII module names" section for export_hook_name
    hook_name = export_hook_name(name)

    # call platform-specific function for loading exported function
    # from shared library
    exportfunc = _find_shared_funcptr(hook_name, path)

    m = exportfunc()
    if isinstance(m, PyModuleDef):
        def = m
        return PyModule_FromDefAndSpec(def, spec)

    module = m

    # fall back to single-phase initialization
    ....

Objects/moduleobject.c:

def PyModule_FromDefAndSpec(def, spec):
    name = spec.name
    create = None
    for slot, value in def.m_slots:
        if slot == Py_mod_create:
            create = value
    if create:
        m = create(spec, def)
    else:
        m = PyModule_New(name)

    if isinstance(m, types.ModuleType):
        m.md_state = None
        m.md_def = def

    if def.m_methods:
        PyModule_AddFunctions(m, def.m_methods)
    if def.m_doc:
        PyModule_SetDocString(m, def.m_doc)

def PyModule_ExecDef(module, def):
    if isinstance(module, types.module_type):
        if module.md_state is NULL:
            # allocate a block of zeroed-out memory
            module.md_state = _alloc(module.md_size)

    if def.m_slots is NULL:
        return

    for slot, value in def.m_slots:
        if slot == Py_mod_exec:
            value(module)

Module Creation Phase

Creation of the module object – that is, the implementation of ExecutionLoader.create_module – is governed by the Py_mod_create slot.

PyObject* (*PyModuleCreateFunction)(PyObject *spec, PyModuleDef *def)

Module Execution Phase

Module execution – that is, the implementation of ExecutionLoader.exec_module – is governed by “execution slots”. This PEP only adds one, Py_mod_exec, but others may be added in the future.

The execution phase is done on the PyModuleDef associated with the module object. For objects that are not a subclass of PyModule_Type (for which PyModule_GetDef would fail), the execution phase is skipped.

Execution slots may be specified multiple times, and are processed in the order they appear in the slots array. When using the default import machinery, they are processed after import-related attributes specified in PEP 451 1 (such as __name__ or __loader__) are set and the module is added to sys.modules.

int (*PyModuleExecFunction)(PyObject* module)

Legacy Init

The backwards-compatible single-phase initialization continues to be supported. In this scheme, the PyInit function returns a fully initialized module rather than a PyModuleDef object. In this case, the PyInit hook implements the creation phase, and the execution phase is a no-op.

Modules that need to work unchanged on older versions of Python should stick to single-phase initialization, because the benefits it brings can’t be back-ported. Here is an example of a module that supports multi-phase initialization, and falls back to single-phase when compiled for an older version of CPython. It is included mainly as an illustration of the changes needed to enable multi-phase init:

#include <Python.h>

static int spam_exec(PyObject *module) {
    PyModule_AddStringConstant(module, "food", "spam");
    return 0;
}

#ifdef Py_mod_exec
static PyModuleDef_Slot spam_slots[] = {
    {Py_mod_exec, spam_exec},
    {0, NULL}
};
#endif

static PyModuleDef spam_def = {
    PyModuleDef_HEAD_INIT,                      /* m_base */
    "spam",                                     /* m_name */
    PyDoc_STR("Utilities for cooking spam"),    /* m_doc */
    0,                                          /* m_size */
    NULL,                                       /* m_methods */
#ifdef Py_mod_exec
    spam_slots,                                 /* m_slots */
#else
    NULL,
#endif
    NULL,                                       /* m_traverse */
    NULL,                                       /* m_clear */
    NULL,                                       /* m_free */
};

PyMODINIT_FUNC
PyInit_spam(void) {
#ifdef Py_mod_exec
    return PyModuleDef_Init(&spam_def);
#else
    PyObject *module;
    module = PyModule_Create(&spam_def);
    if (module == NULL) return NULL;
    if (spam_exec(module) != 0) {
        Py_DECREF(module);
        return NULL;
    }
    return module;
#endif
}

Built-In modules

Any extension module can be used as a built-in module by linking it into the executable, and including it in the inittab (either at runtime with PyImport_AppendInittab, or at configuration time, using tools like freeze).

To keep this possibility, all changes to extension module loading introduced in this PEP will also apply to built-in modules. The only exception is non-ASCII module names, explained below.

Subinterpreters and Interpreter Reloading

Extensions using the new initialization scheme are expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly, avoiding the issues mentioned in Python documentation 9. The mechanism is designed to make this easy, but care is still required on the part of the extension author. No user-defined functions, methods, or instances may leak to different interpreters. To achieve this, all module-level state should be kept in either the module dict, or in the module object’s storage reachable by PyModule_GetState. A simple rule of thumb is: Do not define any static data, except built-in types with no mutable or user-settable class attributes.

Functions incompatible with multi-phase initialization

The PyModule_Create function will fail when used on a PyModuleDef structure with a non-NULL m_slots pointer. The function doesn’t have access to the ModuleSpec object necessary for multi-phase initialization.

The PyState_FindModule function will return NULL, and PyState_AddModule and PyState_RemoveModule will also fail on modules with non-NULL m_slots. PyState registration is disabled because multiple module objects may be created from the same PyModuleDef.

Module state and C-level callbacks

Due to the unavailability of PyState_FindModule, any function that needs access to module-level state (including functions, classes or exceptions defined at the module level) must receive a reference to the module object (or the particular object it needs), either directly or indirectly. This is currently difficult in two situations:

• Methods of classes, which receive a reference to the class, but not to the class’s module
• Libraries with C-level callbacks, unless the callbacks can receive custom data set at callback registration

Fixing these cases is outside of the scope of this PEP, but will be needed for the new mechanism to be useful to all modules. Proper fixes have been discussed on the import-sig mailing list 7.

As a rule of thumb, modules that rely on PyState_FindModule are, at the moment, not good candidates for porting to the new mechanism.

New Functions

A new function and macro implementing the module creation phase will be added. These are similar to PyModule_Create and PyModule_Create2, except they take an additional ModuleSpec argument, and handle module definitions with non-NULL slots:

PyObject * PyModule_FromDefAndSpec(PyModuleDef *def, PyObject *spec)
PyObject * PyModule_FromDefAndSpec2(PyModuleDef *def, PyObject *spec,
                                    int module_api_version)

A new function implementing the module execution phase will be added. This allocates per-module state (if not allocated already), and always processes execution slots. The import machinery calls this method when a module is executed, unless the module is being reloaded:

PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def)

Another function will be introduced to initialize a PyModuleDef object. This idempotent function fills in the type, refcount, and module index. It returns its argument cast to PyObject*, so it can be returned directly from a PyInit function:

PyObject * PyModuleDef_Init(PyModuleDef *);

Additionally, two helpers will be added for setting the docstring and methods on a module:

int PyModule_SetDocString(PyObject *, const char *)
int PyModule_AddFunctions(PyObject *, PyMethodDef *)

Export Hook Name

As portable C identifiers are limited to ASCII, module names must be encoded to form the PyInit hook name.

For ASCII module names, the import hook is named PyInit_<modulename>, where <modulename> is the name of the module.

For module names containing non-ASCII characters, the import hook is named PyInitU_<encodedname>, where the name is encoded using CPython’s “punycode” encoding (Punycode 4 with a lowercase suffix), with hyphens (“-”) replaced by underscores (“_”).

In Python:

def export_hook_name(name):
    try:
        suffix = b'_' + name.encode('ascii')
    except UnicodeEncodeError:
        suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
    return b'PyInit' + suffix

Examples:

For modules with non-ASCII names, single-phase initialization is not supported.

In the initial implementation of this PEP, built-in modules with non-ASCII names will not be supported.

Module Reloading

Reloading an extension module using importlib.reload() will continue to have no effect, except re-setting import-related attributes.

Due to limitations in shared library loading (both dlopen on POSIX and LoadModuleEx on Windows), it is not generally possible to load a modified library after it has changed on disk.

Use cases for reloading other than trying out a new version of the module are too rare to require all module authors to keep reloading in mind. If reload-like functionality is needed, authors can export a dedicated function for it.

Multiple modules in one library

To support multiple Python modules in one shared library, the library can export additional PyInit* symbols besides the one that corresponds to the library’s filename.

Note that this mechanism can currently only be used to load extra modules, but not to find them. (This is a limitation of the loader mechanism, which this PEP does not try to modify.) To work around the lack of a suitable finder, code like the following can be used:

import importlib.machinery
import importlib.util
loader = importlib.machinery.ExtensionFileLoader(name, path)
spec = importlib.util.spec_from_loader(name, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module

On platforms that support symbolic links, these may be used to install one library under multiple names, exposing all exported modules to normal import machinery.

Testing and initial implementations

For testing, a new built-in module _testmultiphase will be created. The library will export several additional modules using the mechanism described in “Multiple modules in one library”.

The _testcapi module will be unchanged, and will use single-phase initialization indefinitely (or until it is no longer supported).

The array and xx* modules will be converted to use multi-phase initialization as part of the initial implementation.

Backwards Compatibility

Existing modules will continue to be source- and binary-compatible with new versions of Python. Modules that use multi-phase initialization will not be compatible with versions of Python that do not implement this PEP.

The functions init_builtin and load_dynamic will be removed from the _imp module (but not from the imp module).

All changed loaders (BuiltinImporter and ExtensionFileLoader) will remain backwards-compatible; the load_module method will be replaced by a shim.

Internal functions of Python/import.c and Python/importdl.c will be removed. (Specifically, these are _PyImport_GetDynLoadFunc, _PyImport_GetDynLoadWindows, and _PyImport_LoadDynamicModule.)