The C API blocks CPython evolutions

Adding or removing members of C structures is causing multiple backward compatibility issues.

Adding a new member breaks the stable ABI (PEP 384), especially for types declared statically (e.g. static PyTypeObject MyType = {...};). In Python 3.4, the PEP 442 “Safe object finalization” added the tp_finalize member at the end of the PyTypeObject structure. For ABI backward compatibility, a new Py_TPFLAGS_HAVE_FINALIZE type flag was required to announce if the type structure contains the tp_finalize member. The flag was removed in Python 3.8 (bpo-32388).

The PyTypeObject.tp_print member, deprecated since Python 3.0 released in 2009, has been removed in the Python 3.8 development cycle. But the change broke too many C extensions and had to be reverted before 3.8 final release. Finally, the member was removed again in Python 3.9.

C extensions rely on the ability to access structure members, indirectly through the C API, or even directly. Modifying structures like PyListObject cannot be even considered.

The PyTypeObject structure is the one which evolved the most, simply because there was no other way to evolve CPython than modifying it.

A C extension can technically dereference a PyObject* pointer and access PyObject members. This prevents experiments like tagged pointers (storing small values as PyObject* which does not point to a valid PyObject structure).

Replacing Python garbage collector with a tracing garbage collector would also need to remove PyObject.ob_refcnt reference counter, whereas currently Py_INCREF() and Py_DECREF() macros access directly to PyObject.ob_refcnt.

Same CPython design since 1990: structures and reference counting

When the CPython project was created, it was written with one principle: keep the implementation simple enough so it can be maintained by a single developer. CPython complexity grew a lot and many micro-optimizations have been implemented, but CPython core design has not changed.

Members of PyObject and PyTupleObject structures have not changed since the “Initial revision” commit (1990):

#define OB_HEAD \
    unsigned int ob_refcnt; \
    struct _typeobject *ob_type;

typedef struct _object {
    OB_HEAD
} object;

typedef struct {
    OB_VARHEAD
    object *ob_item[1];
} tupleobject;

Only names changed: object was renamed to PyObject and tupleobject was renamed to PyTupleObject.

CPython still tracks Python objects lifetime using reference counting internally and for third party C extensions (through the Python C API).

All Python objects must be allocated on the heap and cannot be moved.

Why is PyPy more efficient than CPython?

The PyPy project is a Python implementation which is 4.2x faster than CPython on average. PyPy developers chose to not fork CPython, but start from scratch to have more freedom in terms of optimization choices.

PyPy does not use reference counting, but a tracing garbage collector which moves objects. Objects can be allocated on the stack (or even not at all), rather than always having to be allocated on the heap.

Objects layouts are designed with performance in mind. For example, a list strategy stores integers directly as integers, rather than objects.

Moreover, PyPy also has a JIT compiler which emits fast code thanks to the efficient PyPy design.

PyPy bottleneck: the Python C API

While PyPy is way more efficient than CPython to run pure Python code, it is as efficient or slower than CPython to run C extensions.

Since the C API requires PyObject* and allows to access directly structure members, PyPy has to associate a CPython object to PyPy objects and maintain both consistent. Converting a PyPy object to a CPython object is inefficient. Moreover, reference counting also has to be implemented on top of PyPy tracing garbage collector.

These conversions are required because the Python C API is too close to the CPython implementation: there is no high-level abstraction. For example, structures members are part of the public C API and nothing prevents a C extension to get or set directly PyTupleObject.ob_item[0] (the first item of a tuple).

See Inside cpyext: Why emulating CPython C API is so Hard (Sept 2018) by Antonio Cuni for more details.

Hide implementation details

Hiding implementation details from the C API has multiple advantages:

• It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations. For example, tagged pointers, and replace the garbage collector with a tracing garbage collector which can move objects.
• Adding new features in CPython becomes easier.
• PyPy should be able to avoid conversions to CPython objects in more cases: keep efficient PyPy objects.
• It becomes easier to implement the C API for a new Python implementation.
• More C extensions will be compatible with Python implementations other than CPython.

Relationship with the limited C API

The PEP 384 “Defining a Stable ABI” is implemented in Python 3.4. It introduces the “limited C API”: a subset of the C API. When the limited C API is used, it becomes possible to build a C extension only once and use it on multiple Python versions: that’s the stable ABI.

The main limitation of the PEP 384 is that C extensions have to opt-in for the limited C API. Only very few projects made this choice, usually to ease distribution of binaries, especially on Windows.

This PEP moves the C API towards the limited C API.

Ideally, the C API will become the limited C API and all C extensions will use the stable ABI, but this is out of this PEP scope.

Summary

• (Completed) Reorganize the C API header files: create Include/cpython/ and Include/internal/ subdirectories.
• (Completed) Move private functions exposing implementation details to the internal C API.
• (Completed) Convert macros to static inline functions.
• (Completed) Add new functions Py_SET_TYPE(), Py_SET_REFCNT() and Py_SET_SIZE(). The Py_TYPE(), Py_REFCNT() and Py_SIZE() macros become functions which cannot be used as l-value.
• (Completed) New C API functions must not return borrowed references.
• (In Progress) Provide pythoncapi_compat.h header file.
• (In Progress) Make structures opaque, add getter and setter functions.
• (Not Started) Deprecate PySequence_Fast_ITEMS().
• (Not Started) Convert PyTuple_GET_ITEM() and PyList_GET_ITEM() macros to static inline functions.

Reorganize the C API header files

The first consumer of the C API was Python itself. There is no clear separation between APIs which must not be used outside Python, and API which are public on purpose.

Header files must be reorganized in 3 API:

• Include/ directory is the limited C API: no implementation details, structures are opaque. C extensions using it get a stable ABI.
• Include/cpython/ directory is the CPython C API: less “portable” API, depends more on the Python version, expose some implementation details, few incompatible changes can happen.
• Include/internal/ directory is the internal C API: implementation details, incompatible changes are likely at each Python release.

The creation of the Include/cpython/ directory is fully backward compatible. Include/cpython/ header files cannot be included directly and are included automatically by Include/ header files when the Py_LIMITED_API macro is not defined.

The internal C API is installed and can be used for specific usage like debuggers and profilers which must access structures members without executing code. C extensions using the internal C API are tightly coupled to a Python version and must be recompiled at each Python version.

STATUS: Completed (in Python 3.8)

The reorganization of header files started in Python 3.7 and was completed in Python 3.8:

• bpo-35134: Add a new Include/cpython/ subdirectory for the “CPython API” with implementation details.
• bpo-35081: Move internal headers to Include/internal/

Move private functions to the internal C API

Private functions which expose implementation details must be moved to the internal C API.

If a C extension relies on a CPython private function which exposes CPython implementation details, other Python implementations have to re-implement this private function to support this C extension.

STATUS: Completed (in Python 3.9)

Private functions moved to the internal C API in Python 3.8:

• _PyObject_GC_TRACK(), _PyObject_GC_UNTRACK()

Macros and functions excluded from the limited C API in Python 3.9:

• _PyObject_SIZE(), _PyObject_VAR_SIZE()
• PyThreadState_DeleteCurrent()
• PyFPE_START_PROTECT(), PyFPE_END_PROTECT()
• _Py_NewReference(), _Py_ForgetReference()
• _PyTraceMalloc_NewReference()
• _Py_GetRefTotal()

Private functions moved to the internal C API in Python 3.9:

• GC functions like _Py_AS_GC(), _PyObject_GC_IS_TRACKED() and _PyGCHead_NEXT()
• _Py_AddToAllObjects() (not exported)
• _PyDebug_PrintTotalRefs(), _Py_PrintReferences(), _Py_PrintReferenceAddresses() (not exported)

Public “clear free list” functions moved to the internal C API and renamed to private functions in Python 3.9:

• PyAsyncGen_ClearFreeLists()
• PyContext_ClearFreeList()
• PyDict_ClearFreeList()
• PyFloat_ClearFreeList()
• PyFrame_ClearFreeList()
• PyList_ClearFreeList()
• PyTuple_ClearFreeList()
• Functions simply removed:
  • PyMethod_ClearFreeList() and PyCFunction_ClearFreeList(): bound method free list removed in Python 3.9.
  • PySet_ClearFreeList(): set free list removed in Python 3.4.
  • PyUnicode_ClearFreeList(): Unicode free list removed in Python 3.3.

Convert macros to static inline functions

Converting macros to static inline functions has multiple advantages:

• Functions have well defined parameter types and return type.
• Functions can use variables with a well defined scope (the function).
• Debugger can be put breakpoints on functions and profilers can display the function name in the call stacks. In most cases, it works even when a static inline function is inlined.
• Functions don’t have macros pitfalls.

Converting macros to static inline functions should only impact very few C extensions that use macros in unusual ways.

For backward compatibility, functions must continue to accept any type, not only PyObject*, to avoid compiler warnings, since most macros cast their parameters to PyObject*.

Python 3.6 requires C compilers to support static inline functions: the PEP 7 requires a subset of C99.

STATUS: Completed (in Python 3.9)

Macros converted to static inline functions in Python 3.8:

• Py_INCREF(), Py_DECREF()
• Py_XINCREF(), Py_XDECREF()
• PyObject_INIT(), PyObject_INIT_VAR()
• _PyObject_GC_TRACK(), _PyObject_GC_UNTRACK(), _Py_Dealloc()

Macros converted to regular functions in Python 3.9:

• Py_EnterRecursiveCall(), Py_LeaveRecursiveCall() (added to the limited C API)
• PyObject_INIT(), PyObject_INIT_VAR()
• PyObject_GET_WEAKREFS_LISTPTR()
• PyObject_CheckBuffer()
• PyIndex_Check()
• PyObject_IS_GC()
• PyObject_NEW() (alias to PyObject_New()), PyObject_NEW_VAR() (alias to PyObject_NewVar())
• PyType_HasFeature() (always call PyType_GetFlags())
• Py_TRASHCAN_BEGIN_CONDITION() and Py_TRASHCAN_END() macros now call functions which hide implementation details, rather than accessing directly members of the PyThreadState structure.

Make structures opaque

The following structures of the C API become opaque:

• PyInterpreterState
• PyThreadState
• PyGC_Head
• PyTypeObject
• PyObject and PyVarObject
• PyTypeObject
• All types which inherit from PyObject or PyVarObject

C extensions must use getter or setter functions to get or set structure members. For example, tuple->ob_item[0] must be replaced with PyTuple_GET_ITEM(tuple, 0).

To be able to move away from reference counting, PyObject must become opaque. Currently, the reference counter PyObject.ob_refcnt is exposed in the C API. All structures must become opaque, since they “inherit” from PyObject. For, PyFloatObject inherits from PyObject:

typedef struct {
    PyObject ob_base;
    double ob_fval;
} PyFloatObject;

Making PyObject fully opaque requires converting Py_INCREF() and Py_DECREF() macros to function calls. This change has an impact on performance. It is likely to be one of the very last changes when making structures opaque.

Making PyTypeObject structure opaque breaks C extensions declaring types statically (e.g. static PyTypeObject MyType = {...};). C extensions must use PyType_FromSpec() to allocate types on the heap instead. Using heap types has other advantages like being compatible with subinterpreters. Combined with PEP 489 “Multi-phase extension module initialization”, it makes a C extension behavior closer to a Python module, like allowing to create more than one module instance.

Making PyThreadState structure opaque requires adding getter and setter functions for members used by C extensions.

STATUS: In Progress (started in Python 3.8)

The PyInterpreterState structure was made opaque in Python 3.8 (bpo-35886) and the PyGC_Head structure (bpo-40241) was made opaque in Python 3.9.

Issues tracking the work to prepare the C API to make following structures opaque:

• PyObject: bpo-39573
• PyTypeObject: bpo-40170
• PyFrameObject: bpo-40421
  • Python 3.9 adds PyFrame_GetCode() and PyFrame_GetBack() getter functions, and moves PyFrame_GetLineNumber to the limited C API.
• PyThreadState: bpo-39947
  • Python 3.9 adds 3 getter functions: PyThreadState_GetFrame(), PyThreadState_GetID(), PyThreadState_GetInterpreter().

Disallow using Py_TYPE() as l-value

The Py_TYPE() function gets an object type, its PyObject.ob_type member. It is implemented as a macro which can be used as an l-value to set the type: Py_TYPE(obj) = new_type. This code relies on the assumption that PyObject.ob_type can be modified directly. It prevents making the PyObject structure opaque.

New setter functions Py_SET_TYPE(), Py_SET_REFCNT() and Py_SET_SIZE() are added and must be used instead.

The Py_TYPE(), Py_REFCNT() and Py_SIZE() macros must be converted to static inline functions which can not be used as l-value.

For example, the Py_TYPE() macro:

#define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)

becomes:

#define _PyObject_CAST_CONST(op) ((const PyObject*)(op))

static inline PyTypeObject* _Py_TYPE(const PyObject *ob) {
    return ob->ob_type;
}

#define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob))

STATUS: Completed (in Python 3.10)

New functions Py_SET_TYPE(), Py_SET_REFCNT() and Py_SET_SIZE() were added to Python 3.9.

In Python 3.10, Py_TYPE(), Py_REFCNT() and Py_SIZE() can no longer be used as l-value and the new setter functions must be used instead.

New C API functions must not return borrowed references

When a function returns a borrowed reference, Python cannot track when the caller stops using this reference.

For example, if the Python list type is specialized for small integers, store directly “raw” numbers rather than Python objects, PyList_GetItem() has to create a temporary Python object. The problem is to decide when it is safe to delete the temporary object.

The general guidelines is to avoid returning borrowed references for new C API functions.

No function returning borrowed references is scheduled for removal by this PEP.

STATUS: Completed (in Python 3.9)

In Python 3.9, new C API functions returning Python objects only return strong references:

• PyFrame_GetBack()
• PyFrame_GetCode()
• PyObject_CallNoArgs()
• PyObject_CallOneArg()
• PyThreadState_GetFrame()

Avoid functions returning PyObject**

The PySequence_Fast_ITEMS() function gives a direct access to an array of PyObject* objects. The function is deprecated in favor of PyTuple_GetItem() and PyList_GetItem().

PyTuple_GET_ITEM() can be abused to access directly the PyTupleObject.ob_item member:

PyObject **items = &PyTuple_GET_ITEM(0);

The PyTuple_GET_ITEM() and PyList_GET_ITEM() macros are converted to static inline functions to disallow that.

STATUS: Not Started

New pythoncapi_compat.h header file

Making structures opaque requires modifying C extensions to use getter and setter functions. The practical issue is how to keep support for old Python versions which don’t have these functions.

For example, in Python 3.10, it is no longer possible to use Py_TYPE() as an l-value. The new Py_SET_TYPE() function must be used instead:

#if PY_VERSION_HEX >= 0x030900A4
    Py_SET_TYPE(&MyType, &PyType_Type);
#else
    Py_TYPE(&MyType) = &PyType_Type;
#endif

This code may ring a bell to developers who ported their Python code base from Python 2 to Python 3.

Python will distribute a new pythoncapi_compat.h header file which provides new C API functions to old Python versions. Example:

#if PY_VERSION_HEX < 0x030900A4
static inline void
_Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
{
    ob->ob_type = type;
}
#define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
#endif  // PY_VERSION_HEX < 0x030900A4

Using this header file, Py_SET_TYPE() can be used on old Python versions as well.

Developers can copy this file in their project, or even to only copy/paste the few functions needed by their C extension.

STATUS: In Progress (implemented but not distributed by CPython yet)

The pythoncapi_compat.h header file is currently developed at: https://github.com/pythoncapi/pythoncapi_compat