Python Enhancement Proposals

PEP 576 – Rationalize Built-in function classes

PEP
576
Title
Rationalize Built-in function classes
Author
Mark Shannon <mark at hotpy.org>
BDFL-Delegate
Petr Viktorin
Status
Withdrawn
Type
Standards Track
Created
10-May-2018
Python-Version
3.8
Post-History
17-May-2018 23-June-2018 08-July-2018 29-Mar-1028

Contents

Abstract

Expose the “FastcallKeywords” convention used internally by CPython to third-party code, and make the inspect module use duck-typing. In combination this will allow third-party C extensions and tools like Cython to create objects that use the same calling conventions as built-in and Python functions, thus gaining performance parity with built-in functions like len or print.

A small improvement in the performance of existing code is expected.

Motivation

Currently third-party module authors face a dilemma when implementing functions in C. Either they can use one of the pre-existing built-in function or method classes or implement their own custom class in C. The first choice causes them to lose the ability to access the internals of the callable object. The second choice is an additional maintenance burden and, more importantly, has a significant negative impact on performance.

This PEP aims to allow authors of third-party C modules, and tools like to Cython, to utilize the faster calling convention used internally by CPython for built-in functions and methods, and to do so without a loss of capabilities relative to a function implemented in Python.

Introspection

The inspect module will fully support duck-typing when introspecting callables.

The inspect.Signature.from_callable() function computes the signature of a callable. If an object has a __signature__ property, then inspect.Signature.from_callable() simply returns that. To further support duck-typing, if a callable has a __text_signature__ then the __signature__ will be created from that.

This means that 3rd party builtin-functions can implement __text_signature__ if sufficient, and the more expensive __signature__ if necessary.

Efficient calls to third-party callables

Currently the majority of calls are dispatched to functions and method_descriptors in custom code, using the “FastcallKeywords” internal calling convention. This PEP proposes that this calling convention is implemented via a C function pointer. Third-party callables which implement this binary interface will have the potential to be called as fast as a built-in function.

Continued prohibition of callable classes as base classes

Currently any attempt to use function, method or method_descriptor as a base class for a new class will fail with a TypeError. This behaviour is desirable as it prevents errors when a subclass overrides the __call__ method. If callables could be sub-classed then any call to a function or a method_descriptor would need an additional check that the __call__ method had not been overridden. By exposing an additional call mechanism, the potential for errors becomes greater. As a consequence, any third-party class implementing the addition call interface will not be usable as a base class.

New classes and changes to existing classes

Python visible changes

  1. A new built-in class, builtin_function, will be added.
  2. types.BuiltinFunctionType will refer to builtin_function not builtin_function_or_method.
  3. Instances of the builtin_function class will retain the __module__ property of builtin_function_or_method and gain the func_module and func_globals properties. The func_module allows access to the module to which the function belongs. Note that this is different from the __module__ property which merely returns the name of the module. The func_globals property is equivalent to func_module.__dict__ and is provided to mimic the Python function property of the same name.
  4. When binding a method_descriptor instance to an instance of its owning class, a bound_method will be created instead of a builtin_function_or_method. This means that the method_descriptors now mimic the behaviour of Python functions more closely. In other words, [].append becomes a bound_method instead of a builtin_function_or_method.

C API changes

  1. A new function PyBuiltinFunction_New(PyMethodDef *ml, PyObject *module) is added to create built-in functions.
  2. PyCFunction_NewEx() and PyCFunction_New() are deprecated and will return a PyBuiltinFunction if able, otherwise a builtin_function_or_method.

Retaining backwards compatibility in the C API and ABI

The proposed changes are fully backwards and forwards compatible at both the API and ABI level.

Internal C changes

Two new flags will be allowed for the typeobject.tp_flags field. These are Py_TPFLAGS_EXTENDED_CALL and Py_TPFLAGS_FUNCTION_DESCRIPTOR

Py_TPFLAGS_EXTENDED_CALL

For any built-in class that sets Py_TPFLAGS_EXTENDED_CALL The C struct corresponding to this built-in class must begin with the struct PyExtendedCallable which is defined as follows:

typedef PyObject *(*extended_call_ptr)(PyObject *callable, PyObject** args,
                   int positional_argcount, PyTupleObject* kwnames);

typedef struct {
    PyObject_HEAD
    extended_call_ptr ext_call;
} PyExtendedCallable;

Any class that sets the Py_TPFLAGS_EXTENDED_CALL cannot be used as a base class and a TypeError will be raised if any Python code tries to use it a base class.

Py_TPFLAGS_FUNCTION_DESCRIPTOR

If this flag is set for a built-in class F, then instances of that class are expected to behave the same as a Python function when used as a class attribute. Specifically, this mean that the value of c.m where C.m is an instanceof the built-in class F (and c is an instance of C) must be a bound-method binding C.m and c. Without this flag, it would be impossible for custom callables to behave like Python functions and be efficient as Python or built-in functions.

Changes to existing C structs

The function, method_descriptor and method classes will have their corresponding structs changed to start with the PyExtendedCallable struct.

Third-party built-in classes using the new extended call interface

To enable call performance on a par with Python functions and built-in functions, third-party callables should set the Py_TPFLAGS_EXTENDED_CALL bit of tp_flags and ensure that the corresponding C struct starts with the PyExtendedCallable. Any built-in class that has the Py_TPFLAGS_EXTENDED_CALL bit set must also implement the tp_call function and make sure its behaviour is consistent with the ext_call function.

Performance implications of these changes

Adding a function pointer to each callable, rather than each class of callable, enables the choice of dispatching function (the code to shuffle arguments about and do error checking) to be made when the callable object is created rather than when it is called. This should reduce the number of instructions executed between the call-site in the interpreter and the execution of the callee.

Alternative Suggestions

PEP 580 <https://www.python.org/dev/peps/pep-0580/> is an alternative approach to solving the same problem as this PEP.

Reference implementation

A draft implementation can be found at https://github.com/markshannon/cpython/tree/pep-576-minimal

Source: https://github.com/python/peps/blob/master/pep-0576.rst

Last modified: 2019-07-03 18:20:45 GMT