Python Enhancement Proposals

PEP 599 – The manylinux2014 Platform Tag

PEP
599
Title
The manylinux2014 Platform Tag
Author
Dustin Ingram <di at python.org>
Sponsor
Paul Moore <p.f.moore at gmail.com>
BDFL-Delegate
Paul Moore <p.f.moore at gmail.com>
Discussions-To
https://discuss.python.org/t/the-next-manylinux-specification/
Status
Superseded
Type
Informational
Created
29-Apr-2019
Post-History
29-April-2019
Superseded-By
600
Resolution
https://discuss.python.org/t/the-next-manylinux-specification/1043/199

Contents

Abstract

This PEP proposes the creation of a manylinux2014 platform tag to succeed the manylinux2010 tag introduced by PEP 513. It also proposes that PyPI and pip both be updated to support uploading, downloading, and installing manylinux2014 distributions on compatible platforms.

Rationale

CentOS 6 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020, 1 at which point it will reach end-of-life, and no further updates such as security patches will be made available. All wheels built under the manylinux2010 images will remain at obsolete versions after that point.

Therefore, we propose the continuation of the existing manylinux standard, and that a new PEP 425-style 2 platform tag called manylinux2014 be derived from CentOS 7 and that the manylinux toolchain, PyPI, and pip be updated to support it.

Similar to how PEP 571 and PEP 513 drew allowed shared libraries and their symbol versions from CentOS 5.11 and CentOS 6, respectively, a manylinux2014 platform tag will draw its libraries and symbol versions from CentOS 7, which will reach end-of-life on June 30th, 2024. 1

The manylinuxYYYY pattern has a number of advantages that motivate continuing with the current status quo:

  • Well-defined Docker images with clearly specified compatible libraries;
  • No need to survey for compatibility issues across multiple releases;
  • A single build image and auditwheel profile per architecture.

There are also some disadvantages:

  • Requires drafting a new PEP for every new standard;
  • Requires adding the new platform tag to installers (e.g., pip);
  • Installers are unable to install a platform tag which predates a given release.

There are also challenges which would exist for any proposal, including the time and effort it takes to define, prepare and release the Docker images and corresponding auditwheel profiles. These challenges were experienced in the long rollout period for manylinux2010, which took approximately 1 year from PEP acceptance to compatible build environment published. 3

However, if this PEP can be an indicator, the process is now well-defined and easily repeatable, which should increase the timeline for rollout of a newer, updated platform tag.

The manylinux2014 policy

The following criteria determine a linux wheel’s eligibility for the manylinux2014 tag:

  1. The wheel may only contain binary executables and shared objects compiled for one of the following architectures supported by CentOS 7, or a CentOS 7 compatible base image (such as ubi7): 4
    x86_64
    i686
    aarch64
    armv7l
    ppc64
    ppc64le
    s390x
    

    This list adds support for ARMv7 (armv7l), ARMv8 (aarch64) and PowerPC (ppc64, ppc64le) architectures supported by the CentOS Alternative Architecture Special Interest Group, as well as the IBM Z (s390x) architecture. 5

  2. The wheel’s binary executables or shared objects may not link against externally-provided libraries except those in the following list:
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libresolv.so.2
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0
    

    This list is identical to the externally-provided libraries originally allowed for manylinux2010, with one exception: libcrypt.so.1 was removed due to being deprecated in Fedora 30. libpythonX.Y remains ineligible for inclusion for the same reasons outlined in PEP 513.

    On Debian-based systems, these libraries are provided by the packages:

    Package Libraries
    libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1
    libgcc1 libgcc_s.so.1
    libgl1 libGL.so.1
    libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
    libice6 libICE.so.6
    libsm6 libSM.so.6
    libstdc++6 libstdc++.so.6
    libx11-6 libX11.so.6
    libxext6 libXext.so.6
    libxrender1 libXrender.so.1

    On RPM-based systems, they are provided by these packages:

    Package Libraries
    glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
    glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
    libICE libICE.so.6
    libX11 libX11.so.6
    libXext: libXext.so.6
    libXrender libXrender.so.1
    libgcc: libgcc_s.so.1
    libstdc++ libstdc++.so.6
    mesa libGL.so.1
  3. If the wheel contains binary executables or shared objects linked against any allowed libraries that also export versioned symbols, they may only depend on the following maximum versions:
    GLIBC_2.17
    CXXABI_1.3.7, CXXABI_TM_1 is also allowed
    GLIBCXX_3.4.19
    GCC_4.8.0
    

    As an example, manylinux2014 wheels may include binary artifacts that require glibc symbols at version GLIBC_2.12, because this an earlier version than the maximum of GLIBC_2.17.

  4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it must include a CPython ABI tag indicating its Unicode ABI. A manylinux2014 wheel built against Python 2, then, must include either the cpy27mu tag indicating it was built against an interpreter with the UCS-4 ABI or the cpy27m tag indicating an interpreter with the UCS-2 ABI. 6 7
  5. A wheel must not require the PyFPE_jbuf symbol. This is achieved by building it against a Python compiled without the --with-fpectl configure flag.

Compilation of Compliant Wheels

Like manylinux1, the auditwheel tool adds manylinux2014 platform tags to linux wheels built by pip wheel or bdist_wheel in a manylinux2014 Docker container.

Docker Images

A manylinux2014 Docker image based on CentOS 7 x86_64 should be provided for building binary linux wheels that can reliably be converted to manylinux2014 wheels. This image will come with a full compiler suite installed (gcc, g++, and gfortran 4.8.5) as well as the latest releases of Python and pip.

Auditwheel

The auditwheel tool will also be updated to produce manylinux2014 wheels. 8 Its behavior and purpose will be otherwise unchanged from PEP 513.

Platform Detection for Installers

Platforms may define a manylinux2014_compatible boolean attribute on the _manylinux module described in PEP 513. A platform is considered incompatible with manylinux2014 if the attribute is False.

If the _manylinux module is not found, or it does not have the attribute manylinux2014_compatible, tools may fall back to checking for glibc. If the platform has glibc 2.17 or newer, it is assumed to be compatible unless the _manylinux module says otherwise.

Specifically, the algorithm we propose is:

def is_manylinux2014_compatible():
    # Only Linux, and only supported architectures
    from distutils.util import get_platform

    if get_platform() not in [
        "linux-x86_64",
        "linux-i686",
        "linux-aarch64",
        "linux-armv7l",
        "linux-ppc64",
        "linux-ppc64le",
        "linux-s390x",
    ]:
        return False

    # Check for presence of _manylinux module
    try:
        import _manylinux

        return bool(_manylinux.manylinux2014_compatible)
    except (ImportError, AttributeError):
        # Fall through to heuristic check below
        pass

    # Check glibc version. CentOS 7 uses glibc 2.17.
    # PEP 513 contains an implementation of this function.
    return have_compatible_glibc(2, 17)

Backwards compatibility with manylinux2010 wheels

As explained in PEP 513, the specified symbol versions for manylinux1 allowed libraries constitute an upper bound. The same is true for the symbol versions defined for manylinux2014 in this PEP. As a result, manylinux1 and manylinux2010 wheels are considered manylinux2014 wheels. A pip that recognizes the manylinux2014 platform tag will thus install manylinux2010 wheels for manylinux2014 platforms – even when explicitly set – when no manylinux2014 wheels are available.

PyPI Support

PyPI should permit wheels containing the manylinux2014 platform tag to be uploaded in the same way that it permits manylinux2010.

If technically feasible, PyPI should attempt to verify the compatibility of manylinux2014 wheels, but that capability is not a requirement for adoption of this PEP.

Package authors should not upload non-compliant manylinux2014 wheels to PyPI, and should be aware that PyPI may begin blocking non-compliant wheels from being uploaded.

References

1 (1, 2)
CentOS Product Specifications (https://wiki.centos.org/About/Product)
2
PEP 425: Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/)
3
Tracking issue for manylinux2010 rollout (https://github.com/pypa/manylinux/issues/179)
4
Red Hat Universal Base Image 7 (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7)
5
The CentOS Alternative Architecture Special Interest Group (https://wiki.centos.org/SpecialInterestGroup/AltArch)
6
PEP 3149: ABI version tagged .so files (https://www.python.org/dev/peps/pep-3149/)
7
SOABI support for Python 2.X and PyPy (https://github.com/pypa/pip/pull/3075)
8
auditwheel (https://github.com/pypa/auditwheel/)

Acceptance

PEP 599 was accepted by Paul Moore on July 31, 2019.

Source: https://github.com/python/peps/blob/master/pep-0599.rst

Last modified: 2021-06-26 16:11:00 GMT