Python Enhancement Proposals

PEP 650 – Specifying Installer Requirements for Python Projects

PEP
650
Title
Specifying Installer Requirements for Python Projects
Author
Vikram Jayanthi <vikramjayanthi at google.com>, Dustin Ingram <di at python.org>, Brett Cannon <brett at python.org>
Discussions-To
https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657
Status
Draft
Type
Process
Created
16-Jul-2020
Post-History
2021-01-14

Contents

Abstract

Python package installers are not completely interoperable with each other. While pip is the most widely used installer and a de facto standard, other installers such as Poetry 8 or Pipenv 7 are popular as well due to offering unique features which are optimal for certain workflows and not directly in line with how pip operates.

While the abundance of installer options is good for end-users with specific needs, the lack of interoperability between them makes it hard to support all potential installers. Specifically, the lack of a standard requirements file for declaring dependencies means that each tool must be explicitly used in order to install dependencies specified with their respective format. Otherwise tools must emit a requirements file which leads to potential information loss for the installer as well as an added export step as part of a developer’s workflow.

By providing a standardized API that can be used to invoke a compatible installer, we can solve this problem without needing to resolve individual concerns, unique requirements, and incompatibilities between different installers and their lock files.

Installers that implement the specification can be invoked in a uniform way, allowing users to use their installer of choice as if they were invoking it directly.

Terminology

Installer interface
The interface by which an installer backend and a universal installer interact.
Universal installer
An installer that can invoke an installer backend by calling the optional invocation methods of the installer interface. This can also be thought of as the installer frontend, à la the build 1 project for PEP 517.
Installer backend
An installer that implements the installer interface, allowing it to be invoked by a universal installer. An installer backend may also be a universal installer as well, but it is not required. In comparison to PEP 517, this would be Flit 4. Installer backends may be wrapper packages around a backing installer, e.g. Poetry could choose to not support this API, but a package could act as a wrapper to invoke Poetry as appropriate to use Poetry to perform an installation.
Dependency group
A set of dependencies that are related and required to be installed simultaneously for some purpose. For example, a “test” dependency group could include the dependencies required to run the test suite. How dependency groups are specified is up to the installer backend.

Motivation

This specification allows anyone to invoke and interact with installer backends that implement the specified interface, allowing for a universally supported layer on top of existing tool-specific installation processes.

This in turn would enable the use of all installers that implement the specified interface to be used in environments that support a single universal installer, as long as that installer implements this specification as well.

Below, we identify various use-cases applicable to stakeholders in the Python community and anyone who interacts with Python package installers. For developers or companies, this PEP would allow for increased functionality and flexibility with Python package installers.

Providers

Providers are the parties (organization, person, community, etc.) that supply a service or software tool which interacts with Python packaging and consequently Python package installers. Two different types of providers are considered:

Platform/Infrastructure Providers

Platform providers (cloud environments, application hosting, etc.) and infrastructure service providers need to support package installers for their users to install Python dependencies. Most only support pip, however there is user demand for other Python installers. Most providers do not want to maintain support for more than one installer because of the complexity it adds to their software or service and the resources it takes to do so.

Via this specification, we can enable a provider-supported universal installer to invoke the user-desired installer backend without the provider’s platform needing to have specific knowledge of said backend. What this means is if Poetry implemented the installer backend API proposed by this PEP (or some other package wrapped Poetry to provide the API), then platform providers would support Poetry implicitly.

IDE Providers

Integrated development environments may interact with Python package installation and management. Most only support pip as a Python package installer, and users are required to find work arounds to install their dependencies using other package installers. Similar to the situation with PaaS & IaaS providers, IDE providers do not want to maintain support for N different Python installers. Instead, implementers of the installer interface (installer backends) could be invoked by the IDE by it acting as a universal installer.

Developers

Developers are teams, people, or communities that code and use Python package installers and Python packages. Three different types of developers are considered:

Developers using PaaS & IaaS providers

Most PaaS and IaaS providers only support one Python package installer: pip 6. (Some exceptions include Heroku’s Python buildpack 2, which supports pip and Pipenv 7). This dictates the installers that developers can use while working with these providers, which might not be optimal for their application or workflow.

Installers adopting this PEP to become installer backends would allow users to use third party platforms/infrastructure without having to worry about which Python package installer they are required to use as long as the provider uses a universal installer.

Developers using IDEs

Most IDEs only support pip or a few Python package installers. Consequently, developers must use workarounds or hacky methods to install their dependencies if they use an unsupported package installer.

If the IDE uses/provides a universal installer it would allow for any installer backend that the developer wanted to be used to install dependencies, freeing them of any extra work to install their dependencies in order to integrate into the IDE’s workflow more closely.

Developers working with other developers

Developers want to be able to use the installer of their choice while working with other developers, but currently have to synchronize their installer choice for compatibility of dependency installation. If all preferred installers instead implemented the specified interface, it would allow for cross use of installers, allowing developers to choose an installer regardless of their collaborator’s preference.

Upgraders & Package Infrastructure Providers

Package upgraders and package infrastructure in CI/CD such as Dependabot 3, PyUP 9, etc. currently support a few installers. They work by parsing and editing the installer-specific dependency files directly (such as requirements.txt or poetry.lock) with relevant package information such as upgrades, downgrades, or new hashes. Similar to Platform and IDE providers, most of these providers do not want to support N different Python package installers as that would require supporting N different file types.

Currently, these services/bots have to implement support for each package installer individually. Inevitably, the most popular installers are supported first, and less popular tools are often never supported. By implementing this specification, these services/bots can support any (compliant) installer, allowing users to select the tool of their choice. This will allow for more innovation in the space, as platforms and IDEs are no longer forced to prematurely select a “winner”.

Open Source Community

Specifying installer requirements and adopting this PEP will reduce the friction between Python package installers and people’s workflows. Consequently, it will reduce the friction between Python package installers and 3rd party infrastructure/technologies such as PaaS or IDEs. Overall, it will allow for easier development, deployment and maintenance of Python projects as Python package installation becomes simpler and more interoperable.

Specifying requirements and creating an interface for installers can also increase the pace of innovation around installers. This would allow for installers to experiment and add unique functionality without requiring the rest of the ecosystem to do the same. Support becomes easier and more likely for a new installer regardless of the functionality it adds and the format in which it writes dependencies, while reducing the developer time and resources needed to do so.

Specification

Similar to how PEP 517 specifies build systems, the install system information will live in the pyproject.toml file under the install-system table.

[install-system]

The install-system table is used to store install-system relevant data and information. There are multiple required keys for this table: requires and install-backend. The requires key holds the minimum requirements for the installer backend to execute and which will be installed by the universal installer. The install-backend key holds the name of the install backend’s entry point. This will allow the universal installer to install the requirements for the installer backend itself to execute (not the requirements that the installer backend itself will install) as well as invoke the installer backend.

If either of the required keys are missing or empty then the universal installer SHOULD raise an error.

All package names interacting with this interface are assumed to follow PEP 508’s “Dependency specification for Python Software Packages” format.

An example install-system table:

#pyproject.toml
[install-system]
#Eg : pipenv
requires = ["pipenv"]
install-backend = "pipenv.api:main"

Installer Requirements:

The requirements specified by the requires key must be within the constraints specified by PEP 517. Specifically, that dependency cycles are not permitted and the universal installer SHOULD refuse to install the dependencies if a cycle is detected.

Additional parameters or tool specific data

Additional parameters or tool (installer backend) data may also be stored in the pyproject.toml file. This would be in the “tool.*” table as specified by PEP 518. For example, if the installer backend is Poetry and you wanted to specify multiple dependency groups, the tool.poetry tables could look like this:

[tool.poetry.dev-dependencies]
dependencies = "dev"

[tool.poetry.deploy]
dependencies = "deploy"

Data may also be stored in other ways as the installer backend sees fit (e.g. separate configuration file).

Installer interface:

The installer interface contains mandatory and optional hooks. Compliant installer backends MUST implement the mandatory hooks and MAY implement the optional hooks. A universal installer MAY implement any of the installer backend hooks itself, to act as both a universal installer and installer backend, but this is not required.

All hooks take **kwargs arbitrary parameters that a installer backend may require that are not already specified, allowing for backwards compatibility. If unexpected parameters are passed to the installer backend, it should ignore them.

The following information is akin to the corresponding section in PEP 517. The hooks may be called with keyword arguments, so installer backends implementing them should be careful to make sure that their signatures match both the order and the names of the arguments above.

All hooks MAY print arbitrary informational text to stdout and stderr. They MUST NOT read from stdin, and the universal installer MAY close stdin before invoking the hooks.

The universal installer may capture stdout and/or stderr from the backend. If the backend detects that an output stream is not a terminal/console (e.g. not sys.stdout.isatty()), it SHOULD ensure that any output it writes to that stream is UTF-8 encoded. The universal installer MUST NOT fail if captured output is not valid UTF-8, but it MAY not preserve all the information in that case (e.g. it may decode using the replace error handler in Python). If the output stream is a terminal, the installer backend is responsible for presenting its output accurately, as for any program running in a terminal.

If a hook raises an exception, or causes the process to terminate, then this indicates an error.

Mandatory hooks:

invoke_install

Installs the dependencies:

def invoke_install(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> int:
    ...
  • path : An absolute path where the installer backend should be invoked from (e.g. the directory where pyproject.toml is located).
  • dependency_group : An optional flag specifying a dependency group that the installer backend should install. The install will error if the dependency group doesn’t exist. A user can find all dependency groups by calling get_dependency_groups() if dependency groups are supported by the installer backend.
  • **kwargs : Arbitrary parameters that a installer backend may require that are not already specified, allows for backwards compatibility.
  • Returns : An exit code (int). 0 if successful, any positive integer if unsuccessful.

The universal installer will use the exit code to determine if the installation is successful and SHOULD return the exit code itself.

Optional hooks:

invoke_uninstall

Uninstall the specified dependencies:

def invoke_uninstall(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> int:
    ...
  • path : An absolute path where the installer backend should be invoked from (e.g. the directory where pyproject.toml is located).
  • dependency_group : An optional flag specifying a dependency group that the installer backend should uninstall.
  • **kwargs : Arbitrary parameters that a installer backend may require that are not already specified, allows for backwards compatibility.
  • Returns : An exit code (int). 0 if successful, any positive integer if unsuccessful.

The universal installer MUST invoke the installer backend at the same path that the universal installer itself was invoked.

The universal installer will use the exit code to determine if the uninstall is successful and SHOULD return the exit code itself.

get_dependencies_to_install

Returns the dependencies that would be installed by invoke_install(...). This allows package upgraders (e.g., Dependabot) to retrieve the dependencies attempting to be installed without parsing the dependency file:

def get_dependencies_to_install(
    path: Union[str, bytes, PathLike[str]],
    *,
    dependency_group: str = None,
    **kwargs
) -> Sequence[str]:
    ...
  • path : An absolute path where the installer backend should be invoked from (e.g. the directory where pyproject.toml is located).
  • dependency_group : Specify a dependency group to get the dependencies invoke_install(...) would install for that dependency group.
  • **kwargs : Arbitrary parameters that a installer backend may require that are not already specified, allows for backwards compatibility.
  • Returns: A list of dependencies (PEP 508 strings) to install.

If the group is specified, the installer backend MUST return the dependencies corresponding to the provided dependency group. If the specified group doesn’t exist, or dependency groups are not supported by the installer backend, the installer backend MUST raise an error.

If the group is not specified, and the installer backend provides the concept of a default/unspecified group, the installer backend MAY return the dependencies for the default/unspecified group, but otherwise MUST raise an error.

get_dependency_groups

Returns the dependency groups available to be installed. This allows universal installers to enumerate all dependency groups the installer backend is aware of:

def get_dependency_groups(
    path: Union[str, bytes, PathLike[str]],
    **kwargs
) -> AbstractSet[str]:
    ...
  • path : An absolute path where the installer backend should be invoked from (e.g. the directory where pyproject.toml is located).
  • **kwargs : Arbitrary parameters that a installer backend may require that are not already specified, allows for backwards compatibility.
  • Returns: A set of known dependency groups, as strings The empty set represents no dependency groups.

update_dependencies

Outputs a dependency file based on inputted package list:

def update_dependencies(
    path: Union[str, bytes, PathLike[str]],
    dependency_specifiers: Iterable[str],
    *,
    dependency_group=None,
    **kwargs
) -> int:
    ...
  • path : An absolute path where the installer backend should be invoked from (e.g. the directory where pyproject.toml is located).
  • dependency_specifiers : An iterable of dependencies as PEP 508 strings that are being updated, for example : ["requests==2.8.1", ...]. Optionally for a specific dependency group.
  • dependency_group : The dependency group that the list of packages is for.
  • **kwargs : Arbitrary parameters that a installer backend may require that are not already specified, allows for backwards compatibility.
  • Returns : An exit code (int). 0 if successful, any positive integer if unsuccessful.

Example

Let’s consider implementing an installer backend that uses pip and its requirements files for dependency groups. An implementation may (very roughly) look like the following:

import subprocess
import sys


def invoke_install(path, *, dependency_group=None, **kwargs):
    try:
        return subprocess.run(
            [
                sys.executable,
                "-m",
                "pip",
                "install",
                "-r",
                dependency_group or "requirements.txt",
            ],
            cwd=path,
        ).returncode
    except subprocess.CalledProcessError as e:
        return e.returncode

If we named this package pep650pip, then we could specify in pyproject.toml:

[install-system]
  #Eg : pipenv
  requires = ["pep650pip", "pip"]
  install-backend = "pep650pip:main"

Rationale

All hooks take **kwargs to allow for backwards compatibility and allow for tool specific installer backend functionality which requires a user to provide additional information not required by the hook.

While installer backends must be Python packages, what they do when invoked is an implementation detail of that tool. For example, an installer backend could act as a wrapper for a platform package manager (e.g., apt).

The interface does not in any way try to specify how installer backends should function. This is on purpose so that installer backends can be allowed to innovate and solve problem in their own way. This also means this PEP takes no stance on OS packaging as that would be an installer backend’s domain.

Defining the API in Python does mean that some Python code will eventually need to be executed. That does not preclude non-Python installer backends from being used, though (e.g. mamba 5), as they could be executed as a subprocess from Python code.

Backwards Compatibility

This PEP would have no impact on pre-existing code and functionality as it only adds new functionality to a universal installer. Any existing installer should maintain its existing functionality and use cases, therefore having no backwards compatibility issues. Only code aiming to take advantage of this new functionality will have motivation to make changes to their pre existing code.

Security Implications

A malicious user has no increased ability or easier access to anything with the addition of standardized installer specifications. The installer that could be invoked by a universal installer via the interface specified in this PEP would be explicitly declared by the user. If the user has chosen a malicious installer, then invoking it with a universal installer is no different than the user invoking the installer directly. A malicious installer being an installer backend doesn’t give it additional permissions or abilities.

Rejected Ideas

A standardized lock file

A standardized lock file would solve a lot of the same problems that specifying installer requirements would. For example, it would allow for PaaS/IaaS to just support one installer that could read the standardized lock file regardless of the installer that created it. The problem with a standardized lock file is the difference in needs between Python package installers as well as a fundamental issue with creating reproducible environments via the lockfile (one of the main benefits).

Needs and information stored in dependency files between installers differ significantly and are dependent on installer functionality. For example, a Python package installer such as Poetry requires information for all Python versions and platforms and calculates appropriate hashes while pip doesn’t. Additionally, pip would not be able to guarantee recreating the same environment (install the exact same dependencies) as it is outside the scope of its functionality. This makes a standardized lock file harder to implement and makes it seem more appropriate to make lock files tool specific.

Have installer backends support creating virtual environments

Because installer backends will very likely have a concept of virtual environments and how to install into them, it was briefly considered to have them also support creating virtual environments. In the end, though, it was considered an orthogonal idea.

Open Issues

Should the dependency_group argument take an iterable?

This would allow for specifying non-overlapping dependency groups in a single call, e.g. “docs” and “test” groups which have independent dependencies but which a developer may want to install simultaneously while doing development.

Is the installer backend executed in-process?

If the installer backend is executed in-process then it greatly simplifies knowing what environment to install for/into, as the live Python environment can be queried for appropriate information.

Executing out-of-process allows for minimizing potential issues of clashes between the environment being installed into and the installer backend (and potentially universal installer).

Enforce that results from the proposed interface feed into other parts?

E.g. the results from get_dependencies_to_install() and get_dependency_groups() can be passed into invoke_install(). This would prevent drift between the results of various parts of the proposed interface, but it makes more of the interface required instead of optional.

Raising exceptions instead of exit codes for failure conditions

It has been suggested that instead of returning an exit code the API should raise exceptions. If you view this PEP as helping to translate current installers into installer backends, then relying on exit codes makes sense. There’s is also the point that the APIs have no specific return value, so passing along an exit code does not interfere with what the functions return.

Compare that to raising exceptions in case of an error. That could potentially provide a more structured approach to error raising, although to be able to capture errors it would require specifying exception types as part of the interface.

References

1
https://github.com/pypa/build
2
https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python
3
https://dependabot.com/
4
https://flit.readthedocs.io
5
https://github.com/mamba-org/mamba
6
https://pip.pypa.io
7 (1, 2)
https://pipenv-fork.readthedocs.io/en/latest/
8
https://python-poetry.org/
9
https://pyup.io/

Source: https://github.com/python/peps/blob/master/pep-0650.rst

Last modified: 2021-02-05 16:09:26 GMT