What Additional Repository Files are Required on PyPI?

In order for package managers like pip to download and verify distributions with TUF, a few extra files MUST be added to PyPI. These extra repository files are called TUF metadata, and they contain such information as which keys can be trusted, the cryptographic hashes 33 of files, signatures, metadata version numbers, and the date after which the metadata should be considered expired.

When a package manager wants to check for updates, it asks TUF to do the work. That is, a package manager never has to deal with this additional metadata or understand what’s going on underneath. If TUF reports back that there are updates available, a package manager can then ask TUF to download these files from PyPI. TUF downloads them and checks them against the TUF metadata that it also downloads from the repository. If the downloaded target files are trustworthy, TUF then hands them over to the package manager.

The Metadata 34 document provides information about each type of required metadata and its expected content. The next section covers the different kinds of metadata RECOMMENDED for PyPI.

In addition, all target files SHOULD be available on disk at least two times. Once under their original filename, to provide backwards compatibility, and once with their SHA-512 hash included in their filename. This is required to produce Consistent Snapshots.

Depending on the used file system different data deduplication mechanisms MAY be employed to avoid storage increase from hard copies of target files.

Signing Metadata and Repository Management

The top-level root role signs for the keys of the top-level timestamp, snapshot, targets, and root roles. The timestamp role signs for every new snapshot of the repository metadata. The snapshot role signs for root, targets, and all delegated targets roles. The delegated targets role bins further delegates to the bin-n roles, which sign for all distribution files belonging to registered PyPI projects.

Figure 1 provides an overview of the roles available within PyPI, which includes the top-level roles and the roles delegated to by targets. The figure also indicates the types of keys used to sign each role, and which roles are trusted to sign for files available on PyPI. The next two sections cover the details of signing repository files and the types of keys used for each role.

Figure 1: An overview of the role metadata available on PyPI.

The roles that change most frequently are timestamp, snapshot and roles delegated to by bins (i.e., bin-n). The timestamp and snapshot metadata MUST be updated whenever root, targets or delegated metadata are updated. Observe, though, that root and targets metadata are much less likely to be updated as often as delegated metadata. Similarly, the bins role will only be updated when a bin-n role is added, updated, or removed. Therefore, timestamp, snapshot, and bin-n metadata will most likely be updated frequently (possibly every minute) due to delegated metadata being updated frequently in order to support continuous delivery of projects. Continuous delivery is a set of processes that PyPI uses to produce snapshots that can safely coexist and be deleted independent of other snapshots 18.

Every year, PyPI administrators SHOULD sign for root and targets role keys. Automation will continuously sign for a timestamped snapshot of all projects. A repository management 39 tool is available that can sign metadata files, generate cryptographic keys, and manage a TUF repository.

In standard operation, the bin-n metadata will be updated and signed as new distributions are uploaded to PyPI. However, there will also need to be a one-time online initialization mechanism to create and sign bin-n metadata for all existing distributions that are part of the PyPI repository every time PyPI is re-initialized.

How to Establish Initial Trust in the PyPI Root Keys

Package managers like pip MUST ship the root metadata file with the installation files that users initially download. This includes information about the keys trusted for all top-level roles (including the root keys themselves). Package managers must also bundle a TUF client library. Any new version of root metadata that the TUF client library may download is verified against the root keys initially bundled with the package manager. If a root key is compromised, but a threshold of keys are still secured, then PyPI administrators MUST push new root metadata that revokes trust in the compromised keys. If a threshold of root keys are compromised, then the root metadata MUST be updated out-of-band. (However, the threshold of root keys should be chosen so that this event is extremely unlikely.) Package managers do not necessarily need to be updated immediately if root keys are revoked or added between new releases of the package manager, as the TUF update process automatically handles cases where a threshold of previous root keys sign for new root keys (assuming no backwards-incompatibility in the TUF specification used). So, for example, if a package manager was initially shipped with version 1 of the root metadata, and a threshold of root keys in version 1 signed version 2 of the root metadata, and a threshold of root keys in version 2 signed version 3 of the root metadata, then the package manager should be able to transparently update its copy of the *root metadata from version 1 to 3 using its TUF client library.

Thus, to repeat, the latest good copy of root metadata and a TUF client library MUST be included in any new version of pip shipped with CPython (via ensurepip). The TUF client library inside the package manager then loads the root metadata and downloads the rest of the roles, including updating the root metadata if it has changed. An outline of the update process 40 is available.

Minimum Security Model

There are two security models to consider when integrating TUF into PyPI. The one proposed in this PEP is the minimum security model, which supports verification of PyPI distributions signed with private cryptographic keys stored on PyPI. Distributions uploaded by developers are signed by PyPI and immediately available for download. A possible future extension to this PEP, discussed in PEP 480 21, proposes the maximum security model and allows a developer to sign for his/her project. Developer keys are not stored online: therefore, projects are safe from PyPI compromises.

The minimum security model requires no action from a developer and protects against malicious CDNs 19 and public mirrors. To support continuous delivery of uploaded distributions, PyPI signs for projects with an online key. This level of security prevents projects from being accidentally or deliberately tampered with by a mirror or a CDN because neither will have any of the keys required to sign for projects. However, it does not protect projects from attackers who have compromised PyPI, since they can then manipulate TUF metadata using the keys stored online.

This PEP proposes that the bin-n roles sign for all PyPI projects with online keys. These bin-n roles MUST all be delegated by the upper-level bins role, which is signed with an offline key, and in turn MUST be delegated by the top-level targets role, which is also signed with an offline key. This means that when a package manager such as pip (i.e., using TUF) downloads a distribution file from a project on PyPI, it will consult the targets role about the TUF metadata for that distribution file. If ultimately no bin-n roles delegated by targets via bins specify the distribution file, then it is considered to be non-existent on PyPI.

Note, the reason why targets does not directly delegate to bin-n, but instead uses the intermediary bins role, is so that other delegations can easily be added or removed, without affecting the bins-to-bin-n mapping. This is crucial for the implementation of PEP 480 21.

Metadata Expiry Times

The metadata for the root, targets, and bins roles SHOULD each expire in one year, because these metadata files are expected to change very rarely.

The timestamp, snapshot, and bin-n metadata SHOULD each expire in one day because a CDN or mirror SHOULD synchronize itself with PyPI every day. Furthermore, this generous time frame also takes into account client clocks that are highly skewed or adrift.

Metadata Scalability

As the number of projects and distributions on a repository grows, TUF metadata will need to grow correspondingly. For example, consider the bins role. In August 2013, it was found that the size of the bins metadata was about 42MB if the bins role itself signed for about 220K PyPI targets (which are simple indices and distributions). This PEP does not delve into the details, but TUF features a so-called “lazy bin walk 41” scheme that splits a large targets metadata file into many small ones. This allows a TUF client updater to intelligently download only a small number of TUF metadata files in order to update any project signed for by the bins role. For example, applying this scheme to the previous repository resulted in pip downloading between 1.3KB and 111KB to install or upgrade a PyPI project via TUF.

Based on our findings as of the time this document was updated for implementation (Nov 7 2019), summarized in Tables 2-3, PyPI SHOULD split all targets in the bins role by delegating them to 16,384 bin-n roles (see C10 in Table 2). Each bin-n role would sign for the PyPI targets whose SHA2-512 hashes fall into that bin (see Figure 1 and Consistent Snapshots). It was found that this number of bins would result in a 5-9% metadata overhead (relative to the average size of downloaded distribution files; see V13 and V15 in Table 3) for returning users, and a 69% overhead for new users who are installing pip for the first time (see V17 in Table 3).

A few assumptions used in calculating these metadata overhead percentages:

1. We are ignoring root, timestamp, and top-level targets metadata.
2. pip will always be bundled with the latest good copy of metadata for all roles.

C8 was computed by querying the number of release files. C9 was derived by taking the average between a rough estimate of the average size of release files downloaded over the past 31 days (1,628,321 bytes), and the average size of releases files on disk (2,740,465 bytes). Ee W. Durbin III helped to provide these numbers on November 7, 2019.

Table 2: A list of constants used to calculate metadata overhead.

Table 3: Estimated metadata overheads for new and returning users.

The interested reader may find an interactive version of the metadata overhead calculator here 42:

This number of bins SHOULD increase when the metadata overhead for returning users exceeds 50%. Presently, this SHOULD happen when the number of targets increase at least 10x from over 2M to over 22M, at which point the metadata overhead for returning and new users would be around 50-54% and 114% respectively, assuming that the number of bins stay fixed. If the number of bins is increased, then the cost for all users would effectively be the cost for new users, because their cost would be dominated by the (once-in-a-while) cost of downloading the large number of delegations in the bins metadata. If the cost for new users should prove to be too much, primarily due to the overhead of downloading the bins metadata, then this subject SHOULD be revisited before that happens.

Note that changes to the number of bins on the server are transparent to the client. The package manager will be required to download a fresh set of metadata, as though it were a new user, but this operation will not require any explicit code logic or user interaction in order to do so.

It is possible to make TUF metadata more compact by representing it in a binary format, as opposed to the JSON text format. Nevertheless, a sufficiently large number of projects and distributions will introduce scalability challenges at some point, and therefore the bins role will still need delegations (as outlined in Figure 1) in order to address the problem. The JSON format is an open and well-known standard for data interchange, which is already supported by the TUF reference implementation, and therefore the recommended data format by this PEP. However, due to the large number of delegations, compressed versions of all metadata SHOULD also be made available to clients via the existing Warehouse mechanisms for HTTP compression. In addition, the JSON metadata could be compressed before being sent to clients. The TUF reference implementation does not currently support downloading compressed JSON metadata, but this could be added to reduce the metadata size.

Number and Type Of Keys Recommended

The root role key is critical for security and should very rarely be used. It is primarily used for key revocation, and it is the locus of trust for all of PyPI. The root role signs for the keys that are authorized for each of the top-level roles (including its own). Keys belonging to the root role are intended to be very well-protected and used with the least frequency of all keys. It is RECOMMENDED that the PSF board determine the current set of trusted root key holders, each of whom will own a (strong) root key. A majority of them can then constitute a quorum to revoke or endow trust in all top-level keys. Alternatively, the system administrators of PyPI could be given responsibility for signing for the root role. Therefore, the root role SHOULD require (t, n) keys, where n is the number of key holders determined by the PSF board, and t > 1 (so that at least two members must sign the root role).

The targets role will be used only to sign for the static delegation of all targets to the bins role. Since these target delegations must be secured against attacks in the event of a compromise, the keys for the targets role MUST be offline and independent of other keys. For simplicity of key management, without sacrificing security, it is RECOMMENDED that the keys of the targets role be permanently discarded as soon as they have been created and used to sign for the role. Therefore, the targets role SHOULD require (2, 2) keys. Again, this is because the keys are going to be permanently discarded, and more offline keys will not help resist key recovery attacks 20 unless the diversity of cryptographic algorithms is maintained.

For similar reasons, the keys for the bins role SHOULD be set up similar to the keys for the targets role.

In order to support continuous delivery, the keys for the timestamp, snapshot, and all bin-n roles MUST be online. There is little benefit in requiring all of these roles to use different online keys, since attackers would presumably be able to compromise all of them if they compromise PyPI. Therefore, it is reasonable to use one online key for all of them.

Managing online keys

The online key shared by the timestamp, snapshot, and all bin-n roles MAY be stored, encrypted or not, on the Python infrastructure. For example, the key MAY be kept on a self-hosted key management service (e.g. Hashicorp Vault 43), or a third-party one (e.g. AWS KMS 44, Google Cloud KMS 45, or Azure Key Vault 46).

Some of these key management services allow keys to be stored on Hardware Security Modules (HSMs) (e.g., Hashicorp Vault 47, AWS CloudHSM 48, Google Cloud HSM 49, Azure Key Vault 50). This prevents attackers from exfiltrating the online private key (albeit not from using it, although their actions may now be cryptographically auditable). However, this requires modifying the reference TUF implementation to support HSMs (WIP 51).

Regardless of where and how this online key is kept, its use SHOULD be carefully logged, monitored, and audited, ideally in such a manner that attackers who compromise PyPI are unable to immediately turn off this logging, monitoring, and auditing.

Managing offline keys

As explained in the previous section, the root, targets, and bins role keys MUST be offline for maximum security. These keys will be offline in the sense that their private keys MUST NOT be stored on PyPI, though some of them MAY be online in the private infrastructure of the project.

There SHOULD be an offline key ceremony to generate, backup, and store these keys in such a manner that the private keys can be read only by the Python administrators when necessary (e.g., such as rotating the keys for the top-level TUF roles). Thus, keys SHOULD be generated, preferably in a physical location where side-channel attacks 52 are not a concern, using:

1. A trusted, airgapped 53 computer with a true random number generator 54, and with no data persisting after the ceremony
2. A trusted operating system
3. A trusted set of third-party packages (such as updated versions of cryptographic libraries or the TUF reference implementation, where the versions provided by the trusted operating system are not recent enough)

In order to avoid the persistence of sensitive data (e.g., private keys) other than on backup media after the ceremony, offline keys SHOULD be generated encrypted using strong passwords, either on (in decreasing order of trust): private HSMs (e.g., YubiHSM 55), cloud-based HSMs (e.g., those listed above), in volatile memory (e.g., RAM), or in nonvolatile memory (e.g., SSD or microSD). If keys must be generated on nonvolatile memory, then this memory MUST be irrecoverably destroyed after having securely backed up the keys.

Passwords used to encrypt keys SHOULD be stored somewhere durable and trustworthy to which only Python admins have access.

In order to minimize OPSEC 56 errors during the ceremony, scripts SHOULD be written, for execution on the trusted key-generation computer, to automate tedious steps of the ceremony, such as:

• Exporting to sneakernet 57 all code and data (previous TUF metadata and root keys) required to generate new keys and replace old ones
• Tightening the firewall, updating the entire operating system in order to fix security vulnerabilities, and airgapping the computer
• Exporting all new TUF metadata and keys to encrypted backup media. This backup provides a complete copy of the data required to restore the PyPI TUF repository
• Exporting only new TUF metadata and online keys to encrypted backup media. This backup provides all online data for import into the PyPI infrastructure and is useful, e.g., when the online data needs to be restored from a previous archived state
• Printing and saving cryptographic hashes of new TUF metadata. This printed copy provides an additional offline paper backup, which can be used as a comparison in the case of a compromise

Note the one-time keys for the targets and bins roles MAY be safely generated, used, and deleted during the offline key ceremony. Furthermore, the root keys MAY not be generated during the offline key ceremony itself. Instead, a threshold t of n Python administrators, as discussed above, MAY independently sign the root metadata after the offline key ceremony used to generate all other keys.

Consistent Snapshots

To keep TUF metadata on PyPI consistent with the highly volatile target files, consistent snapshots SHOULD be used. Each consistent snapshot captures the state of all known projects at a given time and MAY safely coexist with any other snapshot, or be deleted independently, without affecting any other snapshot.

To maintain consistent snapshots, all TUF metadata MUST, when written to disk, include a version number in their filename:

VERSION_NUMBER.ROLENAME.json,

where VERSION_NUMBER is an incrementing integer, and ROLENAME is one of the top-level metadata roles – root, snapshot or targets – or one of the delegated targets roles – bins or bin-n.

The only exception is the timestamp metadata file, whose version would not be known in advance when a client performs an update. The timestamp metadata lists the version of the snapshot metadata, which in turn lists the versions of the targets and delegated targets metadata, all as part of a given consistent snapshot.

In normal usage, version number overflow is unlikely to occur. An 8-byte integer, for instance, can be incremented once per millisecond and last almost 300 million years. If an attacker increases the version number arbitrarily, the repository can recover by revoking the compromised keys and resetting the version number as described in the TUF specification 58.

The targets or delegated targets metadata refer to the actual target files, including their cryptographic hashes as specified above. Thus, to mark a target file as part of a consistent snapshot it MUST, when written to disk, include its hash in its filename:

HASH.FILENAME

where HASH is the hex digest 59 of the hash of the file contents and FILENAME is the original filename.

This means that there MAY be multiple copies of every target file, one for each of the cryptographic hash functions specified above.

Assuming infinite disk space, strictly incrementing version numbers, and no hash collisions 60, a client may safely read from one snapshot while PyPI produces another snapshot.

Clients, such as pip, that use the TUF protocol MUST be modified to download every metadata and target file, except for timestamp metadata. This is done by including, in the file request, the version of the file (for metadata), or the cryptographic hash of the file (for target files) in the filename.

In this simple but effective manner, PyPI is able to capture a consistent snapshot of all projects and the associated metadata at a given time. The next subsection provides implementation details of this idea.

Note: This PEP does not prohibit using advanced file systems or tools to produce consistent snapshots. There are two important reasons for proposing a simple solution in this PEP. First, the solution does not mandate that PyPI use any particular file system or tool. Second, the generic file-system based approach allows mirrors to use extant file transfer tools, such as rsync, to efficiently transfer consistent snapshots from PyPI.

Producing Consistent Snapshots

When a new distribution file is uploaded to PyPI, PyPI MUST update the responsible bin-n metadata. Remember that all target files are sorted into bins by their filename hashes. PyPI MUST also update snapshot to account for the updated bin-n metadata, and timestamp to account for the updated snapshot metadata. These updates SHOULD be handled by an automated snapshot process.

File uploads MAY be handled in parallel, however, consistent snapshots MUST be produced in a strictly sequential manner. Furthermore, as long as distribution files are self-contained, a consistent snapshot MAY be produced for each uploaded file. To do so upload processes place new distribution files into a concurrency-safe FIFO queue and the snapshot process reads from that queue one file at a time and performs the following tasks:

First, it adds the new file path to the relevant bin-n metadata, increments its version number, signs it with the bin-n role key, and writes it to VERSION_NUMBER.bin-N.json.

Then, it takes the most recent snapshot metadata, updates its bin-n metadata version numbers, increments its own version number, signs it with the snapshot role key, and writes it to VERSION_NUMBER.snapshot.json.

And finally, the snapshot process takes the most recent timestamp metadata, updates its snapshot metadata hash and version number, increments its own version number, sets a new expiration time, signs it with the timestamp role key, and writes it to timestamp.json.

When updating bin-n metadata for a consistent snapshot, the snapshot process SHOULD also include any new or updated hashes of simple index pages in the relevant bin-n metadata. Note that, simple index pages may be generated dynamically on API calls, so it is important that their output remains stable throughout the validity of a consistent snapshot.

Since the snapshot process MUST generate consistent snapshots in a strictly sequential manner it constitutes a bottleneck. Fortunately, the operation of signing is fast enough that this may be done a thousand or more times per second.

Moreover, PyPI MAY serve distribution files to clients before the corresponding consistent snapshot metadata is generated. In that case the client software SHOULD inform the user that full TUF protection is not yet available but will be shortly.

PyPI SHOULD use a transaction log 61 to record upload processes and the snapshot queue for auditing and to recover from errors after a server failure.

Cleaning up old metadata

To avoid running out of disk space due to the constant production of new consistent snapshots, PyPI SHOULD regularly delete old consistent snapshots, i.e. metadata and target files that were obsoleted some reasonable time in the past, such as 1 hour.

In order to preserve the latest consistent snapshot PyPI MAY use a “mark-and-sweep” algorithm. That is, walk from the root of the latest consistent snapshot, i.e. timestamp over snapshot over targets and delegated targets until the target files, marking all visited files, and delete all unmarked files. The last few consistent snapshots may be preserved in a similar fashion.

Deleting a consistent snapshot will cause clients to see nothing except HTTP 404 responses to any request for a file within that consistent snapshot. Clients SHOULD then retry their requests (as before) with the latest consistent snapshot.

Note that root metadata, even though versioned, is not part of any consistent snapshot. PyPI MUST NOT delete old versions of root metadata. This guarantees that clients can update to the latest root role keys, no matter how outdated their local root metadata is.

In the Event of a Key Compromise

A key compromise means that a threshold of keys (belonging to the metadata roles on PyPI), as well as the PyPI infrastructure have been compromised and used to sign new metadata on PyPI.

If a threshold number of timestamp, snapshot, targets, bins or bin-n keys have been compromised, then PyPI MUST take the following steps:

1. Revoke the timestamp, snapshot and targets role keys from the root role. This is done by replacing the compromised timestamp, snapshot and targets keys with newly issued keys.
2. Revoke the bins keys from the targets role by replacing their keys with newly issued keys. Sign the new targets role metadata and discard the new keys (because, as explained earlier, this increases the security of targets metadata).
3. All targets of the bin-n roles SHOULD be compared with the last known good consistent snapshot in which none of the timestamp, snapshot, bins or bin-n keys were known to have been compromised. Added, updated or deleted targets in the compromised consistent snapshot that do not match the last known good consistent snapshot MAY be restored to their previous versions. After ensuring the integrity of all bin-n targets, their keys should be renewed in the bins metadata.
4. The bins and bin-n metadata MUST have their version numbers incremented, expiry times suitably extended, and signatures renewed.
5. A new timestamped consistent snapshot MUST be issued.

Following these steps would preemptively protect all of these roles, even if only one of them may have been compromised.

If a threshold number of root keys have been compromised, then PyPI MUST take the above steps and also replace all root keys in the root role.

It is also RECOMMENDED that PyPI sufficiently document compromises with security bulletins. These security bulletins will be most informative when users of pip-with-TUF are unable to install or update a project because the keys for the timestamp, snapshot or root roles are no longer valid. They could then visit the PyPI web site to consult security bulletins that would help to explain why they are no longer able to install or update, and then take action accordingly. When a threshold number of root keys have not been revoked due to a compromise, then new root metadata may be safely updated because a threshold number of existing root keys will be used to sign for the integrity of the new root metadata. TUF clients will be able to verify the integrity of the new root metadata with a threshold number of previously known root keys. This will be the common case. Otherwise, in the worst case, in which a threshold number of root keys have been revoked due to a compromise, an end-user may choose to update new root metadata with out-of-band 62 mechanisms.

Auditing Snapshots

If a malicious party compromises PyPI, they can sign arbitrary files with any of the online keys. The roles with offline keys (i.e., root, targets and bins) are still protected. To safely recover from a repository compromise, snapshots should be audited to ensure files are only restored to trusted versions.

When a repository compromise has been detected, the integrity of three types of information must be validated:

1. If the online keys of the repository have been compromised, they can be revoked by having the targets role sign new metadata delegating to a new key.
2. If the role metadata on the repository has been changed, this would impact the metadata that is signed by online keys. Any role information created since the last period should be discarded. As a result, developers of new projects will need to re-register their projects.
3. If the target files themselves may have been tampered with, they can be validated using the stored hash information for target files that existed at the time of the last period.

In order to safely restore snapshots in the event of a compromise, PyPI SHOULD maintain a small number of its own mirrors to copy PyPI snapshots according to some schedule. The mirroring protocol can be used immediately for this purpose. The mirrors must be secured and isolated such that they are responsible only for mirroring PyPI. The mirrors can be checked against one another to detect accidental or malicious failures.

Another approach is to generate the cryptographic hash of snapshot periodically and tweet it. Perhaps a user comes forward with the actual metadata and the repository maintainers can verify the metadata file’s cryptographic hash. Alternatively, PyPI may periodically archive its own versions of snapshot rather than rely on externally provided metadata. In this case, PyPI SHOULD take the cryptographic hash of every target file on the repository and store this data on an offline device. If any target file hash has changed, this indicates an attack.

As for attacks that serve different versions of metadata, or freeze a version of a distribution at a specific version, they can be handled by TUF with techniques like implicit key revocation and metadata mismatch detection 2.

Hash Algorithm Transition Plan

If the algorithm used to hash target and metadata files becomes vulnerable, it SHOULD be replaced by a stronger hash algorithm.

The TUF metadata format allows to list digests from different hash algorithms alongside each other, together with an algorithm identifier, so that clients can seamlessly switch between algorithms.

However, once support for an old algorithm is turned off, clients that don’t support the new algorithm will only be able to install or update packages, including the client itself, by disabling TUF verification. To allow clients to transition without temporarily losing TUF security guarantees, we recommend the following procedure.

1. Implement new algorithm in Warehouse.
2. Regenerate existing, unexpired TUF metadata to include hashes using both the old and new algorithms. All new metadata going forward shall list both hash algorithms. Note, only TUF metadata that lists hash digests for target files or other metadata needs to be renewed, that is bin-n, snapshot and timestamp. Thus, only online keys are required to sign renewed metadata.
3. Announce transition on high-visibility channels, such as packaging on Python Discourse 64 and the PyPI changes mailing list 65.
4. Give popular clients such as pip and bandersnatch the chance to adopt new hash algorithm.
5. Give end-users the chance to update clients.
6. Get rough consensus to remove old hash algorithm from PyPI maintainers.
7. Remove Warehouse support for old algorithm and only support new algorithm.