Cryptographic Signature Scheme: Ed25519

The package manager (pip) shipped with CPython MUST work on non-CPython interpreters and cannot have dependencies that have to be compiled (i.e., the PyPI+TUF integration MUST NOT require compilation of C extensions in order to verify cryptographic signatures). Verification of signatures MUST be done in Python, and verifying RSA 11 signatures in pure-Python may be impractical due to speed. Therefore, PyPI MAY use the Ed25519 14 signature scheme.

Ed25519 12 is a public-key signature system that uses small cryptographic signatures and keys. A pure-Python implementation 15 of the Ed25519 signature scheme is available. Verification of Ed25519 signatures is fast even when performed in Python.

Cryptographic Key Files

The implementation MAY encrypt key files with AES-256-CTR-Mode and strengthen passwords with PBKDF2-HMAC-SHA256 (100K iterations by default, but this may be overridden by the developer). The current Python implementation of TUF can use any cryptographic library (support for PyCA cryptography will be added in the future), may override the default number of PBKDF2 iterations, and the KDF tweaked to taste.

Key Management: miniLock

An easy-to-use key management solution is needed. One solution is to derive a private key from a password so that developers do not have to manage cryptographic key files across multiple computers. miniLock 16 is an example of how this can be done. Developers may view the cryptographic key as a secondary password. miniLock also works well with a signature scheme like Ed25519, which only needs a very small key.

Third-party Upload Tools: Twine

Third-party tools like Twine 17 MAY be modified (if they wish to support distributions that include TUF metadata) to sign and upload developer projects to PyPI. Twine is a utility for interacting with PyPI that uses TLS to upload distributions, and prevents MITM attacks on usernames and passwords.

Build backends

Build backends MAY be modified to sign metadata and to upload signed distributions to PyPI.

Automated Signing Solution

An easy-to-use key management solution is RECOMMENDED for developers. One approach is to generate a cryptographic private key from a user password, akin to miniLock. Although developer signatures can remain optional, this approach may be inadequate due to the great number of potentially unsigned dependencies each distribution may have. If any one of these dependencies is unsigned, it negates any benefit the project gains from signing its own distribution (i.e., attackers would only need to compromise one of the unsigned dependencies to attack end-users). Requiring developers to manually sign distributions and manage keys is expected to render key signing an unused feature.

A default, PyPI-mediated key management and package signing solution that is transparent 18 to developers and does not require a key escrow (sharing of encrypted private keys with PyPI) is RECOMMENDED for the signing tools. Additionally, the signing tools SHOULD circumvent the sharing of private keys across multiple machines of each developer.

The following outlines an automated signing solution that a new developer MAY follow to upload a distribution to PyPI:

1. Register a PyPI project.
2. Enter a secondary password (independent of the PyPI user account password).
3. Optional: Add a new identity to the developer’s PyPI user account from a second machine (after a password prompt).
4. Upload project.

Step 1 is the normal procedure followed by developers to register a PyPI project 19.

Step 2 generates an encrypted key file (private), uploads an Ed25519 public key to PyPI, and signs the TUF metadata that is generated for the distribution.

Optionally adding a new identity from a second machine, by simply entering a password, in step 3 also generates an encrypted private key file and uploads an Ed25519 public key to PyPI. Separate identities MAY be created to allow a developer, or other project maintainers, to sign releases on multiple machines. An existing verified identity (its public key is contained in project metadata or has been uploaded to PyPI) signs for new identities. By default, project metadata has a signature threshold of “1” and other verified identities may create new releases to satisfy the threshold.

Step 4 uploads the distribution file and TUF metadata to PyPI. The “Snapshot Process” section discusses in detail the procedure followed by developers to upload a distribution to PyPI.

Generation of cryptographic files and signatures is transparent to the developers in the default case: developers need not be aware that packages are automatically signed. However, the signing tools should be flexible; a single project key may also be shared between multiple machines if manual key management is preferred (e.g., ssh-copy-id).

The repository 20 and developer 21 TUF tools currently support all of the recommendations previously mentioned, except for the automated signing solution, which SHOULD be added to Distlib, Twine, and other third-party signing tools. The automated signing solution calls available repository tool functions to sign metadata and to generate the cryptographic key files.

Snapshot Process

The snapshot process is fairly simple and SHOULD be automated. The snapshot process MUST keep in memory the latest working set of root, targets, and delegated roles. Every minute or so the snapshot process will sign for this latest working set. (Recall that project uploads continuously inform the snapshot process about the latest delegated metadata in a concurrency-safe manner. The snapshot process will actually sign for a copy of the latest working set while the latest working set in memory will be updated with information that is continuously communicated by the project transaction processes.) The snapshot process MUST generate and sign new timestamp metadata that will vouch for the metadata (root, targets, and delegated roles) generated in the previous step. Finally, the snapshot process MUST make available to clients the new timestamp and snapshot metadata representing the latest snapshot.

A claimed or recently-claimed project will need to upload in its transaction to PyPI not just targets (a simple index as well as distributions) but also TUF metadata. The project MAY do so by uploading a ZIP file containing two directories, /metadata/ (containing delegated targets metadata files) and /targets/ (containing targets such as the project simple index and distributions that are signed by the delegated targets metadata).

Whenever the project uploads metadata or target files to PyPI, PyPI SHOULD check the project TUF metadata for at least the following properties:

• A threshold number of the developers keys registered with PyPI by that project MUST have signed for the delegated targets metadata file that represents the “root” of targets for that project (e.g. metadata/targets/ project.txt).
• The signatures of delegated targets metadata files MUST be valid.
• The delegated targets metadata files MUST NOT have expired.
• The delegated targets metadata MUST be consistent with the targets.
• A delegator MUST NOT delegate targets that were not delegated to itself by another delegator.
• A delegatee MUST NOT sign for targets that were not delegated to itself by a delegator.

If PyPI chooses to check the project TUF metadata, then PyPI MAY choose to reject publishing any set of metadata or target files that do not meet these requirements.

PyPI MUST enforce access control by ensuring that each project can only write to the TUF metadata for which it is responsible. It MUST do so by ensuring that project upload processes write to the correct metadata as well as correct locations within those metadata. For example, a project upload process for an unclaimed project MUST write to the correct target paths in the correct delegated unclaimed metadata for the targets of the project.

On rare occasions, PyPI MAY wish to extend the TUF metadata format for projects in a backward-incompatible manner. Note that PyPI will NOT be able to automatically rewrite existing TUF metadata on behalf of projects in order to upgrade the metadata to the new backward-incompatible format because this would invalidate the signatures of the metadata as signed by developer keys. Instead, package managers SHOULD be written to recognize and handle multiple incompatible versions of TUF metadata so that claimed and recently-claimed projects could be offered a reasonable time to migrate their metadata to newer but backward-incompatible formats. One mechanism for handling this version change is described in TAP 14 22.

If PyPI eventually runs out of disk space to produce a new consistent snapshot, then PyPI MAY then use something like a “mark-and-sweep” algorithm to delete sufficiently outdated consistent snapshots. That is, only outdated metadata like timestamp and snapshot that are no longer used are deleted. Specifically, in order to preserve the latest consistent snapshot, PyPI would walk objects – beginning from the root (timestamp) – of the latest consistent snapshot, mark all visited objects, and delete all unmarked objects. The last few consistent snapshots may be preserved in a similar fashion. Deleting a consistent snapshot will cause clients to see nothing except HTTP 404 responses to any request for a target of the deleted consistent snapshot. Clients SHOULD then retry (as before) their requests with the latest consistent snapshot.

All package managers that support TUF metadata MUST be modified to download every metadata and target file (except for timestamp metadata) by including, in the request for the file, the cryptographic hash of the file in the filename. Following the filename convention RECOMMENDED in the next subsection, a request for the file at filename.ext will be transformed to the equivalent request for the file at digest.filename.

Finally, PyPI SHOULD use a transaction log 23 to record project transaction processes and queues so that it will be easier to recover from errors after a server failure.

Producing Consistent Snapshots

PyPI is responsible for updating, depending on the project, either the claimed, recently-claimed, or unclaimed metadata and associated delegated metadata. Every project MUST upload its set of metadata and targets in a single transaction. The uploaded set of files is called the “project transaction.” How PyPI MAY validate files in a project transaction is discussed in a later section. The focus of this section is on how PyPI will respond to a project transaction.

Every metadata and target file MUST include in its filename the hex digest 24 of its BLAKE2b-256 25 hash, which PyPI may prepend to filenames after the files have been uploaded. For this PEP, it is RECOMMENDED that PyPI adopt a simple convention of the form: digest.filename, where filename is the original filename without a copy of the hash, and digest is the hex digest of the hash.

When an unclaimed project uploads a new transaction, a project transaction process MUST add all new target files and relevant delegated unclaimed metadata. The project upload process MUST inform the snapshot process about new delegated unclaimed metadata.

When a recently-claimed project uploads a new transaction, a project upload process MUST add all new target files and delegated targets metadata for the project. If the project is new, then the project upload process MUST also add new recently-claimed metadata with the public keys (which MUST be part of the transaction) for the project. recently-claimed projects have a threshold value of “1” set by the upload process. Finally, the project upload process MUST inform the snapshot process about new recently-claimed metadata, as well as the current set of delegated targets metadata for the project.

The upload process for a claimed project is slightly different in that PyPI administrators periodically move (a manual process that MAY occur every two weeks to a month) projects from the recently-claimed role to the claimed role. (Moving a project from recently-claimed to claimed is a manual process because PyPI administrators have to use an offline key to sign the claimed project’s distribution.) A project upload process MUST then add new recently-claimed and claimed metadata to reflect this migration. As is the case for a recently-claimed project, the project upload process MUST always add all new target files and delegated targets metadata for the claimed project. Finally, the project upload process MUST inform the consistent snapshot process about new recently-claimed or claimed metadata, as well as the current set of delegated targets metadata for the project.

Project upload processes SHOULD be automated, except when PyPI administrators move a project from the recently-claimed role to the claimed role. Project upload processes MUST also be applied atomically: either all metadata and target files – or none of them – are added. The project transaction processes and snapshot process SHOULD work concurrently. Finally, project upload processes SHOULD keep in memory the latest claimed, recently-claimed, and unclaimed metadata so that they will be correctly updated in new consistent snapshots.

The queue MAY be processed concurrently in order of appearance, provided that the following rules are observed:

1. No pair of project upload processes may concurrently work on the same project.
2. No pair of project upload processes may concurrently work on unclaimed projects that belong to the same delegated unclaimed role.
3. No pair of project upload processes may concurrently work on new recently-claimed projects.
4. No pair of project upload processes may concurrently work on new claimed projects.
5. No project upload process may work on a new claimed project while another project upload process is working on a new recently-claimed project and vice versa.

These rules MUST be observed to ensure that metadata is not read from or written to inconsistently.

Auditing Snapshots

If a malicious party compromises PyPI, they can sign arbitrary files with any of the online keys. The roles with offline keys (i.e., root and targets) are still protected. To safely recover from a repository compromise, snapshots should be audited to ensure that files are only restored to trusted versions.

When a repository compromise has been detected, the integrity of three types of information must be validated:

1. If the online keys of the repository have been compromised, they can be revoked by having the targets role sign new metadata, delegated to a new key.
2. If the role metadata on the repository has been changed, this will impact the metadata that is signed by online keys. Any role information created since the compromise should be discarded. As a result, developers of new projects will need to re-register their projects.
3. If the packages themselves may have been tampered with, they can be validated using the stored hash information for packages that existed in trusted metadata before the compromise. Also, new distributions that are signed by developers in the claimed role may be safely retained. However, any distributions signed by developers in the recently-claimed or unclaimed roles should be discarded.

In order to safely restore snapshots in the event of a compromise, PyPI SHOULD maintain a small number of its own mirrors to copy PyPI snapshots according to some schedule. The mirroring protocol can be used immediately for this purpose. The mirrors must be secured and isolated such that they are responsible only for mirroring PyPI. The mirrors can be checked against one another to detect accidental or malicious failures.

Another approach is to periodically generate the cryptographic hash of each snapshot and tweet it. For example, upon receiving the tweet, a user comes forward with the actual metadata and the repository maintainers are then able to verify the metadata’s cryptographic hash. Alternatively, PyPI may periodically archive its own versions of snapshots rather than rely on externally provided metadata. In this case, PyPI SHOULD take the cryptographic hash of every package on the repository and store this data on an offline device. If any package hash has changed, this indicates an attack has occurred.

Attacks that serve different versions of metadata or that freeze a version of a package at a specific version can be handled by TUF with techniques such as implicit key revocation and metadata mismatch detection 2.

In the Event of a Key Compromise

A key compromise means that a threshold of keys belonging to developers or the roles on PyPI, as well as the PyPI infrastructure, have been compromised and used to sign new metadata on PyPI.

If a threshold number of developer keys of a project have been compromised, the project MUST take the following steps:

1. The project metadata and targets MUST be restored to the last known good consistent snapshot where the project was not known to be compromised. This can be done by developers repackaging and resigning all targets with the new keys.
2. The project’s metadata MUST have its version numbers incremented, expiry times suitably extended, and signatures renewed.

Whereas PyPI MUST take the following steps:

1. Revoke the compromised developer keys from the recently-claimed or claimed role. This is done by replacing the compromised developer keys with newly issued developer keys.
2. A new timestamped consistent snapshot MUST be issued.

If a threshold number of timestamp, snapshot, recently-claimed, or unclaimed keys have been compromised, then PyPI MUST take the following steps:

1. Revoke the timestamp, snapshot, and targets role keys from the root role. This is done by replacing the compromised timestamp, snapshot, and targets keys with newly issued keys.
2. Revoke the recently-claimed and unclaimed keys from the targets role by replacing their keys with newly issued keys. Sign the new targets role metadata and discard the new keys (because, as we explained earlier, this increases the security of targets metadata).
3. Clear all targets or delegations in the recently-claimed role and delete all associated delegated targets metadata. Recently registered projects SHOULD register their developer keys again with PyPI.
4. All targets of the recently-claimed and unclaimed roles SHOULD be compared with the last known good consistent snapshot where none of the timestamp, snapshot, recently-claimed, or unclaimed keys were known to have been compromised. Added, updated, or deleted targets in the compromised consistent snapshot that do not match the last known good consistent snapshot SHOULD be restored to their previous versions. After ensuring the integrity of all unclaimed targets, the unclaimed metadata MUST be regenerated.
5. The recently-claimed and unclaimed metadata MUST have their version numbers incremented, expiry times suitably extended, and signatures renewed.
6. A new timestamped consistent snapshot MUST be issued.

This would preemptively protect all of these roles even though only one of them may have been compromised.

If a threshold number of the targets or claimed keys have been compromised, then there is little that an attacker would be able do without the timestamp and snapshot keys. In this case, PyPI MUST simply revoke the compromised targets or claimed keys by replacing them with new keys in the root and targets roles, respectively.

If a threshold number of the timestamp, snapshot, and claimed keys have been compromised, then PyPI MUST take the following steps in addition to the steps taken when either the timestamp or snapshot keys are compromised:

1. Revoke the claimed role keys from the targets role and replace them with newly issued keys.
2. All project targets of the claimed roles SHOULD be compared with the last known good consistent snapshot where none of the timestamp, snapshot, or claimed keys were known to have been compromised. Added, updated, or deleted targets in the compromised consistent snapshot that do not match the last known good consistent snapshot MAY be restored to their previous versions. After ensuring the integrity of all claimed project targets, the claimed metadata MUST be regenerated.
3. The claimed metadata MUST have their version numbers incremented, expiry times suitably extended, and signatures renewed.

Following these steps would preemptively protect all of these roles even though only one of them may have been compromised.

If a threshold number of root keys have been compromised, then PyPI MUST take the steps taken when the targets role has been compromised. All of the root keys must also be replaced.

It is also RECOMMENDED that PyPI sufficiently document compromises with security bulletins. These security bulletins will be most informative when users of pip-with-TUF are unable to install or update a project because the keys for the timestamp, snapshot, or root roles are no longer valid. Users could then visit the PyPI web site to consult security bulletins that would help to explain why users are no longer able to install or update, and then take action accordingly. When a threshold number of root keys have not been revoked due to a compromise, then new root metadata may be safely updated because a threshold number of existing root keys will be used to sign for the integrity of the new root metadata. TUF clients will be able to verify the integrity of the new root metadata with a threshold number of previously known root keys. This will be the common case. In the worst case, where a threshold number of root keys have been revoked due to a compromise, an end-user may choose to update new root metadata with out-of-band 27 mechanisms.