Contents

• Abstract
• PEP Withdrawal
• Rationale for changes to the core development workflow
• Current Tools
• Proposal
• Deferred Proposals
• Suggested Variants
• Perceived Benefits
• Technical Challenges
  • Kallithea vs Gerrit
  • Mercurial vs Gerrit/git
  • Buildbot vs Jenkins
  • Handling of maintenance branches
  • Handling of security branches
  • Handling of NEWS file updates
  • Stability of “stable” Buildbot slaves
  • Intermittent test failures
  • Custom Mercurial client workflow support
• Social Challenges
• Practical Challenges
• Open Questions
• Next Steps
• Acknowledgements
• References
• Copyright

Abstract

This PEP proposes investing in automation of several of the tedious, time-consuming activities that are currently required for the core development team to incorporate changes into CPython. This proposal is intended to allow core developers to make more effective use of the time they have available to contribute to CPython, which should also result in an improved experience for other contributors that are reliant on the core team to get their changes incorporated.

PEP Withdrawal

This PEP has been withdrawn by the author in favour of the GitLab based proposal in PEP 507.

If anyone else would like to take over championing this PEP, contact the core-workflow mailing list

Rationale for changes to the core development workflow

The current core developer workflow to merge a new feature into CPython on a POSIX system “works” as follows:

1. If applying a change submitted to bugs.python.org by another user, first check they have signed the PSF Contributor Licensing Agreement. If not, request that they sign one before continuing with merging the change.
2. Apply the change locally to a current checkout of the main CPython repository (the change will typically have been discussed and reviewed as a patch on bugs.python.org first, but this step is not currently considered mandatory for changes originating directly from core developers).
3. Run the test suite locally, at least make test or ./python -m test (depending on system specs, this takes a few minutes in the default configuration, but substantially longer if all optional resources, like external network access, are enabled).
4. Run make patchcheck to fix any whitespace issues and as a reminder of other changes that may be needed (such as updating Misc/ACKS or adding an entry to Misc/NEWS)
5. Commit the change and push it to the main repository. If hg indicates this would create a new head in the remote repository, run hg pull --rebase (or an equivalent). Theoretically, you should rerun the tests at this point, but it’s very tempting to skip that step.
6. After pushing, monitor the stable buildbots for any new failures introduced by your change. In particular, developers on POSIX systems will often break the Windows buildbots, and vice-versa. Less commonly, developers on Linux or Mac OS X may break other POSIX systems.

The steps required on Windows are similar, but the exact commands used will be different.

Rather than being simpler, the workflow for a bug fix is more complicated than that for a new feature! New features have the advantage of only being applied to the default branch, while bug fixes also need to be considered for inclusion in maintenance branches.

• If a bug fix is applicable to Python 2.7, then it is also separately applied to the 2.7 branch, which is maintained as an independent head in Mercurial
• If a bug fix is applicable to the current 3.x maintenance release, then it is first applied to the maintenance branch and then merged forward to the default branch. Both branches are pushed to hg.python.org at the same time.

Documentation patches are simpler than functional patches, but not hugely so - the main benefit is only needing to check the docs build successfully rather than running the test suite.

I would estimate that even when everything goes smoothly, it would still take me at least 20-30 minutes to commit a bug fix patch that applies cleanly. Given that it should be possible to automate several of these tasks, I do not believe our current practices are making effective use of scarce core developer resources.

There are many, many frustrations involved with this current workflow, and they lead directly to some undesirable development practices.

• Much of this overhead is incurred on a per-patch applied basis. This encourages large commits, rather than small isolated changes. The time required to commit a 500 line feature is essentially the same as that needed to commit a 1 line bug fix - the additional time needed for the larger change appears in any preceding review rather than as part of the commit process.
• The additional overhead of working on applying bug fixes creates an additional incentive to work on new features instead, and new features are already inherently more interesting to work on - they don’t need workflow difficulties giving them a helping hand!
• Getting a preceding review on bugs.python.org is additional work, creating an incentive to commit changes directly, increasing the reliance on post-review on the python-checkins mailing list.
• Patches on the tracker that are complete, correct and ready to merge may still languish for extended periods awaiting a core developer with the time to devote to getting it merged.
• The risk of push races (especially when pushing a merged bug fix) creates a temptation to skip doing full local test runs (especially after a push race has already been encountered once), increasing the chance of breaking the buildbots.
• The buildbots are sometimes red for extended periods, introducing errors into local test runs, and also meaning that they sometimes fail to serve as a reliable indicator of whether or not a patch has introduced cross platform issues.
• Post-conference development sprints are a nightmare, as they collapse into a mire of push races. It’s tempting to just leave patches on the tracker until after the sprint is over and then try to clean them up afterwards.

There are also many, many opportunities for core developers to make mistakes that inconvenience others, both in managing the Mercurial branches and in breaking the buildbots without being in a position to fix them promptly. This both makes the existing core development team cautious in granting new developers commit access, as well as making those new developers cautious about actually making use of their increased level of access.

There are also some incidental annoyances (like keeping the NEWS file up to date) that will also be necessarily addressed as part of this proposal.

One of the most critical resources of a volunteer-driven open source project is the emotional energy of its contributors. The current approach to change incorporation doesn’t score well on that front for anyone:

• For core developers, the branch wrangling for bug fixes is delicate and easy to get wrong. Conflicts on the NEWS file and push races when attempting to upload changes add to the irritation of something most of us aren’t being paid to spend time on (and for those that are, contributing to CPython is likely to be only one of our responsibilities). The time we spend actually getting a change merged is time we’re not spending coding additional changes, writing or updating documentation or reviewing contributions from others.
• Red buildbots make life difficult for other developers (since a local test failure may not be due to anything that developer did), release managers (since they may need to enlist assistance cleaning up test failures prior to a release) and for the developers themselves (since it creates significant pressure to fix any failures we inadvertently introduce right now, rather than at a more convenient time, as well as potentially making hg bisect more difficult to use if hg annotate isn’t sufficient to identify the source of a new failure).
• For other contributors, a core developer spending time actually getting changes merged is a developer that isn’t reviewing and discussing patches on the issue tracker or otherwise helping others to contribute effectively. It is especially frustrating for contributors that are accustomed to the simplicity of a developer just being able to hit “Merge” on a pull request that has already been automatically tested in the project’s CI system (which is a common workflow on sites like GitHub and BitBucket), or where the post-review part of the merge process is fully automated (as is the case for OpenStack).

Current Tools

The following tools are currently used to manage various parts of the CPython core development workflow.

• Mercurial (hg.python.org) for version control
• Roundup (bugs.python.org) for issue tracking
• Rietveld (also hosted on bugs.python.org) for code review
• Buildbot (buildbot.python.org) for automated testing

This proposal suggests replacing the use of Rietveld for code review with the more full-featured Kallithea-based forge.python.org service proposed in PEP 474. Guido has indicated that the original Rietveld implementation was primarily intended as a public demonstration application for Google App Engine, and switching to Kallithea will address some of the issues with identifying intended target branches that arise when working with patch files on Roundup and the associated reviews in the integrated Rietveld instance.

It also suggests the addition of new tools in order to automate additional parts of the workflow, as well as a critical review of the remaining tools to see which, if any, may be candidates for replacement.

Proposal

The essence of this proposal is that CPython aim to adopt a “core reviewer” development model, similar to that used by the OpenStack project.

The workflow problems experienced by the CPython core development team are not unique. The OpenStack infrastructure team have come up with a well designed automated workflow that is designed to ensure:

• once a patch has been reviewed, further developer involvement is needed only if the automated tests fail prior to merging
• patches never get merged without being tested relative to the current state of the branch
• the main development branch always stays green. Patches that do not pass the automated tests do not get merged

If a core developer wants to tweak a patch prior to merging, they download it from the review tool, modify and upload it back to the review tool rather than pushing it directly to the source code repository.

The core of this workflow is implemented using a tool called Zuul 1, a Python web service created specifically for the OpenStack project, but deliberately designed with a plugin based trigger and action system to make it easier to adapt to alternate code review systems, issue trackers and CI systems. James Blair of the OpenStack infrastructure team provided an excellent overview of Zuul at linux.conf.au 2014.

While Zuul handles several workflows for OpenStack, the specific one of interest for this PEP is the “merge gating” workflow.

For this workflow, Zuul is configured to monitor the Gerrit code review system for patches which have been marked as “Approved”. Once it sees such a patch, Zuul takes it, and combines it into a queue of “candidate merges”. It then creates a pipeline of test runs that execute in parallel in Jenkins (in order to allow more than 24 commits a day when a full test run takes the better part of an hour), and are merged as they pass (and as all the candidate merges ahead of them in the queue pass). If a patch fails the tests, Zuul takes it out of the queue, cancels any test runs after that patch in the queue, and rebuilds the queue without the failing patch.

If a developer looks at a test which failed on merge and determines that it was due to an intermittent failure, they can then resubmit the patch for another attempt at merging.

To adapt this process to CPython, it should be feasible to have Zuul monitor Kallithea for approved pull requests (which may require a feature addition in Kallithea), submit them to Buildbot for testing on the stable buildbots, and then merge the changes appropriately in Mercurial. This idea poses a few technical challenges, which have their own section below.

For CPython, I don’t believe we will need to take advantage of Zuul’s ability to execute tests in parallel (certainly not in the initial iteration - if we get to a point where serial testing of patches by the merge gating system is our primary bottleneck rather than having the people we need in order to be able to review and approve patches, then that will be a very good day).

However, the merge queue itself is a very powerful concept that should directly address several of the issues described in the Rationale above.

Deferred Proposals

The OpenStack team also use Zuul to coordinate several other activities:

• Running preliminary “check” tests against patches posted to Gerrit.
• Creation of updated release artefacts and republishing documentation when changes are merged
• The Elastic recheck 2 feature that uses ElasticSearch in conjunction with a spam filter to monitor test output and suggest the specific intermittent failure that may have caused a test to fail, rather than requiring users to search logs manually

While these are possibilities worth exploring in the future (and one of the possible benefits I see to seeking closer coordination with the OpenStack Infrastructure team), I don’t see them as offering quite the same kind of fundamental workflow improvement that merge gating appears to provide.

However, if we find we are having too many problems with intermittent test failures in the gate, then introducing the “Elastic recheck” feature may need to be considered as part of the initial deployment.

Suggested Variants

Terry Reedy has suggested doing an initial filter which specifically looks for approved documentation-only patches (~700 of the 4000+ open CPython issues are pure documentation updates). This approach would avoid several of the issues related to flaky tests and cross-platform testing, while still allowing the rest of the automation flows to be worked out (such as how to push a patch into the merge queue).

The key downside to this approach is that Zuul wouldn’t have complete control of the merge process as it usually expects, so there would potentially be additional coordination needed around that.

It may be worth keeping this approach as a fallback option if the initial deployment proves to have more trouble with test reliability than is anticipated.

It would also be possible to tweak the merge gating criteria such that it doesn’t run the test suite if it detects that the patch hasn’t modified any files outside the “Docs” tree, and instead only checks that the documentation builds without errors.

As yet another alternative, it may be reasonable to move some parts of the documentation (such as the tutorial and the HOWTO guides) out of the main source repository and manage them using the simpler pull request based model described in PEP 474.

Perceived Benefits

The benefits of this proposal accrue most directly to the core development team. First and foremost, it means that once we mark a patch as “Approved” in the updated code review system, we’re usually done. The extra 20-30 minutes (or more) of actually applying the patch, running the tests and merging it into Mercurial would all be orchestrated by Zuul. Push races would also be a thing of the past - if lots of core developers are approving patches at a sprint, then that just means the queue gets deeper in Zuul, rather than developers getting frustrated trying to merge changes and failing. Test failures would still happen, but they would result in the affected patch being removed from the merge queue, rather than breaking the code in the main repository.

With the bulk of the time investment moved to the review process, this also encourages “development for reviewability” - smaller, easier to review patches, since the overhead of running the tests multiple times will be incurred by Zuul rather than by the core developers.

However, removing this time sink from the core development team should also improve the experience of CPython development for other contributors, as it eliminates several of the opportunities for patches to get “dropped on the floor”, as well as increasing the time core developers are likely to have available for reviewing contributed patches.

Another example of benefits to other contributors is that when a sprint aimed primarily at new contributors is running with just a single core developer present (such as the sprints at PyCon AU for the last few years), the merge queue would allow that developer to focus more of their time on reviewing patches and helping the other contributors at the sprint, since accepting a patch for inclusion would now be a single click in the Kallithea UI, rather than the relatively time-consuming process that it is currently. Even when multiple core developers are present, it is better to enable them to spend their time and effort on interacting with the other sprint participants than it is on things that are sufficiently mechanical that a computer can (and should) handle them.

With most of the ways to make a mistake when committing a change automated out of existence, there are also substantially fewer new things to learn when a contributor is nominated to become a core developer. This should have a dual benefit, both in making the existing core developers more comfortable with granting that additional level of responsibility, and in making new contributors more comfortable with exercising it.

Finally, a more stable default branch in CPython makes it easier for other Python projects to conduct continuous integration directly against the main repo, rather than having to wait until we get into the release candidate phase of a new release. At the moment, setting up such a system isn’t particularly attractive, as it would need to include an additional mechanism to wait until CPython’s own Buildbot fleet indicated that the build was in a usable state. With the proposed merge gating system, the trunk always remains usable.

Technical Challenges

Adapting Zuul from the OpenStack infrastructure to the CPython infrastructure will at least require the development of additional Zuul trigger and action plugins, and may require additional development in some of our existing tools.

Social Challenges

The primary social challenge here is getting the core development team to change their practices. However, the tedious-but-necessary steps that are automated by the proposal should create a strong incentive for the existing developers to go along with the idea.

I believe three specific features may be needed to assure existing developers that there are no downsides to the automation of this workflow:

• Only requiring approval from a single core developer to incorporate a patch. This could be revisited in the future, but we should preserve the status quo for the initial rollout.
• Explicitly stating that core developers remain free to approve their own patches, except during the release candidate phase of a release. This could be revisited in the future, but we should preserve the status quo for the initial rollout.
• Ensuring that at least release managers have a “merge it now” capability that allows them to force a particular patch to the head of the merge queue. Using a separate clone for release preparation may be sufficient for this purpose. Longer term, automatic merge gating may also allow for more automated preparation of release artefacts as well.

Practical Challenges

The PSF runs its own directly and indirectly sponsored workflow infrastructure primarily due to past experience with unacceptably poor performance and inflexibility of infrastructure provided for free to the general public. CPython development was originally hosted on SourceForge, with source control moved to self hosting when SF was both slow to offer Subversion support and suffering from CVS performance issues (see PEP 347), while issue tracking later moved to the open source Roundup issue tracker on dedicated sponsored hosting (from Upfront Systems), due to a combination of both SF performance issues and general usability issues with the SF tracker at the time (the outcome and process for the new tracker selection were captured on the python.org wiki rather than in a PEP).

Accordingly, proposals that involve setting ourselves up for “SourceForge usability and reliability issues, round two” will face significant opposition from at least some members of the CPython core development team (including the author of this PEP). This proposal respects that history by recommending only tools that are available for self-hosting as sponsored or PSF funded infrastructure, and are also open source Python projects that can be customised to meet the needs of the CPython core development team.

However, for this proposal to be a success (if it is accepted), we need to understand how we are going to carry out the necessary configuration, customisation, integration and deployment work.

The last attempt at adding a new piece to the CPython support infrastructure (speed.python.org) has unfortunately foundered due to the lack of time to drive the project from the core developers and PSF board members involved, and the difficulties of trying to bring someone else up to speed to lead the activity (the hardware donated to that project by HP is currently in use to support PyPy instead, but the situation highlights some of the challenges of relying on volunteer labour with many other higher priority demands on their time to steer projects to completion).

Even ultimately successful past projects, such as the source control migrations from CVS to Subversion and from Subversion to Mercurial, the issue tracker migration from SourceForge to Roundup, the code review integration between Roundup and Rietveld and the introduction of the Buildbot continuous integration fleet, have taken an extended period of time as volunteers worked their way through the many technical and social challenges involved.

Fortunately, as several aspects of this proposal and PEP 474 align with various workflow improvements under consideration for Red Hat’s Beaker open source hardware integration testing system and other work-related projects, I have arranged to be able to devote ~1 day a week to working on CPython infrastructure projects.

Together with Rackspace’s existing contributions to maintaining the pypi.python.org infrastructure, I personally believe this arrangement is indicative of a more general recognition amongst CPython redistributors and major users of the merit in helping to sustain upstream infrastructure through direct contributions of developer time, rather than expecting volunteer contributors to maintain that infrastructure entirely in their spare time or funding it indirectly through the PSF (with the additional management overhead that would entail). I consider this a positive trend, and one that I will continue to encourage as best I can.

Open Questions

Pretty much everything in the PEP. Do we want to adopt merge gating and Zuul? How do we want to address the various technical challenges? Are the Kallithea and Zuul development communities open to the kind of collaboration that would be needed to make this effort a success?

While I’ve arranged to spend some of my own work time on this, do we want to approach the OpenStack Foundation for additional assistance, since we’re a key dependency of OpenStack itself, Zuul is a creation of the OpenStack infrastructure team, and the available development resources for OpenStack currently dwarf those for CPython?

Are other interested folks working for Python redistributors and major users also in a position to make a business case to their superiors for investing developer time in supporting this effort?

Next Steps

If pursued, this will be a follow-on project to the Kallithea-based forge.python.org proposal in PEP 474. Refer to that PEP for more details on the discussion, review and proof-of-concept pilot process currently under way.

Acknowledgements

Thanks to Jesse Noller, Alex Gaynor and James Blair for providing valuable feedback on a preliminary draft of this proposal, and to James and Monty Taylor for additional technical feedback following publication of the initial draft.

Thanks to Bradley Kuhn, Mads Kiellerich and other Kallithea developers for the discussions around PEP 474 that led to a significant revision of this proposal to be based on using Kallithea for the review component rather than the existing Rietveld installation.

References

1

http://ci.openstack.org/zuul/

2 (1, 2)

http://status.openstack.org/elastic-recheck/

Copyright

This document has been placed in the public domain.