Quarterly Branching Plan

(This page is on the pkgsrc wiki so that it is viewable by all.)

pkgsrc has a longstanding practice of creating quarterly stable branches with names like pkgsrc-2020Q1 every quarter. There has typically been a freeze during the last 2 weeks of the quarter, and then a new branch is created.

This wiki page outlines details and rationale around the stable branch process. It is primarily intended for those who have commit access to pkgsrc.

Nothing here applies to pkgsrc-wip.

Goals and Rationale

pkgsrc has a tension between rapidly updating packages to the most recent releases and stability. Stability problems include the failure of newly-updated packages to build, and the failure of packages depending on the newly-updated package to build. Some people use pkgsrc-current and tend to rapid updates, and some people use the quarterly branches and especially their associated binary package sets.

pkgsrc has historically balanced these goals by having updates most of the time, and a 2-week freeze of no updates and extra attention to resolving problems, with the result that for every quarterly branch, essentially all packages (that ever build) build on almost all platforms (where they ever could build). There has sometimes been a notion of "careful mode" before a freeze, which is a notion of avoiding updates unless it is clear that there will be no resulting problems from those updates at the start of the freeze.

PMC has decided that stability of quarterly branches is very important, and that a reasonable measure of this is the success of bulk builds of stable branches across well-supported platforms. In an ideal world, the ~14 day freeze would be sufficient to produce a quality branch. Historically, it has worked well, but as of late 2019 and early 2020 there have been issues with some packages, leading to defective binary package sets. Sometimes this is due to issues of upstream stability, sometimes due to interaction with new language features, and sometimes due to platform bugs. To address this, PMC is setting early freeze times for specific packages.

General Policy

(This is explicitly stated, but believed to be already universally agreed on.)

Bulk build machines should not require special configuration; the checked-in packages should be suitable.

Packages should build reliably even with high MAKE_JOBS (and if not, MAKE_JOBS_SAFE=no should be set, conditioned by OS if the problem is only on one OS).

When an update is committed, there should be, unless there is discussion and consensus, an expectation that the package will build and work on all platforms where it worked before. Additionally, packages that depend on the updated package should continue to work. This is unrealistic given current practices, and a substitute expectation is that all breakage can be found and fixed by the next freeze start, except for a few packages that are deletion candidates because they are abandoned by upstream and unused.

Pre-Freeze Update Policy

To reach the stability goal, the "careful mode" notion is extended and formalized. Packages listed under a date may not be updated on or after that date without PMC approval. Dates are written in terms of the first quarter, and apply to the other quarters.

Packages are sorted into these categories based on a combination of how important they are to users (and hence pkgsrc reputation) and the history of past updates. Ideally, when an update is committed, the package builds on all platforms where it built before, and any depending package with a functioning upstream also builds. If there are problems, they should be resolvable and resolved quickly (order of a week).

Certainly the hope is that after a pattern emerges of repeated uneventful updates (no breakage, or all breakage fixed in a few days) packages can be taken off the special rules list.

The two dates are about a month before the normal freeze start, and two weeks before the normal freeze start.

Always, the following updates require PMC approval:

  • gmake (There are problems with the current release of gmake, and updates are basically on hold until they are resolved. Even then, this is scary enough to warrant discussion before update.)

  • qt5 (Updates have caused breakage and while this is certainly a reflection on the broader qt5 ecosystem, we would like to understand what will, if anything, break, and have results of test-building sentinel packages before an update happens.)

On or after February 15, the following updates require PMC approval:

  • any change in mk that is intended to result in build failures (typically for things that were warnings before)

  • boost (boost has frequent API changes and typically results in many packages not building. There does not seem to be a notion of boost-using upstreams issuing versions compatible with new boost releases before the boost releases.)

  • clang/llvm (New versions almost always are backwards incompatible.)

  • poppler* (poppler has frequent API changes and typically results in many packages not building.)

  • qt5 (always needs approval, but the notion is that updates should not happen after this date)

On or after March 1, the following updates require PMC approval:

  • rust (rust did not build reliably on NetBSD bulk build machines for 2020Q1, and while some say that updates don't make things worse, in general rust is not free of trouble. The situation with rust and whether it should be on this list is particularly dynamic.)

  • firefox (firefox has a particular need to be recent, but there have been serious problems recently, e.g. firefox 74 still has not built on 2020Q1 for either NetBSD-8 i386 or amd64, while 2019Q1 contained 71. New versions often have surprising platform requirements. This does not apply to firefox ESR packages, which do not have a history of trouble.)

  • cmake (While cmake has not been particularly troublesome lately, changes have the potential for widespread breakage.)

  • non-micro updates to widely-used non-versioned languages (e.g. perl 5.X->5.X+1)

  • changing the default version of versioned languages (e.g. go, guile, php, python, ruby)

Future thoughts

It could be that packages with difficulties should be versioned, but versioning brings its own pain.

It could be that packages with difficulties should have updates staged for testing (perhaps in wip).

The overall problem is difficult, and we should perhaps consider the notion of non-bugfix pullups to stable branches, when upstream packages have security fixes but do not release the expected security bugfix releases (e.g., firefox).

It may be that we should keep boost behind the latest release on purpose, because of how boost-using packages and boost relate to each other.

Bulk Build Instructions

It is expected that bulk builds will simply work with the contents of pkgsrc for all reasonable situations, including

  • RELEASE or STABLE of an OS version
  • MAKE_JOBS possibly high
  • with or without entropy (as lack of entropy is common in emulation/VMs)

It is also expected that pkgsrc will build for -current of an OS. Of course, only current -curent is expected to work.

This section documents any differences from those default expectations, and thus defines the line between "the package is broken" and "the bulk build is not set up correctly".

NetBSD

On all versions, (apparent) entropy is required because rust fails without it. There is not currently an articulated reason why anything bad would happen if the entropy is low quality. Therefore:

  • Configure sources with rndctl, even if unsound.
  • Move aside /dev/random and symlink it to /dev/urandom.

NetBSD 8

There are bugs in ld.so that cause problems with building rust, but the fixes have not been pulled up. There are no known reasons to depart from RELEASE for bulk builds. (There are no known reasons to avoid STABLE either.)

NetBSD 9

Rather than 9.0_RELEASE, 9.0_STABLE must be used to get the ld.so fixes needed for rust. The fixes were pulled up on 2020-05-13, so a build from after that date must be used. See lang/rust/Makefile for details.

Binary packages recency

Bulk builds are done either regularly or ad hoc, and this results in their being package sets for some combinations of OS, OS version, CPU arch, and pkgsrc branch. This section talks about what we consider "current" binary package sets (vs. "old" binary package sets), with respect to storage on NetBSD Foundation servers.

Current package sets

Current sets are those which are built

  • for a stable branch or release of an OS (e.g. NetBSD 9_STABLE, not NetBSD-current) that is currently receiving security maintenance
  • from the current pkgsrc stable branch, or the previous one.
  • (For slow CPU architectures where a bulk build takes more than a month, the second previous stable branch is also considered current. E.g., a week after 2020Q2 is cut, any of 2019Q4, 2020Q1, and 2020Q2 count as current for something like sparc.)

Old package sets

Old package sets are those which aren't current, because of e.g.

  • They are built for -current and not a release
  • They are built for an OS branch that is not receiving maintenance
  • They are built for an old pkgsrc branch
  • They are built for pkgsrc-current.