Quarterly Branching Plan

pkgsrc has a longstanding practice of creating quarterly stable branches with names like pkgsrc-2020Q1 every quarter. There has typically been a freeze during the last 2 weeks of the quarter, and then a new branch is created.

This wiki page outlines details and rationale around the stable branch process. It is primarily intended for those who have commit access to pkgsrc.

Nothing here applies to pkgsrc-wip.

Goals and Rationale

pkgsrc has a tension between rapidly updating packages to the most recent releases and stability. Stability problems include the failure of newly-updated packages to build, and the failure of packages depending on the newly-updated package to build. Some people use pkgsrc-current and tend to rapid updates, and some people use the quarterly branches and especially their associated binary package sets.

pkgsrc has historically balanced these goals by having updates most of the time, and a 2-week freeze of no updates and extra attention to resolving problems, with the result that for every quarterly branch, essentially all packages (that ever build) build on almost all platforms (where they ever could build). There has sometimes been a notion of "careful mode" before a freeze, which is a notion of avoiding updates unless it is clear that there will be no resulting problems from those updates at the start of the freeze.

PMC has decided that stability of quarterly branches is very important, and that a reasonable measure of this is the success of bulk builds of stable branches across well-supported platforms. In an ideal world, the ~14 day freeze would be sufficient to produce a quality branch. Historically, it has worked well, but as of late 2019 and early 2020 there have been issues with some packages, leading to defective binary package sets. Sometimes this is due to issues of upstream stability, sometimes due to interaction with new language features, and sometimes due to platform bugs. To address this, PMC is setting early freeze times for specific packages.

While quarterly branches should build on NetBSD-current (and probably other OS's -current) at branch cut time, we do not update stable branches to accomodate changes in -current. Therefore, NetBSD-current tends to require pkgsrc-current.

General Policy

(This is explicitly stated, but believed to be already universally agreed on.)

Bulk build machines should not require special configuration; the checked-in packages should be suitable.

Packages should build reliably even with high MAKE_JOBS (and if not, MAKE_JOBS_SAFE=no should be set, conditioned by OS if the problem is only on one OS).

When an update is committed, there should be, unless there is discussion and consensus, an expectation that the package will build and work on all platforms where it worked before. Additionally, packages that depend on the updated package should continue to work. This is unrealistic given current practices, and a substitute expectation is that all breakage can be found and fixed by the next freeze start, except for a few packages that are deletion candidates because they are abandoned by upstream and unused.

The general expectation that (non-abandoned) depending packages should continue to work implies consideration of whether there are API changes in an update. If the package with API changes has not done an adequate job of documenting and communicating those changes, such that depending packages do not have a compatible release, then pkgsrc as a group should decide whether to hold the offending update until the rest of the software community copes with the breakage, and there is a strong presumption that we should wait. This logically follows from the expectation that depending packages do not break.

Pre-Freeze Update Policy

To reach the stability goal, the "careful mode" notion is extended and formalized. Packages listed under a date may not be updated on or after that date without PMC approval. Dates are written in terms of the first quarter, and apply to the other quarters.

Packages are sorted into these categories based on a combination of how important they are to users (and hence pkgsrc reputation) and the history of past updates. Ideally, when an update is committed, the package builds on all platforms where it built before, and any depending package with a functioning upstream also builds. If there are problems, they should be resolvable and resolved quickly (order of a week).

Certainly the hope is that after a pattern emerges of repeated uneventful updates (no breakage, or all breakage fixed in a few days) packages can be taken off the special rules list.

The two dates are about a month before the normal freeze start, and two weeks before the normal freeze start.

Always, the following updates require PMC approval:

By "require PMC approval", we intend to decide as a group that an update is good for pkgsrc, allowing those who think it's too soon to be part of the discussion.

Boost (Boost has frequent API breaks. Many things depend on boost, and it is typical for a number of those to fail to build after a boost update. Depending packages typically lag boost in that one needs a release after a boost release to use that new boost release. It is not feasible to have multiple boost versions. Before a boost update is committed, there should be an explanation of why we believe it will not cause significant breakage. A report that meta-pkgs/bulk-test-boost builds would be an excellent way to show that the new update is ok! It is expected that this may result in pkgsrc typically getting new boost version some number of months after their release.)
gmake (While we have moved beyond the previous scary update, any future non-micro update has enough consequences that a group decision is warranted..)
icu (frequent API changes)
qt5 (Updates have caused breakage and while this is certainly a reflection on the broader qt5 ecosystem, we would like to understand what will, if anything, break, and have results of test-building sentinel packages before an update happens.)

On or after February 15 (May 15, August 15, November 15), the following updates require PMC approval:

any package always needing approval (boost, gmake, qt5) - the standard answer will be wait until after the branch
any change in mk that is intended to result in build failures (typically for things that were warnings before)
clang/llvm (New versions almost always are backwards incompatible.)
poppler* (frequent API changes)
Any package with a history of a troubled update in the last 3 years: glib2 py-sip6

On or after March 1 (June 1, September 1, December 1), the following updates require PMC approval:

rust (There have been issues with rust, and we have not yet had a 4-branch period where there are no issues. Most recently, an update to 1.54.0 in 2021-09 broke librsvg on NetBSD/aarch64.) See the rust update page
firefox (firefox has a particular need to be recent, but there have been serious problems recently, e.g. firefox 74 still has not built on 2020Q1 for either NetBSD-8 i386 or amd64, while 2019Q1 contained 71. New versions often have surprising platform requirements. This does not apply to firefox ESR packages, which do not have a history of trouble.)
gtk3
wayland
cmake (While cmake has not been particularly troublesome lately, changes have the potential for widespread breakage.)
non-micro updates to pbulk, distcc, ccache, and similar things that have non-trivial use to build packages at scale
non-micro updates to widely-used non-versioned languages (e.g. perl 5.X->5.X+1, ocaml)
changing the default version of versioned languages and systems (e.g. go, guile, php, python, ruby, postgresql, mysql)
non-micro updates to packages with more than 500 (recursive) dependencies

Future thoughts

It could be that packages with difficulties should have updates staged for testing (perhaps in wip). (This is not currently a requirement.)

The overall problem is difficult, and we should perhaps consider the notion of non-bugfix pullups to stable branches, when upstream packages have security fixes but do not release the expected security bugfix releases (e.g., firefox).

Freeze Rules

In general, each freeze has different rules.

By definition, a "micro update" is one that contains only bugfixes. It cannot contain build system changes, or anything that could cause it to fail on a platform it has not been tested on. (Typically, a release labeled micro from an upstream is a micro release, but some upstreams make bigger changes; our definition is about actual content, not upstream labeling.)

gdt's rules

No new packages
Build fixes are welcome
Bug fixes (especially security fixes) are very welcome
Micro updates that would be eligible and appropriate for stable branch pullups, because they fix a security bug or something serious, are ok
Micro updates for packages whose recursive total depending package count is < 25 are ok
For any other updates (especially under mk or pkgtools), please ask, explaining why the update furthers pkgsrc's goals of stabilty and usefulness for users. While there's no intent to completely prohibit changes, I'm concerned about impacts to platforms not tested, and I value working packages from the branch over packages being slightly newer.
No large-scale cleanup of things that are not causing failure of a package to build, package, and run, e.g. HOMEPAGE and MASTER_SITES.

Bulk Build Instructions

It is expected that bulk builds will simply work with the contents of pkgsrc for all reasonable situations, including

RELEASE or STABLE of an OS version
MAKE_JOBS possibly high
with or without entropy (as lack of entropy is common in emulation/VMs)

This means that any comment like "to make a bulk build work, you must change X" is a statement that there is a bug in pkgsrc, which should be addressed.

It is also expected that pkgsrc will build for -current of an OS. Of course, only current -curent is expected to work.

This section documents any differences from those default expectations, and thus defines the line between "the package is broken" and "the bulk build is not set up correctly".

NetBSD

Some packages require random numbers during the build process. If they use /dev/random or getrandom, they may block waiting for entropy.

Ensure you have at least 256 bits of entropy in rndctl -l. See entropy(7).

On NetBSD 8-9, /dev/random should be a symlink to /dev/urandom in the build environment.

NetBSD 8

There are bugs in ld.so that cause problems with building rust, but the fixes have not been pulled up. There are no known reasons to depart from RELEASE for bulk builds. (There are no known reasons to avoid STABLE either.) \todo Still valid in 2022?

NetBSD 9

Rather than 9.0_RELEASE, 9.0_STABLE must be used to get the ld.so fixes needed for rust. The fixes were pulled up on 2020-05-13, so a build from after that date must be used. See lang/rust/Makefile for details.

Promotion of binary package sets

There is a tension between having security updates and being able to run desired programs. Upstream packages sometimes have problems which make them fail to build or work on various platforms.

Package sets are each available at a URL named by OS, OS version, CPU, and branch. New package sets arrive, and old sets are removed. This is straightforward and people that want a particular set for some reason can choose that set by using the URL, manually or via pkgin.

There are also symlinks that are intended to refer to the most recent pacakage set recommended for general use, so that upgrading via pkgin will simply move branches when that's the right thing to do. Many users have this configuration.

As each branch is created, some packages will fail to build on some platforms, even when the build environment is healthy and has high memory. Additionally, some packages will fail to build in some bulk environments because of various issues that can be worked around. And, some packages will build, or even have been added, that were not present before.

After a branch, binary package sets are produced. For a given OS/version/arch, the newer branch's set will have newer packages (including security fixes), but packages may be missing if they do not build on the new branch. Packages can be missing in two main cases:

A. because of memory, randomness, or other issues on the bulk build machine, which can be worked around for the next build cycle. In this case there is an expectation that the package can become available reasonably quickly, typically within a week.
B. because of upstream changes, platform bugs, or other reasons which are in the branch timeframe intractable, the package does not build (for that OS/version/cpu) for anyone. In this case there is no real hope for it building soon.

In case A, it is better to wait for the resolvable problems to be resolved, and then move the symlink, so that users don't upgrade and e.g. end up with firefox missing.

In case B, the choice is between the previous branch which has old versions of packages and is no longer receiving security updates, and the new branch which is maintained but missing some packages that people need. The only satisfactory paths forward are to fix the underlying issues (but they have not been fixed in the branch), for people to stop using those packages (because they are in practice unreliable), or to upgrade their operating systems to newer versions. In cases where a system is not used for desktop purposes, the new sets are often entirely satisfactory.

To address this, our plan is to change the symlink to the new build when issues in category A have been resolved.

Those concerned about category B issues should change their pkgin URL to one with an explicit branch and make their own case-by-case decision.

(Users are cautioned that NetBSD 8 is no longer generally viable for large desktop-type programs.)

Current vs. old binary package sets

Bulk builds are done either regularly or ad hoc, and this results in their being package sets for some combinations of OS, OS version, CPU arch, and pkgsrc branch. Without removal, these grow without bound and exhaust available disk space. Therefore we define what is "current", and should be kept on the NetBSD Foundation ftp server, vs. "old", which should be deleted on a regular basis.

Current package sets

Current sets are those which are built:

for a stable branch or release of an OS (e.g. NetBSD 9_STABLE, not NetBSD-current) that is currently receiving security maintenance.
from the current pkgsrc stable branch, or the previous one (one month after a branch is cut, two branches ago becomes non-current).
(For slow CPU architectures where a bulk build takes more than a month, the second previous stable branch is also considered current. This only applies when the current branch hasn't been built. E.g. Once a valid build of 2022Q2 arrives, 2021Q4 is no longer current.)

Old package sets

Old package sets are those which don't meet the above definitions. Examples:

They are built for NetBSD-current.
They are built for an OS branch that is not receiving maintenance.
They are built for an old pkgsrc branch.
They are built for pkgsrc-current.

For now, we do not host packages for NetBSD-current, unmaintained OS branches, or old pkgsrc branches.

We do allow two builds of pkgsrc-current for NetBSD 9.

Maintenance

Old package sets may be removed at any time, although typically the maintainer of the build will be asked to prune. (Pruning should happen regularly without being asked.)