Another year of pkgsrc

News since last pkgsrcCon

Joerg Sonnenberger <joerg@NetBSD.org>

The NetBSD Foundation

Preamble

• Focus on infrastructure changes since last pkgsrcCon
• No credits — I'm not responsible for many of the changes
• No specific order — I tried to group them semantically
• Contact me if I missed something important

Performance improvements

• Ever ktrace(1)'d bmake in pkgsrc?
• bsd.prefs.mk gets included all over the place
• Include guards, but still 16 KB to parse by bmake
• Idea: include guard + include of real file == fast
• bsd.fast.prefs.mk: 10% time saving for kdebase3

Apache tribes

• Apache version 2 as new default
• More consistent naming for Apache and PHP/PEAR modules
  • Inspired by Python handling
  • Apache prefix is ap13 vs. ap2 vs. ap22
  • PHP packages prefixed with php4/php5
  • ap-php dependency selects both Apache and PHP version
• Apache Portable Runtime (APR) still a mess

Splitting BUILD_DEFS

• List of settings affecting the build
• Problem: some are used early by the infrastructure
• Split into package settable files and infrastructure parameters
• LOCALBASE, PKGINFODIR, PKGMANDIR are the latter
• Added BUILD_DEFS_EFFECTS to describe the resulting changes

Unphase build

• Concept of barriers replaces concept of build phases
• Simplifies handling of su targets
• Eliminates most recursive makes
• Bug: NO_CONFIGURE killed the wrappers
• Drop .ORDER, pkgsrc does not support -j
• RECURSIVE_MAKE vs. MAKE, BUILD_ENV vs. PKGSRC_MAKE_ENV
• Resort processing, depends -> tools -> extract -> wrapper

Output consistency

• Before:
  • Echos for warnings
  • More echos for errors
  • ...and mixed between build output
• After:
  • STEP_MSG, PHASE_MSG, WARNING_MSG, ERROR_MSG
  • DELAYED_*_MSG for later inspection
• Improve the config.h parsing to make debug more useful
• Print the original arguments after failing wrapped commands

CHECK_FILES improvements

• Fixed with PREFIX as symlink
• Default for developers (and most bulk builds)
• Make it optional for platforms that can't support it
• Sort file lists first, avoids false positive
• Skip PACKAGES, DISTDIR, fontconfig cache
• Deobscure output for overwritten files

Package consistency (1)

• Stricter PLIST entries for font packages:
  • fonts.alias is static content, must be in PLIST
  • fonts.encoding is handled by the infrastructure
  • To Do: fontconfig handling
• Pass MAKECONF down, su-targets didn't get it
• Fixes to run with set -u and set -e

Package consistency (2)

• Optional check for TOOLS_DIR references
• Permission check added, not active by default (yet)
• SHLIB_HANDLING -> CHECK_SHLIBS_SUPPORTED, controlled by CHECK_SHLIBS, active for developers
• Check for unexpanded ${prefix} and ${exec_prefix} in headers

mtree

• MTREE_FILE and MTREE_ARGS are internal now
• NO_MTREE as default post-2007Q1
• to be gone after 2007Q2

DESTDIR support (1)

• Don't use DESTDIR internally
• Removal of MAKE_PACKAGE_AS_ROOT, DESTDIR replaces this mostly
• _INSTALL_ROOT_USER -> REAL_ROOT_USER, used e.g. for SETUID_ROOT_PERMS
• Added _SU_ROOT_USER to allow unprivileged builds mixed with su-targets
• Honour it in warning/error messages
• Fixed check-shlibs to deal with relative paths

DESTDIR support (2)

• Install to area in WRKDIR
• Conditionally run install target via su-target
• For USE_DESTDIR=yes run make clean as root
• package-install target
• Use @owner / @group to set ownership of files with USE_DESTDIR=full
• Problem with tar and intermediate directories -> pkg_create changes pending to fix issues with this

Modular Xorg support (1)

• kill commands to make X11BASE if it didn't exist, it was never used anyway
• Add include guards in mk/x11.b3.mk as this is now included quite often
• Fail properly if no valid X11_TYPE is set
• No ${WRKDIR}/.x11-buildlink for modular Xorg
• No magic mk/x11.b3.mk include for USE_X11BASE

Modular Xorg support (2)

• No xpkgwedge
• USE_XPKGWEDGE moved to bsd.prefs.mk -> overriding of X11_TYPE per platform
• Use xdg-dirs directly for modular Xorg
• Fix bugs with Xaw3d and friends: don't use Xaw headers and possibly even wrong lib
• Handle LOCALBASE before X11BASE – more specific first and X11_TYPE=xorg is going away

Modular Xorg support (3)

• For mk/motif.b3.mk, move mk/x11.b3.mk include into non-openmotif, non-lesstif case
• mkfontscale vs type1inst
• Include modular Xorg b3.mk files, redirect to mk/x11.b3.mk for non-modular builds

Bulk build system (1)

• Remove old .bulk-not_available
• BULK_BUILD_ID_FILE is using BULK_ID now
• Avoid zombies in printindex on NetBSD
• Hack sort-packages to speed-up vulnerable search, 60% improvement
• Don't try to pkg_delete -f
• Remove packages with pkg_delete -r "*", no iteration

Bulk build system (2)

• Cleanup by using awk in some places instead of sed
• Optionally preserve WRKDIR on failure, print a message in that case
• Call make once in pkgtools/pkglint to extract variables in one go
• Optionally run only post-build steps
• Replace wc with awk, saving two forks/package

Bulk build system (3)

• Include PKGSRC_COMPILER in the report
• Better handle *SPECIFIC_PKGS, e.g. non-existing directories
• Add lang/nawk and sysutils/install-sh to the precious package list (about to change)
• Refix the upload script to never upload restricted packages
• Create the pkg_summary file
• With current pkg_install, record all packages used as dependency

Refactoring of the infrastructure

• Implementation of most parts split into separate files and direcorties
• Documentation of public and private variables and targets
• More code moved from .mk files to separate shell/awk scripts
• Abstraction of low-level pkg_install access to allow different backends
• Overall aim: smaller, better manageable pieces

Refactoring examples

• show-all-depends-dirs and show-all-depends-dirs-excl replaced by awk scripts
• Fetch code moved into a shell script
• Checksum handling moved into a shell script
• Locking uses make macros, split lock into WRKDIR and PKGDB lock
• Add ${RUN} to replace ${_PKG_SILENT}${_PKG_DEBUG}

Fetching

• FETCH_MESSAGE is public now
• Renamed DYNAMIC_SITE_* to DYNAMIC_SITES_*
• Prefer ${FILESDIR}/getsite.sh over SITE_<file> over MASTER_SITES
• Randomize MASTER_SITES for normal users

Extracting

• Handle new cases: .gem, .cpio, -tar.gz
• EXTRACT_DIR when WRKDIR is not the right target directory
• Replace gzcat with gzip -cd, latter more portable

Package changes

• Added PostgreSQL 8.2, removed 7.3 and 7.4
• Removed teTeX 2
• SuSE 7.3 gone, SuSE 9.1 legacy
• NetBSD before 2.x requires bootstrap
• XFree86 removed

Compiler changes (1)

• Enumerate languages explicitly, don't add to USE_LANGUAGES (nice breakage!)
• Generic C99 support
  • Annotate it as language
  • For GCC, require 3.x or newer
  • Solaris difficult due to broken system headers

Compiler changes (2)

• Missing languages get ${FALSE} wrappers
• f2c-f77 aliased as g77 and f77 for libtool
• f2c as Fortran backend adds C requirement
• Allow full paths for CPP, e.g. Solaris and IRIX need it

Compiler changes (3)

• More GCC options to filter out / translate for other compilers
• Many SunPro fixes, e.g. -O0/-O1 -> -xO0/-xO1
• Allow ccache to use cache outside of ${WRKDIR}

Tools changes

• intltool and itools as full tools, reduces hackery
• Many tools get a broken ${FALSE} version by default
  • Better catches missing dependencies in normal environments
  • Some fallout, e.g. pkg-config breaks GTK detection

Wrapper changes

• Turn off WRAPPER_DEBUG by default, work.log could grow to over 100 MB
• Make the wrapper cache parallel build safe by using copy+rename
• imake-check uses real imake, less issues with the wrappers
• Reordered to be back between patch and configure

Portability fixes (1)

• Normalize ${TEST} vs [ ], use the later
• Use test -f instead of ls to check for libtool, config.*, detects broken symlinks
• CHECK_PORTABILITY
  • Warn about $RANDOM
  • Error out on test == and [ == ] (bash vs coreutils)

Portability fixes (1)

• Many sh implementations are reading byte-wise, avoid it in some cases
• /dev/stderr in awk scripts is bad, e.g. IRIX doesn't have it
• MAKEFILE -> MAKE_FILE, not useful otherwise
• Use find -o, not find -or

MAKE_JOBS

• Allows parallel building within a package
• Works by passing down -jX to make
• Dangerous and prone to failures

pkgsrc documentation

• Refactored help subsystem: make help topic=pkgname
• Now recognises "# variable" comments e.g. in mk.conf
• make show-all
• Split many docs into user and developer parts
• When printing supported options, also show the current package
• Explain errors from check framework

SUBST framework

• SUBST_VARS.* for replacement of @FOO@ by ${FOO}
• SUBST_SKIP_TEXT_CHECK.* disables file(1) check
• Warn about unmodified files
• Even simpler for awk, bash and sh: REPLACE_*

Buildlink changes (1)

• BUILDLINK_ORDER changes to allows tree-like output
• BUILDLINK_RECOMMENDED.foo renamed to BUILDLINK_ABI_DEPENDS.foo
• BUILDLINK_DEPENDS.foo renamed to BUILDLINK_API_DEPENDS.foo
• IGNORE_RECOMMENDED renamed to USE_ABI_DEPENDS with inverted meaning

Buildlink changes (2)

• BUILDLINK_TRANSFORM.* renamed to BUILDLINK_FNAME_TRANSFORM.*
• Common handling for build options

Other cleanup (1)

• If running "make replace", it replaces install
• Silence rm of the work symlink
• Better test infrastructure, similiar to normal build variables
• Deprecate PKG_SKIP_REASON
• Override public targets if PKG_FAIL_REASON is set
• Kill trailing spaces
• Build dependencies of build dependencies no longer work

Other cleanup (2)

• Bootstrap dependency phase
• OVERRIDE_DEPTH for highly nested autoconf users
• NOT_FOR_UNPRIVILEGED / ONLY_FOR_UNPRIVILEGED
• BSD_MAKE_ENV to pass down BINOWN, BINGRP, ...
• No interpreter check in share/doc and share/examples
• More quoting fixes
• NO_DEPENDS renamed to SKIP_DEPENDS

The end

Questions?