Packaging Complex Software

Amitai Schlair

The NetBSD Project

What's simple software?

• Software that:
• is portable
• is licensed in simple, commonly understood terms
• follows Unix idioms and practices
• "wants" to be packaged
• ./configure && make && make install
• see also jmmv's "Making Packager-Friendly Software" [Part 1] and [Part 2]

Packaging simple software

• url2pkg http://master.site/path
• pkgvi, mkpatches, make mps, make install, make print-PLIST, etc.

What can add complexity?

• compile-time build options
• multiple build targets with differing requirements
• atypical build process
• atypical installation paths
• atypical run scripts
• atypical use of UIDs and GIDs
• atypical license

Examples?

Example: qmail

• PKG_OPTIONS: 12 (as of this writing).
• Run scripts: don't daemonize, don't syslog, run in nondeterministic order from /service.
• UIDs and GIDs: compiled into binaries, instead of runtime lookup with getpwnam().
• License: none at all. There are restrictions on redistribution: distributing modified versions of qmail is prohibited. This makes packaging difficult.
• Build: make && make setup check. At least something's simple!

qmail redistribution terms

From DJB's Information for distributors:

You are permitted to distribute a precompiled var-qmail package if(1) installing the package produces exactlythe same /var/qmail hierarchy as a user would obtainby downloading, compiling, and installingqmail-1.03.tar.gz, fastforward-0.51.tar.gz, and dot-forward-0.71.tar.gz;(2) the package behaves correctly,i.e.,the same way as normal qmail+fastforward+dot-forward installationson all other systems; and(3) the package's creatorwarrants that he has made a good-faith attempt to ensure that thepackage behaves correctly.

What to do?

• The package doesn't behave well
• I want it to behave well
• I want to be able to make binary packages that work and can be redistributed
• Some creativity will be required!

qmail install paths

• /var/qmail/{alias,bin,boot,control,doc,man,queue,users}
• Can't force it under ${PREFIX}. Still wouldn't follow hier(7).
• Mustn't patch it!
• With an INSTALL script, pre-create those paths as symlinks pointing to the proper locations. Default qmail paths still work, yet files go where pkgtools (and pkgsrc users) expect.
• A few filename conflicts ensued with other mail software: courier-maildirmake, pulsar, mutt. Mustn't modify qmail. Resolved with cooperation from maintainers of those packages.

qmail users and groups

• UIDs and GIDs are compiled in.
• Mustn't patch it to use getpwnam() or a simple file lookup.
• PKG_USERS and PKG_GROUPS run too late.
• Create users and groups in a separate qmail-users package, then depend on it.
• A binary package of qmail would requires consistent UIDs and GIDs across systems. Our other binary packages don't need this. DJB's suggested workaround: change the compiled-in IDs with the included idedit tool. Of course, this renders package checksums invalid. This is the only reason we don't have a fully functional qmail binary package.

qmail patches

• Modifications are distributed as patches.
• Mustn't apply any patches by default.
• Many useful patches exist. Some better than others.
• Different people want different features.
• Hooray for bsd.options.mk. qmail has a carefully chosen subset of common patches as PKG_OPTIONS.
• Not all combinations of patches apply cleanly. Can't test all combinations. Instead, heuristically provide a helpful message when the patch phase fails (PKGSRC_PATCH_FAIL).

qmail run scripts

• Provided run scripts stay in the foreground, logging to stdout. No dependency ordering.
• Mustn't add custom scripts.
• Write hairy rc.d scripts for qmail-send, qmail-pop3d, qmail-smtpd such that they background and log to syslog. Put them in a separate qmail-run package that depends on qmail.
• While there, provide a few more scripts: qmail-procmail, qmail-qfilter-queue, qmail-qread-client, qmail.
• Some basic config files need to be created before qmail can start. qmail-run creates them if needed.

Results

• Installing qmail and qmail-run from pkgsrc is easy and (mostly) matches the expectations of a pkgsrc user. Install, set a couple variables in rc.conf, edit mailer.conf, et voila.
• Binary packages almost work. And when they do, we can distribute them.

Example 2: ce (Chet's Emacs)

• curses, X11 GUI, and documentation
• Small package, but needed to be split
• Let's look closer...

Discussion

• Existing packages (in pkgsrc or pkgsrc-wip) that face similar problems? Different problems?
• Software you've tried packaging and had trouble?

FIN