xbstrap: A tool to build OS distributions

[Korona]

Hi everyone,

some time ago, I found that building "large" OSes that depend on multiple interdependent packages¹ with traditional build system like make is a pain: Cross compilers need to be built, external packages need to be downloaded, patched, built and installed into the sysroot in the correct order (that satisfied their dependencies). Often, those packages need to be modified (e.g. for debugging reasons) and recompiled. Rebuilding the whole system from scratch can take hours -- when building managarm, about 5 GiB of object files are generated. Furthermore, modeling all inter-package dependencies in systems like make is complicated: Take for example a cross-GCC. To regenerate the build system of GCC specific versions of autoconf and automake are necessary. The target libraries like libgcc depend on libc and building libc in turn first requires building the compiler itself.

To automate this process and simplify rebuilding single packages, as well as the whole system, I wrote a new build system called xbstrap. This is an announcement as well as a RFC about this build system. xbstrap is specifically adapted to building (hobby) OS distributions. It does not replace build systems for individual packages (such a make, cmake or meson); rather, it augments them. Regarding usability and maturity, xbstrap is powerful is enough to build the entire kernel and userspace of managarm, including it's Wayland stack and Weston; however, it's command-line interface might still have some rough edges.

How does xbstrap work? It takes as input a YAML file to describe the tools (things running on the build machine) and packages (things running on the target machine) that need to be built. For example, to build libpng (which depends on zlib), the following YAML fragment is used:

Code: Select all

packages:
  - name: libpng
    source:
      subdir: 'ports'
      git: 'http://git.code.sf.net/p/libpng/code'
      tag: 'v1.6.34'
      tools_required:
        - host-autoconf-v2.69
        - host-automake-v1.15
        - host-libtool
      regenerate:
        - args: ['./autogen.sh']
    tools_required:
      - system-gcc
    pkgs_required:
      - zlib
    configure:
      - args:
        - '@THIS_SOURCE_DIR@/configure'
        - '--host=x86_64-managarm'
        - '--prefix=/usr'
    build:
      - args: ['make', '-j@PARALLELISM@']
      - args: ['make', 'install']
        environ:
          DESTDIR: '@THIS_COLLECT_DIR@'

For a full example, see managarm's YAML file. I am looking forward to hear your comments on this tool -- and if turns out that you want to try it to build your own OS, I'd be happy to help you and/or hear about your experience.

Best,
Alexander

¹ Such as my own OS, managarm.

[kzinti]

This is very interesting. I bookmarked it for future use.

Does your system support applying patches to downloaded packages? For example I see that you are using patches on bootstrap-managarm and I am wondering if you handle them with xbstrap?

https://github.com/managarm/bootstrap-m ... er/patches

[Korona]

Yes, those patches are applied automatically. It goes through this cycle for all packages:
fetch (e.g. pull from git or download a tar file) -> checkout -> patch -> regenerate (e.g. by running autoconf) -> configure -> build (installs the package to a temporary dir and packs it into a tar) -> install (installs the tar file into the sysroot).

[davidv1992]

Out of interest, how does it discover those patches to apply?

[Korona]

Right, it just scans the patches/<package> directory relative to the configuration YAML file. Patches are applied in lexicographical order, which interacts nicely with git format-patch that outputs patches exactly in that order. The discovery procedure could be more configurable -- and I'd be happy to do that if someone needs it, but for my own OS project, it's not necessary.

[sydbarrett74]

Korona,

Have you ever checked out the OSKit project at the University of Utah? https://www.cs.utah.edu/flux/oskit/

It's old (last updated in 2002) but you might be able to mine some ideas from it.

sydbarrett74

[Korona]

Hi,

I did not look into OSKit yet. However, from a quick glance, it seems somewhat unrelated to xbstrap. xbstrap is a build system but does not try to provide libraries for OS development.

Managarm, my OS, is developed based on a different philosophy (it uses a smallish microkernel and its own set of userspace drivers).

Best,
Alexander

[Korona]

Hi everyone,

after more than a year, I feel that an update is due. xbstrap is currently at version 0.13. It has gained many new capabilities since its initial launch and several OS projects other than Managarm adopted it, including quack, qword, Sigma and Vineyard.

New features since the initial post include:

• Automated updates. xbstrap can now automatically detect whether repositories have changed and - if that is the case - rebuild packages automatically. This can be also done recursively, for example such that downstream packages have the chance to pick up new functions added by upstream dependencies in their configure scripts.
• Mercurial support. hg is now supported in addition to Git and plain tar/zip archives.
• Custom tasks. Custom tasks can be specified directly in the bootstrap.yml file. For example, this is useful to run tests on a compiled package or to build a disk image that depends a certain set of packages. Like packages, custom talks are fully integrated into the dependency resolution mechanism (e.g., they can depend on packages or other tasks and can be forced to run in a fixed order compared to other build steps).
• Importing packages. The bootstrap.yml file can now be split into multiple files and package descriptions can be imported from these additional files.
• Cross pkg-config support. xbstrap can now automatically wrap a pkg-config command for the target system. This is useful for packages that need both a pkg-config for the build system and a pkg-config for the target system.
• xbps support. xbstrap can automatically generate and install xbps binary packages for the target system. We also consider adding support for other package managers such as .deb or .rpm, should anyone be interested in that.
• Better CI integration. We now provide a second CLI interface to the same bootstrap.yml file that is better suited for CI and automated build bots. While xbstrap's CLI is focused on building and installing packages, the new xbstrap-pipeline tool provides a view on the bootstrap.yml file that is centered around jobs and artifacts built by these jobs. Using this better CI integration does not require any source changes to bootstrap.yml files.

A big thanks to the contributors who made these features possible.

Last but not least, we also have a Discord: https://discord.gg/7WB6Ur3 server. If you decide to give xbstrap a try, feel free to join and chat with us about your use case and your requirements.

Best,
Alexander