OSDev.org
https://forum.osdev.org/

xbstrap: A tool to build OS distributions
https://forum.osdev.org/viewtopic.php?f=2&t=33417
Page 1 of 1

Author:  Korona [ Fri Jan 04, 2019 9:04 am ]
Post subject:  xbstrap: A tool to build OS distributions

Hi everyone,

some time ago, I found that building "large" OSes that depend on multiple interdependent packages┬╣ with traditional build system like make is a pain: Cross compilers need to be built, external packages need to be downloaded, patched, built and installed into the sysroot in the correct order (that satisfied their dependencies). Often, those packages need to be modified (e.g. for debugging reasons) and recompiled. Rebuilding the whole system from scratch can take hours -- when building managarm, about 5 GiB of object files are generated. Furthermore, modeling all inter-package dependencies in systems like make is complicated: Take for example a cross-GCC. To regenerate the build system of GCC specific versions of autoconf and automake are necessary. The target libraries like libgcc depend on libc and building libc in turn first requires building the compiler itself.

To automate this process and simplify rebuilding single packages, as well as the whole system, I wrote a new build system called xbstrap. This is an announcement as well as a RFC about this build system. xbstrap is specifically adapted to building (hobby) OS distributions. It does not replace build systems for individual packages (such a make, cmake or meson); rather, it augments them. Regarding usability and maturity, xbstrap is powerful is enough to build the entire kernel and userspace of managarm, including it's Wayland stack and Weston; however, it's command-line interface might still have some rough edges.

How does xbstrap work? It takes as input a YAML file to describe the tools (things running on the build machine) and packages (things running on the target machine) that need to be built. For example, to build libpng (which depends on zlib), the following YAML fragment is used:
Code:
packages:
  - name: libpng
    source:
      subdir: 'ports'
      git: 'http://git.code.sf.net/p/libpng/code'
      tag: 'v1.6.34'
      tools_required:
        - host-autoconf-v2.69
        - host-automake-v1.15
        - host-libtool
      regenerate:
        - args: ['./autogen.sh']
    tools_required:
      - system-gcc
    pkgs_required:
      - zlib
    configure:
      - args:
        - '@THIS_SOURCE_DIR@/configure'
        - '--host=x86_64-managarm'
        - '--prefix=/usr'
    build:
      - args: ['make', '-j@PARALLELISM@']
      - args: ['make', 'install']
        environ:
          DESTDIR: '@THIS_COLLECT_DIR@'


For a full example, see managarm's YAML file. I am looking forward to hear your comments on this tool -- and if turns out that you want to try it to build your own OS, I'd be happy to help you and/or hear about your experience.

Best,
Alexander

┬╣ Such as my own OS, managarm.

Author:  kzinti [ Fri Jan 04, 2019 10:56 pm ]
Post subject:  Re: xbstrap: A tool to build OS distributions

This is very interesting. I bookmarked it for future use.

Does your system support applying patches to downloaded packages? For example I see that you are using patches on bootstrap-managarm and I am wondering if you handle them with xbstrap?

https://github.com/managarm/bootstrap-m ... er/patches

Author:  Korona [ Sat Jan 05, 2019 4:15 am ]
Post subject:  Re: xbstrap: A tool to build OS distributions

Yes, those patches are applied automatically. It goes through this cycle for all packages:
fetch (e.g. pull from git or download a tar file) -> checkout -> patch -> regenerate (e.g. by running autoconf) -> configure -> build (installs the package to a temporary dir and packs it into a tar) -> install (installs the tar file into the sysroot).

Author:  davidv1992 [ Mon Jan 07, 2019 4:44 am ]
Post subject:  Re: xbstrap: A tool to build OS distributions

Out of interest, how does it discover those patches to apply?

Author:  Korona [ Mon Jan 07, 2019 1:32 pm ]
Post subject:  Re: xbstrap: A tool to build OS distributions

Right, it just scans the patches/<package> directory relative to the configuration YAML file. Patches are applied in lexicographical order, which interacts nicely with git format-patch that outputs patches exactly in that order. The discovery procedure could be more configurable -- and I'd be happy to do that if someone needs it, but for my own OS project, it's not necessary.

Author:  sydbarrett74 [ Sun Mar 17, 2019 4:32 pm ]
Post subject:  Re: xbstrap: A tool to build OS distributions

Korona,

Have you ever checked out the OSKit project at the University of Utah? https://www.cs.utah.edu/flux/oskit/

It's old (last updated in 2002) but you might be able to mine some ideas from it. :D

sydbarrett74

Author:  Korona [ Mon Mar 18, 2019 10:53 am ]
Post subject:  Re: xbstrap: A tool to build OS distributions

Hi,

I did not look into OSKit yet. However, from a quick glance, it seems somewhat unrelated to xbstrap. xbstrap is a build system but does not try to provide libraries for OS development.

Managarm, my OS, is developed based on a different philosophy (it uses a smallish microkernel and its own set of userspace drivers).

Best,
Alexander

Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/