OSDev.org https://forum.osdev.org/ |
|
xbstrap: A tool to build OS distributions https://forum.osdev.org/viewtopic.php?f=2&t=33417 |
Page 1 of 1 |
Author: | Korona [ Fri Jan 04, 2019 9:04 am ] |
Post subject: | xbstrap: A tool to build OS distributions |
Hi everyone, some time ago, I found that building "large" OSes that depend on multiple interdependent packages¹ with traditional build system like make is a pain: Cross compilers need to be built, external packages need to be downloaded, patched, built and installed into the sysroot in the correct order (that satisfied their dependencies). Often, those packages need to be modified (e.g. for debugging reasons) and recompiled. Rebuilding the whole system from scratch can take hours -- when building managarm, about 5 GiB of object files are generated. Furthermore, modeling all inter-package dependencies in systems like make is complicated: Take for example a cross-GCC. To regenerate the build system of GCC specific versions of autoconf and automake are necessary. The target libraries like libgcc depend on libc and building libc in turn first requires building the compiler itself. To automate this process and simplify rebuilding single packages, as well as the whole system, I wrote a new build system called xbstrap. This is an announcement as well as a RFC about this build system. xbstrap is specifically adapted to building (hobby) OS distributions. It does not replace build systems for individual packages (such a make, cmake or meson); rather, it augments them. Regarding usability and maturity, xbstrap is powerful is enough to build the entire kernel and userspace of managarm, including it's Wayland stack and Weston; however, it's command-line interface might still have some rough edges. How does xbstrap work? It takes as input a YAML file to describe the tools (things running on the build machine) and packages (things running on the target machine) that need to be built. For example, to build libpng (which depends on zlib), the following YAML fragment is used: Code: packages: - name: libpng source: subdir: 'ports' git: 'http://git.code.sf.net/p/libpng/code' tag: 'v1.6.34' tools_required: - host-autoconf-v2.69 - host-automake-v1.15 - host-libtool regenerate: - args: ['./autogen.sh'] tools_required: - system-gcc pkgs_required: - zlib configure: - args: - '@THIS_SOURCE_DIR@/configure' - '--host=x86_64-managarm' - '--prefix=/usr' build: - args: ['make', '-j@PARALLELISM@'] - args: ['make', 'install'] environ: DESTDIR: '@THIS_COLLECT_DIR@' For a full example, see managarm's YAML file. I am looking forward to hear your comments on this tool -- and if turns out that you want to try it to build your own OS, I'd be happy to help you and/or hear about your experience. Best, Alexander ¹ Such as my own OS, managarm. |
Author: | kzinti [ Fri Jan 04, 2019 10:56 pm ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
This is very interesting. I bookmarked it for future use. Does your system support applying patches to downloaded packages? For example I see that you are using patches on bootstrap-managarm and I am wondering if you handle them with xbstrap? https://github.com/managarm/bootstrap-m ... er/patches |
Author: | Korona [ Sat Jan 05, 2019 4:15 am ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
Yes, those patches are applied automatically. It goes through this cycle for all packages: fetch (e.g. pull from git or download a tar file) -> checkout -> patch -> regenerate (e.g. by running autoconf) -> configure -> build (installs the package to a temporary dir and packs it into a tar) -> install (installs the tar file into the sysroot). |
Author: | davidv1992 [ Mon Jan 07, 2019 4:44 am ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
Out of interest, how does it discover those patches to apply? |
Author: | Korona [ Mon Jan 07, 2019 1:32 pm ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
Right, it just scans the patches/<package> directory relative to the configuration YAML file. Patches are applied in lexicographical order, which interacts nicely with git format-patch that outputs patches exactly in that order. The discovery procedure could be more configurable -- and I'd be happy to do that if someone needs it, but for my own OS project, it's not necessary. |
Author: | sydbarrett74 [ Sun Mar 17, 2019 4:32 pm ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
Korona, Have you ever checked out the OSKit project at the University of Utah? https://www.cs.utah.edu/flux/oskit/ It's old (last updated in 2002) but you might be able to mine some ideas from it. sydbarrett74 |
Author: | Korona [ Mon Mar 18, 2019 10:53 am ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
Hi, I did not look into OSKit yet. However, from a quick glance, it seems somewhat unrelated to xbstrap. xbstrap is a build system but does not try to provide libraries for OS development. Managarm, my OS, is developed based on a different philosophy (it uses a smallish microkernel and its own set of userspace drivers). Best, Alexander |
Author: | Korona [ Fri May 15, 2020 10:30 am ] |
Post subject: | Re: xbstrap: A tool to build OS distributions |
Hi everyone, after more than a year, I feel that an update is due. xbstrap is currently at version 0.13. It has gained many new capabilities since its initial launch and several OS projects other than Managarm adopted it, including quack, qword, Sigma and Vineyard. New features since the initial post include:
A big thanks to the contributors who made these features possible. Last but not least, we also have a Discord: https://discord.gg/7WB6Ur3 server. If you decide to give xbstrap a try, feel free to join and chat with us about your use case and your requirements. Best, Alexander |
Page 1 of 1 | All times are UTC - 6 hours |
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |