OSDev.org

The Place to Start for Operating System Developers
It is currently Sun Mar 24, 2019 10:30 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: xbstrap: A tool to build OS distributions
PostPosted: Fri Jan 04, 2019 9:04 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 540
Hi everyone,

some time ago, I found that building "large" OSes that depend on multiple interdependent packages┬╣ with traditional build system like make is a pain: Cross compilers need to be built, external packages need to be downloaded, patched, built and installed into the sysroot in the correct order (that satisfied their dependencies). Often, those packages need to be modified (e.g. for debugging reasons) and recompiled. Rebuilding the whole system from scratch can take hours -- when building managarm, about 5 GiB of object files are generated. Furthermore, modeling all inter-package dependencies in systems like make is complicated: Take for example a cross-GCC. To regenerate the build system of GCC specific versions of autoconf and automake are necessary. The target libraries like libgcc depend on libc and building libc in turn first requires building the compiler itself.

To automate this process and simplify rebuilding single packages, as well as the whole system, I wrote a new build system called xbstrap. This is an announcement as well as a RFC about this build system. xbstrap is specifically adapted to building (hobby) OS distributions. It does not replace build systems for individual packages (such a make, cmake or meson); rather, it augments them. Regarding usability and maturity, xbstrap is powerful is enough to build the entire kernel and userspace of managarm, including it's Wayland stack and Weston; however, it's command-line interface might still have some rough edges.

How does xbstrap work? It takes as input a YAML file to describe the tools (things running on the build machine) and packages (things running on the target machine) that need to be built. For example, to build libpng (which depends on zlib), the following YAML fragment is used:
Code:
packages:
  - name: libpng
    source:
      subdir: 'ports'
      git: 'http://git.code.sf.net/p/libpng/code'
      tag: 'v1.6.34'
      tools_required:
        - host-autoconf-v2.69
        - host-automake-v1.15
        - host-libtool
      regenerate:
        - args: ['./autogen.sh']
    tools_required:
      - system-gcc
    pkgs_required:
      - zlib
    configure:
      - args:
        - '@THIS_SOURCE_DIR@/configure'
        - '--host=x86_64-managarm'
        - '--prefix=/usr'
    build:
      - args: ['make', '-j@PARALLELISM@']
      - args: ['make', 'install']
        environ:
          DESTDIR: '@THIS_COLLECT_DIR@'


For a full example, see managarm's YAML file. I am looking forward to hear your comments on this tool -- and if turns out that you want to try it to build your own OS, I'd be happy to help you and/or hear about your experience.

Best,
Alexander

┬╣ Such as my own OS, managarm.

_________________
managarm: A microkernel-based OS that is capable of running a Wayland desktop


Top
 Profile  
 
 Post subject: Re: xbstrap: A tool to build OS distributions
PostPosted: Fri Jan 04, 2019 10:56 pm 
Offline
Member
Member

Joined: Mon Feb 02, 2015 7:11 pm
Posts: 293
This is very interesting. I bookmarked it for future use.

Does your system support applying patches to downloaded packages? For example I see that you are using patches on bootstrap-managarm and I am wondering if you handle them with xbstrap?

https://github.com/managarm/bootstrap-m ... er/patches

_________________
https://github.com/kiznit/rainbow-os


Top
 Profile  
 
 Post subject: Re: xbstrap: A tool to build OS distributions
PostPosted: Sat Jan 05, 2019 4:15 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 540
Yes, those patches are applied automatically. It goes through this cycle for all packages:
fetch (e.g. pull from git or download a tar file) -> checkout -> patch -> regenerate (e.g. by running autoconf) -> configure -> build (installs the package to a temporary dir and packs it into a tar) -> install (installs the tar file into the sysroot).

_________________
managarm: A microkernel-based OS that is capable of running a Wayland desktop


Top
 Profile  
 
 Post subject: Re: xbstrap: A tool to build OS distributions
PostPosted: Mon Jan 07, 2019 4:44 am 
Offline
Member
Member

Joined: Thu Jul 05, 2007 8:58 am
Posts: 217
Out of interest, how does it discover those patches to apply?


Top
 Profile  
 
 Post subject: Re: xbstrap: A tool to build OS distributions
PostPosted: Mon Jan 07, 2019 1:32 pm 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 540
Right, it just scans the patches/<package> directory relative to the configuration YAML file. Patches are applied in lexicographical order, which interacts nicely with git format-patch that outputs patches exactly in that order. The discovery procedure could be more configurable -- and I'd be happy to do that if someone needs it, but for my own OS project, it's not necessary.

_________________
managarm: A microkernel-based OS that is capable of running a Wayland desktop


Top
 Profile  
 
 Post subject: Re: xbstrap: A tool to build OS distributions
PostPosted: Sun Mar 17, 2019 4:32 pm 
Offline

Joined: Sun Mar 17, 2019 4:27 pm
Posts: 2
Korona,

Have you ever checked out the OSKit project at the University of Utah? https://www.cs.utah.edu/flux/oskit/

It's old (last updated in 2002) but you might be able to mine some ideas from it. :D

sydbarrett74


Top
 Profile  
 
 Post subject: Re: xbstrap: A tool to build OS distributions
PostPosted: Mon Mar 18, 2019 10:53 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 540
Hi,

I did not look into OSKit yet. However, from a quick glance, it seems somewhat unrelated to xbstrap. xbstrap is a build system but does not try to provide libraries for OS development.

Managarm, my OS, is developed based on a different philosophy (it uses a smallish microkernel and its own set of userspace drivers).

Best,
Alexander

_________________
managarm: A microkernel-based OS that is capable of running a Wayland desktop


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group