Using mkosi for kernel tinkering

If you have not heard of it yet, systemd/mkosi is a great tool by the systemd folks that can generate disk images. To quote their description:

mkosi is a tool for easily building customized OS images. It’s a fancy wrapper around dnf --installroot, apt, pacman and zypper that may generate disk images with a number of bells and whistles.

Definitely go check it out if you ever need to build images. It can even be used to create OCI images. I have only used mkosi to create an image for a digital signage appliance but I am quite happy with it so far. At some point I will likely write another blog post about that project.

One of the mkosi maintainers, @DaanDeMeyer, has created a collection of mkosi definitions and build scripts called DaanDeMeyer/mkosi-kernel which contains profiles useful for kernel development. As I enrolled in a Linux kernel programming course this semester in university I though to myself that this would be a great opportunity to check out that project. Our professor had initially given us a VM image containing Arch Linux and wrapper scripts for QEMU which we could use for working on our assignments but being myself I wanted to make things more complicated. I invite you to join my journey into solving the first few tasks using mkosi-kernel.

Task 1: Setting up your environment

The goal of this task was to get the VM running with QEMU. There are a few deliberate shortcoming in the provided VM image which one was supposed to catch and explain. My own image would not have those errors so instead, I had to resort to downloading the VM image and starting it using QEMU.

I did, however, need to set up mkosi to generate me an Arch image. As mkosi is fast-moving at times and the last release was a few months back I wanted to get the latest version from git. To do this I first created a virtual environment for Python and installed mkosi into it.

python3 -m venv .venv # create virtual environment under .venv
source .venv/bin/activate # activate the virtual environment
pip install git+https://github.com/systemd/mkosi.git@a60dade823319be71f028b78d1482a351207c60b # install mkosi from git

I wanted to stay close to the provided image so I needed to tell mkosi that it should create an Arch image. By default, mkosi detects the distribution of your machine and creates an image matching that distribution but I am using Fedora so we have to be explicit here. To do this, let’s create a config file and add the following content:

[Distribution]
Distribution=arch

To create an Arch image, mkosi also needs to call pacman which I do not have it installed. The simple solution would be to dnf install pacman but I have already established that I generally avoid simple solutions. Luckily mkosi can help us out here, again! The solution I am encountering here is not uncommon. There might also be other tools that mkosi depends on which I do not have installed so the developers of mkosi added a way to create a “tools tree”. This is basically just another image mkosi creates and automatically uses when invoking tools during the build process. To set it up we just need to add two more lines to our config:

[Build]
ToolsTree=default

If mkosi needs pacman to create the first image, how should things be different now? Well, the good thing is that while we set the desired distribution for our image to arch, the tools tree still defaults to using the host distribution. This means that on my machine, mkosi will create a basic Fedora image which contains pacman (amongst some other stuff).

Now we can invoke mkosi and get an image that we can boot using mkosi vm. Hm, that is weird, QEMU does not like our image.

BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x8,0x0): Not Found
BdsDxe: No bootable option or device was found.
BdsDxe: Press any key to enter the Boot Manager Menu.

To exit this screen you can press Ctrl + A , X .

Maybe it’s QEMU that is wrong. Let’s try to boot the image with mkosi boot (which uses systemd-nspawn under the hood):

$ mkosi boot
No machine ID set, using randomized partition UUIDs.
No changes.
execv(/usr/lib/systemd/systemd, /lib/systemd/systemd, /sbin/init) failed: No such file or directory

Oh, well, silly me! We do not have an init system (/sbin/init) installled. That certainly explains our problems. To be precise, we haven’t installed anything, not even a kernel. Instead of specifying every single package we need, mkosi provides a mkosi-vm definition for us with all necessary packages listed (like the kernel, an init system, and a shell). We can simply include this in our mkosi.conf file:

[Include]
Include=mkosi-vm

Let’s rebuild our image by calling mkosi --force and see what happens when we call mkosi vm now. Indeed, after waiting a few seconds, and seeing the green [ OK ]s fly by we have gotten accustomed to, we get to a login prompt:

Asciinema Recording
ASCII representation

...
[  OK  ] Reached target Login Prompts.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.

Arch Linux 6.14.6-arch1-1 (hvc0)

archlinux login:

We did not specify any root password so logging in is not really possible right now. Let’s exit the VM again using the aforementioned keyboard shortcut and resolve the situation by adding a root password:

[Content]
# Prefixing our password with "hashed:"
# means that it will be treated as an already hashed password.
# Choosing the empty hash results in an unlocked account without a password.
RootPassword=hashed:

Finally, we can rebuild our image, boot it again, and get a shell after logging in as root:

...
Arch Linux 6.14.6-arch1-1 (hvc0)

archlinux login: root
[root@archlinux ~]# uname -a
Linux archlinux 6.14.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 09 May 2025 17:36:18 +0000 x86_64 GNU/Linux

Our final config:

[Distribution]
Distribution=arch

[Build]
ToolsTree=default

[Include]
Include=mkosi-vm

[Content]
RootPassword=hashed:

As you might be able guess from the uname output above, we are still using the kernel from the Arch repositories. This is not what we want though. To change that we need to proceed with the next task.

Task 2: Building your own kernel 🚧

Finally, we arrived at the point where we will start to utilize mkosi-kernel. First, let’s download it:

git clone [email protected]:DaanDeMeyer/mkosi-kernel.git ../mkosi-kernel
git -C ../mkosi-kernel/ reset --hard 7beb959e51354077ded4333d2c9951909ea46c75

Here I chose to download it into a sibling directory but you are free to put it anywhere else (or use a git submodule). You just need to adjust the paths accordingly.

To start using it we also need to include it, similar to mkosi-vm before. The mkosi-vm was a builtin preset and we could simply specify it by name. For mkosi-kernel though we need to specify the path to it. I can also already tell you that mkosi-kernel specifies all the required packages we need for booting so we can get rid of the mkosi-vm include.

[Include]
Include=mkosi-vm
Include=../mkosi-kernel

If we were to create a new image with this configuration we would get a bootable image but it would still use the kernel from the official Arch repositories. To change that we need to tell mkosi-kernel to build a custom kernel and provide it with the sources to do so. The university course is using Linux 6.5.7 so let’s download the sources for that first.

wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.7.tar.xz
# We could compare the downloaded archive against a checksum and signature
# but I do not see any reason to do so when using HTTPS.
tar xf linux-6.5.7.tar.xz --directory ..

Again, I chose to extract the archive into a sibling directory. More specifically, I told tar to extract the archive into the parent directory (..) but the archive itself contains all files in a directory called linux-6.5.7/. This means that our the kernel source are now under ../linux-6.5.7/. Now, to get mkosi-kernel to build us a custom kernel we need to provide it the kernel sources and enable a profile containing the build scripts and other configuration.

[Build]
ToolsTree=default
# Make the kernel source available to build scripts as kernel/
BuildSources=../linux-6.5.7:kernel

[Config]
# Enable the kernel profile provided by mkosi-kernel
Profiles=kernel

When building older kernels (such as the version we are using in the course - 6.5.7) we might encounter some build errors. These happen if you are compiling using GCC 15 or later which raised the default C standard from C11 to C23. The latter made bool, true, and false reserve keywords which breaks the kernel build. To resolve this issue we can “backport” the fixes from the kernel sources by running following commands in the directory where the kernel sources are located:

for commit in \
  8ba14d9f490aef9fd535c04e9e62e1169eb7a055 \
  ee2ab467bddfb2d7f68d996dbab94d7b88f8eaf7 \
  b3bee1e7c3f2b1b77182302c7b2131c804175870 \
; do
  curl https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=$commit | patch -p1
done

Let’s try to boot this:

...
Arch Linux 6.5.7 (hvc0)

archlinux login: root (automatic login)

[root@archlinux ~]# uname -a
Linux archlinux 6.5.7 #1 SMP PREEMPT_DYNAMIC Fri Jun  5 15:58:00 CEST 2015 x86_64 GNU/Linux

Our professor also provided us with a custom Kconfig file which sets up a few config variables for easier development. I downloaded it from the course website and told mkosi-kernel to use it by setting the $CONFIG environment variable.

[Build]
ToolsTree=default
BuildSources=../linux-6.5.7:kernel
# Make the current directory available to build scripts under lkp/
BuildSources=.:lkp
Environment=CONFIG=lkp/config-lkp

As we changed the kernel config we need to compile the kernel again. Once that is done, and we are trying to boot the new image, we run into a new problem…a kernel panic.

[    1.032151] List of all bdev filesystems:
[    1.032288]  ext3
[    1.032289]  ext2
[    1.032411]  ext4
[    1.032501]  vfat
[    1.032605]  msdos
[    1.032708]  iso9660
[    1.032811]
[    1.033019] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    1.033334] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.5.7 #2
[    1.033400] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-4.fc42 04/01/2014
[    1.033400] Call Trace:
[    1.033400]  <TASK>
[    1.033400]  dump_stack_lvl+0x36/0x50
[    1.033400]  panic+0x185/0x340
[    1.033400]  mount_root_generic+0x27a/0x330
[    1.033400]  prepare_namespace+0x61/0x250
[    1.033400]  kernel_init_freeable+0x289/0x2d0
[    1.033400]  ? __pfx_kernel_init+0x10/0x10
[    1.033400]  kernel_init+0x15/0x1b0
[    1.033400]  ret_from_fork+0x2c/0x50
[    1.033400]  ? __pfx_kernel_init+0x10/0x10
[    1.033400]  ret_from_fork_asm+0x1b/0x30
[    1.033400]  </TASK>
[    1.033400] Kernel Offset: disabled
[    1.033400] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

mkosi-kernel generates the image as a directory and wants to mount it using virtiofs. My professor’s kernel config on the other hand assumes that the kernel is started from a disk image and thus does not enable the functions required for virtiofs. Therefore we need to change some options. At this point I will also enable the kernel modules for AF_VSOCK as mkosi sets these up to communicate with the booted image (and vice versa). Furthermore, I will also enable the dmi_sysfs module. This exposes DMI entries which mkosi passes to QEMU in the sysfs of the guest. Systemd reads these to extract credentials which again, mkosi uses for some things.

# CONFIG_FUSE_FS is not set
CONFIG_FUSE_FS=y
# CONFIG_CUSE is not set
CONFIG_VIRTIO_FS=y
...
# CONFIG_VSOCKETS is not set
CONFIG_VSOCKETS=y
# CONFIG_VSOCKETS_DIAG is not set
CONFIG_VSOCKETS_LOOPBACK=y
CONFIG_VIRTIO_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS_COMMON=y
...
# CONFIG_DMI_SYSFS is not set
CONFIG_DMI_SYSFS=y

Another recompilation later we can try successfully boot the image again. We can also inspect the kernel image that was generated in our working directory:

$ file --brief ../mkosi-kernel/mkosi.output/image.vmlinuz
Linux kernel x86 boot executable, bzImage, version 6.14.6-arch1-1 (linux@archlinux) #1 SMP PREEMPT_DYNAMIC Fri...

Task 3: The init process

Now that we have successfully build and booted our own kernel we are tasked with investigating the Linux boot process and the role of the init process.

(Not a) Question 1

Our first ~~question~~ task is to create a simple “Hello World” program and put it inside the image. Here is the code:

#include <stdio.h>
#include <unistd.h>

int main(void) {
    printf("Hello World\n");
    sleep(5);
    return 0;
}

Now, how do we get the compiled binary into the image? There is a plethora of ways I can think of but many of these have some drawbacks. Normally I would turn towards scp to copy the file over but I would like to utilize mkosi to its fullest. The easiest solution is to let mkosi take care of it and create a small build script. Remeber how we added the BuildSources option to our configuration to access the lkp-config file? This also means that the helloWorld.c file is already available under lkp/ within the build environment. So our build script only needs to invoke the C compiler on the source file and put the generated binary under $DESTDIR/helloWorld where mkosi will automatically take care of including it in the image.

#!/usr/bin/sh
cc lkp/helloWorld.c -o "$DESTDIR"/helloWorld

Mkosi automatically picks up this file due to its name, just need to remember to mark it as executable (chmod u+x mkosi.build.chroot). The extension .chroot is important here because normally mkosi executes the build scripts using binaries from the tools tree. However, our tools tree does not contain a C compiler. We could add one to it but by using the .chroot extension, we instead instruct mkosi to run the script in the context of the target image instead. This means we now have access to all the binaries and libraries that will be installed in the final image. It just so happens that the default set of packages listed by mkosi-kernel includes a C compiler. With this out of the way let’s recreate the image and see if the compiled file is preset:

$ mkosi -f
$ mkosi vm
...
[root@archlinux ~]# time /helloWorld
Hello World

real    0m5.002s
user    0m0.001s
sys     0m0.001s

Fantasic. So far so good. Onto the next step…

(Not a) Question 2

We are told about the init=... boot parameter and instructed to change our QEMU wrapper script such that our helloWorld program is executed as the init script. Now…we do not have such a wrapper script and instead use mkosi. Lucky for is this is even easier to do with mkosi. We can just add KernelCommandLine= to our configuration to include some text in our command line. Even better we can use KernelCommandLineExtra= to archive the same thing without needing to build a new image. We’re gonna play around with that setting a bit more in the following tasks so instead of modifying the config file ever time I will instead use the --kernel-command-line-extra= flag when invoking mkosi vm.

$ mkosi --kernel-command-line-extra="init=/helloWorld" vm
Hello World
[    6.129068] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[    6.129528] CPU: 1 PID: 1 Comm: helloWorld Not tainted 6.5.7 #3
[    6.129870] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-4.fc42 04/01/2014
[    6.129981] Call Trace:
[    6.129981]  <TASK>
[    6.129981]  dump_stack_lvl+0x36/0x50
[    6.129981]  panic+0x185/0x340
[    6.129981]  do_exit+0x94d/0xab0
[    6.129981]  do_group_exit+0x29/0xb0
[    6.129981]  __x64_sys_exit_group+0x13/0x20
[    6.129981]  do_syscall_64+0x3c/0x90
[    6.129981]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[    6.129981] RIP: 0033:0x7f7f5fc03f68
[    6.129981] Code: ff 64 89 02 eb c4 67 e8 d6 1d 04 00 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 35 95 1d 10 00 eb 04 0f 1f 00 f4 b8 e7 00 00 00 0f 05 <48> 3d 00 f0 ff ff 76 f0 f7 d8 64 89 06 eb e9 66 0f 1f 84 00 00 00
[    6.129981] RSP: 002b:00007ffe8c6386a8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
[    6.129981] RAX: ffffffffffffffda RBX: 00007f7f5fd07fe8 RCX: 00007f7f5fc03f68
[    6.129981] RDX: 00007f7f5fb1da48 RSI: ffffffffffffff88 RDI: 0000000000000000
[    6.129981] RBP: 00007ffe8c638700 R08: 00007ffe8c638640 R09: 0000000000000000
[    6.129981] R10: 00007ffe8c638550 R11: 0000000000000202 R12: 0000000000000001
[    6.129981] R13: 0000000000000000 R14: 00007f7f5fd06680 R15: 00007f7f5fd08000
[    6.129981]  </TASK>
[    6.129981] Kernel Offset: disabled
[    6.129981] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 ]---

Perfect! Except…we get a kernel panic. Let’s ignore that for now and see what the next task offers for us.

(Almost a) Question 3

Now we shall explain why the panic occurs. Quite convenient that I ignore that in the last task… almost as if I knew that where coming (inception SFX).

Question 4

Now we should launch a shell as our init program. Nothing easier than that. We just need to know the path to our shell (/usr/bin/bash in this case) and pass it as the value for the init= boot parameter:

$ mkosi --kernel-command-line-extra="init=/usr/bin/bash" vm
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
[root@(none) /]#

We should also run ps and explain why it does not function as expected.

(Not a) Question 5

Let’s go ahead and fix the problem - which is exactly what we are instructed to do in this task. Probably shouldn’t hurt to just try out the thing ps is already telling us to do.

[root@(none) /]# # 🪄 MAGIC COMMAND YOU SHALL FIGURE OUT ON YOUR OWN 🪄
[root@(none) /]# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 bash
    2 ?        00:00:00 kthreadd
    3 ?        00:00:00 rcu_gp
62 collapsed lines
    4 ?        00:00:00 rcu_par_gp
    5 ?        00:00:00 slub_flushwq
    6 ?        00:00:00 netns
    7 ?        00:00:00 kworker/0:0-rcu_gp
    8 ?        00:00:00 kworker/0:0H-events_highpri
    9 ?        00:00:00 kworker/0:1-events
   10 ?        00:00:00 kworker/u4:0-events_unbound
   11 ?        00:00:00 mm_percpu_wq
   12 ?        00:00:00 rcu_tasks_kthread
   13 ?        00:00:00 ksoftirqd/0
   14 ?        00:00:00 rcu_preempt
   15 ?        00:00:00 migration/0
   16 ?        00:00:00 cpuhp/0
   17 ?        00:00:00 cpuhp/1
   18 ?        00:00:00 migration/1
   19 ?        00:00:00 ksoftirqd/1
   20 ?        00:00:00 kworker/1:0-mm_percpu_wq
   21 ?        00:00:00 kworker/1:0H-events_highpri
   22 ?        00:00:00 kdevtmpfs
   23 ?        00:00:00 inet_frag_wq
   24 ?        00:00:00 kauditd
   25 ?        00:00:00 khungtaskd
   26 ?        00:00:00 kworker/u4:1-events_unbound
   27 ?        00:00:00 oom_reaper
   28 ?        00:00:00 writeback
   29 ?        00:00:00 kcompactd0
   30 ?        00:00:00 kworker/1:1
   31 ?        00:00:00 kblockd
   32 ?        00:00:00 ata_sff
   33 ?        00:00:00 md
   34 ?        00:00:00 md_bitmap
   35 ?        00:00:00 kworker/0:1H
   36 ?        00:00:00 rpciod
   37 ?        00:00:00 xprtiod
   38 ?        00:00:00 cfg80211
   39 ?        00:00:00 kswapd0
   40 ?        00:00:00 nfsiod
   41 ?        00:00:00 acpi_thermal_pm
   42 ?        00:00:00 khvcd
   43 ?        00:00:00 kworker/1:2
   44 ?        00:00:00 scsi_eh_0
   45 ?        00:00:00 scsi_tmf_0
   46 ?        00:00:00 scsi_eh_1
   47 ?        00:00:00 scsi_tmf_1
   48 ?        00:00:00 scsi_eh_2
   49 ?        00:00:00 scsi_tmf_2
   50 ?        00:00:00 scsi_eh_3
   51 ?        00:00:00 scsi_tmf_3
   52 ?        00:00:00 scsi_eh_4
   53 ?        00:00:00 scsi_tmf_4
   54 ?        00:00:00 scsi_eh_5
   55 ?        00:00:00 scsi_tmf_5
   56 ?        00:00:00 scsi_eh_6
   57 ?        00:00:00 scsi_tmf_6
   58 ?        00:00:00 kworker/u4:2-events_unbound
   59 ?        00:00:00 kworker/u4:3-events_unbound
   60 ?        00:00:00 kworker/u4:4-events_unbound
   61 ?        00:00:00 kworker/u4:5-events_unbound
   62 ?        00:00:00 kworker/u4:6-events_unbound
   63 ?        00:00:00 kworker/0:2-pm
   64 ?        00:00:00 kworker/u4:7-events_unbound
   65 ?        00:00:00 kworker/1:1H
   66 ?        00:00:00 mld
   67 ?        00:00:00 ipv6_addrconf
   73 ?        00:00:00 ps
[root@(none) /]# pstree 0
?-+-bash---pstree
  `-kthreadd-+-acpi_thermal_pm
             |-ata_sff
             |-cfg80211
             |-cpuhp/0
58 collapsed lines
             |-cpuhp/1
             |-inet_frag_wq
             |-ipv6_addrconf
             |-kauditd
             |-kblockd
             |-kcompactd0
             |-kdevtmpfs
             |-khungtaskd
             |-khvcd
             |-ksoftirqd/0
             |-ksoftirqd/1
             |-kswapd0
             |-kworker/0:0-rcu_gp
             |-kworker/0:0H-events_highpri
             |-kworker/0:1-events
             |-kworker/0:1H-kblockd
             |-kworker/0:2-pm
             |-kworker/1:0-events_power_efficient
             |-kworker/1:0H-events_highpri
             |-kworker/1:1
             |-kworker/1:1H
             |-kworker/1:2-events_power_efficient
             |-kworker/u4:0-events_unbound
             |-kworker/u4:1-events_unbound
             |-kworker/u4:2-events_unbound
             |-kworker/u4:3-events_unbound
             |-kworker/u4:4-events_unbound
             |-kworker/u4:5-events_unbound
             |-kworker/u4:6-events_unbound
             |-kworker/u4:7-events_unbound
             |-md
             |-md_bitmap
             |-migration/0
             |-migration/1
             |-mld
             |-mm_percpu_wq
             |-netns
             |-nfsiod
             |-oom_reaper
             |-rcu_gp
             |-rcu_par_gp
             |-rcu_preempt
             |-rcu_tasks_kthread
             |-rpciod
             |-scsi_eh_0
             |-scsi_eh_1
             |-scsi_eh_2
             |-scsi_eh_3
             |-scsi_eh_4
             |-scsi_eh_5
             |-scsi_eh_6
             |-scsi_tmf_0
             |-scsi_tmf_1
             |-scsi_tmf_2
             |-scsi_tmf_3
             |-scsi_tmf_4
             |-scsi_tmf_5
             |-scsi_tmf_6
             |-slub_flushwq
             |-writeback
             `-xprtiod

Phew, that was easy.

(Not a) Question 6

Finally, we should try to continue the boot process as usual by starting the original init process from here.

Task 4: Understanding the initramfs

Question 1

Instead of inspecting the initrd of our host machine, let’s inspect the one from the image we just created. There is just one problem: It does not have an initrd. Why? Because mkosi-kernel uses direct kernel boot without an initrd.

But fear not, we can still create one with just a few config additions. Mkosi supports building so called submimages which can be used to create artifacts required for the main image (or other subimages). In our case we want to create an initrd artifact. Mkosi provides a preset for that called mkosi-initrd thus this is as easy as creating a new subimage under mkosi.images/initrd/ with the following content:

[Include]
Include=mkosi-initrd

Because we did not specify any explicit subimage Dependencies= for the main image, mkosi will assume all subimage are required for the main image. Thus, simply rebuilding the main image will also build the initrd (though it will still not be used for booting).

After a quick mkosi -f we can check the generated initrd.

$ file --brief ../mkosi-kernel/mkosi.output/initrd.cpio.zst
Zstandard compressed data (v0.8+), Dictionary ID: None
$ zstdcat ../mkosi-kernel/mkosi.output/initrd.cpio.zst | cpio -tv
...

Question 2

You will soon find out that -static is not actually needed though in that case we need perform some other adjustments. Why, and how do we do that? I will get to that, first, however, let’s see what happens if we follow the instructions normally and compile the binary with -static.

Question 3 - Creating the initramfs

To include the helloWorld binary in the initrd we again need to create a build script. I choose to simply copy the one we previously created for the main image to mkosi.images/initrd/, add the -static flag to the compiler invocation, and adjust the destination path to $DESTDIR/init:

#!/usr/bin/sh
# We still have access to the same build sources under lkp/
cc -static lkp/helloWorld.c -o "$DESTDIR"/init

This time, however, you will notice that the compilation will fail because the C compiler is missing. This is because subimages do not inherit the package list from the main image, and the mkosi-initrd preset does not include a C compiler. However, we also do not want to include a C compiler in our initrd by adding it to the Packages= of our initrd config as we want to only include the init binary, nothing else. We have two alternative options here:

Add it to the BuildPackages= of our initrd subimage config, or
add it to the ToolsTreePackages= of our main config.

I chose to go with the first option here. In the same step I will reset the list of packages set by the mkosi-initrd preset. Due to the way pacman and mkosi preset work, there will still be a few other files left in the initrd. To get rid of most of them I will add some more options as you can see below.

[Include]
Include=mkosi-initrd

[Content]
BuildPackages=gcc
# Reset the package list from mkosi-initrd
Packages=
# Explicitly remove the filesystem package
RemovePackages=filesystem
RemoveFiles=
    /boot
    /dev
    /efi
    /etc
    /proc
    /run
    /tmp
    /usr
    /var

Indeed, the generated initrd now contains only our init binary:

$ zstdcat ../mkosi-kernel/mkosi.output/initrd | cpio -tv
-rwxr-xr-x   1 root     root       778456 May 21 23:02 init
1521 blocks

Approach A
Approach B

Question 4 - Booting the initramfs

Now we can try to boot our new initrd. For this, we need to tell mkosi to use the initrd:

...
[Content]
RootPassword=hashed:
# %O gets expanded to the output directory of our build artifacts,
# we the initrd file created by our subimage is located.
Initrds=%O/initrd

This is already everything we need to do. Booting the image using mkosi will now greet us with our “Hello World” message.

Static compilation, revisited ⏪

Previously I mentioned that we do not necessarily need to static compilation. First, let’s see what happens if we do not use the -static flag:

[    0.828548] Failed to execute /init (error -2)
[    0.828927] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    0.829482] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.5.7 #3
[    0.829482] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-4.fc42 04/01/2014
[    0.829482] Call Trace:
[    0.829482]  <TASK>
[    0.829482]  dump_stack_lvl+0x36/0x50
[    0.829482]  panic+0x185/0x340
[    0.829482]  ? __pfx_kernel_init+0x10/0x10
[    0.829482]  kernel_init+0x168/0x1b0
[    0.829482]  ret_from_fork+0x2c/0x50
[    0.829482]  ? __pfx_kernel_init+0x10/0x10
[    0.829482]  ret_from_fork_asm+0x1b/0x30
[    0.829482]  </TASK>
[    0.829482] Kernel Offset: disabled
[    0.829482] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

We get a failure when trying to execute with errno -2. Looking up the error code (errno 2 if you have moreutils installed) we can see that this corresponds to ENOENT 2 No such file or directory… which is weird because we got it while trying to execute the file. If the file did not exist the kernel would not even try to execute it (which is the case for the other fallback locations like /sbin/init). So there must be something else going on here. Indeed, we can reproduce the same error by unpacking the initrd and entering it using unshare:

$ mkdir initrd
$ zstdcat ../mkosi-kernel/mkosi.output/initrd | cpio -tv -D initrd
$ unshare --user --root initrd /init
unshare: failed to execute /init: No such file or directory

My usual tool to debug similar issues is strace so maybe it can provide some insights:

$ # Set LANG=C to suppress unnecessary syscalls from looking up translations
$ strace --env LANG=C -e %file unshare --user --root initrd /init
execve("/usr/bin/unshare", ["unshare", "--user", "--root", "initrd", "/init"], 0x55ac07d41820 /* 89 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
chroot("initrd")                        = 0
chdir("/")                              = 0
execve("/init", ["/init"], 0x7ffdf5f04f38 /* 89 vars */) = -1 ENOENT (No such file or directory)
unshare: failed to execute /init: No such file or directory
+++ exited with 127 +++

The problem we are encountering is that the kernel can provide no context about errors. In the highlighted line we can see that the kernel tries to execute the file and returns a -1 to indicate a failure. Futher information is placed in the errno variable which strace kindly tells us contains ENOENT (No such file or directory). This does not really help us. We are falling victim to something else. Dynamically linked programs require a loader (ld.so) which, well, is responsible for loading those libraries. However, this file is not present in our initrd resulting in the ENOENT error. The kernel is trying to run the executable but cannot find the loader, and there is simply no better way for it to tell us that. This problem does not exist for static compilation, simply because it does not need a loader.

That being said, we can also certainly resolve this issue without resorting to static compilation. We just need to add the loader to our initrd. I will also tell you that there is one other file that is required. While the loader is responsible for loading the libraries, we still have to add the libraries themselves. To see which libraries are required we can use ldd command:

$ ldd initrd/init
        linux-vdso.so.1 (0x00007fdde77ea000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fdde75c3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdde77ec000)

This lists the libraries required by our binary. The first entry is the Linux vDSO. This is a special shared library that the kernel itself provides to user space and is not required to be included in the initrd. Next, we have an entry for “libc”, the standard C library. This is definitely something we need to include. In theory we could also write our hello world program completely without using libc by using -nostdlib, a bit of assembly, and using the syscall interface directly. However, I also like to have some free time so I will not do that. But don’t let me stop you from trying that out if you really want to. And finally, the last entry is the dynamic loader itself mentioned above.

You will also notice that all three files are displayed a bit different regarding the path of the library. For the vDSO only the filename is given. ldd does not resolve this to anything because it is provided by the kernel. The libc has a small arrow behind it showing the path from which the loader would link the library during runtime. For this, it searches a set of predefined paths (here, /usr/lib64 and /lib64). Lastly we have the dynamic loader itself again which is listed with an absolute path. This is because the loader cannot use the search algorithm for itself, thus the absolute path is embedded which the kernel just invokes.

Now, the easiest way to get the binary running is to just include those files in our initrd. However, we have set RemovePackages=filesystem which, amongst most paths, also results in the removal of /lib64. One way around this would be to remove that setting from our config and instead add a few more lines to our RemoveFiles= list. The way I am going to do this though will be different. During compilation we have a few options we can play around with that influence the paths embedded inside the binary, as well as the search algorithm of the loader. These are -Wl,--dynamic-loader= which we can provide a path that will be stored in the binary as the path for the loader, and -Wl,-rpath= to which we can assign additional runtime paths for the loader to look through when searching libraries.

Here is now our new build script which sets the correct values for the flags, and copies the libraries to the root directory:

#!/usr/bin/sh
# Provide absolute path to ld.so to allow putting it in the root dir
# and add / to the rpath to allow putting libc in the root dir as well.
cc \
    -Wl,--dynamic-linker=/ld-linux-x86-64.so.2 \
    -Wl,-rpath=/ \
    lkp/helloWorld.c \
    -o "$DESTDIR"/init
cp /lib64/libc.so.6 "$DESTDIR"/
cp /lib64/ld-linux-x86-64.so.2 "$DESTDIR"/

Let’s confirm that everything is in place as expected after a rebuild:

$ zstdcat ../mkosi-kernel/mkosi.output/initrd | cpio -tv
-rwxr-xr-x   1 root     root        15472 May 22 16:17 init
-rwxr-xr-x   1 root     root       226904 May 22 16:17 ld-linux-x86-64.so.2
-rwxr-xr-x   1 root     root      2006328 May 22 16:17 libc.so.6
4393 blocks

You may also check that the loader path is correctly set using either ldd or objdump -s -j .interp.

And booting this image we are able to confirm that this indeed also works as expected. So, hooray, we can boot our initrd even without static compilation 🎉.

Task 5: Dynamic library loading

As this does not benefit anymore from mkosi-kernel I will not be delving into this task. However, if I have awaken your interest in mkosi in general, you can still use it to set up a build environment for this task. For me though this is where I will leave you to explore the world of mkosi on your own.

Some final words

I hope you enjoyed this little journey into the world of mkosi. There are still many facets I have left unexplored and I am sure you will find many more interesting things to do with mkosi.

If you want to continue using it, especially with kernel development you might also want to look into mkosi --rerun-build-scripts build -- -i which mkosi-kernel has set up to allow live-reload of kernel modules. And if you need to reload the kernel itself, maybe because you worked on some built-in kernel module, you can also check out kexec.

Going the extra mile: Copycat 🐱

Here we try to make our image as close to the provided image… including the (hopefully) deliberately included shortcomings. I will also say how I would improve the image without changing the spirit of those shortcomings.

There is not much content here but the first thing to bringing the image closer to the provided one would be to adjust the set of installed Packages=. Additionally, one can point mkosi to an Arch archive mirror with the old packages. Generally, using an older kernel with newer programs presents no problem unless the kernel is really old or the program really new. There can, however still be surprises caused by this. E.g. in the course you will eventually implement a new syscall. Properly writting programs (like systemd) check for the existence of syscalls and gracefully fallback to alternative means if the syscall does not exist. If, however, the syscall now exists but does something completely different, this can, and eventually will, lead to problems.

Regarding improvements to the provided image (and instructions) I have a few more thoughts:

Use fallocate instead of dd to allocate/grow the disk image.
Set up systemd-repart to automatically allocate and grow partitions on boot.
Using virtiofsd instead of virtio for improved performance.
Provide a more pristine machine, without any leftover files, .bash_history etc.
…

Mix ‘n’ match: Combining this with virtme-ng

If you intend to mix mkosi with arighi/virtme-ng , (which is certainly possible) but by no means necessary, you might need to enable a few more kernel options. In my case I did not want to use it for development but instead wanted to replicate the submissions system locally. I have not yet narrowed down which kernel options are really required for this, but below is at least a set of config options that should get you going.

## Values required by virtme-ng
48 collapsed lines
## TODO slim them down and eliminate the ones that are not required
CONFIG_UEVENT_HELPER=n
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_NET=y
CONFIG_NET_CORE=y
CONFIG_NETDEVICES=y
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_INET=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=y
CONFIG_VIRTIO_NET=y
CONFIG_CMDLINE_OVERRIDE=n
CONFIG_BINFMT_SCRIPT=y
CONFIG_SHMEM=y
CONFIG_TMPFS=y
CONFIG_UNIX=y
CONFIG_MODULE_SIG_FORCE=n
CONFIG_DEVTMPFS=y
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_UNIX98_PTYS=y
CONFIG_EARLY_PRINTK=y
CONFIG_INOTIFY_USER=y
CONFIG_BLOCK=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI=y
CONFIG_SCSI_VIRTIO=y
CONFIG_BLK_DEV_SD=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_I6300ESB_WDT=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_OVERLAY_FS=y
CONFIG_DAX=y
CONFIG_DAX_DRIVER=y
CONFIG_FS_DAX=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_ZONE_DEVICE=y
CONFIG_FUSE_FS=y
CONFIG_VIRTIO_FS=y
CONFIG_VSOCKETS=y
CONFIG_VIRTIO_VSOCKETS=y