Phase 214 — first kernel module/kmod payload

At a glance

Phase 214 answers the next problem discovered by Phase 212 after Phase 213.

Phase 213 made this part work:

kernel
  -> initramfs
  -> ONIX root filesystem
  -> /usr/lib/systemd/systemd
  -> multi-user.target

That is a real boot milestone. systemd became PID 1 and reached the normal multi-user target.

But the boot log also showed warnings like:

Unable to locate executable 'modprobe': No such file or directory

and the /boot or /efi mounts could still fail or time out.

Those two things are connected.

What modprobe is

The Linux kernel can have functionality in two broad forms:

built into the kernel image
loadable as a kernel module

If something is built in, it is already available as soon as the kernel starts.

If something is a module, userspace may need to ask the kernel to load it.

The normal tool for that is:

modprobe

modprobe is not magic by itself. It reads the module metadata under:

/lib/modules/<kernel-release>/

and then asks the kernel to load the right .ko module files.

Because ONIX uses a merged /usr layout:

/lib -> /usr/lib

the module tree can physically live at:

/usr/lib/modules/<kernel-release>/

and still be visible through:

/lib/modules/<kernel-release>/

Why this matters for /boot and /efi

The ONIX image uses:

ONIX-ESP   -> /efi   -> vfat
ONIX-BOOT  -> /boot  -> vfat

vfat is the common FAT filesystem variant used by EFI system partitions.

Mounting a vfat filesystem may require these kernel modules:

fat.ko
vfat.ko
nls_cp437.ko
nls_iso8859-1.ko

During the initramfs stage, these modules already exist. That is why the early boot environment can understand the disk well enough to find and mount the real root filesystem.

But after switch_root, the old initramfs filesystem disappears.

The real ONIX root then needs its own copy of:

modprobe
/lib/modules/<kernel-release>

otherwise systemd can ask for module loading and fail because the tool or module tree is missing.

Why this phase extracts from the initramfs

Phase 211 already installed this pair:

/boot/ONIX/vmlinuz
/boot/ONIX/initramfs.img

Those two artifacts came from the forge VM together. The initramfs contains the module tree that matches the kernel it boots.

That means the safest bootstrap source for Phase 214 is not a random package download. It is the already-booted initramfs:

vm/state/initramfs-virt

Phase 214 extracts from that initramfs and stages the relevant files into the real image root.

The key copied paths are:

/usr/bin/kmod
/usr/bin/modprobe -> kmod
/usr/sbin/modprobe -> ../bin/kmod
/etc/systemd/system/modprobe@.service.d/10-onix-modprobe.conf
/usr/lib/modules/6.18.38-0-virt

The exact kernel release is discovered from the initramfs module tree. The script expects exactly one release directory so it does not accidentally mix modules from multiple kernels.

The source modules in the Alpine-derived initramfs are gzip-compressed:

vfat.ko.gz
fat.ko.gz
fuse.ko.gz
configfs.ko.gz

Phase 214 does not leave them compressed in the ONIX root. It decompresses them to plain ELF kernel objects:

vfat.ko
fat.ko
fuse.ko
configfs.ko

Then it regenerates modules.dep with depmod.

That extra step matters because the temporary Nix-provided systemd closure links against a libkmod build that supports xz and zstd compression, but not gzip. Plain .ko files are the common format both sides can read.

Why kmod needs libraries too

Inside this Alpine-derived initramfs, modprobe is a symlink:

/usr/sbin/modprobe -> ../bin/kmod

and /usr/bin/kmod is a dynamically linked musl executable.

Phase 214 also exposes both common command names:

/usr/bin/modprobe -> kmod
/usr/sbin/modprobe -> ../bin/kmod

But there is a second subtle problem.

The upstream modprobe@.service unit uses a relative command:

ExecStart=-modprobe -abq %i

That only works if systemd’s executable search path contains the directory where modprobe lives. Our bootstrap systemd comes from a Nix musl closure, so its search behavior is not something ONIX should leave implicit.

Phase 214 therefore adds a small drop-in:

/etc/systemd/system/modprobe@.service.d/10-onix-modprobe.conf

with:

[Service]
ExecStart=
ExecStart=-/usr/sbin/modprobe -abq %i

The first ExecStart= line clears the inherited command. The second line adds the same command back, but with an absolute ONIX path.

That means copying only modprobe is not enough. We also need systemd to know exactly which modprobe to run during this bootstrap stage.

The executable also needs its dynamic loader and libraries, such as:

/usr/lib/ld-musl-x86_64.so.1
/usr/lib/libc.musl-x86_64.so.1
/usr/lib/libzstd.so.1
/usr/lib/liblzma.so.5
/usr/lib/libz.so.1
/usr/lib/libcrypto.so.3

Because ONIX has:

/lib -> /usr/lib

the interpreter path:

/lib/ld-musl-x86_64.so.1

is satisfied by installing:

/usr/lib/ld-musl-x86_64.so.1

This is still a bootstrap payload. It is not the final package story.

What this phase verifies

make phase 214 verifies:

• the Phase 211 kernel/initramfs payload exists
• the Phase 213 systemd payload exists
• the initramfs contains exactly one module release directory
• usr/bin/kmod exists in the initramfs
• usr/sbin/modprobe points to ../bin/kmod
• the module tree contains vfat, fat, and the common FAT NLS modules
• the image receives /usr/bin/kmod
• the image receives /usr/bin/modprobe
• the image receives /usr/sbin/modprobe
• the image receives the modprobe@.service drop-in
• the image receives the matching /usr/lib/modules/<release> tree
• gzip-compressed source modules are converted to plain .ko files
• depmod regenerates modules.dep for the converted module tree
• /lib/modules/<release> works through the merged /usr symlink
• a chrooted kmod --version can execute
• chrooted /usr/bin/modprobe, /usr/sbin/modprobe, and /sbin/modprobe can resolve vfat module dependencies
• the Nix-provided bootstrap modprobe, if available from the systemd closure, can also resolve vfat against the ONIX root

That last check is important. It does not load a module on the host. It only asks modprobe to show what it would load for the ONIX kernel release.

Why /boot and /efi still use nofail

Phase 214 gives systemd a real chance to mount the FAT boot partitions cleanly.

But ONIX is still in bootstrap territory. We are proving one boot layer at a time.

The fstab entries currently include:

nofail,x-systemd.device-timeout=10s

That means a failure to mount /boot or /efi is recorded, but it does not prevent the machine from reaching:

multi-user.target

This is intentional while we are still building the base.

Once the module stack and device timing are consistently clean, a later phase can decide whether /boot and /efi should become strict mounts again.

What this phase does not solve yet

Phase 214 does not create the final ONIX kernel package.

It does not create the final ONIX kmod package.

It does not create the final ONIX initramfs generator.

It imports a known-good bootstrap payload so we can keep moving:

prove boot behavior first
package it properly second

Later ONIX should replace this imported payload with ONIX-owned stones, likely something like:

onix-kernel
onix-kernel-modules
onix-kmod
onix-initramfs

The exact names can change. The ownership boundary is the important part.

Expected result

After running:

make phase 214

the image should contain the first real module-loading payload and the boot menu default should move to:

ONIX (phase-214)

Then run:

make phase 212

to boot-probe the result.

The next thing to look for in the serial log is whether /boot and /efi mount more cleanly, and whether the previous modprobe missing warnings disappear.