Gentoo on LUKS and Btrfs with remote unlock

This is the procedure I used to install Gentoo on a VPS: the root volume fully encrypted, Btrfs with subvolumes on top, and the disk unlocked over SSH from the initramfs, so the machine can be rebooted without a console session. The installation runs in a chroot started from the provider's rescue system or any live medium, without an interactive installer.

Why the commands before the chroot are Debian's. VPS providers rarely offer a Gentoo image, and few let you boot your own installation medium. What they almost always offer is a rescue mode: a small Debian-based system booted from a separate disk, with the machine's own disk attached but not in use. That is enough for Gentoo, which installs from any running Linux with a shell, tar and chroot. Hence apt in Stages 0, 12 and 13: it installs the tools the rescue system lacks. Inside the chroot, everything is Gentoo.

It covers the base system together with the boot loader, initramfs and network configuration. It doesn't cover the services you build on top; the kernel configuration below already contains what containers, a firewall and WireGuard need, because adding it later means another kernel rebuild and another manual unlock.

Assumptions

Departing from any of these needs adjustments in the places marked Variant.

Parameters

Decide these before you start and substitute them consistently. The values are examples.

IPMASK doesn't have to match PREFIX. The kernel refuses to add a default route when the gateway, after applying the mask, doesn't fall into the same network as the address, and prints Gateway not on directly connected network. With a /32 address, give ip= a mask that covers the gateway, for example 255.255.255.0. That configuration lives for a dozen seconds in the initramfs and is replaced by the real one once the target system takes over.

Stage 0: Reconnaissance

Boot the rescue system and collect the facts the rest of the procedure depends on:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
lspci -nnk
ip -br addr
ip route
nproc
lscpu | grep -E 'Model name|Flags'
grep . /sys/devices/system/cpu/vulnerabilities/*

Read off the output:

• the boot mode, which Stage 11 depends on;
• the PCI IDs of the disk controller and the network card, which Stage 8 depends on;
• the CPU extensions, which MARCH depends on;
• address, netmask and gateway;
• the list of active CPU vulnerability mitigations, as a reference point.

Pick the CPU extensions conservatively. On a virtual machine that can be migrated to another node, -march=native gives a result that depends on the moment of compilation; a named microarchitecture level survives migration. A CPU reported as a named QEMU model (for example Haswell without TSX) is a strong hint that the provider has unified the instruction set across its fleet exactly for that reason.

Finding the target disk

Disk numbering in a rescue system isn't stable between boots. The target disk may be /dev/sda one time and /dev/sdb the next. Determine it again after every entry:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
DISK=/dev/sda
lsblk "$DISK"
mount | grep "^$DISK"
sgdisk -p "$DISK"

The target disk must not be mounted anywhere. If the rescue system's root lives on it, stop. In the partition table, look for a bios_grub partition on GPT; it has to survive Stage 1.

Once the file systems exist, refer to them by UUID, not by device path.

Tools

apt update
apt install -y tmux cryptsetup-bin lvm2 btrfs-progs
tmux new -s gentoo

Working inside a terminal multiplexer is mandatory. Rebuilding the base system and building the kernel take hours, and a dropped SSH connection kills emerge at a random point. Reattach with tmux attach -t gentoo.

cryptsetup-bin rather than cryptsetup: the latter pulls in initramfs hooks for the rescue system, which are useless here and costly when space is tight. Rescue systems usually run from a small disk with a few hundred megabytes free, which is also why the stage3 archive is downloaded straight onto the target disk in Stage 4.

Stage 1: Partitioning

Target layout:

p1        1 GB   /boot, Btrfs, unencrypted
p2      the rest LUKS2 -> LVM PV -> VG
                     swap     4 GB
                     system   the rest -> Btrfs
                          @           /
                          @lxc        /var/lib/lxc
                          @log        /var/log
                          @portage    /var/tmp/portage
                          @snapshots  /.snapshots

Keep any helper partitions created by the provider's image, in particular bios_grub on a GPT disk. Without it GRUB has nowhere to put core.img.

sgdisk -p "$DISK"

sgdisk -d 1 "$DISK"
sgdisk -n 1:0:+1G -t 1:8300 -c 1:boot "$DISK"
sgdisk -n 2:0:0   -t 2:8309 -c 2:luks "$DISK"

sgdisk -p "$DISK"

Type 8300 is a plain Linux file system, 8309 is Linux LUKS. A start of 0 means the first sector of the largest free area.

Variant UEFI: instead of deleting partition 1, use the existing EFI system partition, or create one of type ef00 and 512 MB formatted with mkfs.vfat -F32. /boot can then live on it.

Syncing with the kernel

The kernel keeps its own copy of the partition table, read when the device was detected. Until it's refreshed, the device files describe the old layout, and formatting would write a file system at the wrong offset.

blockdev --rereadpt "$DISK"
udevadm settle
lsblk "$DISK"

blockdev --rereadpt fails with EBUSY if anything on the disk is mounted. udevadm settle waits until the device files exist.

The lsblk output decides, not the exit codes of the commands before it. The partition sizes must match what you intended. If they don't, don't format.

Stage 2: LUKS and LVM

cryptsetup luksFormat --type luks2 --pbkdf argon2id "${DISK}2"
cryptsetup open --allow-discards "${DISK}2" cryptroot

pvcreate /dev/mapper/cryptroot
vgcreate "$VG" /dev/mapper/cryptroot

lvcreate -L 4G       -n swap   "$VG"
lvcreate -l 100%FREE -n system "$VG"

vgs
lvs

Choose a passphrase that is safe on a US keyboard layout. Providers' management consoles send key codes according to the US layout regardless of your local settings, and dracut doesn't echo what you type. Letters, digits, spaces, dots and dashes are safe on every layout. A long passphrase made of several words is both easier and stronger here than a short one full of symbols.

No space needs to be reserved for snapshots: Btrfs takes them inside the volume, without allocating anything up front.

TRIM

By default cryptsetup doesn't pass TRIM down to the layer below. Enabling it reveals which blocks are in use to someone with access to the medium, but not their contents. On thin-provisioned provider storage, keeping it off means freed blocks never return to the host's pool.

TRIM has to be enabled in three places, and all three must agree:

Forgetting the middle one is the most common mistake: the installation works, and after the first boot from the initramfs TRIM silently stops reaching the disk. LVM passes TRIM from the file system through without any configuration. Releasing space on lvremove and lvreduce is a separate matter, controlled by issue_discards = 1 in /etc/lvm/lvm.conf, optional and independent of the above.

Stage 3: File systems

mkfs.btrfs -L boot   "${DISK}1"
mkfs.btrfs -L system "/dev/$VG/system"
mkswap     -L swap   "/dev/$VG/swap"

/boot gets no subvolumes and no compression. GRUB has its own Btrfs reader, developed independently of the kernel and not supporting every feature of the format. Kernels and initramfs images are compressed already, so skipping compression costs nothing and removes one feature from the list the boot loader's reader has to understand.

Subvolumes

mount "/dev/$VG/system" /mnt
btrfs subvolume create /mnt/@
btrfs subvolume create /mnt/@lxc
btrfs subvolume create /mnt/@log
btrfs subvolume create /mnt/@portage
btrfs subvolume create /mnt/@snapshots
btrfs subvolume list /mnt
umount /mnt

@log and @portage are separate so that a snapshot of the root taken before an update doesn't drag along logs and build directories.

A snapshot doesn't include nested subvolumes, and some get created without you asking. On OpenRC systems systemd-tmpfiles ships the rule q /srv in /usr/lib/tmpfiles.d/home.conf, which on Btrfs creates /srv as a subvolume at first boot. Anything you later keep in /srv shows up in a root snapshot as an empty directory. Check what a snapshot leaves out with btrfs subvolume list /.

Mounting in the target layout

OPTS="noatime,compress=zstd:3,ssd,discard=async"

mkdir -p /mnt/gentoo
mount -o "$OPTS,subvol=@" "/dev/$VG/system" /mnt/gentoo

mkdir -p /mnt/gentoo/{boot,var/lib/lxc,var/log,var/tmp/portage,.snapshots}
mount -o "$OPTS,subvol=@lxc"       "/dev/$VG/system" /mnt/gentoo/var/lib/lxc
mount -o "$OPTS,subvol=@log"       "/dev/$VG/system" /mnt/gentoo/var/log
mount -o "$OPTS,subvol=@snapshots" "/dev/$VG/system" /mnt/gentoo/.snapshots
mount -o "noatime,discard=async,subvol=@portage" "/dev/$VG/system" /mnt/gentoo/var/tmp/portage
mount -o noatime,discard=async "${DISK}1" /mnt/gentoo/boot

swapon "/dev/$VG/swap"
findmnt -R /mnt/gentoo

@portage goes without compression and with copy-on-write disabled; build directories are pure transient traffic:

chattr +C /mnt/gentoo/var/tmp/portage
lsattr -d /mnt/gentoo/var/tmp/portage

chattr +C only works on an empty directory and applies to files created after the attribute is set. The same rule holds later for the data directories of databases living on this file system.

Identifiers

mkdir -p /mnt/gentoo/root
cat > /mnt/gentoo/root/install.env << EOF
UUID_BOOT=$(blkid -s UUID -o value "${DISK}1")
UUID_LUKS=$(blkid -s UUID -o value "${DISK}2")
UUID_SYSTEM=$(blkid -s UUID -o value "/dev/$VG/system")
UUID_SWAP=$(blkid -s UUID -o value "/dev/$VG/swap")
VG=$VG
HOST=$HOST
IFACE=$IFACE
ADDR=$ADDR
PREFIX=$PREFIX
GW=$GW
IPMASK=$IPMASK
SSHPORT=$SSHPORT
ROOTDEV=/dev/mapper/$(printf '%s' "$VG" | sed 's/-/--/g')-system
EOF

cat /mnt/gentoo/root/install.env

The EOF delimiter is unquoted on purpose: the substitutions have to happen when the file is written.

ROOTDEV is the device-mapper name, in which LVM doubles every dash in the volume group name. A group srv-vg with the volume system becomes /dev/mapper/srv--vg-system.

All subvolumes live on one device and share one UUID; in fstab they are told apart by subvol=. The most important value is UUID_LUKS, the UUID of the LUKS container rather than of the decrypted device. That one goes into rd.luks.uuid.

The file lives inside the encrypted volume, so keep a copy off the server as well. Unpacking the stage3 won't remove it.

Stage 4: Stage3

cd /mnt/gentoo
BASE=https://distfiles.gentoo.org/releases/amd64/autobuilds

STAGE=$(curl -s "$BASE/latest-stage3-amd64-openrc.txt" \
        | grep -oE '[0-9]{8}T[0-9]{6}Z/stage3-amd64-openrc-[0-9]{8}T[0-9]{6}Z\.tar\.xz' \
        | head -1)
FILE=$(basename "$STAGE")
DIR=$(dirname "$STAGE")
echo "$FILE"

latest-stage3-*.txt is PGP-signed and contains -----BEGIN PGP SIGNED MESSAGE----- blocks. Filtering out comments doesn't remove them, which is why the pattern matches the path directly instead of listing what to skip.

curl -s "$BASE/$DIR/" | grep -oE "${FILE}[.A-Za-z0-9]*" | sort -u

curl -L -O "$BASE/$STAGE"
curl -L -O "$BASE/$STAGE.DIGESTS"

The digest file is split into sections introduced by # <ALGORITHM> HASH lines. Gentoo uses BLAKE2B and SHA512 there, not SHA256.

awk -v f="$FILE" '/^# /{alg=$2; next} $2==f && alg=="SHA512"{print}' "$FILE.DIGESTS" \
    | sha512sum -c -

The result must be OK. Otherwise stop.

A checksum fetched from the same server as the archive only proves the transfer wasn't damaged. Verifying the signature needs gnupg and the Gentoo Release Engineering key; take the key ID from the Gentoo Handbook, not from this page.

tar xpf "$FILE" --xattrs-include='*.*' --numeric-owner -C /mnt/gentoo
rm "$FILE" "$FILE.DIGESTS"
ls /mnt/gentoo

Variant systemd: replace openrc with systemd in the file name and pick the matching profile in Stage 6.

Stage 5: Configuration before the chroot

5.1. make.conf

cat > /mnt/gentoo/etc/portage/make.conf << 'EOF'
COMMON_FLAGS="-march=MARCH -O2 -pipe"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"

CHOST="x86_64-pc-linux-gnu"

MAKEOPTS="-jJOBS -lJOBS_PLUS_ONE"
EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=JOBS_PLUS_ONE"

USE="-X -gtk -gnome -kde -qt5 -qt6 -wayland -bluetooth -cups -pulseaudio -systemd nftables"

GRUB_PLATFORMS="pc"
VIDEO_CARDS=""
INPUT_DEVICES=""

ACCEPT_LICENSE="-* @FREE"
L10N="en"
LC_MESSAGES="C.utf8"

PORTAGE_SCHEDULING_POLICY="idle"
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

PORTAGE_LOGDIR="/var/log/portage"
PORTAGE_ELOG_CLASSES="warn error log"
PORTAGE_ELOG_SYSTEM="save_summary"
FEATURES="split-log clean-logs"
EOF

Substitute MARCH, JOBS and JOBS_PLUS_ONE. Use GRUB_PLATFORMS="efi-64" for UEFI boot.

\${PID} must keep its backslash: the heredoc delimiter is quoted, and Portage substitutes the variable itself.

PORTAGE_SCHEDULING_POLICY and PORTAGE_IONICE_COMMAND lower the priority of build processes for the CPU and for the disk. They do nothing during installation; they start to matter when a multi-hour emerge competes with production services.

The elog settings send ewarn and einfo messages to /var/log/portage/elog/summary.log. Without them, messages about required manual intervention scroll away in the terminal history. FEATURES is incremental, so these values are added to the ones set by the profile.

No --keep-going in EMERGE_DEFAULT_OPTS. When building a system from scratch, a failed package usually means a configuration error you want to see immediately, not two hours later. Add the switch to individual invocations when you mean it.

5.2. Name servers

cat > /mnt/gentoo/etc/resolv.conf << 'EOF'
nameserver 9.9.9.9
nameserver 1.1.1.1
EOF

5.3. fstab

. /mnt/gentoo/root/install.env

cat > /mnt/gentoo/etc/fstab << EOF
UUID=$UUID_SYSTEM  /                btrfs  noatime,compress=zstd:3,ssd,discard=async,subvol=@           0 0
UUID=$UUID_SYSTEM  /var/lib/lxc     btrfs  noatime,compress=zstd:3,ssd,discard=async,subvol=@lxc        0 0
UUID=$UUID_SYSTEM  /var/log         btrfs  noatime,compress=zstd:3,ssd,discard=async,subvol=@log        0 0
UUID=$UUID_SYSTEM  /.snapshots      btrfs  noatime,compress=zstd:3,ssd,discard=async,subvol=@snapshots  0 0
UUID=$UUID_SYSTEM  /var/tmp/portage btrfs  noatime,discard=async,subvol=@portage                        0 0
UUID=$UUID_BOOT    /boot            btrfs  noatime,discard=async                                        0 0
UUID=$UUID_SWAP    none             swap   sw,discard=once                                              0 0
EOF

cat /mnt/gentoo/etc/fstab

Swap gets discard=once: the whole area is released once at activation instead of reporting every freed page separately.

The last column is 0 everywhere. Btrfs has no consistency check run at boot; checksums do that job.

5.4. System mounts and chroot

mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys  /mnt/gentoo/sys && mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev  /mnt/gentoo/dev && mount --make-rslave /mnt/gentoo/dev
mount --bind  /run  /mnt/gentoo/run && mount --make-slave  /mnt/gentoo/run

chroot /mnt/gentoo /bin/bash
source /etc/profile
export PS1="(chroot) ${PS1}"
. /root/install.env

From here until Stage 12 every command runs inside the chroot. The (chroot) marker in the prompt isn't cosmetic: without it, running a command in the rescue system instead of the new one is easy.

Stage 6: Ebuild repository and profile

emerge-webrsync
eselect news read
emerge -1av sys-apps/portage

eselect profile list
eselect profile set default/linux/amd64/23.0/no-multilib
eselect profile show

Actually read the news before marking it read. A fresh tree has dozens of items, and some of them require action when migrating profiles or when default USE flags change.

Rebuild the whole world for the new profile, USE flags and CFLAGS:

emerge -eavuDN --with-bdeps=y @world

--emptytree is required: the stage3 was built with generic settings, so without it only packages with changed USE flags would be rebuilt, and the rest would stay compiled for the default -march. One pass is enough; a second one is only recommended after changing CHOST, which doesn't happen here. This is the longest step of the procedure, so run it only inside the multiplexer.

emerge -av --depclean
emerge -av @preserved-rebuild

Review the removal list. The profile change and your own USE flags make some packages from the stage3 unnecessary, but something you use directly may have come in only as a dependency.

No binary packages. Besides the principle, a binary host would make -march meaningless here: Portage matches binary packages by version and USE flags, and CFLAGS aren't a criterion for rejecting one.

Stage 7: Time zone, locale, accounts

ln -sf ../usr/share/zoneinfo/TZ /etc/localtime
date

cat > /etc/locale.gen << 'EOF'
en_US.UTF-8 UTF-8
EOF
locale-gen
eselect locale list
eselect locale set en_US.utf8

env-update && source /etc/profile && export PS1="(chroot) ${PS1}"

echo "$HOST" > /etc/hostname
sed -i "s/^hostname=.*/hostname=\"$HOST\"/" /etc/conf.d/hostname
cat /etc/hostname /etc/conf.d/hostname

passwd root

Substitute TZ and adjust the locale to taste. The time zone link must be relative: created inside the chroot, an absolute link would point into the rescue system's directory.

/etc/hostname is the mechanism that counts. /etc/conf.d/hostname describes itself as a fallback used when the main file is missing, and contains localhost in the stage3. Setting both makes a vanished main file show up legibly instead of as a machine called localhost. The hostname service is in the boot runlevel by default.

source /etc/profile overwrites PS1, which is why the chroot marker is restored on the same line.

A side effect of a non-English locale. Locales like pl_PL change the decimal separator to a comma and the sort order in LC_COLLATE, which affects sort and ranges in shell globs. Scripts that parse other commands' output should start with export LC_ALL=C. LC_MESSAGES="C.utf8" in make.conf only affects package builds, so that build logs stay usable for bug reports.

User account

Root login over SSH gets disabled in Stage 10, so the user account has to exist before that. Without it, the provider's console is all that's left after the first reboot.

useradd -c "Full Name" -g users -G wheel -m -s /bin/bash USERNAME
passwd USERNAME

install -d -m 700 -o USERNAME -g users /home/USERNAME/.ssh
cat > /home/USERNAME/.ssh/authorized_keys << 'EOF'
YOUR_SSH_PUBLIC_KEY
EOF
chown USERNAME:users /home/USERNAME/.ssh/authorized_keys
chmod 600 /home/USERNAME/.ssh/authorized_keys

The primary group is users, not a private per-user group. Private groups exist so that umask 002 is safe; Gentoo defaults to umask 022.

wheel is the conventional administrative group. To let only its members use su, uncomment the pam_wheel line in /etc/pam.d/su; by default su accepts anyone who knows the root password. With a single administrator, su from shadow is enough and sudo is one more configuration to maintain.

The user's password matters for the same reason as root's: the provider's console doesn't accept SSH keys.

Stage 8: Kernel

emerge -av sys-kernel/gentoo-sources sys-apps/pciutils
eselect kernel list
eselect kernel set 1
cd /usr/src/linux
make defconfig
make menuconfig

Required options

With a kernel without modules, every missing item means a system that can't see its own disk or network. Everything =y.

Guest virtualization
  CONFIG_HYPERVISOR_GUEST=y
  CONFIG_PARAVIRT=y
  CONFIG_KVM_GUEST=y
  CONFIG_VIRTIO=y
  CONFIG_VIRTIO_PCI=y
  CONFIG_VIRTIO_PCI_LEGACY=y
  CONFIG_VIRTIO_BALLOON=y

Disk, virtio-scsi controller [1af4:1004]
  CONFIG_SCSI=y
  CONFIG_BLK_DEV_SD=y
  CONFIG_SCSI_VIRTIO=y

Network, virtio-net card [1af4:1000]
  CONFIG_VIRTIO_NET=y
  CONFIG_PACKET=y
  CONFIG_UNIX=y
  CONFIG_INET=y
  CONFIG_IPV6=y
  # CONFIG_DUMMY is not set
  # CONFIG_IPV6_SIT is not set

Provider console
  CONFIG_VT=y
  CONFIG_VT_CONSOLE=y
  CONFIG_VGA_CONSOLE=y
  CONFIG_SERIAL_8250=y
  CONFIG_SERIAL_8250_CONSOLE=y

Encryption and LVM
  CONFIG_BLK_DEV_DM=y
  CONFIG_DM_CRYPT=y
  CONFIG_CRYPTO_XTS=y
  CONFIG_CRYPTO_AES=y
  CONFIG_CRYPTO_AES_NI_INTEL=y
  CONFIG_CRYPTO_SHA256=y
  CONFIG_CRYPTO_USER_API_SKCIPHER=y

File systems and partitions
  CONFIG_BTRFS_FS=y
  CONFIG_BTRFS_FS_POSIX_ACL=y
  CONFIG_EXT4_FS=y
  CONFIG_VFAT_FS=y
  CONFIG_PARTITION_ADVANCED=y
  CONFIG_EFI_PARTITION=y
  CONFIG_DEVTMPFS=y
  CONFIG_DEVTMPFS_MOUNT=y
  CONFIG_TMPFS=y

Compression and config visibility
  CONFIG_KERNEL_ZSTD=y
  CONFIG_IKCONFIG=y
  CONFIG_IKCONFIG_PROC=y

Initramfs and networking for dropbear
  CONFIG_BLK_DEV_INITRD=y
  CONFIG_RD_ZSTD=y
  CONFIG_RD_GZIP=y
  CONFIG_IP_PNP=y
  CONFIG_IP_PNP_DHCP=y

Queueing and congestion control
  CONFIG_NET_SCHED=y
  CONFIG_NET_SCH_FQ=y
  CONFIG_TCP_CONG_ADVANCED=y
  CONFIG_TCP_CONG_BBR=y

Socket diagnostics, needed by ss
  CONFIG_INET_DIAG=y
  CONFIG_INET_TCP_DIAG=y
  CONFIG_INET_UDP_DIAG=y
  CONFIG_UNIX_DIAG=y
  CONFIG_PACKET_DIAG=y
  CONFIG_NETLINK_DIAG=y

Firewall, tunnel and containers
  CONFIG_NETFILTER=y
  CONFIG_NF_TABLES=y
  CONFIG_NF_CONNTRACK=y
  CONFIG_NF_NAT=y
  CONFIG_NFT_NAT=y             # dnat and snat, for port forwarding
  CONFIG_NFT_MASQ=y            # masquerading traffic leaving the containers
  CONFIG_NFT_CT=y              # connection state matching
  CONFIG_NFT_LIMIT=y           # rate limiting
  CONFIG_NFT_LOG=y
  CONFIG_NFT_REJECT=y
  CONFIG_WIREGUARD=y
  CONFIG_BRIDGE=y
  CONFIG_VETH=y
  CONFIG_NAMESPACES=y
  CONFIG_USER_NS=y
  CONFIG_NET_NS=y
  CONFIG_PID_NS=y
  CONFIG_IPC_NS=y
  CONFIG_UTS_NS=y
  CONFIG_CGROUPS=y
  CONFIG_CGROUP_DEVICE=y
  CONFIG_MEMCG=y

Match the disk and network blocks to the lspci -nnk output from Stage 0. On physical hardware, enable the drivers for the actual controller and card instead of virtio.

CONFIG_VIRTIO_PCI_LEGACY is required when devices report transitional IDs (1af4:1000, 1af4:1004) instead of modern ones (1af4:1041, 1af4:1048). Without it the kernel sees neither the disk nor the network card.

CONFIG_EFI_PARTITION handles reading GPT, not UEFI boot. Without it the kernel sees no partitions at all on a GPT disk, even when booting via BIOS.

CONFIG_BTRFS_FS pulls in its own crypto and compression dependencies. CONFIG_BTRFS_FS_POSIX_ACL is needed if the system has to restore permissions from archives, for example with containers. CONFIG_EXT4_FS and CONFIG_VFAT_FS aren't needed by this layout, but adding them later on a kernel without modules means a full rebuild and a manual unlock, so they are cheap insurance for attaching foreign media.

CONFIG_NET_SCH_FQ and CONFIG_TCP_CONG_BBR are required by the sysctl settings in Stage 10. Without them the corresponding files in /proc/sys don't exist and the sysctl service fails at every boot. defconfig gives you only cubic.

Disable CONFIG_DUMMY and CONFIG_IPV6_SIT. They create dummy0 and sit0, on which dracut makes failed configuration attempts when the interface field in ip= is empty, delaying the boot by a dozen seconds.

Keep CONFIG_IPV6 even if you disable IPv6 through sysctl. Removing IPv6 from the kernel breaks software that assumes AF_INET6 sockets exist.

CONFIG_RD_ZSTD is the initramfs decompressor and must match compress= in the dracut configuration from Stage 9. A mismatch shows up as a kernel panic complaining about a missing root file system. CONFIG_KERNEL_ZSTD is the compression of the kernel image itself, independent of the above, because the image decompresses itself.

CONFIG_IKCONFIG_PROC exposes the running kernel's configuration as /proc/config.gz. On a remote machine that tells you what the system actually booted with, and it's also the best starting point for the next kernel (see Operations).

The console block keeps the emergency path from Stage 13 open. When trimming a configuration, it's easy to disable CONFIG_VT and lose the ability to type a passphrase on the provider's console.

CPU vulnerability mitigations

Leave them all enabled. Without page table isolation, a process in a container can read kernel memory without escaping the container, and that memory holds the credentials of every other service.

With a configuration carried over from another machine, check the whole family at once. CONFIG_MITIGATION_PAGE_TABLE_ISOLATION and CONFIG_MITIGATION_RETPOLINE default to y, but the parent switch CONFIG_CPU_MITIGATIONS turns off the whole submenu, and make oldconfig would carry that state over without asking.

grep -E 'CONFIG_(CPU_MITIGATIONS|MITIGATION_)' .config

Retpoline really is a compile-time decision: the compiler generates different code for indirect calls. It can be turned off at boot with spectre_v2=off, but it can't be turned on if it wasn't compiled in.

Don't put mitigations=off on the kernel command line. That is the right place for such a decision, should it ever be made, precisely because it's reversible without a rebuild.

Checking the configuration

Run this after make oldconfig, not before. oldconfig silently drops symbols whose dependencies aren't met, which is exactly why looking through menuconfig isn't enough.

cd /usr/src/linux

for s in \
  CONFIG_HYPERVISOR_GUEST CONFIG_PARAVIRT CONFIG_KVM_GUEST \
  CONFIG_VIRTIO CONFIG_VIRTIO_PCI CONFIG_VIRTIO_PCI_LEGACY CONFIG_VIRTIO_BALLOON \
  CONFIG_SCSI CONFIG_BLK_DEV_SD CONFIG_SCSI_VIRTIO \
  CONFIG_VIRTIO_NET CONFIG_PACKET CONFIG_UNIX CONFIG_INET \
  CONFIG_VT CONFIG_VT_CONSOLE CONFIG_VGA_CONSOLE \
  CONFIG_SERIAL_8250 CONFIG_SERIAL_8250_CONSOLE \
  CONFIG_BLK_DEV_DM CONFIG_DM_CRYPT \
  CONFIG_CRYPTO_XTS CONFIG_CRYPTO_AES CONFIG_CRYPTO_AES_NI_INTEL \
  CONFIG_CRYPTO_SHA256 CONFIG_CRYPTO_USER_API_SKCIPHER \
  CONFIG_BTRFS_FS CONFIG_BTRFS_FS_POSIX_ACL \
  CONFIG_PARTITION_ADVANCED CONFIG_EFI_PARTITION \
  CONFIG_DEVTMPFS CONFIG_DEVTMPFS_MOUNT CONFIG_TMPFS \
  CONFIG_BLK_DEV_INITRD CONFIG_RD_ZSTD CONFIG_RD_GZIP \
  CONFIG_KERNEL_ZSTD CONFIG_IKCONFIG CONFIG_IKCONFIG_PROC \
  CONFIG_IP_PNP CONFIG_IP_PNP_DHCP \
  CONFIG_NET_SCHED CONFIG_NET_SCH_FQ CONFIG_TCP_CONG_ADVANCED CONFIG_TCP_CONG_BBR \
  CONFIG_INET_DIAG CONFIG_INET_TCP_DIAG CONFIG_INET_UDP_DIAG \
  CONFIG_UNIX_DIAG CONFIG_PACKET_DIAG CONFIG_NETLINK_DIAG \
  CONFIG_NETFILTER CONFIG_NF_TABLES CONFIG_NF_CONNTRACK CONFIG_NF_NAT \
  CONFIG_NFT_NAT CONFIG_NFT_MASQ CONFIG_NFT_CT CONFIG_NFT_LIMIT \
  CONFIG_NFT_LOG CONFIG_NFT_REJECT \
  CONFIG_WIREGUARD CONFIG_BRIDGE CONFIG_VETH \
  CONFIG_NAMESPACES CONFIG_USER_NS CONFIG_NET_NS \
  CONFIG_PID_NS CONFIG_IPC_NS CONFIG_UTS_NS \
  CONFIG_CGROUPS CONFIG_CGROUP_DEVICE CONFIG_MEMCG \
; do
    v=$(sed -n "s/^${s}=//p" .config)
    case "$v" in
        y)  ;;
        m)  printf 'MODULE  %s\n' "$s" ;;
        "") printf 'MISSING %s\n' "$s" ;;
        *)  printf '=%-6s %s\n' "$v" "$s" ;;
    esac
done

Silence means everything is =y. Only problems are printed: MISSING is a symbol that is absent or disabled, MODULE is =m. Run the same loop after every future kernel update, after make oldconfig and before make install.

Building

At this stage, build only. Installation happens in Stage 11, after dracut and the boot loader are configured.

make -jJOBS
ls -lh arch/x86/boot/bzImage

sys-kernel/installkernel

Set the USE flags before installing: the package pulls in dracut, LVM2 and GRUB, which are exactly the packages these entries concern.

cat > /etc/portage/package.use/50-installkernel << 'EOF'
sys-kernel/installkernel dracut grub -efistub -refind -systemd -systemd-boot -ugrd -uki -ukify
sys-fs/lvm2 lvm
EOF

cat > /etc/portage/package.use/50-grub << 'EOF'
sys-boot/grub -themes -fonts -branding -truetype -nls
EOF

emerge -av sys-kernel/installkernel

The lvm flag on sys-fs/lvm2 is mandatory and not enabled by default. Without it the package only provides the device-mapper library, without lvchange, vgchange and the other tools. Dracut's lvm module needs those binaries, and the root file system lives on a logical volume.

GRUB's graphical theme is pointless with GRUB_TERMINAL="console serial" from Stage 11. Keep the device-mapper flag: grub-mkconfig needs it to recognise that root lives under LUKS and LVM.

The package provides /sbin/installkernel, which make install calls automatically. Without it, the kernel's install script copies the image to a fixed name /boot/vmlinuz, keeping exactly one backup. With it you get versioned names and, with the flags above, the initramfs and a regenerated boot loader configuration as part of the same make install. A dracut failure aborts make install, which is what you want: the alternative is a kernel without an initramfs, i.e. a machine that can't find its encrypted root.

Variant UEFI: switch the flags to -grub efistub or -grub systemd-boot, depending on the boot loader, and skip the GRUB installation in Stage 11.

Stage 9: Initramfs with remote unlock

emerge -av sys-kernel/dracut net-misc/dropbear sys-fs/cryptsetup sys-fs/lvm2 sys-fs/btrfs-progs

Every dracut module needs its tool in the target system. crypt needs cryptsetup, lvm needs lvm2, btrfs needs the btrfs command. The file systems were created from the rescue system, so the new system doesn't have these yet. A missing one ends with Module 'X' cannot be installed and aborts make install.

9.1. The crypt-ssh module

The module is in the main tree as sys-kernel/dracut-crypt-ssh and only needs its keyword accepted; it's soft-masked with ~amd64.

cat > /etc/portage/package.accept_keywords/50-dracut-crypt-ssh << 'EOF'
sys-kernel/dracut-crypt-ssh ~amd64
EOF

The ebuild declares an any-of dependency:

|| ( net-misc/connman  net-misc/dhcp[client]  net-misc/networkmanager  sys-apps/systemd )

Portage picks the first entry, connman, which drags in wpa_supplicant, D-Bus, GLib and part of the Python stack: twenty-odd packages to install a few dozen kilobytes of shell scripts. Naming the ISC DHCP client explicitly in the same command makes Portage satisfy the alternative with it instead.

cat > /etc/portage/package.use/50-dracut-crypt-ssh << 'EOF'
net-misc/dhcp client -server
EOF

emerge -av net-misc/dhcp sys-kernel/dracut-crypt-ssh
ls -l /usr/lib/dracut/modules.d/*crypt-ssh*/

The DHCP client never runs. Networking in the initramfs is configured statically by the ip= parameter, handled by the 35network-legacy module that ships with dracut. The package is installed only to satisfy the ebuild's declaration. This is also a good example of why emerge should always run with -a.

Read the module's scripts before you use them. They end up in the initramfs with root privileges, listening on a port exposed to the internet. It's a few hundred lines, a quarter of an hour of reading.

This is the only package from the testing branch in this system. It gets less testing than the rest and sits exactly where a failure cuts off access to the machine. After every update of it, go through the initramfs checks from Stage 11 before rebooting.

9.2. Keys

Keep the keys allowed to unlock the disk in a separate file, not in the target system's /root/.ssh/authorized_keys. These are two different privileges: unlocking the disk before the system starts, and working as root in the running system.

mkdir -p /etc/dropbear/initramfs
cat > /etc/dropbear/initramfs/authorized_keys << 'EOF'
YOUR_SSH_PUBLIC_KEY
SECOND_SSH_PUBLIC_KEY
EOF
chmod 700 /etc/dropbear/initramfs
chmod 600 /etc/dropbear/initramfs/authorized_keys

You can list several keys, one per line. A machine that can't be unlocked can't be fixed remotely in any other way.

Revoking access requires rebuilding the initramfs. The keys are baked into the image when it's built, not read from disk at boot. Removing a line from the file takes nothing away from anyone until the image is rebuilt.

Point the module at an explicit host key. Otherwise it generates a new one on every initramfs rebuild, i.e. after every kernel update, which trains you to ignore host key change warnings on a port exposed to the internet.

ssh-keygen -t ed25519 -N "" -C "initramfs unlock" -f /etc/dropbear/initramfs/ssh_host_ed25519_key
chmod 600 /etc/dropbear/initramfs/ssh_host_ed25519_key
ssh-keygen -lf /etc/dropbear/initramfs/ssh_host_ed25519_key.pub

The key must be in OpenSSH format, not dropbear's. The module runs dropbearconvert openssh dropbear itself when building the initramfs. A file generated by dropbearkey fails with dropbearconvert for ED25519 key failed, and the result is an image without a host key.

Write down the fingerprint printed by the last command. The conversion doesn't change the key material, so it's the fingerprint your SSH client will show on the first connection, and your only chance to verify it without trust on first use.

9.3. /etc/crypttab

cat > /etc/crypttab << EOF
luks-$UUID_LUKS UUID=$UUID_LUKS none luks,discard
EOF

cat /etc/crypttab

The file is needed by the module, not by the system. Under OpenRC it does nothing in the running system, which uses /etc/conf.d/dmcrypt instead, so it looks like a systemd leftover and is easy to "clean up" one day. Dracut in hostonly mode copies it into the image, and unlock reads it to find out what to open. Without it the initramfs shell answers /etc/crypttab is empty or invalid, even though dropbear is running. The module's README documents this; the Gentoo wiki article on unlocking over SSH describes a different solution and doesn't mention crypttab at all.

The third field, none, means "ask for a passphrase" instead of using a key file. discard matches the TRIM decision from Stage 2.

9.4. Dracut configuration

cat > /etc/dracut.conf.d/10-local.conf << 'EOF'
hostonly="yes"
hostonly_cmdline="no"
add_dracutmodules+=" crypt lvm btrfs network crypt-ssh "
omit_dracutmodules+=" plymouth i18n "
compress="zstd"
no_kernel="yes"
install_items+=" /bin/ps "
kernel_cmdline=""
EOF

cat > /etc/dracut.conf.d/20-crypt-ssh.conf << EOF
dropbear_port="$SSHPORT"
dropbear_acl="/etc/dropbear/initramfs/authorized_keys"
dropbear_keytypes="ed25519"
dropbear_ed25519_key="/etc/dropbear/initramfs/ssh_host_ed25519_key"
EOF

no_kernel="yes" is mandatory with CONFIG_MODULES=n. Without modules, make modules_install has nothing to install, so /lib/modules/<version> never exists. Dracut assumes it does and refuses with Cannot find module directory, aborting the whole make install. Creating an empty directory would only fool the check and would have to be repeated for every kernel version.

install_items+=" /bin/ps " adds ps, which dracut doesn't pull in by itself. After a successful unlock, unlock runs unlock-reap-success, which uses ps to find the cryptroot-ask process still waiting for a passphrase on the console, and kills it. Without ps the script reports ps: command not found.

The i18n module handles the keyboard layout in the initramfs and errors out unless i18n_vars is set. It's unnecessary if the passphrase was chosen for the US layout.

Module defaults overridden above: port 222, the key file /root/.ssh/authorized_keys, and three host key types at once. Limiting it to ed25519 gives one key to look after and a smaller image.

The initramfs is created by make install in Stage 11; this section only configures it. To rebuild it separately, for example after changing the dropbear keys without touching the kernel:

cd /usr/src/linux
KV=$(make -s kernelrelease)
dracut --force --kver "$KV" /boot/initramfs-"$KV".img

Take the version from make -s kernelrelease, not from the list in /lib/modules. With CONFIG_MODULES=n that directory doesn't exist, and reading it would return nothing, or the rescue system's kernel version.

Stage 10: Network, SSH, hardening

10.1. Network

emerge -av net-misc/netifrc

cat > /etc/conf.d/net << EOF
config_$IFACE="$ADDR/$PREFIX"
routes_$IFACE="$GW/32 dev $IFACE
default via $GW"

preup() {
    [ "\${IFACE}" = "$IFACE" ] || return 0
    ip link set dev "\${IFACE}" down
    ip addr flush dev "\${IFACE}"
    return 0
}

postup() {
    [ "\${IFACE}" = "$IFACE" ] || return 0
    tc qdisc replace dev "\${IFACE}" root fq
    return 0
}
EOF

cd /etc/init.d
ln -s net.lo "net.$IFACE"
rc-update add "net.$IFACE" default
cat /etc/conf.d/net

\${IFACE} must keep its backslash: the heredoc delimiter is unquoted, and netifrc substitutes the variable at run time. The right-hand $IFACE has no backslash on purpose, so that the interface name is written into the file literally.

The interface name check at the start of both functions is required. preup and postup are global, and netifrc calls them for every interface it brings up, including lo and, later, bridges and tunnels. Without the guard, ip link set dev lo down would wipe 127.0.0.1. || return 0 also gives the other interfaces a zero exit code, which matters because a non-zero result from preup aborts the interface's configuration.

The config_ and routes_ entries are written for a gateway outside the address's mask. The route order matters: the gateway first has to become reachable through an on-link route, and only then can it be the default. The other way round ends with Network is unreachable. With the gateway inside the mask, this is enough:

config_IFACE="ADDR/PREFIX"
routes_IFACE="default via GW"

preup cleans up after the kernel's own configuration. CONFIG_IP_PNP processes ip= before user space starts in the initramfs, so the interface enters the target system already up, with an address and a default route set by the kernel. netifrc adds its own configuration but doesn't remove what it finds, leaving two addresses and two default routes with different metrics.

The symptom is quiet, because the machine works. But an address with a mask wider than intended creates a direct route to that whole network, so the host treats the other addresses in it as on-link and ARPs for them instead of routing through the gateway. At a provider that shares a subnet between customers, traffic to those addresses is lost. The kernel's route, with the lower metric, also takes precedence over /etc/conf.d/net, so changing the gateway there has no effect until the next reboot.

Taking the interface down fixes both: the kernel removes all routes of a device that goes down, and ip addr flush removes the address.

postup sets the queueing discipline explicitly, because sysctl isn't enough here. net.core.default_qdisc is read only when a device is first activated after registration. The kernel brings the interface up while processing ip=, before sysctl in the boot runlevel sets fq, so the device gets pfifo_fast, and BBR runs without the pacing it was meant to be paired with.

Bringing the interface down and up in preup doesn't fix that: on deactivation the kernel replaces the discipline on the transmit queues but doesn't reset the device's root qdisc pointer, so the same one comes back. A mismatch between sysctl net.core.default_qdisc and tc qdisc show is the rule here, not a contradiction: the former is the default for devices registered from now on, the latter is the state of the existing interface.

The sysctl service doesn't show up in a bare rc-status. It lives in the boot runlevel, and rc-status without arguments only lists the current one. Use rc-status boot or rc-status -a.

10.2. SSH

emerge -av net-misc/openssh

sed -i \
  -e 's/^#\?PermitRootLogin.*/PermitRootLogin no/' \
  -e 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' \
  -e 's/^#\?KbdInteractiveAuthentication.*/KbdInteractiveAuthentication no/' \
  /etc/ssh/sshd_config

sshd -t && grep -E '^(PermitRootLogin|PasswordAuthentication|KbdInteractiveAuthentication)' /etc/ssh/sshd_config
rc-update add sshd default

PasswordAuthentication no alone isn't enough. Keyboard-interactive authentication is a separate method, enabled by default, and through PAM it accepts the same account password. You can tell by the prompt: (user@host) Password: in parentheses comes from PAM, plain password login looks like user@host's password:. This check catches it, and must be refused:

ssh -o PreferredAuthentications=keyboard-interactive -o PubkeyAuthentication=no USERNAME@ADDR

Check with a pattern that includes comments. All three directives are present in the default sshd_config, but commented out. A pattern without # finds nothing and looks like a missing entry, while the default value silently applies. That's why one sed with ^#\? replaces all three instead of appending to the end of the file.

Consider adding AuthenticationMethods publickey. It isn't in the default file at all, so it has to be added. It declares what is allowed instead of listing what is disabled, and a list of exclusions goes stale as OpenSSH defaults change, exactly as it did with keyboard-interactive.

Root doesn't log in over SSH. /root/.ssh/authorized_keys is deliberately never created: the way in is the user account from Stage 7, and privileges are raised with su. The provider's console with the root password remains the emergency path.

None of this applies to dropbear in the initramfs: only root exists there, so you connect as root on SSHPORT to unlock the disk. Two different SSH servers, two different policies.

10.3. Kernel parameters

cat > /etc/sysctl.d/99-hardening.conf << 'EOF'
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_syncookies = 1
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.ip_forward = 1
EOF

net.ipv4.ip_forward is only needed with containers or tunnels. Leave it off if the machine doesn't forward traffic.

10.4. /etc/rc.conf

rc_cgroup_mode is required if containers with systemd inside are going to run on an OpenRC host. rc_autostart_user disables OpenRC user services.

sed -i \
  -e 's/^#\?rc_cgroup_mode=.*/rc_cgroup_mode="unified"/' \
  -e 's/^#\?rc_autostart_user=.*/rc_autostart_user="NO"/' \
  /etc/rc.conf

grep -E '^rc_(cgroup_mode|autostart_user)' /etc/rc.conf

Without rc_autostart_user="NO", every login leaves a user.<login> service in the failed state. Since sys-auth/pambase-20250228, the PAM stack contains an unconditional -session optional pam_openrc.so, which tries to start an OpenRC user service at login. That service needs XDG_RUNTIME_DIR, provided by sys-auth/elogind, sys-apps/systemd or manually through pam_env, and an installation following this procedure has none of them.

The effect is cosmetic, since the entry is optional and prefixed with a dash, so logins work. But the service appears in rc-status --all as [ failed ], clutters the system log and is retried by supervise-daemon, while the actual message, XDG_RUNTIME_DIR unset, user services cannot start, never reaches the terminal.

rc_autostart_user="NO" is the way the authors intended. It disables the PAM module without touching /etc/pam.d, so the change doesn't come back as a file to merge on the next pambase update. Don't edit system-login by hand, and don't rebuild sys-apps/openrc with -pam: that flag also controls PAM session handling in start-stop-daemon. If you want user services instead, add elogind to the boot runlevel, not default, or it races with logins. Source: the news item 2025-09-04-openrc-user-services.

10.5. Basic services

emerge -av app-admin/sysklogd net-misc/chrony sys-process/cronie app-admin/logrotate
rc-update add sysklogd default
rc-update add chronyd default
rc-update add cronie default

Without logrotate, nothing in the system rotates any log. It isn't a dependency of anything, yet packages keep installing configurations for it into /etc/logrotate.d/. Without the tool those files sit idle, which is easy to miss, because their presence suggests a working mechanism.

sysklogd ships no rotation configuration for the files it writes itself:

cat > /etc/logrotate.d/sysklogd << 'EOF'
/var/log/syslog
/var/log/messages
/var/log/kern.log
/var/log/auth.log
/var/log/mail.log
/var/log/mail.err
{
    weekly
    maxsize 20M
    rotate 8
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /etc/init.d/sysklogd --quiet reload >/dev/null 2>&1 || true
    endscript
}
EOF

logrotate -d /etc/logrotate.d/sysklogd
ls /etc/cron.daily/ | grep -i logrotate

maxsize next to weekly rotates every week, or earlier when a file exceeds the size. weekly alone doesn't protect against a sudden burst, size alone loses regularity.

The postrotate reload is required. sysklogd keeps its files open, so after a plain rename it would keep writing to the deleted inode. The symptom is a log that stops growing while the service seems fine.

logrotate -d is a dry run. The last command confirms the daily job exists; without a running cron daemon nothing rotates, however correct the configuration.

Firewall rules that log dropped packets can dominate log volume. A kernel message at level info lands, with a typical syslog.conf, in three files at once: kern.log, syslog and messages. Even rate-limited to five entries a minute, that's on the order of five megabytes a day. Rotation handles it, but know the multiplier before picking a limit.

Stage 11: Boot loader

emerge -av sys-boot/grub

cat > /etc/default/grub << EOF
GRUB_DISTRIBUTOR="Gentoo"

GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
GRUB_DISABLE_SUBMENU=y

GRUB_CMDLINE_LINUX="rd.luks.uuid=$UUID_LUKS rd.luks.options=discard rd.lvm.vg=$VG root=$ROOTDEV rootflags=subvol=@ ip=$ADDR::$GW:$IPMASK::eth0:off console=ttyS0,115200 console=tty0"
GRUB_CMDLINE_LINUX_DEFAULT=""

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
EOF

cat /etc/default/grub

Write the whole file rather than appending to it. The packaged version is mostly a commented-out cheat sheet, and appending leaves two descriptions of the same thing in one file.

GRUB_DISABLE_SUBMENU=y together with GRUB_TIMEOUT_STYLE=menu keeps older kernel versions visible right away, without navigating nested menus on the provider's console. installkernel keeps previous versions precisely so you can go back to one.

Two console= entries, and their order. Kernel messages go to every listed console, but /dev/console, where dracut's passphrase prompt appears, is the last one. Set it to what the provider's panel shows: a graphical console is tty0, a serial one is ttyS0. The other entry makes sure the full boot log ends up there as well.

ip= names eth0, not the name from the running system. The initramfs has no udev rules for predictable interface names, so the kernel gives the card its default name. /etc/conf.d/net keeps the predictable name. The two are supposed to differ.

The name is given explicitly rather than left empty. An empty field makes dracut configure every interface it sees with the same static address: an address conflict with two cards, and failed attempts plus a dozen seconds of delay with dummy0 and sit0. eth0 means the first detected card and doesn't depend on the PCI slot, so it survives migration better than ensN.

The residual risk: if a future dracut starts pulling naming rules into the initramfs, eth0 stops existing and remote unlock stops working. That would show up at the first reboot after a kernel update, with the provider's console as the way in.

In the generated boot entry root= and rootflags= will appear twice, and that's intended. grub-mkconfig generates its own, derived from the detected device; the ones above are extra. The values must be identical, and the kernel uses the last occurrence. The redundancy protects against grub-probe getting the device wrong from inside the chroot. GRUB_DISABLE_LINUX_UUID doesn't avoid it: that option changes the parameter's form, it doesn't remove it.

Install in BIOS mode, to the disk, not to a partition:

grub-install --target=i386-pc "$DISK"

Variant UEFI: grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=gentoo, with GRUB_PLATFORMS="efi-64" in make.conf.

With /boot on Btrfs, grub-install builds the btrfs module into core.img, which goes into the partition flagged bios_grub.

Installing the kernel

installkernel isn't a command you type. make install runs the kernel's install script, which checks for /sbin/installkernel and hands control over to it. One command copies the image under a versioned name, runs dracut with the configuration from Stage 9, and regenerates grub.cfg.

cd /usr/src/linux
make install
ls -lh /boot

/boot must now contain vmlinuz-<version>, initramfs-<version>.img, System.map-<version> and config-<version>.

Checking the initramfs

Mandatory. Any missing item means a machine you can't get into after a reboot.

KV=$(make -s kernelrelease)
lsinitrd /boot/initramfs-"$KV".img | grep -E 'dropbear|authorized_keys|cryptsetup|lvm|crypttab|btrfs'
lsinitrd /boot/initramfs-"$KV".img | grep dropbear_ed25519_host_key
file /boot/initramfs-"$KV".img

You must see the dropbear binary, the authorized_keys file, cryptsetup, the LVM tools and etc/crypttab. Without the last one remote unlock won't work, even though dropbear starts and lets you into a shell.

The host key in the image must be the one generated in 9.2. A different one means the module generated its own, and the fingerprint will change on every rebuild. The file output must match the decompressor compiled into the kernel; a mismatch only shows up at boot, as a kernel panic about a missing root.

Checking the boot loader configuration

grep -E 'linux|initrd|insmod btrfs' /boot/grub/grub.cfg

Check that:

• both occurrences of root= have identical values;
• console=tty0 comes last if the provider's panel offers a graphical console;
• rd.luks.uuid, rd.luks.options=discard, rd.lvm.vg, rootflags=subvol=@ and ip= are present;
• the file contains insmod btrfs;
• the recovery entry additionally has single.

rootflags=subvol=@ is essential: without it the top-level subvolume gets mounted, with @ and the others as subdirectories, and the system won't find /sbin/init. Without insmod btrfs, GRUB can't read its own boot partition.

Stage 12: Test in an emulator

Optional, but it lets you catch a mistake while still in the rescue system, with the file system mounted and every tool at hand, instead of debugging through the provider's console. If you skip it, open the console in the panel before rebooting the machine.

Before leaving the chroot

grep -E '^(root|USERNAME):' /etc/shadow | cut -d: -f1,2 | sed 's/:\(.\).*/: \1.../'

Both accounts must show the start of a password hash, not ! or *, which mean a locked account.

Releasing the devices

exit
cd /
umount -l /mnt/gentoo/dev /mnt/gentoo/proc /mnt/gentoo/sys /mnt/gentoo/run
umount -R /mnt/gentoo
swapoff "/dev/$VG/swap"

systemctl restart systemd-udevd

dmsetup info -c
vgchange -an "$VG"
cryptsetup close cryptroot

lsblk "$DISK"

Restarting services is necessary. Without it vgchange -an refuses with Logical volume in use, even though findmnt and /proc/mounts show no mounts.

The cause is copies of the mount table in other namespaces. Services running with their own mount namespace in slave mode receive our mounts through propagation, but unmounting in our namespace doesn't remove them from theirs. To find them, search the mount tables of every process, which findmnt doesn't do:

grep -l "$VG" /proc/[0-9]*/mountinfo 2>/dev/null
ls /sys/fs/btrfs/
ps -o pid,comm,args -p FOUND_PIDS

On Debian-based rescue images the usual culprit is systemd-udevd; providers add their own monitoring services, vnstatd being a common one. Restart whatever you find; that rebuilds their namespaces and releases the reference. If nothing helps, rebooting the rescue system settles it in two minutes: without the passphrase no mapped devices come back.

btrfs device scan --forget isn't the fix. While the file system is mounted in another namespace, it refuses with Device or resource busy. After restarting the services the device is already unregistered, and the same command returns No such file or directory, which looks like an error and is actually confirmation.

In dmsetup info -c, the Open column of the root volume must be 0. umount -l unmounts lazily, so its exit code says little; cryptsetup close decides. If anything still uses the volumes, it fails with Device busy. Then check lsof +D /mnt/gentoo or fuser -vm /mnt/gentoo; the most common reason is a forgotten chroot shell in another multiplexer window.

Starting it

apt install -y qemu-system-x86
apt clean

qemu-system-x86_64 \
    -enable-kvm \
    -cpu Haswell-noTSX \
    -m 2048 -smp 2 \
    -serial file:/root/qemu-boot.log \
    -no-reboot \
    -device virtio-scsi-pci,id=scsi0,addr=0x4 \
    -drive file="$DISK",format=raw,if=none,id=disk0,cache=none \
    -device scsi-hd,drive=disk0,bus=scsi0.0 \
    -netdev user,id=net0,hostfwd=tcp:127.0.0.1:2222-:2222,hostfwd=tcp:127.0.0.1:2223-:22 \
    -device virtio-net-pci,netdev=net0,addr=0x3 \
    -vnc 127.0.0.1:0 \
    -boot c

The CPU model must cover the instruction set used for compiling. The default qemu64 has no AVX, and user space built with -march=x86-64-v3 needs AVX2. The first vector instruction in init raises an invalid opcode exception and the kernel panics with Attempted to kill init! exitcode=0x00000004, with asm_exc_invalid_op and machine code starting with the VEX prefix c5 in the log. Pass a named model matching the target machine's CPU, or -cpu host.

The PCI slots are pinned explicitly to mirror the target machine's layout. Otherwise the emulator assigns them in declaration order, which gives a different interface name and a netifrc complaint about a non-existent interface with a perfectly correct configuration. ensN comes straight from the slot number.

-serial file: and -no-reboot. The graphical console only shows the last screen, and with a panic what matters is what scrolled by earlier: tail -n 120 /root/qemu-boot.log. -no-reboot stops the machine on a panic instead of letting it loop.

Networking in the emulator is NATed and the guest gets an address from 10.0.2.0/24, so the static configuration on the kernel command line won't work there. For the test, press e in the GRUB menu, replace the ip=... part with ip=:::::eth0:dhcp and boot with Ctrl+X.

Connecting

Connect to dropbear from your workstation, not from the rescue system: the private key isn't there and shouldn't be.

ssh -N -L 12222:127.0.0.1:2222 -L 12223:127.0.0.1:2223 -L 5900:127.0.0.1:5900 root@ADDR

In a second window, also on the workstation:

ssh -p 12222 root@127.0.0.1

The local ports are shifted so they don't collide with anything running on the workstation. 5900 carries the emulator's graphical console, 12223 the SSH server of the booted system, so you can test that too.

Permission denied (publickey) when trying from the rescue system is a good result. It proves the key list baked into the initramfs is enforced.

Compare the host key fingerprint with the one written down in 9.2.

Unlocking

The module installs three tools:

Use unlock. It reads the passphrase from standard input and runs cryptsetup luksOpen itself on the volumes from /etc/crypttab, so it doesn't depend on whether the console prompt has appeared yet.

console_auth sends the passphrase to the console and introduces a race: if the prompt doesn't exist yet, the text lands on the console as ordinary input with echo on, so the passphrase is displayed in clear and nothing gets unlocked. Treat it as an emergency tool, preceded by console_peek.

The proper call, from the workstation:

read -rsp 'LUKS passphrase: ' PASS && echo && printf '%s' "$PASS" | ssh -p 12222 root@127.0.0.1 unlock; unset PASS

read -rsp reads the passphrase without echo and keeps it out of shell history. printf '%s' adds no newline. Closing the pipe gives end of input.

unlock reads until end of input, not until a newline. Typing the passphrase and pressing Enter isn't enough; the program keeps waiting and looks hung. Close the input with Ctrl+D.

The warning Warning: Unable to lock memory, are you root? is normal: unlock tries to lock its memory so that the passphrase can't end up in swap, and there's no swap in the initramfs yet.

After the passphrase is accepted, the connection drops. That's correct: the hook pre-pivot/05-dropbear-stop.sh stops dropbear before control passes to the target system. If unlock answers /etc/crypttab is empty or invalid, the file didn't make it into the image; check lsinitrd and go back to 9.3.

Shut the emulator down with poweroff inside the guest.

Stage 13: First boot

Switch the machine back to normal boot and restart it. After a minute, from the workstation:

read -rsp 'LUKS passphrase: ' PASS && echo && printf '%s' "$PASS" | ssh -p SSHPORT root@ADDR unlock; unset PASS

A dozen seconds later:

ssh USERNAME@ADDR

Verification

uname -a
lsblk
ip -br addr
ip route
tc qdisc show dev IFACE
sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control
findmnt -R /
free -h
cat /proc/cmdline
rc-status
dmsetup table "luks-$UUID_LUKS" | grep -o allow_discards
lsblk -D
grep . /sys/devices/system/cpu/vulnerabilities/*

Expected: one address on the interface, one default route, qdisc fq, every subvolume mounted in its place, swap active, no services in a failed state.

Check the congestion control algorithm with sysctl, not ss -i, until you've confirmed CONFIG_INET_DIAG is in the kernel. Without the netlink diagnostic socket, ss falls back to reading /proc/net/tcp and shows only the old fields: rto, ato, cwnd, reordering, without the algorithm name, rtt or wscale, and with rto in kernel ticks instead of milliseconds. It looks like a broken tool and is actually a missing kernel option.

rc-status without arguments only shows the current runlevel, so net.lo and sysctl won't appear there; they belong to boot.

dmsetup table must print allow_discards. If it doesn't, rd.luks.options=discard didn't make it onto the kernel command line. In lsblk -D, the DISC-GRAN and DISC-MAX columns must be non-zero along the whole path, from the disk through the encryption layer to the logical volume.

Save the CPU vulnerabilities list as a reference. After the machine is migrated between the provider's nodes the picture may change, and this directory will be the only trace.

If the machine doesn't respond

First, check whether it's simply waiting for the passphrase. A machine sitting in the initramfs looks half alive from the outside: it answers ping, because the kernel assigns the address already in the initramfs, but the regular SSH port and every service refuse connections. The dropbear port tells you for sure:

timeout 5 bash -c 'exec 3<>/dev/tcp/ADDR/SSHPORT && head -1 <&3'

SSH-2.0-dropbear_… means unlock is all you need. Only if that port stays silent while ping answers did dropbear fail to start.

Then open the console in the provider's panel. You'll see the boot loader menu, kernel messages and dracut's passphrase prompt on it, which tells you within seconds whether the problem is in booting, in the initramfs or in the network. You can also type the passphrase there and fix dropbear later from the running system, provided you are willing to send it through the provider's web infrastructure.

The console lets you edit the GRUB entry with e, for example to append rd.break=pre-mount or temporarily change ip= to dhcp. rd.break=pre-mount stops dracut in a shell before mounting the root. Check ip a, whether dropbear is running (ps -ef), and whether the network card was detected at all; that separates a network configuration problem from a missing driver.

Likely causes, in order: the key missing from authorized_keys baked into the initramfs, a typo in ip=, an initramfs built before the configuration was in place, a missing network driver in the kernel. The first three are checked with lsinitrd and cat /proc/cmdline, the last one with zgrep VIRTIO_NET /proc/config.gz after a successful boot.

Only when the console shows nothing, or the boot never reaches the menu, go back to the rescue system:

apt update && apt install -y cryptsetup-bin lvm2 btrfs-progs tmux

cryptsetup open --allow-discards /dev/disk/by-uuid/UUID_LUKS cryptroot
vgchange -ay VG

mkdir -p /mnt/gentoo
mount -o subvol=@    /dev/VG/system /mnt/gentoo
mount -o subvol=@log /dev/VG/system /mnt/gentoo/var/log
mount /dev/disk/by-uuid/UUID_BOOT /mnt/gentoo/boot

mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys && mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev && mount --make-rslave /mnt/gentoo/dev
mount --bind  /run /mnt/gentoo/run && mount --make-slave  /mnt/gentoo/run

chroot /mnt/gentoo /bin/bash
source /etc/profile && export PS1="(chroot) ${PS1}"
. /root/install.env

Use UUIDs: disk numbering in the rescue system changes between boots. Without -o subvol=@ you mount the top-level subvolume and see @, @lxc and the rest as directories instead of the root file system.

Stage 14: Workstation

The configuration on the computer you unlock the machine from.

~/.ssh/config

Host server-unlock
    HostName ADDR
    Port SSHPORT
    User root

Host server
    HostName ADDR
    User USERNAME

Add IdentityFile together with IdentitiesOnly yes to server-unlock if a non-default key is used for unlocking. Without IdentitiesOnly, the client first offers every key from the agent, and dropbear drops the connection after ten failed authentication attempts.

Pinning the host key

known_hosts entries are keyed by address and port, so the dropbear key doesn't collide with the running system's SSH key. Pin it out of band instead of accepting it on first connection: that connection is where you send the disk passphrase. On the server, after a normal login:

printf '[ADDR]:SSHPORT %s\n' "$(ssh-keygen -y -f /etc/dropbear/initramfs/ssh_host_ed25519_key)"

Append the output to ~/.ssh/known_hosts on the workstation.

Shell functions

server-unlock() {
    local pass
    read -rsp 'LUKS passphrase: ' pass; echo
    printf '%s' "$pass" | ssh server-unlock unlock
    unset pass
}

server-boot() {
    local h=ADDR
    printf 'Waiting for dropbear' >&2
    until (exec 3<>"/dev/tcp/$h/SSHPORT") 2>/dev/null; do printf '.' >&2; sleep 2; done
    echo >&2
    server-unlock || return
    printf 'Waiting for the system' >&2
    until (exec 3<>"/dev/tcp/$h/22") 2>/dev/null; do printf '.' >&2; sleep 2; done
    echo >&2
    ssh server
}

The passphrase travels through a variable and a pipe, never through the argument list, so it doesn't show up in ps or in shell history. The ( ) subshell closes the descriptor by itself. Call server-boot right after rebooting the machine. While the initramfs runs, the regular SSH port is closed, so the condition is reliable.

With a password manager, without typing:

pass show path/to/passphrase | tr -d '\n' | ssh server-unlock unlock

tr -d '\n' is required: unlock reads until end of input, and the password manager appends a newline that would become part of the passphrase.

Restricting the unlock key

Prefixing an authorized_keys entry with command="unlock",no-port-forwarding,no-agent-forwarding,no-pty means the key gives no shell in the initramfs and only passes the passphrase on. The call shrinks to ssh server-unlock without a command, but needs RequestTTY no in the config: a session without a command requests a terminal by default, and unlock, reading until end of input, would wait for Ctrl+D. The restriction takes away diagnostics from the initramfs, so use it for a key meant for automation and keep the administrative key unrestricted.

Operations

Every reboot needs a manual unlock, including the one after a kernel update. The machine won't come back by itself until someone connects to dropbear and provides the passphrase.

The dropbear port is exposed to the internet during the boot window. It runs before any firewall, so no rule protects it. Hence key authentication only.

Updating dracut-crypt-ssh deserves the same care as a kernel update: it's the only package from the testing branch, and its failure cuts off access to the machine.

Updating the kernel

Start from the configuration of the kernel that is actually running, the one known to boot and unlock, not from whatever is in an old source tree:

eselect kernel set N
cd /usr/src/linux
zcat /proc/config.gz > .config
make oldconfig

Then the option check loop from Stage 8, make -jJOBS and make install. Before rebooting, check the new initramfs, using the version from the source tree:

KV=$(make -s kernelrelease)
ls -l /boot/vmlinuz-"$KV" /boot/initramfs-"$KV".img /boot/config-"$KV"
lsinitrd /boot/initramfs-"$KV".img | grep -E 'dropbear|crypttab'
grep -c "$KV" /boot/grub/grub.cfg

Not uname -r. Before the reboot it returns the running kernel's version, so a check based on it would inspect the image that already booted once and wave the new one through, with or without dropbear. The last command must print a number greater than zero, otherwise GRUB has no entry for the new kernel.

If the new sources come from sys-kernel/gentoo-sources, emerge --depclean may later remove the old source tree. That's harmless: the old kernel stays in /boot and in the GRUB menu, and its configuration lives in /proc/config.gz for as long as it runs.

Updating the system

• Take a root snapshot before emerge -uDN @world (see below). It won't roll back the whole system in one step, because the boot always mounts @, but it lets you recover single files from /etc, /usr or /var/db/pkg after a bad configuration merge or --depclean, which are the most likely kinds of damage.
• Read the news before updating, and read every configuration merge. On a machine reachable only over SSH, the dangerous ones are sshd_config and the firewall configuration. Before closing the session after such a change, run sshd -t, restart the service and open a second connection. Only when that works, close the first one or reboot.
• If you keep hand-written nftables rules in /etc/nftables.conf, set SAVE_ON_STOP="no" in /etc/conf.d/nftables and reject every update that proposes the package defaults again. The defaults (SAVE_ON_STOP="yes" with NFTABLES_SAVE pointing at your file) would make the init script dump the live ruleset over your configuration on the next stop, i.e. the next reboot, losing comments, defines and layout. Pointing NFTABLES_SAVE back to /var/lib/nftables/rules-save alone would make the service load a different file than the one you maintain.
• If emerge runs inside tmux and updates tmux itself, the running server keeps the old version while the client on disk is new. tmux ls still works, but tmux attach from another terminal fails with open terminal failed: not a terminal. tmux display -p '#{version}' shows the server's version. Attach with the old binary, which the kernel keeps in memory although it's gone from disk: /proc/$(pgrep -o -u "$USER" tmux)/exe attach. Don't use pgrep -x: tmux renames its processes to tmux: server and tmux: client.

Snapshots

btrfs subvolume snapshot -r / /.snapshots/root-YYYY-MM-DD
btrfs subvolume list -s /
btrfs filesystem usage /
btrfs subvolume delete /.snapshots/root-YYYY-MM-DD

-r makes the snapshot read-only, which is right for a restore point. To roll back, take a new writable snapshot from it and swap the subvolumes, instead of mounting the read-only one directly.

Snapshots reserve no space up front, but they grow as data diverges. A forgotten snapshot holds every block overwritten since it was taken and turns into a slow leak. Delete them at the end of the session in which you made them, once everything has been confirmed to work.

@log and @portage are separate subvolumes so that they don't end up in the root snapshot; nested subvolumes created later, like /srv, don't either.

Periodic checks

btrfs scrub start /
btrfs scrub status /
grep . /sys/devices/system/cpu/vulnerabilities/*
cat /var/log/portage/elog/summary.log

Why it's built this way

Btrfs with subvolumes instead of separate logical volumes. A fixed split between root and data means guessing proportions up front, and every guess is wrong: spare room in one volume is useless when the other fills up. Subvolumes share one pool. In return you lose the damage containment separate volumes give, since one runaway directory can now fill everything; disk usage monitoring is the right control for that, not a wall between volumes. Btrfs adds instant per-subvolume snapshots (per container, too, if every container root is its own subvolume), native LXC support, and data checksums, which matter on storage you don't control.

LVM only for swap. A swap file on Btrfs works, but needs copy-on-write and compression disabled and has to stay out of every snapshot: three conditions, each easily broken by the first root snapshot. A swap volume avoids all of them and leaves room to carve out a raw volume later.

A plain /boot. GRUB can unlock LUKS2, but it computes argon2id in software, which can take minutes, and the passphrase would have to be given twice or a key embedded in the initramfs. A plain /boot means someone with access to the disk could replace the kernel or initramfs and capture the passphrase at the next boot. On a virtual machine, where the provider's hypervisor can read memory anyway, that vector is secondary: the encryption here protects data on decommissioned media, not against the provider itself. GRUB's Btrfs reader is independent of the kernel's, so /boot stays without compression and without subvolumes, and the emulator test in Stage 12 is the cheapest way to find out early if a new btrfs-progs default ever makes it unreadable.

No installer, no emulator for the installation. Gentoo installs with shell commands in a chroot, so there's no need to run the target system in an emulator, and a whole class of problems with environment differences (device names, interface names, MAC addresses) disappears. QEMU comes back only as a test bench for the initramfs, before the first real reboot.