Home Blag Links Wireguard About

Encrypted remote rootserver

2021-03-13

After figuring out how to remote unlock (initramfs, tinyssh) a homeserver, I'll do the same for a Hetzner-hosted rootserver.

Approach

  • install unencrypted Archlinux (per Hetzner default tools)
  • boot into unencrypted Archlinux
  • install extra/arch-install-scripts
  • run installation process as usual
    • create encrypted LUKS volume
    • create filesystem
    • install base system
  • install all packages that were present in Hetzner's install
  • copy Hetzner configuration
    • copy all configuration (/etc) into encrypted LUKS volume
    • manually resolve any conflicts
  • install and configure initramfs/tinyssh
    • core/mkinitcpio-busybox
    • community/mkinitcpio-netconf
    • community/mkinitcpio-tinyssh
    • community/mkinitcpio-utils
  • configure GRUB
    • add ip settings to GRUB_CMDLINE
    • add encrypted filesystem's details to GRUB_CMDLINE

Preparation

Install Archlinux from Hetzner's rescue system, according to the standard documentation.

I chose three partitions (boot, root and home). This was mostly to use Hetzner's mechanisms to take care of setting up mdraid. Partition boot will be kept (but mostly overwritten), partition root will (only) be used during the installation process and partition home will become the new cryptroot.

Cryptroot installation

Start by adding zsh, grml-zsh-config and tmux for convenience, then switch shell to zsh: chsh -s /bin/zsh

Identify where and how networking got auto-configured by Hetzner's install script, then preserve those settings for later re-use:

root@archlinux ~ # ip addr show
(...)
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 60:a4:4c:24:0d:ab brd ff:ff:ff:ff:ff:ff
    inet 144.76.71.150 peer 144.76.71.129/32 scope global enp4s0
       valid_lft forever preferred_lft forever
    inet6 2a01:4f8:191:8495::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::62a4:4cff:fe24:dab/64 scope link
       valid_lft forever preferred_lft forever
root@archlinux ~ # grep 144.76.71.150 /etc --recursive
/etc/systemd/network/10-enp4s0.network:Address=144.76.71.150
/etc/hosts:144.76.71.150 vetinari.oepkes.net

Some trouble:

root@archlinux ~ # cryptsetup luksFormat /dev/md2
WARNING: Device /dev/md2 already contains a 'ext4' superblock signature.

WARNING!
========
This will overwrite data on /dev/md2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/md2:
Verify passphrase:
Device /dev/md2 is in use. Can not proceed with format operation.

Shoot the trouble (/dev/md was already mounted):

root@archlinux ~ # umount /dev/md2
root@archlinux ~ # cryptsetup luksFormat /dev/md2
WARNING: Device /dev/md2 already contains a 'ext4' superblock signature.

WARNING!
========
This will overwrite data on /dev/md2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/md2:
Verify passphrase:
cryptsetup luksFormat /dev/md2  9.90s user 0.85s system 92% cpu 11.609 total

Open LUKS volume, create btrfs filesystem, mount to /mnt

root@archlinux ~ # cryptsetup open /dev/md2 cryptroot
Enter passphrase for /dev/md2:
cryptsetup open /dev/md2 cryptroot  4.92s user 0.31s system 70% cpu 7.371 total
root@archlinux ~ # mount /dev/mapper/cryptroot /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/cryptroot, missing codepage or helper program
, or other error.
32 root@archlinux ~ # mkfs.btrfs --label root /dev/mapper/cryptroot
btrfs-progs v5.11
See http://btrfs.wiki.kernel.org for more information.

Label:              root
UUID:               cd356fa0-37ee-4ec6-b4c9-462da3bd952e
Node size:          16384
Sector size:        4096
Filesystem size:    2.72TiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Runtime features:
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     2.72TiB  /dev/mapper/cryptroot

root@archlinux ~ # mount /dev/mapper/cryptroot /mnt

root@archlinux ~ # lsblk --fs
NAME            FSTYPE            FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
__sda1          linux_raid_member 1.2   rescue:0 e4550606-7663-74c6-368a-74e058afa4b2
_ __md0         ext3              1.0            53a9ed32-75d1-4697-9e30-01b24427d51e  860.4M     8% /boot
__sda2          linux_raid_member 1.2   rescue:1 c774e188-eae8-ccbc-885d-261077f91210
_ __md1         ext4              1.0            597db7b8-caf1-4967-a7ee-0a896e57ddb1    7.2G    21% /
__sda3          linux_raid_member 1.2   rescue:2 2812faef-fc0a-90d1-11ec-013dbbbba5cc
_ __md2         crypto_LUKS       2              e7417b77-7099-4f2b-8521-0c851a1f50f3
_   __cryptroot btrfs                   root     cd356fa0-37ee-4ec6-b4c9-462da3bd952e    2.7T     0% /mnt
__sda4
sdb
__sdb1          linux_raid_member 1.2   rescue:0 e4550606-7663-74c6-368a-74e058afa4b2
_ __md0         ext3              1.0            53a9ed32-75d1-4697-9e30-01b24427d51e  860.4M     8% /boot
__sdb2          linux_raid_member 1.2   rescue:1 c774e188-eae8-ccbc-885d-261077f91210
_ __md1         ext4              1.0            597db7b8-caf1-4967-a7ee-0a896e57ddb1    7.2G    21% /
__sdb3          linux_raid_member 1.2   rescue:2 2812faef-fc0a-90d1-11ec-013dbbbba5cc
_ __md2         crypto_LUKS       2              e7417b77-7099-4f2b-8521-0c851a1f50f3
_   __cryptroot btrfs                   root     cd356fa0-37ee-4ec6-b4c9-462da3bd952e    2.7T     0% /mnt
__sdb4

Mount boot partition into future chroot:

root@archlinux ~ # mount | grep mnt
/dev/mapper/cryptroot on /mnt type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
root@archlinux ~ # umount /boot
root@archlinux ~ # mkdir /mnt/boot
root@archlinux ~ # mount /dev/md0 /mnt/boot
root@archlinux ~ # mount | grep mnt
/dev/mapper/cryptroot on /mnt type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
/dev/md0 on /mnt/boot type ext3 (rw,relatime)

Initialize system with pacstrap:

root@archlinux ~ # pacstrap /mnt base linux linux-firmware
==> Creating install root at /mnt
==> Installing packages to /mnt
(...)

Store Hetzner's default install's list of packages into future cryptroot:

root@archlinux ~ # pacman -Qq >> /mnt/hetzner-archlinux-packagelist

Now would be a good time to make a btrfs snapshot of the current cryptroot. Install btrfs-progs, create snapshot folder, create snapshot:

root@archlinux ~ # arch-chroot /mnt
[root@archlinux /]# zsh
root@archlinux / # mkdir /.snapshots
root@archlinux / # date "+%F_%H%M%S"
2021-03-13_121927
root@archlinux / # pacman -S btrfs-progs
(...)
pacman -S btrfs-progs  8.86s user 3.75s system 108% cpu 11.653 total
root@archlinux / # btrfs subvolume snapshot / /.snapshots/root_$(date "+%F_%H%M%S")
Create a snapshot of '/' in '/.snapshots/root_2021-03-13_122057'

Install all packages that are present in Hetzner's default install:

root@archlinux / # cat /hetzner-archlinux-packagelist | pacman -S --needed -
warning: acl-2.3.0-1 is up to date -- skipping
warning: archlinux-keyring-20210110-1 is up to date -- skipping
warning: argon2-20190702-3 is up to date -- skipping
(...)
Package (22)                  New Version     Net Change  Download Size

extra/arch-install-scripts    23-2              0.04 MiB       0.01 MiB
core/cronie                   1.5.5-2           0.23 MiB       0.08 MiB
core/dnssec-anchors           20190629-3        0.00 MiB       0.00 MiB
extra/gptfdisk                1.0.7-1           0.79 MiB       0.20 MiB
core/grub                     2:2.04-10        32.94 MiB       6.75 MiB
extra/haveged                 1.9.14-1          0.15 MiB       0.05 MiB
core/ldns                     1.7.1-2           1.85 MiB       0.43 MiB
core/libaio                   0.3.112-2         0.02 MiB       0.01 MiB
core/libedit                  20191231_3.1-3    0.27 MiB       0.11 MiB
core/libinih                  52-2              0.04 MiB       0.01 MiB
core/libnsl                   1.3.0-1           0.19 MiB       0.06 MiB
core/lvm2                     2.03.11-5         6.00 MiB       1.46 MiB
core/mdadm                    4.1-2             0.95 MiB       0.34 MiB
core/net-tools                2.10-1            0.55 MiB       0.14 MiB
core/openssh                  8.5p1-1           5.75 MiB       0.99 MiB
extra/python                  3.9.2-1          79.28 MiB      31.78 MiB
extra/rsync                   3.2.3-3           0.57 MiB       0.30 MiB
core/run-parts                4.8.6.1-2         0.04 MiB       0.03 MiB
core/thin-provisioning-tools  0.9.0-1           1.57 MiB       0.48 MiB
extra/wget                    1.21.1-1          2.99 MiB       0.72 MiB
core/xfsprogs                 5.10.0-2          5.36 MiB       1.05 MiB
community/xxhash              0.8.0-1           0.28 MiB       0.07 MiB

Total Download Size:    45.07 MiB
Total Installed Size:  139.87 MiB

:: Proceed with installation? [Y/n]

Create another snapshot as we'll overwrite a lot of configuration in the next step:

root@archlinux / # btrfs subvolume snapshot / /.snapshots/root_$(date "+%F_%H%M%S")
Create a snapshot of '/' in '/.snapshots/root_2021-03-13_124538'

Check if all worked well with the snapshots:

root@archlinux / # diff /.snapshots/root_2021-03-13_{122057,124538}/etc/pacman.conf
36c36
< #VerbosePkgLists
---
> VerbosePkgLists

Leave arch-chroot, copy all of Hetzner's config into /mnt, fix overwritten /etc/fstab, enter arch-chroot again:

root@archlinux ~ # cp --archive --interactive /etc /mnt
cp: overwrite '/mnt/etc/locale.gen'? y
cp: overwrite '/mnt/etc/inputrc'? y
cp: overwrite '/mnt/etc/mdadm.conf'? y
cp: overwrite '/mnt/etc/mkinitcpio.conf'? y
(...)

Recognize that --interactive is waaaay too annoying and simply overwrite any files instead: cp --archive /etc /mnt. After all, we have the btrfs snapshot if anything goes bad ;-)

Adjust mkinitcpio configuration

Add module btrfs and hooks to /etc/mkinitcpio.conf:

MODULES=(btrfs)
(...)
HOOKS=(base udev autodetect modconf block mdadm_udev lvm2 filesystems keyboard sleep netconf tinyssh encryptssh fsck)

Adjust GRUB configuration

A bit more trouble: translating the Hetzner networking config for use in /etc/default/grub. Hetzner's config doesn't use a normal gateway, but uses a host route to .129. I haven't seen a setup like this before. I'll try to simply set a subnet mask of /32 in the GRUB config and hope it'll work. Another caveat is that in early boot my NIC will still be named eth0 and not enp4s0 which needs to be reflected in GRUB's config. UUID has to be taken from lsblk --fs and cryptroot is the name used by the devicemapper.

(...)
GRUB_CMDLINE_LINUX="ip=144.76.71.150::144.76.71.129:255.255.255.255::eth0:none cryptdevice=UUID=e7417b77-7099-4f2b-8521-0c851a1f50f3:cryptroot"
(...)

After reboot, it turns out that this syntax was exactly right: configure a /32 host route in the GRUB_CMDLINE.

Remote unlock

Contents of my SSH client config:

(...)
# Normal use
Host vetinari.oepkes.net
    RequestTTY yes
    RemoteCommand tmux new-session -As tammo

# If rescue system is required (always autogenerates new SSH keys)
Host vetinari-rescue
    HostKeyAlias vetinari-rescue
    HostName vetinari.oepkes.net
    StrictHostKeyChecking=no
    User root

# For remote unlocking (and tracking of host keys!)
Host vetinari-disk-unlock
    HostName vetinari.oepkes.net
    User root
    HostKeyAlias vetinari-disk-unlock
(...)

This is the footer. Not sure if I'll keep it.

Feed · Xkcd · Webdesign