We can now install K3s onto our server. The way everyone says to do this is like this:

curl -sfL https://get.k3s.io | sh -

but I truly despise this "box of chocolates" approach, so I'll be doing an "air-gapped install"^[1]So that I know exactly what will be installed.

Preparing for the installation

K3s provide air-gapped install images, so get the one appropriate for your architecture, plus the corresponding k3s binary.

You will also need to get the installer script e.g.

curl -o install.sh https://get.k3s.io

Since CoreOS uses SELinux, you will also need the k3s-selinux RPM.

We will be modifying the Ignition script to install K3s every time the system is provisioned, which means that we need to put these files on the data disk^[2]So that they will be available to Ignition on first boot, and don't get blown away the next time the system is provisioned.. So, we need to mount the data disk manually into the pre-installation environment e.g.

sudo mkdir /mnt/data
sudo mount /dev/mapper/vg--data-lv--data /mnt/data

and then use the Python web server trick^[3]Or you can use an NFS mount, if you have one available. to transfer the files in e.g.

sudo mkdir -p /mnt/data/provisioning/k3s-v1.29.2+k3s1
cd /mnt/data/provisioning/k3s-v1.29.2+k3s1
sudo curl -O http://10.2.2.10:8000/k3s-airgap-images-amd64.tar.gz
sudo curl -O http://10.2.2.10:8000/k3s
sudo curl -O http://10.2.2.10:8000/k3s-selinux-1.5-1.coreos.noarch.rpm
sudo curl -O http://10.2.2.10:8000/install.sh
sudo chmod +x k3s install.sh

Installing K3s

Since we want Ignition to automatically install K3s for us when the system is provisioned, we need to write a script.

Note that the installer files we downloaded above are on the data disk, which we will mount at /mnt/data/^[4]Which also appears at /var/mnt/data, due to how CoreOS maps the file system.. For now, we'll mount the drive manually, and then later do it automatically.

First up, we install SELinux support:

PROVISIONING=/var/mnt/data/provisioning/k3s-v1.29.2+k3s1/
rpm-ostree --apply-live -C -y install "$PROVISIONING/k3s-selinux-1.5-1.coreos.noarch.rpm"

We then prepare things for the K3s installer:

cp "$PROVISIONING/k3s" /usr/local/bin/
chmod +x /usr/local/bin/k3s
mkdir -p /var/lib/rancher/k3s/agent/images/
cp "$PROVISIONING/k3s-airgap-images-amd64.tar.gz" /var/lib/rancher/k3s/agent/images/

We can now run the K3s installer:

export K3S_KUBECONFIG_MODE="644"
export INSTALL_K3S_EXEC="--docker"
export INSTALL_K3S_SKIP_DOWNLOAD=true
"$PROVISIONING/install.sh"

Note that K3s uses Podman by default, but because I have some legacy stuff, I configure it to use Docker.

Moving the K3s data files

This is the tricky bit. We want to move the K3s data files off the main file system and onto our data disk, so that it will be preserved even if the system is re-provisioned. K3s does support specifying a data directory, but it was something added late in the game, and seems to be perhaps a little flaky...

So instead, we let K3s create its data files the first time it is run, then move them to the data drive, and then symlink them back to their original locations.

First, we stop K3s:

systemctl stop k3s
/usr/local/bin/k3s-killall.sh

Then we move the data files, and create symlinks to them:

K3S_DATA=/mnt/data/k3s-data/
function move_k3s_data {
    local src=$1
    dest=$( echo "$src" | tr '/' '-' )
    dest="$K3S_DATA/${dest:1}"
    if [ -d "$dest" ]; then
        echo "Already moved: $dest"
        rm -rf "$src"
        # nb: so that we can create the symlink
        mkdir -p "$(dirname "$src")"
    elif [ -d "$src" ]; then
        echo "Moving k3s data: $src -> $dest"
        mv "$src" "$dest"
    else
        echo "k3s data not present: $src"
        mkdir "$dest"
        # nb: so that we can create the symlink
        mkdir -p "$(dirname "$src")"
    fi
    ln -s "$dest" "$src"
}
mkdir -p "$K3S_DATA"
move_k3s_data /var/lib/rancher
move_k3s_data /var/lib/kubelet/pods
move_k3s_data /run/k3s

We apply^[5]It's worth noting that while one would normally manage SELinux labels using semanage fcontext and restorecon, this is a little pointless in CoreOS, since it won't survive re-provisioning. That, plus … Continue reading SELinux labels to data directories that need them:

dname="$K3S_DATA/var-lib-rancher/k3s/storage"
mkdir -p "$dname"
chcon --type container_file_t --recursive "$dname"

Finally, we restart K3s:

systemctl start k3s

Moving the Docker data files

We also want all our Docker files to be on the data disk. This is much easier - all we have to do is get Ignition to set up a Docker config file that points to the new data directory, which will be created the first time Docker is run:

storage:
  files:
    - path: /etc/docker/daemon.json
      contents:
        local: docker-daemon.json

where docker-daemon.json looks like this:

{
    "data-root": "/mnt/data/docker-data"
}

However, if you try to use it, you may get errors of the form:

    open GetImageBlob: no such file or directory

The solution is to restart the service, but since this directory persists across system rebuilds, it's very much a one-time problem.

The Docker data files also need SELinux labels applied:

DOCKER_DATA=/mnt/data/docker-data/
dname="$DOCKER_DATA/overlay2"
mkdir -p "$dname"
chcon --type container_share_t --recursive "$dname"

Note that since we're using Docker, it's worth adding your user to the docker group, so that they can run Docker commands without sudo:

passwd:
  users:
    - name: core
      password_hash: $y$j9T$DUQ4JpHLE86HQpQemBNs3.$W2K2ymcyTaADWJfrZGbBnYFndwfYGtx6fs0lhzQJtu5
      groups:
      - docker

Running the installer script

Everything is now in place, and we'll start off by configuring Ignition to install our script into the file system, but we'll run it manually, to make sure everything's OK.

Download the full installer script, and add the following to your Butane script to provision it:

storage:
  files:
    - path: /usr/local/bin/install-k3s.sh
      mode: 0755
      contents:
        local: install-k3s.sh

Reoot from the Live DVD, re-provision your system^[6]The full Butane script is here., remount the data drive, and manually run install-k3s.sh (using sudo).

Note that you will get an error at the very end of the script, complaining about not being able to disable install-k3s.service - this will go away in the next section.

K3s should be installed and running, with its data files on the secondary data disk:

[core@vm-k3s ~]$ kubectl get pods -A

NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   local-path-provisioner-6c86858495-mlv9v   1/1     Running             0          115s
kube-system   traefik-f4564c4f4-hr5v6                   0/1     ContainerCreating   0          11s
kube-system   svclb-traefik-324fb914-w2q82              0/2     ContainerCreating   0          10s
kube-system   helm-install-traefik-crd-49bqs            0/1     Completed           0          115s
kube-system   helm-install-traefik-8qklq                0/1     Completed           1          115s
kube-system   coredns-6799fbcd5-r4rsg                   1/1     Running             0          115s
kube-system   metrics-server-67c658944b-tkr6h           1/1     Running             0          115s

[core@vm-k3s ~]$ ls -l /var/lib/rancher
lrwxrwxrwx. 1 root root 35 Mar 28 06:27 /var/lib/rancher -> /mnt/data/k3s-data//var-lib-rancher

[core@vm-k3s ~]$ ls -l /var/lib/kubelet/pods
lrwxrwxrwx. 1 root root 40 Mar 28 06:27 /var/lib/kubelet/pods -> /mnt/data/k3s-data//var-lib-kubelet-pods

[core@vm-k3s ~]$ ls -l /run/k3s
lrwxrwxrwx. 1 root root 27 Mar 28 06:27 /run/k3s -> /mnt/data/k3s-data//run-k3s

Running the installer script automatically

Now the K3s install script is working, we can get Ignition to run it automatically when the system is being provisioned.

Since the data disk needs to be mounted for the install script to work, we update the Ignition script to create a systemd service^[7]I originally tried to do this using storage.filesystems, but it didn't work, since the LVM mapper wasn't there, I imagine because it runs after Ignition does. to do that:

systemd:
  units:
  - name: mount-data-volume.service
    enabled: true
    contents_local: mount-data-volume.service

where mount-data-volume.service is a file^[8]In the files/ sub-directory, since that's where the --files-dir parameter points to when we run the butane command. that looks like this:

[Unit]
Description = Mount the external data volume
OnFailure = emergency.target
OnFailureJobMode = replace-irreversibly

[Service]
Type = oneshot
RemainAfterExit = yes
ExecStartPre = mkdir -p /mnt/data
ExecStart = mount /dev/mapper/vg--data-lv--data /mnt/data
StandardOutput = kmsg+console
StandardError = kmsg+console

[Install]
WantedBy = multi-user.target

We can now create another systemd service that runs the install script:

systemd:
  units:
  - name: install-k3s.service
    enabled: true
    contents_local: install-k3s.service

where install-k3s.service looks like this:

[Unit]
Description = Install K3s
Requires = mount-data-volume.service
After = mount-data-volume.service
Before = systemd-user-sessions.service
OnFailure = emergency.target
OnFailureJobMode = replace-irreversibly

[Service]
Type = oneshot
RemainAfterExit = yes
ExecStart = /usr/local/bin/install-k3s.sh
StandardOutput = kmsg+console
StandardError = kmsg+console

[Install]
WantedBy = multi-user.target

When the system first boots, systemd will run the install-k3s.sh script, and if it finishes successfully, it will disable the service, so that it doesn't run again on subsequent reboots.

Go back to your snapshot of a clean machine, re-provision again^[9]The full Butane script is here., and you should see the install script being run automatically during the startup sequence.

References[+]

References

↑1	So that I know exactly what will be installed.
↑2	So that they will be available to Ignition on first boot, and don't get blown away the next time the system is provisioned.
↑3	Or you can use an NFS mount, if you have one available.
↑4	Which also appears at /var/mnt/data, due to how CoreOS maps the file system.
↑5	It's worth noting that while one would normally manage SELinux labels using semanage fcontext and restorecon, this is a little pointless in CoreOS, since it won't survive re-provisioning. That, plus semanage and restorecon are not installed in CoreOS In this particular case, since the data directories live on the external data disk, and will survive a re-provisioning, it only needs to be done once, but we do it here so that it will happen automatically.
↑6	The full Butane script is here.
↑7	I originally tried to do this using storage.filesystems, but it didn't work, since the LVM mapper wasn't there, I imagine because it runs after Ignition does.
↑8	In the files/ sub-directory, since that's where the --files-dir parameter points to when we run the butane command.
↑9	The full Butane script is here.