Managing K3s from another machine

If you want to manage K3s from another machine^[1]For example, by running kubectl or Octant., you will need credentials. Grab the file /etc/rancher/k3s/k3s.yaml, update it with the correct server address, and save it as ~/.kube/config on the remote client.

Updating CoreOS

Manual updates used to be triggered by running the update_engine_client command, but this is no more, and it looks like everything is now managed by Zincati.

I got things going by by editing the file we put in /etc/zincati/config.d/ to disable auto-updates, and re-enabling them. Restart the service, and then follow its log to see what's happening:

[core@vm-k3s ~]$ sudo systemctl restart zincati

[core@vm-k3s ~]$ sudo journalctl -f -xeu zincati
Mar 31 10:17:40 vm-k3s systemd[1]: Starting zincati.service - Zincati Update Agent...
  Subject: A start job for unit zincati.service has begun execution
  Defined-By: systemd
  Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
  
  A start job for unit zincati.service has begun execution.
  
  The job identifier is 3515.
Mar 31 10:17:40 vm-k3s zincati[33247]: [INFO  zincati::cli::agent] starting update agent (zincati 0.0.27)
Mar 31 10:17:40 vm-k3s zincati[33247]: [INFO  zincati::cincinnati] Cincinnati service: https://updates.coreos.fedoraproject.org
Mar 31 10:17:40 vm-k3s zincati[33247]: [INFO  zincati::cli::agent] agent running on node '75681f1d1b1d434ebfadb0bc5ae3b327', in update group 'default'
Mar 31 10:17:40 vm-k3s zincati[33247]: [INFO  zincati::update_agent::actor] registering as the update driver for rpm-ostree
Mar 31 10:17:41 vm-k3s zincati[33247]: [INFO  zincati::update_agent::actor] initialization complete, auto-updates logic enabled
Mar 31 10:17:41 vm-k3s zincati[33247]: [INFO  zincati::strategy] update strategy: immediate
Mar 31 10:17:41 vm-k3s zincati[33247]: [INFO  zincati::update_agent::actor] reached steady state, periodically polling for updates
Mar 31 10:17:41 vm-k3s systemd[1]: Started zincati.service - Zincati Update Agent.
  Subject: A start job for unit zincati.service has finished successfully
  Defined-By: systemd
  Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
  
  A start job for unit zincati.service has finished successfully.
  
  The job identifier is 3515.
Mar 31 10:17:43 vm-k3s zincati[33247]: [INFO  zincati::cincinnati] current release detected as not a dead-end
Mar 31 10:17:44 vm-k3s zincati[33247]: [INFO  zincati::update_agent::actor] target release '39.20240309.3.0' selected, proceeding to stage it

Increasing file descriptors for Sonatype Nexus

When using Docker with Sonatype Nexus, the OS needs to be configured with a lot more file descriptors than normal. While this can be configured in /etc/docker/daemon.json (which we create), it is also configured by default in /etc/sysconfig/docker, and it can't be in both

To work around this, we create a service in the Ignition config that increases the number of descriptors when the system comes up:

systemd:
  units:
  - name: increase-docker-file-descriptors.service
    enabled: true
    contents: |
      [Unit]
      Description = Increase Docker's file description limit.
      Before = docker.service
      [Service]
      Type = oneshot
      RemainAfterExit = yes
      ExecStart = sed --in-place -e 's/--default-ulimit nofile=1024:1024/--default-ulimit nofile=1024:65536/' /etc/sysconfig/docker
      StandardOutput = kmsg+console
      StandardError = kmsg+console
      [Install]
      WantedBy = multi-user.target

Slow uploads

I found that uploading files was incredibly slow. Others have also reported this, and AFAICT, there's no work-around However, here's the weird thing: it's only an issue when doing it from a browser on the same machine!?! Doing an upload using curl or Python is fine, but via a webapp is extremely slow. Disabling my anti-virus (Windows Defender) didn't make a difference, doing it from a Linux VM on the same machine sorta worked^[2]It was much slower, and a bit flaky., so I'm baffled as to what could be causing this.

If the file is small, you could just wait, but uploading files to Sonatype Nexus was problematic, since my files are large, and their webapp is constantly pinging the server, and gets confused because it thinks that the server has gone away. However, if you can't upload from another machine, you can do it manually from a console:

curl --insecure --user {USERNAME}:{PASSWORD} \
    --upload-file ... \
    https://nexus3.k3s/repository/{REPO}/...