Configuring the firewall
Friday 5th April 2024 7:25 PM

My server lives on a private LAN (with a firewall protecting everything), running only container-based applications, so configuring the firewall might be a bit overkill, but not doing so makes the sys-admin in me feel a bit icky, so configure it, we shall.

The mess that Docker makes of your firewall rules, and how it easy it makes it to do the wrong thing, are well-known and documented by now, so I won't rehash the gory details here. The TL;DR is that you need to add your rules to the DOCKER-USER chain, so I was a little surprised to see that adding a rule to block everything:

sudo iptables -A DOCKER-USER -j DROP

had no effect i.e. I could still access my webapps.

After a bit of digging around[1]Use sudo iptables -S to check what rules have been configured., it became clear that K3s adds a bunch of rules itself, that get processed before the DOCKER-USER chain, so my new rule was never getting processed.

Like Docker, it does its thing in the FORWARD chain, so I created a new chain for my rules, and added that to the start of the FORWARD chain, so that it gets processed before any of the Kubernetes stuff:

sudo iptables -N MY-RULES
sudo iptables -I FORWARD -j MY-RULES
sudo iptables -A MY-RULES -i ens33 -j DROP

And cool, it worked, I could no longer access my webapps :cool:

All I have to do now is insert a rule[2]Note that it's important to use RETURN and not ACCEPT, so that iptables continues processing rules, including the ones K3s has created to handle the traffic., before the blocking one, that allows the traffic I want e.g.

sudo iptables -I MY-RULES -i ens33 \
    -p tcp -m tcp -m multiport --dports 80,443 \
    -j RETURN

And not cool, it didn't work, I still couldn't access my webapps :mad:

Turning on logging, we can see the packet arriving, but it's going to port 8000, not 80!

[core@vm-k3s ~]$ sudo iptables -I MY-RULES -i ens33 -j LOG
[core@vm-k3s ~]$ journalctl -f | grep ens33
Mar 31 10:43:48 vm-k3s kernel: IN=ens33 OUT=cni0 MAC=00:0c:29:eb:11:51:04:7c:16:3b:ce:ac:08:00 SRC=10.2.2.10 DST=10.42.0.7 LEN=40 TOS=0x00 PREC=0x00 TTL=127 ID=61351 DF PROTO=TCP SPT=62627 DPT=8000 WINDOW=0 RES=0x00 ACK RST URGP=0

What's happening is that iptables does NAT processing before it does filtering of packets, and K3s has also created a bunch of rules there that send incoming traffic to the corresponding K3s services.

If we check what ports the Traefik service is listening on:

[core@vm-k3s ~]$  kubectl describe service traefik -n kube-system
Name:                     traefik
Namespace:                kube-system
Labels:                   app.kubernetes.io/instance=traefik-kube-system
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-25.0.2_up25.0.0
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         PreferDualStack
IP Families:              IPv4
IP:                       10.43.63.78
IPs:                      10.43.63.78
LoadBalancer Ingress:     10.2.2.33
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  31389/TCP
Endpoints:                10.42.0.7:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  30449/TCP
Endpoints:                10.42.0.7:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

we can see that it's listening for HTTP(S) traffic on ports 8000 and 8443, so that's where iptables is sending port 80 and 443 traffic.

So, we have to use these different port numbers[3]Because by the time our FORWARD rules get processed, these are the port numbers that will be in the packet. when creating the rule, and our final script looks something like this:

iptables -N MY-RULES
iptables -I FORWARD -j MY-RULES
iptables -A MY-RULES -i ens33 \
    -p tcp -m tcp -m multiport --dports 8000,8443 \
    -j RETURN
iptables -A MY-RULES -i ens33 -j DROP

Running this script (as sudo), everything works fine, and if you remove the 8000 and/or 8443, HTTP(S) traffic is blocked correctly.

All we have to do now is modify the Ignition config to install this script, and create a systemd service to run it when the server starts. Which doesn't work... :mad: :mad: :mad:

The script runs correctly at startup, but this happens before K3s has finished initializing. We add our new rules to the start of the FORWARD chain, and then some time later, K3s adds its own rules, also to the start of the FORWARD chain, which means that they get processed before ours, but ours need to be before theirs to work :-| We could still get things to work by running this script after K3s has finished coming up[4]Maybe there's a "ready" hook for the Traefik service., but this is definitely more trouble than it's worth, and I gave up on this, as well :-|



References

References
↑1 Use sudo iptables -S to check what rules have been configured.
↑2 Note that it's important to use RETURN and not ACCEPT, so that iptables continues processing rules, including the ones K3s has created to handle the traffic.
↑3 Because by the time our FORWARD rules get processed, these are the port numbers that will be in the packet.
↑4 Maybe there's a "ready" hook for the Traefik service.