Awasu » Provisioning dynamic storage
Friday 4th March 2022 9:09 PM

Servers as cattle[1]In other words, servers that you're happy to kill off. is fine in theory, but it introduces the need for persistent storage i.e. we need some way to store stuff on disk (e.g. in a database, or files for a website), that will always be there, even as the servers come and go.

Kubernetes lets applications request persistent disk space, and you need to set up at least one storage class to provision these requests. While it's possible to set up a no-provisioner storage class that provisions space on the local file system, the volume won't be available if the node goes down, and this storage class also doesn't support dynamic provisioning.

Clusters will typically use space in the cloud, but since we want to keep everything local, we'll set up a storage class that provisions space on an NFS mount. The underlying file system would normally be kept on a server outside the cluster, but our control plane VM is not doing much, so we'll just put it there :-)

Set up NFS

On the vm-kcontrol server, first enable NFS, and prepare the directory that will be used to hold the shared data:

sudo systemctl start nfs-server
sudo systemctl enable nfs-server

sudo mkdir -p /srv/nfs/kdata
sudo chmod -R 777 /srv/nfs/ # don't do this in a production environment!

To export the directory, add the following line to /etc/exports:

/srv/nfs/kdata  *(rw,sync,no_subtree_check,no_root_squash,insecure)

and run the following command:

sudo exportfs -rv

You can check that the directory is now available for mounting by running showmount -e.

Finally, open up the firewall:

sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --reload

Set up the storage class provisioner

Next, we'll install a storage class provisioner that works with NFS. We'll do that using Helm, so we need to install that first.

On the machine you're managing Kubernetes from[2]In other words, not necessarily the vm-kcontrol server (although it can be)., download and unpack the Helm binary. Then install nfs-subdir-external-provisioner:

helm repo add nfs-subdir-external-provisioner \

helm install nfs-subdir-external-provisioner \
    nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
    --set nfs.server=vm-kcontrol \
    --set nfs.path=/srv/nfs/kdata \
    --set storageClass.onDelete=delete

Note that when volume directories are no longer required, the default behavior is to archive them, which is not really necessary in a test environment, so we configure them to be deleted instead.

Finally, make it the default provisioner:

kubectl patch storageclass nfs-client -p \
    '{"metadata": {"annotations":{"":"true"}}}'

You can confirm everything's been set up by running kubectl get storageclass.

Test the storage class provisioner

We'll test things by creating a persistent volume with an HTML file in it, then serve that file using nginx. We should be able to run an nginx pod on any node, or even reboot the entire cluster, and that file will continue to be served.

First, we need to create the volume by creating a persistent volume claim:

cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
      name: nginx-nfs-test
        - ReadOnlyMany
      storageClassName: nfs-client
          storage: 10Mi

We can confirm that the volume has been created by running kubectl get pvc, and if you check the /srv/nfs/kdata/ directory on vm-kcontrol, you'll see that a new sub-directory has been created for the volume.

Inside that directory, create a file called index.html and put some content in it (this is the file that nginx will serve).

Next, we want to create an nginx deployment that mounts this newly-created volume at /usr/share/nginx/html/, so that when we connect to nginx, we will see the index.html we just created:

cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
        app: nginx-nfs-test
      name: nginx-nfs-test
      replicas: 2
          app: nginx-nfs-test
            app: nginx-nfs-test
          - name: nginx-nfs-test
              claimName: nginx-nfs-test
          - image: nginx
            name: nginx
            - name: nginx-nfs-test
              mountPath: /usr/share/nginx/html
              readOnly: true

Finally, we create a service to make nginx available to the outside:

cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Service
      name: nginx-nfs-test
      type: NodePort
        app: nginx-nfs-test
        - protocol: TCP
          port: 80
          nodePort: 30333

If you open vm-knode1:30333 in a browser, you should see the index.html file you created before. Even if you reboot the nodes, this will still be case, thus demonstrating that the storage is persistent, even after the pods have been deleted and re-created.


1 In other words, servers that you're happy to kill off.
2 In other words, not necessarily the vm-kcontrol server (although it can be).
Have your say