k3s
K3s lightweight Kubernetes: server flags, service management, node operations, kubeconfig, and upgrade procedures for ARM64 clusters.
SKILL.mdskills/k3s/SKILL.md
GitHubK3s Skill
Cluster Context
Single-node K3s cluster on Raspberry Pi CM4 (ARM64), Ubuntu 24.04.
Node: srv-rk1-nvme-01 at 192.168.178.30
Version pinned in: roles/install-k3s/defaults/main.yml
SSH: ssh dalmine@192.168.178.133
Active Server Flags
--flannel-backend=none # Cilium handles CNI
--disable-network-policy # Cilium handles network policy
--disable-kube-proxy # Cilium handles kube-proxy (BPF)
--disable servicelb # MetalLB handles LoadBalancer
--disable traefik # not used
--disable metrics-server # not used
--disable-cloud-controller # bare metal, no cloud
--write-kubeconfig-mode 644
--token <token>
Note:
--disable local-storagewas previously set but has been removed.local-path-provisioneris re-enabled to provide PVC storage for Pi-hole. See "Local Storage (local-path-provisioner)" section below.
Local Storage (local-path-provisioner)
K3s ships with local-path-provisioner built-in. It was previously disabled
but is required for Pi-hole (PVC for DNS data persistence).
StorageClass: local-path (default after re-enabling)
Verify it is active:
kubectl get storageclass
kubectl get pods -n kube-system -l app=local-path-provisioner
If you need to disable it again (e.g., migrating to a different storage solution):
- Add
--disable local-storageback tok3s_server_execinroles/install-k3s/defaults/main.yml - Uninstall + reinstall K3s (flag changes require reinstall — see "Changing Server Flags")
Changing Server Flags
Flags are set via INSTALL_K3S_EXEC env var at install time — they become
part of the systemd unit /etc/systemd/system/k3s.service.
To change flags after install:
- Edit
k3s_server_execinroles/install-k3s/defaults/main.yml - Uninstall and reinstall:
ansible-playbook playbooks/uninstall.yml && ansible-playbook playbooks/bootstrap.yml— or — manually patch the service file andsystemctl daemon-reload && systemctl restart k3s
Service Management (on the node)
# Status
sudo systemctl status k3s
# Restart
sudo systemctl restart k3s
# Live logs
sudo journalctl -u k3s -f
# Last 100 lines
sudo journalctl -u k3s --no-pager -n 100
# K3s CLI (runs kubectl with embedded kubeconfig)
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A
kubeconfig
K3s writes kubeconfig to /etc/rancher/k3s/k3s.yaml (root-owned).
The get-kubeconfig Ansible role fetches it to ~/.kube/config on localhost,
replacing 127.0.0.1 with 192.168.178.133.
# Re-fetch kubeconfig manually
ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.ini \
--start-at-task "Make sure ~/.kube directory exists"
Upgrade Procedure
- Check latest release: https://github.com/k3s-io/k3s/releases
- Verify compatibility matrix (K3s version ↔ Cilium version)
- Update
k3s_versioninroles/install-k3s/defaults/main.yml - Drain node (single-node: skip or
kubectl cordononly) - Uninstall + reinstall:
ansible-playbook playbooks/uninstall.yml -i inventory/hosts.ini ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.ini - Verify:
kubectl get nodesshows new version
Troubleshooting
Node NotReady after K3s start
- Expected if CNI (Cilium) not yet installed —
NetworkUnavailable=True - Check:
kubectl describe node | grep NetworkUnavailable - Fix: run install-cilium role
K3s crashlooping
sudo journalctl -u k3s -n 50 --no-pager
sudo systemctl status k3s
Common causes:
- Port conflict: another process on 6443
- Disk full:
df -hon node - Bad flags: check
/etc/systemd/system/k3s.serviceExecStart line - Leftover state after failed uninstall: check
/var/lib/rancher,/run/k3s
kubectl permission denied on node
- K3s kubeconfig at
/etc/rancher/k3s/k3s.yamlis root-only by default - Fix:
--write-kubeconfig-mode 644is already in server flags - Or use:
sudo k3s kubectl ...
Add a new agent node
- Add new host to
[agent_nodes]ininventory/hosts.ini - Ensure
k3s_tokenmatches the server token - Run:
ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.ini— agent-only tasks are gated on'agent_nodes' in group_names
Useful Commands
# Verify K3s server flags live
sudo systemctl cat k3s
# Check K3s binary version on node
ssh dalmine@192.168.178.133 'k3s --version'
# All running containers via containerd (K3s embeds containerd)
ssh dalmine@192.168.178.133 'sudo k3s ctr containers list'
# Token (needed for joining agents)
ssh dalmine@192.168.178.133 'sudo cat /var/lib/rancher/k3s/server/node-token'