pihole
Pi-hole DNS server in Kubernetes: cluster.home wildcard DNS, Cilium LB-IPAM LoadBalancer for port 53, local-path PVC storage, and split-DNS for workstations (Fedora Silverblue / macOS).
SKILL.mdskills/pihole/SKILL.md
GitHubPi-hole Skill
Cluster Context
Pi-hole runs inside the K3s cluster as the LAN DNS server.
Its only job: resolve *.cluster.home → 192.168.178.200 (Cilium Gateway IP).
Everything else is forwarded upstream (8.8.8.8 / 8.8.4.4).
Helm release: pihole in namespace pihole
Chart: mojo2600/pihole
Chart version pinned in: roles/install-pihole/defaults/main.yml
Chart version: 2.35.0
Pi-hole app version: 2026.04.0
Cilium LB-IPAM IP: 192.168.178.203 (port 53 UDP+TCP)
Web UI: https://pihole.cluster.home/admin (via HTTPRoute on shared Gateway)
Storage: PersistentVolumeClaim via local-path StorageClass, 500Mi
Architecture
Workstation (DNS: 192.168.178.203)
│
▼
Pi-hole (.203, port 53)
│
├── *.cluster.home → 192.168.178.200 (local override via dnsmasq)
│
└── everything else → 8.8.8.8 (upstream)
│
Cilium Gateway (.200)
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
argocd.cluster.home pihole.cluster.home app.cluster.home
Pi-hole vs cert-manager — No Relationship
Pi-hole and cert-manager are completely independent. They do not communicate.
- Pi-hole resolves DNS names to IPs
- cert-manager issues TLS certificates
- The Gateway uses both: Pi-hole for name resolution (client-side), cert-manager for TLS
Key Helm Values
# roles/install-pihole/defaults/main.yml
pihole_chart_version: "2.35.0"
pihole_namespace: pihole
pihole_dns_ip: "192.168.178.203" # Cilium LB-IPAM IP for DNS
pihole_storage_size: "500Mi"
pihole_storage_class: "local-path" # MUST be local-path — SQLite incompatible with SMB
# Helm values (roles/install-pihole/tasks/main.yml)
# DNS: two separate LoadBalancer services (UDP + TCP), sharing same Cilium LB-IPAM IP
# lbipam sharing-key ensures both services get the same IP
# externalTrafficPolicy must be Cluster — L2 Announcements incompatible with Local
serviceDns:
type: LoadBalancer
annotations:
lbipam.cilium.io/ips: "192.168.178.203"
lbipam.cilium.io/sharing-key: "pihole-dns"
externalTrafficPolicy: Cluster
serviceDnsTCP:
type: LoadBalancer
annotations:
lbipam.cilium.io/ips: "192.168.178.203"
lbipam.cilium.io/sharing-key: "pihole-dns"
externalTrafficPolicy: Cluster
# Web UI: ClusterIP — exposed via HTTPRoute on the shared Gateway
serviceWeb:
type: ClusterIP
http:
enabled: true
port: 80
persistentVolumeClaim:
enabled: true
storageClass: local-path # MUST be local-path — SQLite incompatible with SMB/CIFS
size: 500Mi
dnsmasq:
# Wildcard: *.cluster.home and cluster.home itself → shared Gateway IP
customDnsEntries:
- address=/cluster.home/192.168.178.200
# Pi-hole 6.x (chart v2.35.0+): extraEnvVars is a MAP not a list.
# v2.34.x and older used: [{name: ..., value: ...}] format.
extraEnvVars:
FTLCONF_dns_listeningMode: "ALL"
# Pi-hole 6.x FTL: 30s initial delay (not 120) — subsequent boots from PVC take ~15s.
# failureThreshold: 18 × periodSeconds: 10 = 180s tolerance → 210s total window.
probes:
liveness:
initialDelaySeconds: 30
failureThreshold: 18
periodSeconds: 10
readiness:
initialDelaySeconds: 30
failureThreshold: 18
periodSeconds: 10
# initContainer to remove 0-byte gravity.db before FTL starts (power-cut protection).
# Volume name in chart is 'config' (not 'pihole') — volumeMount must use name: config.
extraInitContainers:
- name: fix-gravity-db
image: busybox
command: [sh, -c, "for f in /etc/pihole/gravity.db ...; do [ -f $f ] && [ ! -s $f ] && rm -f $f; done"]
volumeMounts:
- name: config # chart volume is named 'config' — not 'pihole'
mountPath: /etc/pihole
The address=/cluster.home/192.168.178.200 dnsmasq syntax matches ALL subdomains
of cluster.home (wildcard), not just the apex.
Pi-hole 6.x Gotchas
FTLCONF_dns_listeningMode=ALL (required)
Pi-hole 6.x FTL defaults to listeningMode=LOCAL, which maps to dnsmasq's local-service
option. If you also set local-service in customSettings, dnsmasq crashes with:
CRIT: Duplicate keyword 'local-service' in configuration file
Fix: Do NOT add local-service to customSettings. Instead, override the listening
mode entirely via the env var FTLCONF_dns_listeningMode=ALL. This makes FTL bind port 53
on all interfaces and accept queries from external networks (e.g. workstation at .178.x).
Liveness probe: FTL slow start on first boot
Pi-hole 6.x FTL loads the gravity blocklist database on first boot. On ARM64 with a large blocklist, this can take >110 seconds before port 80 is ready. On subsequent boots with a PVC (data already exists), FTL starts in ~15s.
Fix (applied in role): use short initial delay with high failure threshold:
probes:
liveness:
initialDelaySeconds: 30 # not 120 — only 15s needed on normal reboots
failureThreshold: 18 # 18×10s = 180s tolerance → 210s total window
periodSeconds: 10
Do NOT use initialDelaySeconds: 120 — it wastes 90s on every pod restart.
Workstation DNS
Fedora Silverblue (primary workstation)
Split-DNS via systemd-resolved drop-in. Run on HOST (not toolbox — sudo inside toolbox writes to container fs, not host).
# From infra repo on HOST:
bash scripts/setup-dns-split.sh
Drop-in created: /etc/systemd/resolved.conf.d/cluster-home.conf
[Resolve]
DNS=192.168.178.203
Domains=~cluster.home
This routes only *.cluster.home queries to Pi-hole. All other queries use the default resolver.
Why .203 works and .200 does not
192.168.178.203 is the DNS listener. 192.168.178.200 is only the shared HTTP/HTTPS Gateway.
That means:
dig @192.168.178.203 <name>should answer immediatelycurl https://<name>.cluster.homegoes to.200after DNS resolvesarp -n 192.168.178.200can stay empty if the Gateway has not been announced yet
If DNS times out on .203, the issue is L2/LAN reachability for the DNS VIP, not the wildcard record itself.
macOS
# Set Pi-hole as DNS for active WiFi interface
networksetup -setdnsservers Wi-Fi 192.168.178.203
# Verify
networksetup -getdnsservers Wi-Fi
# Test
dig argocd.cluster.home @192.168.178.203 # → 192.168.178.200
Adding a New Service (Zero Pi-hole Changes Needed)
Because Pi-hole has a wildcard entry (address=/cluster.home/192.168.178.200),
any new <service>.cluster.home hostname resolves automatically without touching Pi-hole.
New service checklist:
- Deploy service with
ClusterIP(no LoadBalancer) - Create
HTTPRoutepointing to Gateway.200with hostname<service>.cluster.home - cert-manager issues cert automatically (wildcard
*.cluster.homecovers it) - Done — DNS, TLS, and routing all work without manual steps
Upgrade Procedure
- Update
pihole_chart_versioninroles/install-pihole/defaults/main.yml - Check release notes: https://github.com/MoJo2600/pihole-kubernetes/releases
- Re-run:
ansible-playbook playbooks/bootstrap.yml -i inventory/hosts.ini \ --start-at-task "Add Pi-hole Helm repository"
Health Checks
# Pi-hole pods (expect 1/1 Running, RESTARTS=0 or <2 from first boot)
kubectl get pods -n pihole
# DNS services and assigned Cilium LB-IPAM IP
kubectl get svc -n pihole
# Test DNS resolution via Pi-hole
dig argocd.cluster.home @192.168.178.203
# Test wildcard
dig anything.cluster.home @192.168.178.203
# Both should return 192.168.178.200
# Pi-hole query logs (last 20)
kubectl logs -n pihole -l app=pihole --tail=20
Troubleshooting
Pi-hole pod CrashLoopBackOff (first boot)
kubectl describe pod -n pihole -l app=pihole | grep -E 'initialDelay|failureThreshold|Events'
→ If initialDelaySeconds is 60 and pod keeps restarting, the probe config is missing.
→ Fix: ensure probes block is in the Helm values (added in role since Pi-hole 6.x).
Pi-hole not responding on port 53
# Check services and Cilium LB-IPAM IP assignment
kubectl get svc pihole-dns-tcp pihole-dns-udp -n pihole
# Check Cilium L2 announcement
kubectl get l2announcement -A
kubectl get svc pihole-dns-tcp pihole-dns-udp -n pihole -o wide
# Test from node directly
ssh dalmine@192.168.178.133 'dig argocd.cluster.home @192.168.178.203'
If the pod answers queries inside the cluster but the workstation times out, focus on
arp -n 192.168.178.203, the cilium-l2announce-pihole-pihole-dns lease, and the
host-side neighbor cache, not on Pi-hole's DNS records.
DNS not resolving from workstation
# Fedora Silverblue: check resolved config
resolvectl status
# Should show Pi-hole for cluster.home domain
# macOS: check WiFi DNS
networksetup -getdnsservers Wi-Fi
# Test directly (bypass system DNS)
dig cluster.home @192.168.178.203
Host can reach cluster but not Pi-hole VIP
ip neigh show 192.168.178.203
arp -n 192.168.178.203
sudo ip neigh flush to 192.168.178.203
dig +short argocd.cluster.home @192.168.178.203
If Pi-hole works from inside the cluster but the laptop times out, check:
- The VIP is announced on the right interface family (
en*,wl*,end*,eno*,ens*,enp*,enx*) - The laptop has a stale ARP/neighbor cache entry
systemd-resolvedor NetworkManager is still using another DNS path
Pi-hole answers inside cluster but laptop times out
# Check LAN reachability and neighbor cache first
ip neigh show 192.168.178.203
arp -n 192.168.178.203
# Clear stale ARP if needed
sudo ip neigh flush to 192.168.178.203
# Test raw DNS over UDP to the VIP
dig +short argocd.cluster.home @192.168.178.203
If the service answers from inside the cluster but the laptop times out, the usual causes are:
- the Pi-hole VIP is not being announced on the correct host interface
- the host has a stale neighbor entry for
192.168.178.203 - the workstation DNS path is pointed elsewhere, even if Pi-hole itself is healthy
Troubleshoot in this order:
ip neigh show 192.168.178.203arp -n 192.168.178.203dig @192.168.178.203 argocd.cluster.homekubectl -n kube-system get lease | grep cilium-l2announcekubectl get svc pihole-dns-tcp pihole-dns-udp -n pihole -o wide
SQLite corruption / DNS stops working (CRITICAL)
Root cause: Pi-hole's PVC was set to smb-nas → SQLite file locking is incompatible
with SMB/CIFS network filesystems. Symptoms: SQLITE_MISUSE, queries_to_database() failed!,
file unlinked while open, DNS not responding.
Fix: pihole_storage_class MUST be local-path. Never use smb-nas for Pi-hole.
To recover: delete release + PVC, reinstall:
helm uninstall pihole -n pihole
kubectl delete pvc --all -n pihole
kubectl delete namespace pihole
make dns-metrics
PVC not binding (Pending)
kubectl get pvc -n pihole
kubectl describe pvc -n pihole
kubectl get storageclass # must show local-path
Custom DNS entries not working after Pi-hole restart
- Pi-hole persists custom DNS in its SQLite DB via PVC — should survive restarts
- If entries are lost: PVC data was corrupted or PVC was deleted
- Re-run
install-piholerole to reapplycustomDnsEntriesvia Helm values
Useful Commands
# All Pi-hole resources
kubectl get all -n pihole
# Helm values
helm get values pihole -n pihole
# Enter Pi-hole container (for pihole-FTL commands)
kubectl exec -it -n pihole deploy/pihole -- pihole status
kubectl exec -it -n pihole deploy/pihole -- pihole -q argocd.cluster.home
# Check dnsmasq custom config
kubectl exec -it -n pihole deploy/pihole -- cat /etc/dnsmasq.d/02-custom.conf