Cluster Health Check

MANDATORY: Run the script first

When this skill is invoked, your first action must be to run the cluster health check script and reason over its output before doing anything else. Do not improvise individual kubectl calls — the script is the authoritative surface.

cd /home/wizard/code
bash infra/scripts/cluster_healthcheck.sh --json | tee /tmp/cluster-health.json

If the session is rooted elsewhere, fall back to the absolute path:

bash /home/wizard/code/infra/scripts/cluster_healthcheck.sh --json

Then:

1. Parse the JSON. Report the PASS/WARN/FAIL counts + overall verdict.
2. Iterate every FAIL and WARN check, describe what tripped, and propose the remediation path (use the recipes below).
3. Only reach for ad-hoc kubectl commands when investigating a specific failure beyond what the script reported.

Exit codes: 0 = healthy, 1 = warnings only, 2 = failures.

Quick flags

# Human-readable report (default), no auto-fix
bash infra/scripts/cluster_healthcheck.sh

# Machine-readable JSON summary
bash infra/scripts/cluster_healthcheck.sh --json

# Only show WARN + FAIL (suppress PASS noise)
bash infra/scripts/cluster_healthcheck.sh --quiet

# Enable auto-fix (delete evicted pods, kick stuck CrashLoop pods)
bash infra/scripts/cluster_healthcheck.sh --fix

# Combined: quiet JSON without auto-fix
bash infra/scripts/cluster_healthcheck.sh --no-fix --quiet --json

# Custom kubeconfig
bash infra/scripts/cluster_healthcheck.sh --kubeconfig /path/to/config

What It Checks (50 checks)

#	Check	Notes
1	Node Status	NotReady nodes, version drift
2	Node Resources	CPU/mem >80% (warn) / >90% (fail)
3	Node Conditions	MemoryPressure / DiskPressure / PIDPressure
4	Problematic Pods	CrashLoopBackOff / Error / ImagePullBackOff
5	Evicted/Failed Pods	status.phase=Failed
6	DaemonSets	desired == ready
7	Deployments	ready == desired replicas
8	PVC Status	all Bound, plus free space: WARN ≥90% full (the pvc-autoresizer's own 10%-free trigger — still being there means the PVC is unmanaged or has hit its storage_limit ceiling), FAIL ≥97%. Usage from kubelet_volume_stats_* via Prometheus. Coverage caveat: kubelet only exports stats for volumes mounted by a RUNNING pod (~124 of 159 today), so this catches a volume filling but goes blind once its pod is already crashlooping — the unmonitored count is reported, never silently skipped. Added 2026-08-05: Bound alone reported "All Bound" for 27.6h while a 99.91%-full 5Gi PVC held the Technitium DNS primary in CrashLoopBackOff
9	HPA Health	targets not <unknown>, utilization <100%
10	CronJob Failures	job conditions Failed=True in last 24h
11	CrowdSec Agents	all pods Running
12	Ingress Routes	every ingress has an LB IP + Traefik LB
13	Firing Alerts	active warning + critical alerts from Alertmanager, which covers both Prometheus rules and the Loki ruler (silenced, inhibited and info alerts excluded). JSON key stays prometheus_alerts. If Alertmanager cannot be reached it falls back to Prometheus and says so in the detail, since that view misses every Loki-ruler alert. Until 2026-09-27 the Alertmanager call named a Deployment that does not exist (it is a StatefulSet), so the check always used the fallback, and a critical DevvmJournalSilent fired for 14 hours without appearing here
14	Uptime Kuma Monitors	internal + external monitors up
15	ResourceQuota Pressure	any quota >80% used
16	StatefulSets	ready == desired
17	Node Disk Usage	ephemeral-storage <80%
18	Helm Release Health	all deployed (no pending-*)
19	Kyverno Policy Engine	all pods Running
20	NFS Connectivity	192.168.1.127 showmount / port 2049
21	DNS Resolution	Answers "is DNS serving?" — not "is the primary healthy?" (that's check 7). Execs from the first Ready pod behind the dns-server=true selector (primary, secondary or tertiary) and resolves internal + external against the service ClusterIP 10.96.0.53. Rewritten 2026-08-05: it used to take items[0] of -l app=technitium (primary only) and query 127.0.0.1, so any non-serving primary false-FAILed the check while secondary + tertiary served every client fine — twice (2026-06-24 eviction storm, 2026-08-04 crashloop). Note a CrashLoopBackOff pod still reports status.phase=Running, so selection must test the Ready condition, not the phase
22	TLS Certificate Expiry	TLS Secret certs >30d valid
23	GPU Health	nvidia namespace + device-plugin Running
24	Cloudflare Tunnel	pods Running
25	Resource Usage	node CPU/mem headroom
26	HA Sofia — Entity Availability	Home Assistant unavailable/unknown count
27	HA Sofia — Integration Health	config entries setup_error / not_loaded
28	HA Sofia — Automation Status	disabled / stale (>30d) automations
29	HA Sofia — System Resources	HA CPU / mem / disk
30	Hardware Exporters	snmp / idrac-redfish / proxmox / tuya pods + scrapes
31	cert-manager — Certificate Readiness	Certificate CRs with Ready!=True
32	cert-manager — Certificate Expiry (<14d)	notAfter within 14d
33	cert-manager — Failed CertificateRequests	Ready=False, reason=Failed
34	Backup Freshness — Per-DB Dumps	MySQL + PG dumps within 25h
35	Backup Freshness — Offsite Sync	Pushgateway backup_last_success_timestamp <27h
36	Backup Freshness — LVM PVC Snapshots	newest thin snapshot <25h (SSH PVE)
37	Monitoring — Prometheus + Alertmanager	/-/ready + AM pods Running
38	Monitoring — Vault Sealed Status	vault status reports Sealed: false
39	Monitoring — ClusterSecretStore Ready	vault-kv + vault-database Ready
40	External — Cloudflared + Authentik Replicas	deployments fully ready
41	External — ExternalAccessDivergence Alert	alert not firing
42	External — Traefik 5xx Rate (15m)	top-10 services emitting 5xx
43	PVE Host Thermals	package + per-core temps via /sys/class/hwmon (SSH). Baseline 55-65 °C. PASS <65 °C, WARN 65-82 °C (a VM is burning too much CPU), FAIL ≥83 °C (TjMax)
44	PVE Host Load	/proc/loadavg via SSH. PASS 5m <30, WARN 30-37, FAIL ≥38 of 44 threads
45	HA Sofia — Status Dashboard	emo's curated Барзини → Статус view (dashboard-barzini / path status). Pulls the lovelace config via WS, batch-renders every custom:mushroom-template-card secondary template against /api/template, classifies each rendered line: FAIL on Offline / Disconnected / Разкачен / — No data; WARN on ⚠️ / Abnormal / Trouble ( / (ниска) / Пълен резервоар / Грешка / attention / Внимание. Verdict rolls up across the 8 sections (Сигурност, Мрежа & IT, Енергия, Климат, Уреди, Мултимедия, Осветление, Поливна)
46	Immich Smart Search	clip_index residency in PG shared_buffers + representative ANN probe latency (in immich-postgresql). FAIL >1.5s or <50% resident; WARN >0.5s or <90% resident. Cold cache → check the immich-search-probe CronJob (it runs the prewarm on :00/:30 ticks since 2026-08-16, when clip-index-prewarm + clip-keepalive were merged into it)
47	Proxmox CSI — Ghost-Disk + Hotplug-Wedge Drift	Per node, FOUR-way diff: CSI disks in qm config <vmid> (SSH PVE) vs the LIVE QEMU runtime (info block via qm monitor) vs QEMU's throttle-drive-scsiN objects (qom-list /objects) vs attached proxmox-CSI VolumeAttachments k8s tracks. Catches (a) orphaned "ghost" disks left by failed detaches (query-pci QMP timeouts) the scheduler's 28-LUN guard can't see, (b) WEDGED HOTPLUG — config-present but runtime-absent disks (the 2026-07-12 n8n/node5 class): every new mount on that node fails "device not found" until the VM reboots, and (c) LEAKED THROTTLE OBJECT — a throttle-drive-scsiN whose scsiN is absent from the config, i.e. a POISONED LUN SLOT: QEMU rejects every future attach there with object-add failed - attempt to add duplicate property, and proxmox-csi swallows that 400 and reports success (the 2026-08-04 stirling-pdf / node3-scsi2 class — invisible to (a) and (b), which all agreed). PASS all four match; WARN ghosts>0 or real 20-24; FAIL wedged>0, leaked>0, or real ≥25. Cleanup: ghosts → qm set <vmid> --delete scsiN (frees slot, retains LV); wedged → drain + reboot the VM (interim: cordon → delete stuck pod → reschedules elsewhere → uncordon); leaked → drive_del drive-scsiN then object_del throttle-drive-scsiN via qm monitor (data-safe, no reboot; object_del alone fails "in use")
48	Goldmane Edge-Aggregator	goldmane-edge-aggregator Deployment Available condition (the who-talks-to-whom edge trail, ADR-0014). Missing Deployment or not-Available → FAIL (mirrors the AggregatorDown Prometheus alert; the pod has no /metrics to scrape)
49	Slack — #alerts Recent Alerts	Reads the #alerts channel via the Slack Web API (bot token from Vault secret/viktor/slack_bot_token, env override SLACK_BOT_TOKEN; window 2h, HEALTHCHECK_SLACK_WINDOW_HOURS to override). Classifies messages into Alertmanager alert events ([CRITICAL]/[WARNING]/[INFO]/[SECURITY/*] firing vs [RESOLVED]; [INFO] disambiguated by colour) vs other traffic (CI notices, digests, diun, KMS). WARN if any alert is net-FIRING (no later RESOLVED in the window — names listed) or >20 messages in the window (noisy); PASS otherwise. Token missing / API unreachable → WARN + skip
50	Proxmox CSI — Failed Attach Tasks	Reads the PVE task log (pvesh get /nodes/pve/tasks) for qmconfig tasks run by csi@pve!csi-token that did NOT return OK, in the last 24h. This is the ground truth proxmox-csi discards: it logs ControllerPublishVolume: volume published and k8s fires SuccessfulAttachVolume even when Proxmox returned HTTP 400, so a permanent host-side attach failure otherwise shows up only as a pod stuck in ContainerCreating on "device … not found". PASS none; WARN failures in the window; FAIL any within the last hour (an attach is wedged right now and will not self-recover). A duplicate property throttle-drive-scsiN message = poisoned LUN slot → see check 47

Safe Auto-Fix Rules

--fix only performs operations that are genuinely reversible and observable. Nothing here rewrites Terraform state or mutates the cluster beyond "delete pod".

Done automatically by --fix

• Evicted / Failed pods — delete them; the controller recreates.

  kubectl delete pods -A --field-selector=status.phase=Failed
  

• CrashLoopBackOff pods with >10 restarts — delete once to reset backoff timer.

NEVER auto-fix (requires human investigation)

• NotReady nodes
• MemoryPressure / DiskPressure / PIDPressure
• ImagePullBackOff (usually a bad tag / registry credential)
• Deployment ready-replica mismatch
• Pending PVCs
• Node CPU/memory >90%
• CronJob failures
• DaemonSet desired != ready
• Vault sealed
• ClusterSecretStore not Ready
• cert-manager Certificate failures
• Backup freshness regressions
• Any external-reachability failure

Deep-investigation recipes per failure mode

Node Issues (checks 1, 3, 17, 25)

kubectl describe node <node>
kubectl top nodes
kubectl get events --field-selector involvedObject.name=<node> --sort-by='.lastTimestamp'
# SSH to the node
ssh root@10.0.20.10X
systemctl status kubelet
journalctl -u kubelet --since "30 minutes ago" | tail -100
df -h ; free -h

Node IPs: 10.0.20.100 master, .101 node1 (GPU), .102 node2, .103 node3, .104 node4.

Pod Issues (checks 4, 5, 11, 19)

kubectl describe pod -n <ns> <pod>
kubectl logs -n <ns> <pod> --tail=200
kubectl logs -n <ns> <pod> --previous --tail=200
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20

Common failure causes: OOMKilled (raise mem limit in Terraform), bad config / missing env var, DB connection failure (check dbaas pods), NFS mount failure (showmount -e 192.168.1.127), stale imagePullSecret.

Deployment / StatefulSet / DaemonSet (checks 6, 7, 16)

kubectl describe deployment -n <ns> <name>
kubectl rollout status deployment -n <ns> <name>
kubectl rollout history deployment -n <ns> <name>
kubectl get rs -n <ns> -l app=<app>

PVC (check 8)

kubectl describe pvc -n <ns> <pvc>
kubectl get events -n <ns> --field-selector reason=FailedMount --sort-by='.lastTimestamp'
kubectl get pv | grep <pvc>
showmount -e 192.168.1.127

cert-manager (checks 31, 32, 33)

kubectl get certificate -A
kubectl describe certificate -n <ns> <name>
kubectl get certificaterequest -A
kubectl describe certificaterequest -n <ns> <name>
kubectl logs -n cert-manager deploy/cert-manager | tail -50

Common causes: ACME HTTP-01 challenge blocked, ClusterIssuer missing DNS provider secret, rate-limit from Let's Encrypt.

Backups (checks 34, 35, 36)

# Per-DB dumps: last successful run of each backup CronJob (the dumps live
# on NFS, /srv/nfs/{mysql,postgresql}-backup/per-db/, not in the DB pods)
kubectl -n dbaas get cronjob mysql-backup-per-db postgresql-backup-per-db \
    -o custom-columns=NAME:.metadata.name,LAST_OK:.status.lastSuccessfulTime

# Pushgateway metrics
kubectl exec -n monitoring deploy/prometheus-server -- \
    wget -qO- http://prometheus-prometheus-pushgateway:9091/metrics | \
    grep backup_last_success_timestamp

# LVM snapshots on PVE host
ssh -o BatchMode=yes root@192.168.1.127 \
    'lvs -o lv_name,lv_time,lv_size --noheadings | grep snap'

If offsite sync is stale, the common cause is the offsite-sync-backup.service systemd unit on the PVE host failing. ssh root@192.168.1.127 'systemctl status offsite-sync-backup'.

Monitoring stack (checks 37, 38, 39)

# Prometheus
kubectl exec -n monitoring deploy/prometheus-server -- wget -qO- http://localhost:9090/-/ready
kubectl logs -n monitoring deploy/prometheus-server --tail=100

# Alertmanager
kubectl get pods -n monitoring | grep alertmanager
kubectl logs -n monitoring -l app=prometheus-alertmanager --tail=100

# Vault
kubectl exec -n vault vault-0 -- sh -c 'VAULT_ADDR=http://127.0.0.1:8200 vault status'
# If sealed: check raft peers with `vault operator raft list-peers` and unseal.

# ClusterSecretStore
kubectl get clustersecretstore
kubectl describe clustersecretstore vault-kv vault-database
kubectl logs -n external-secrets deploy/external-secrets --tail=100

External reachability (checks 40, 41, 42)

# Cloudflared
kubectl get pods -n cloudflared
kubectl logs -n cloudflared -l app=cloudflared --tail=100

# Authentik (Helm chart names the deployment goauthentik-server)
kubectl get deployment -n authentik goauthentik-server
kubectl logs -n authentik deploy/goauthentik-server --tail=100

# ExternalAccessDivergence alert
kubectl exec -n monitoring deploy/prometheus-server -- \
    wget -qO- 'http://localhost:9090/api/v1/alerts' | \
    python3 -m json.tool | grep -A 5 ExternalAccessDivergence

# Traefik 5xx — find the hot service
kubectl exec -n monitoring deploy/prometheus-server -- \
    wget -qO- 'http://localhost:9090/api/v1/query?query=topk(10,rate(traefik_service_requests_total{code=~%225..%22}%5B15m%5D))' \
    | python3 -m json.tool

OOMKilled remediation

1. kubectl describe pod -n <ns> <pod> | grep -A 5 Limits
2. Edit infra/modules/kubernetes/<service>/main.tf and raise resources.limits.memory.
3. cd /home/wizard/code/infra && scripts/tg apply (Tier 1) or terraform apply -target=module.<service> as appropriate.

ImagePullBackOff remediation

1. kubectl describe pod -n <ns> <pod> | grep -A 5 Events
2. Verify tag exists on the source registry.
3. Check pull-through cache at 10.0.20.10:{5000,5010,5020,5030}.
4. Update the image tag in Terraform + re-apply.

Persistent CrashLoopBackOff after auto-fix

1. kubectl logs -n <ns> <pod> --previous --tail=200
2. kubectl describe pod -n <ns> <pod> and check Last State:
   • OOMKilled → raise memory limit
   • Exit code 137 → OOM or probe killed
   • Exit code 143 → SIGTERM / graceful shutdown failed
3. Cross-check dbaas + NFS + secrets are healthy.

Performance forensics — top consumers + optimization hints

When the cluster is healthy (script returns 0) but the host is hot or load is elevated, switch from "what broke?" to "what's expensive?". Run these in order; stop as soon as the root cause is obvious.

Step 1 — Snapshot top consumers cluster-wide

# Top 15 pods by current CPU
kubectl top pods --all-namespaces --sort-by=cpu --no-headers | head -15

# Top 5 nodes by CPU + memory pressure
kubectl top nodes

# Top 15 by 5-min rolling rate (smoothed — kills noise from one-off spikes)
kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \
  "http://localhost:9090/api/v1/query?query=topk(15,sum%20by%20(namespace,pod)%20(rate(container_cpu_usage_seconds_total%7Bcontainer!%3D''%7D%5B5m%5D)))" \
  | python3 -m json.tool | head -80

Step 2 — For each suspect pod, get the WHY

For every pod in the top-N, gather these BEFORE proposing a fix:

NS=<namespace>; POD=<pod>; CONT=$(kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[0].name}')

# What it does (image + command)
kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[0].image}{"\n"}{.spec.containers[0].args}{"\n"}'

# Resource limits + current usage
kubectl -n $NS top pod $POD --containers
kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[0].resources}'

# Recent logs filtered for reconcile loops, watch storms, slow queries
kubectl -n $NS logs $POD -c $CONT --tail=200 --since=5m 2>&1 \
  | grep -iE 'reconcil|watch|scrape|index|loop|retry|slow|timeout' | tail -20

# Restart count + recent OOM
kubectl -n $NS describe pod $POD | grep -E 'Restart Count|Last State|Reason'

# Self-exported metrics (for apps that publish on /metrics)
kubectl -n $NS exec $POD -c $CONT -- wget -qO- localhost:<port>/metrics 2>/dev/null | head -50

Step 3 — apiserver / etcd specific deep-dive (when control-plane is hot)

# Top request producers by verb+resource (last 30 min)
kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \
  "http://localhost:9090/api/v1/query?query=topk(15,sum%20by%20(resource,verb)%20(rate(apiserver_request_total%5B30m%5D)))" \
  | python3 -m json.tool

# Top user agents (which clients are hammering)
kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \
  "http://localhost:9090/api/v1/query?query=topk(15,sum%20by%20(user_agent)%20(rate(apiserver_request_total%5B30m%5D)))" \
  | python3 -m json.tool

# Long-running requests (WATCH / CONNECT — log streams, pod-watchers)
kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \
  "http://localhost:9090/api/v1/query?query=apiserver_longrunning_requests" \
  | python3 -m json.tool

# etcd write rate + DB size
kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \
  "http://localhost:9090/api/v1/query?query=rate(etcd_disk_wal_fsync_duration_seconds_count%5B5m%5D)" \
  | python3 -m json.tool

Step 4 — PVE host specific deep-dive (when temp / load is high)

Checks 43 + 44 capture package temp + 5-min load avg with PASS/WARN/FAIL thresholds — that's the first stop. When those WARN or FAIL, the follow-up commands below trace which VM / process is the source:

# Per-core temps (broader than the package summary in check 43)
ssh root@192.168.1.127 'for f in /sys/class/hwmon/hwmon0/temp*_input; do
    base=${f%_input}; label=$(cat ${base}_label 2>/dev/null || echo "${base##*/}")
    val=$(cat "$f"); echo "  $label: $((val/1000))°C"
done'

# Per-VM CPU (each VM = one kvm process)
ssh root@192.168.1.127 'top -bn1 -o %CPU | grep kvm | head -10'

# pvestatd anomaly check — bursts > 50% usually mean LV count > 1000
ssh root@192.168.1.127 'lvs --noheadings 2>/dev/null | wc -l'

# Stale snapshots (any '_pre-*' that survived past their rollback window)
ssh root@192.168.1.127 'lvs --noheadings -o lv_name 2>/dev/null | awk "/_pre-/" | head -20'

Step 5 — Optimization decision

For each consumer in the top-N, fill in a row:

Pod / Process	CPU (m)	Why busy	Tunable	Est saving	Trade-off	Effort

Then rank by ROI (saving / effort) and surface the top 3-5. Hold back the ones where saving < 50m unless effort is also < 5 min.

Common causes + tunables (catalogue)

Symptom	Likely cause	Tunable
kube-apiserver > 1 core sustained	CONNECT pods/log streams from alloy/promtail using apiserver-tail; OR Kyverno PolicyReport churn (background+enforce mode); OR VPA fanout (309 VPAs cause ~7 req/s)	Switch alloy/promtail to loki.source.file; raise Kyverno backgroundScanInterval; reduce VPA count
pvestatd 70-100% bursts	LV metadata scan over > 1000 LVs (typically stale _pre-* snapshots from ad-hoc node ops)	Delete stale snapshots; /usr/local/bin/lvm-pvc-snapshot prune
Frigate > 2 cores	Birdseye mode: continuous (16% on frigate.output); LPR debug; debug logging; too many active cameras × detect.fps	birdseye.mode: motion; lpr.debug_save_plates: false; remove debug loggers
vault-0 looping ERRORs every ~10s	DB static-role not in connection's allowed_roles list (drift between role and connection)	Add role to vault_database_secret_backend_connection.*.allowed_roles in TF
Alloy DS > 100m/pod	loki.source.kubernetes (apiserver-tail) instead of loki.source.file	Switch to file-tail (~5× drop per pod)
Prometheus default 1m scrape	Chart default; new sample every minute	Raise server.global.scrape_interval to 2m; pin critical jobs (snmp-ups) to 30s; bump for: 1m alerts to for: 3m
kube-controller-manager periodic ERROR loop	Aggregated APIService discovery fails (calico/metrics-server unreachable, OR stuck Terminating pod still in endpoints)	Force-delete stuck pod; verify APIService Available; check pod runc bug on k8s-master
etcd write > 1 MB/s	PolicyReport thrash, too-frequent secret rotation, or audit log mode = RequestResponse	Trim Kyverno reports config; raise rotation_period; downgrade audit policy to Metadata for noisy resources

What NOT to touch

• calico-node, etcd write rate, kube-controller-manager core work, pg-cluster replication — structural cost, touching them risks correctness.
• Pods doing legitimate request-serving work (web servers, databases under load) — optimize the workload, not the runtime.
• Anything where Goldilocks VPA upperBound is already close to current request — no headroom to cut.

Source-of-truth notes

• All infra mutations go via Terraform (scripts/tg plan/apply). The recipes above are diagnostic; the FIX lives in infra/stacks/<name>/main.tf or chart values.
• Pod-internal config files (e.g., Frigate's /config/config.yml on a PVC) are not TF-managed — edit in-pod and document in infra/docs/runbooks/.
• PVE host-level state (LVM snapshots, pvestatd) — SSH + manual ops; record in memory if the pattern recurs.

Notes on the canonical / hardlink setup

The authoritative copy of this SKILL.md lives at /home/wizard/code/.claude/skills/cluster-health/SKILL.md. A hardlink at /home/wizard/code/infra/.claude/skills/cluster-health/SKILL.md points to the same inode so infra-rooted sessions also discover the skill.

To verify the hardlink is intact:

stat -c '%i %n' \
    /home/wizard/code/.claude/skills/cluster-health/SKILL.md \
    /home/wizard/code/infra/.claude/skills/cluster-health/SKILL.md

Both should print the same inode number. If they diverge (e.g. git checkout replaced the file rather than updating it), re-link:

ln -f /home/wizard/code/.claude/skills/cluster-health/SKILL.md \
      /home/wizard/code/infra/.claude/skills/cluster-health/SKILL.md