k8s-debug
Kubernetes debugging skill: systematic approach to diagnosing pod failures, network issues, resource contention, and cluster-level problems.
SKILL.mdskills/k8s-debug/SKILL.md
GitHubKubernetes Debugging Skill
When diagnosing Kubernetes issues, follow this systematic approach:
Pod not starting
kubectl get pod <name> -o wide— check node, status, restartskubectl describe pod <name>— read Events section firstkubectl logs <name> --previous— if CrashLoopBackOff- Check resource limits: OOMKilled → increase memory limit
- Check image pull: ImagePullBackOff → verify registry auth / image tag
Service not reachable
kubectl get endpoints <svc>— verify pod IPs are listed (selector match?)kubectl run -it --rm debug --image=nicolaka/netshoot -- bash— test from inside cluster- Check NetworkPolicy:
kubectl get netpol -A - Verify port names match between Service and Pod spec
Node pressure
kubectl describe node <name>— check Conditions and Allocated resourceskubectl top nodes/kubectl top pods -A --sort-by=memory- Identify DaemonSets consuming resources on every node
Persistent volume issues
kubectl get pvc -A— check Bound/Pending statuskubectl describe pvc <name>— look for provisioner errors- Verify StorageClass exists:
kubectl get sc
General tools
k9s— interactive real-time viewkubectl events --for pod/<name>(k8s 1.26+)stern <pod-prefix>— multi-pod log tailingkubectl exec -it <pod> -- /bin/sh— shell into container
Hermes MCP troubleshooting
- Working Hermes MCP setup used an in-pod HTTP sidecar exposing
http://127.0.0.1:8080/mcp. urlwas required in Hermes config;type: ssealone did not solve client selection.- Keep the
kubernetes-mcp-servercontainer in the same pod as Hermes. - Mount
/opt/data,/opt/data/.hermes,config.yaml,.env, andgateway.json. - Use
serviceAccountNameon the pod and let Kubernetes mount the API token/CA automatically. - If Hermes falls back to terminal and tries
kubectl, remove or disable the terminal block while testing MCP. - The static manifest that worked used
hermes-agent-mcp+kubernetes-mcp-server:v0.0.60+hermes-config-mcp-test+mcp_servers.kubernetes.url: http://127.0.0.1:8080/mcp. - For Telegram-only access, enforce the bot allowlist with
TELEGRAM_ALLOWED_USERSand the gatewayallowed_usersfield; if either is broad, anyone can talk to the bot.
Common quick fixes
- Stuck namespace terminating: patch finalizers to
[] - ConfigMap/Secret not updating in pod: rolling restart
kubectl rollout restart deploy/<name> - Certificate expired: check cert-manager
kubectl get certificate -A
Observability stack debugging
The cluster runs Prometheus + Grafana + Tempo + Loki + Alloy in monitoring namespace.
Grafana datasource not connected
- Check datasource config:
kubectl get configmap kube-prometheus-stack-grafana-datasource -n monitoring -o yaml - Verify target service is running:
kubectl get pods -n monitoring - Test connectivity from Grafana pod:
kubectl exec -it <grafana-pod> -n monitoring -- curl http://loki-gateway.monitoring:80/loki/api/v1/label - For Loki: ensure
X-Scope-OrgId: fakeheader is set in datasource config
Loki not receiving logs
- Check Loki status:
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki - Verify MinIO is running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=minio - Test push:
kubectl port-forward -n monitoring svc/loki-gateway 3100:80thencurl -XPOST http://localhost:3100/loki/api/v1/push -H "Content-Type: application/json" -H "X-Scope-OrgId:fake" -d '{"streams":[{"stream":{"job":"test"},"values":[["'$(date +%s)000000000'","test log"]]}]}' - Check Alloy config for Loki output:
kubectl get configmap -n monitoring -l app.kubernetes.io/name=alloy loki-chunks-cache-0Pending is normal — optional cache, not required for operation
Tempo not receiving traces
- Check Tempo status:
kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo - Verify Alloy is sending traces:
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep tempo - Test query:
kubectl port-forward -n monitoring svc/tempo 3200:3200thencurl http://localhost:3200/api/search
Alloy not forwarding data
- Check Alloy pods:
kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy - Check Alloy logs:
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy - Verify Alloy config:
kubectl get configmap -n monitoring -l app.kubernetes.io/name=alloy -o yaml
ArgoCD repo-server CrashLoopBackOff (ARM64 CM4)
- Check pod events:
kubectl describe pod -n argocd -l app.kubernetes.io/name=argocd-repo-server - Check logs:
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=50 - If liveness probe fails with "context deadline exceeded": probe timeout too low for CM4
- Fix: set
repoServer.livenessProbe.timeoutSeconds: 5,initialDelaySeconds: 30
Observability stack debugging
The cluster runs Prometheus + Grafana + Tempo + Loki + Alloy in monitoring namespace.
Grafana datasource not connected
- Check datasource config:
kubectl get configmap kube-prometheus-stack-grafana-datasource -n monitoring -o yaml - Verify target service is running:
kubectl get pods -n monitoring - Test connectivity from Grafana pod:
kubectl exec -it <grafana-pod> -n monitoring -- curl http://loki-gateway.monitoring:80/loki/api/v1/label - For Loki: ensure
X-Scope-OrgId: fakeheader is set in datasource config
Loki not receiving logs
- Check Loki status:
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki - Verify MinIO is running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=minio - Test push:
kubectl port-forward -n monitoring svc/loki-gateway 3100:80thencurl -XPOST http://localhost:3100/loki/api/v1/push -H "Content-Type: application/json" -H "X-Scope-OrgId:fake" -d '{"streams":[{"stream":{"job":"test"},"values":[["'$(date +%s)000000000'","test log"]]}]}' - Check Alloy config for Loki output:
kubectl get configmap -n monitoring -l app.kubernetes.io/name=alloy loki-chunks-cache-0Pending is normal — optional cache, not required for operation
Tempo not receiving traces
- Check Tempo status:
kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo - Verify Alloy is sending traces:
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep tempo - Test query:
kubectl port-forward -n monitoring svc/tempo 3200:3200thencurl http://localhost:3200/api/search
Alloy not forwarding data
- Check Alloy pods:
kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy - Check Alloy logs:
kubectl logs -n monitoring -l app.kubernetes.io/name=alloy - Verify Alloy config:
kubectl get configmap -n monitoring -l app.kubernetes.io/name=alloy -o yaml
ArgoCD repo-server CrashLoopBackOff (ARM64 CM4)
- Check pod events:
kubectl describe pod -n argocd -l app.kubernetes.io/name=argocd-repo-server - Check logs:
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=50 - If liveness probe fails with "context deadline exceeded": probe timeout too low for CM4
- Fix: set
repoServer.livenessProbe.timeoutSeconds: 5,initialDelaySeconds: 30