k8s-triage
Use when something is wrong in a cluster and it is not yet clear where: "the cluster is broken", "a lot of pods are failing", "users say the app is down", "something is off after the upgrade", or any
SKILL.mdskills/k8s-triage/SKILL.md
GitHubKubernetes triage
Find out where a cluster is unhealthy when nobody knows yet, rank the problems, and hand each one to the right place. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
Triage is a first look, not a root-cause analysis. For each problem, report it, show the evidence, propose a first fix, and name the skill that investigates it fully. Do not dig deeper than the short first look this skill describes.
The commands use placeholders: my-namespace, my-app, my-pod and my-node. Replace
them with the real names. Never read Secrets. If output contains passwords, tokens or
keys, do not repeat them.
When to use
- The user reports a problem without naming a workload: "the cluster is broken", "everything is slow", "lots of pods are failing", "users say the app is down".
- Something changed cluster-wide (an upgrade, a new node pool, a network change) and "something is off".
- You were asked for a general health check of a cluster.
When the user names one workload that is unhealthy, start with
k8s-workload-troubleshooting instead.
Hand-offs, by skill name. For each issue you find, name the skill that goes deeper. If it is not installed, give the issue, its evidence, a first fix and the short first look below, then tell the user which skill would help, and stop there:
- Pods crash, restart, stay
Pending, fail probes, or a rollout or Job is stuck:k8s-workload-troubleshooting. - A node is
NotReady, under memory, disk or PID pressure, cordoned, or pods are being evicted:k8s-node-troubleshooting. - Pods are healthy but traffic does not arrive (Service with no endpoints, DNS,
NetworkPolicy, Ingress):
k8s-network-troubleshooting. - A PersistentVolumeClaim is
Pending, or volumes fail to attach or mount:k8s-storage-troubleshooting. - The API server refuses requests (
failed calling webhook, an APIService that is notTrue, a namespace stuckTerminating):k8s-api-troubleshooting. - On AKS, the error comes from Azure (
SyncLoadBalancerFailed, aProvisioningFailedwith an Azure error code,AADSTSerrors, a cluster or node poolFailed):k8s-aks-troubleshooting.
Investigate
Work through the steps in order. They are cheap and cluster-wide; do all of them before you rank anything.
1. Check what you can see. A user with access to one namespace cannot list nodes or other namespaces. Know this before you read any result.
kubectl auth can-i list nodes
kubectl auth can-i list pods --all-namespaces
kubectl auth can-i list events --all-namespaces
If a check says no, or a command below fails with Forbidden, say exactly what you
could not see, and narrow the sweep to the namespaces you can read. Start with the
namespace your kubeconfig points at: commands without -n use it, and users with access
to one namespace usually have it set.
kubectl auth can-i list pods
kubectl get pods
kubectl auth can-i list pods -n my-namespace
If none of these works and you know no namespace, ask the user which one to look at. Never report something you could not see as healthy.
2. Nodes. Look at STATUS (Ready, NotReady, and SchedulingDisabled for a
cordoned node), then describe every node that is not plainly Ready, and read its
Conditions, Taints and Allocated resources.
kubectl get nodes -o wide
kubectl describe node my-node
3. Pods that are not running. Across every namespace you can read:
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'
The first command finds Pending, Failed and Unknown pods. The second also catches
pods whose phase is Running but whose STATUS column says CrashLoopBackOff,
ImagePullBackOff, Error or Terminating. The third puts the most restarted pods
last. Also look for Running pods that are not ready, such as 0/1 in the READY
column.
When a node is not Ready, list the pods on it:
kubectl get pods -A --field-selector=spec.nodeName=my-node -o wide
4. Warning events. For each one, check whether the object it is about is still unhealthy now. Only those count as issues; a Warning about an object that has since recovered is history, even if it is recent (a new cluster has several from start-up).
kubectl events -A --types=Warning
5. CPU and memory. kubectl top works only when metrics-server is installed and
running. If it fails, write "metrics not available" in the overall status and go on;
Allocated resources from step 2 still shows requests and limits against capacity.
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl top pods -A --sort-by=cpu
6. Controllers short of replicas. A Deployment, StatefulSet or DaemonSet whose ready count is below the desired count is affected even when its pods look fine one by one.
kubectl get deployments -A
kubectl get statefulsets -A
kubectl get daemonsets -A
7. PersistentVolumeClaims and Services, only if steps 3 and 4 point there (pods
Pending with volume events, or an app that is up but unreachable):
kubectl get pvc -A
kubectl get endpointslices -n my-namespace
8. Pick the area of each issue and its next skill, from this table:
| What you see | Tier | Next skill |
|---|---|---|
Node NotReady or Unknown, a pressure condition True, NetworkUnavailable, SchedulingDisabled |
1 | k8s-node-troubleshooting |
CrashLoopBackOff, OOMKilled, Error, ErrImagePull, ImagePullBackOff, CreateContainerConfigError |
2 | k8s-workload-troubleshooting |
Pending, Running but not ready, a controller short of replicas |
3 | k8s-workload-troubleshooting (or k8s-storage-troubleshooting for an unbound PVC) |
| Warning events about objects that are still unhealthy, a node near its memory or CPU capacity, pods at their limits | 4 | the area the event belongs to |
| Service with no ready endpoints while its pods are healthy, DNS failures | 3 | k8s-network-troubleshooting |
| CoreDNS pods crashing or not Ready, CoreDNS errors in its logs | 2 | k8s-coredns-troubleshooting |
FailedCreate events that name a webhook, an APIService that is not True, a namespace stuck Terminating |
3 | k8s-api-troubleshooting |
On AKS: SyncLoadBalancerFailed, Azure error codes in ProvisioningFailed or FailedAttachVolume, AADSTS errors in logs |
3 | k8s-aks-troubleshooting |
A short first look inside a container can confirm a network issue. Ask first: show the
command, and run it only after the user agrees (pick a running pod in the namespace that
has nslookup; many minimal images do not):
Ask first: the agent runs this only after the user approves it.
kubectl exec my-pod -n my-namespace -- nslookup my-app.my-namespace.svc.cluster.local
Read the results
Rank by severity. Report issues in this order, most urgent first:
- Nodes.
NotReadyorUnknown,MemoryPressure,DiskPressureorPIDPressureTrue,NetworkUnavailable, cordoned (SchedulingDisabled). - Crashing pods.
CrashLoopBackOff,OOMKilled,Error,ErrImagePullorImagePullBackOff,CreateContainerConfigError. - Pods that do not become ready.
Pending,Runningbut not ready, controllers with fewer ready replicas than desired, Services without ready endpoints. - Early warnings. Warning events about objects that are still unhealthy, and resource pressure that has not broken anything yet (a node above about 90% of its memory, pods at their CPU or memory limit).
Within a tier, more affected nodes or pods rank higher. One cause, one issue: when a
node problem explains pod symptoms (the pods on a NotReady node show Unknown,
Terminating, or are being recreated elsewhere), report the node issue once and list
the pods as its evidence. Do not report those pods as separate application failures.
Read nodes. Ready is Unknown when the node controller has not heard from the
node within its grace period (50 seconds by default): the kubelet is down or the node is
unreachable. Ready is False when the kubelet reports the node is not healthy. A
pressure condition True means the kubelet may evict pods from that node. The taints
node.kubernetes.io/not-ready, node.kubernetes.io/unreachable and
node.kubernetes.io/unschedulable (a cordoned node) show the same state to the
scheduler. By default, pods on a NotReady or unreachable node are evicted after five
minutes; until then they look stuck, and pods of a StatefulSet can stay Terminating.
Read events. Typical reasons: FailedScheduling, BackOff, Failed (image pull),
Unhealthy (probes), FailedMount, Evicted, NodeNotReady, OOMKilling. An event
is evidence only when its object is unhealthy now: a Running, ready pod that once
failed a readiness probe is not an issue.
Write the report in this shape:
- Overall status. Nodes (how many, how many
Ready), pods (how many are not healthy, in how many namespaces), metrics (available or not), and what you could not see because of access. - Issues, ranked. For each: one sentence for the issue, the evidence you saw (the status, the condition, the event reason and message, the number), the next skill, and a first fix in the format under Proposed fix.
- When the sweep is clean, do not invent a problem. Say what you checked and that it looked healthy from where you could see, then ask the user to narrow it down: which app, namespace or URL, what exactly they saw (an error message, a timeout, slowness), and since when.
Proposed fix
Triage proposes first fixes only. Show each one to the user first: what it changes and how to undo it. A reversible fix goes under "Ask first" and runs only after the user says yes. Anything destructive or hard to undo is never run by you; say why and hand it over. Prefer fixing the source the workload is deployed from (manifest, Helm values, GitOps repository) when there is one. Write each undo with the values you found.
Uncordon a node that was left cordoned after maintenance and is otherwise healthy:
Ask first: the agent runs this only after the user approves it.
kubectl uncordon my-node
# Undo: kubectl cordon my-node
Roll back a Deployment whose pods started failing right after a rollout. First note the
current revision with kubectl rollout history deployment/my-app -n my-namespace; the
undo below returns to it by number (here revision 3), so it holds even if another rollout
happens in between:
Ask first: the agent runs this only after the user approves it.
kubectl rollout undo deployment/my-app -n my-namespace
# Undo: kubectl rollout undo deployment/my-app -n my-namespace --to-revision=3
Force-delete a pod stuck Terminating on a node that is gone for good. Never run this
yourself: for a StatefulSet pod it can start a second copy while the first still runs
on a node that is only unreachable. A human must first confirm the node is really gone.
Proposed fix: review before running. The agent does not run this.
kubectl delete pod my-pod -n my-namespace --force --grace-period=0
# Undo: none; the pod's controller recreates it on another node
For a node that stays NotReady, a workload that keeps crashing, a Service without
endpoints or a stuck volume, the fix needs the deeper skill named in the issue.
Sources
Checked against the Kubernetes 1.37 documentation:
- Nodes and
Node Status: conditions,
the 50-second grace period before
Unknown, cordoned nodes shown asSchedulingDisabled, eviction five minutes afterUnknown. - Node-pressure Eviction: pressure conditions and eviction signals.
- Taints and Tolerations:
the
node.kubernetes.io/*taints, the default 300-second tolerations, pods still running on a partitioned node. - Pod Lifecycle:
pod phases (
Unknownwhen the node cannot be reached),Terminating,CrashLoopBackOff. - Field Selectors:
status.phaseandspec.nodeNamefor pods. - kubectl events:
--types,-A. - Resource metrics pipeline
and kubectl top:
kubectl topneeds metrics-server. - Troubleshooting Clusters: listing and describing nodes.
- Using RBAC Authorization:
the
viewrole andkubectl auth can-i.