k8s-node-troubleshooting
Use when a Kubernetes node is unhealthy or pods will not go onto it: a node NotReady or Unknown, MemoryPressure, DiskPressure or PIDPressure, pods Evicted, pods Pending with "untolerated taint(s)", "I
SKILL.mdskills/k8s-node-troubleshooting/SKILL.md
GitHubKubernetes node troubleshooting
Find out why a node is unhealthy, or why pods will not be scheduled onto it, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
You see the node only through the Kubernetes API. Checks on the node itself (the kubelet, the container runtime, the disk) need someone with access to the machine, so they are always proposed fixes for a human.
The commands use placeholders: my-node, my-namespace, my-app and my-pod.
Replace them with the real names. Never read Secrets. If output contains passwords,
tokens or keys, do not repeat them.
When to use
kubectl get nodesshows a nodeNotReady,UnknownorSchedulingDisabled.- A node reports
MemoryPressure,DiskPressureorPIDPressure, or pods wereEvicted. - Pods stay
Pendingand theFailedSchedulingevent names taints,Insufficient cpuorInsufficient memory, or unschedulable nodes. - The cluster autoscaler does not add a node for Pending pods, or does not remove an empty one.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- The nodes are healthy and have room, and the pod itself fails (crash, image, probe,
quota):
k8s-workload-troubleshooting. - A volume cannot attach or mount on the node:
k8s-storage-troubleshooting. NetworkUnavailableisTrue, or the CNI pods on the node fail:k8s-network-troubleshooting.- On AKS, a node pool or the cluster is
Failed, or an upgrade or scale fails in Azure (quota, VM capacity, a drain blocked by a PodDisruptionBudget):k8s-aks-troubleshooting.
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. List the nodes. STATUS shows Ready, NotReady or Unknown, plus
SchedulingDisabled when a node is cordoned.
kubectl get nodes -o wide
2. Describe the node. Read Conditions (status, reason, LastHeartbeatTime),
Taints, Unschedulable, Lease (RenewTime), Capacity and Allocatable,
Non-terminated Pods with their requests, and Allocated resources.
kubectl describe node my-node
3. See the pods on the node, in every namespace you can read.
kubectl get pods -A -o wide --field-selector spec.nodeName=my-node
kubectl get pods -A --field-selector spec.nodeName=my-node,status.phase=Failed
The second command lists failed pods there, which include Evicted ones.
4. Read the node's events and its heartbeat. Node events are kept in the default
namespace; the heartbeat is a Lease named after the node. The built-in view role
cannot read Leases; if the second command is forbidden, use LastHeartbeatTime of the
conditions in describe node instead.
kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name=my-node
kubectl get lease my-node -n kube-node-lease
5. For a Pending pod, read why the scheduler refused it, and what it asks for.
kubectl describe pod my-pod -n my-namespace
kubectl get pod my-pod -n my-namespace -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}{"\n"}{.spec.containers[*].resources.requests}'
6. Look at usage, if the cluster has metrics-server. top shows usage, not
requests; the scheduler uses requests. Without metrics-server it fails with "Metrics API
not available"; say so and go on.
kubectl top node my-node
7. Open the matching reference and follow its steps:
| What you see | Reference |
|---|---|
NotReady, Unknown, "Kubelet stopped posting node status" |
references/not-ready.md |
MemoryPressure, DiskPressure, PIDPressure, Evicted pods |
references/pressure.md |
Insufficient cpu or Insufficient memory, requests near Allocatable |
references/capacity.md |
untolerated taint(s), SchedulingDisabled, node(s) were unschedulable |
references/taints.md |
| Pending pods but no new node, or an empty node that stays | references/autoscaler.md |
Read the results
What the common findings mean (Kubernetes 1.37):
ReadyUnknown: the node controller has not heard from the kubelet withinnode-monitor-grace-period(50 seconds by default). The kubelet is down, or the node cannot reach the API server. The Lease inkube-node-lease, normally renewed every 10 seconds, stops changing. The node gets thenode.kubernetes.io/unreachabletaints.ReadyFalse: the kubelet is running and reports the node is not healthy (for example the container runtime or the network plugin is not ready). The reason and message say which. The node gets thenode.kubernetes.io/not-readytaint.- Pods on a
NotReadynode keep their status for a while. Pods tolerate thenot-readyandunreachableNoExecutetaints for 300 seconds by default (added by the DefaultTolerationSeconds admission controller), then they are evicted. DaemonSet pods tolerate them without a time limit and are not evicted. MemoryPressure,DiskPressureorPIDPressureTrue: the kubelet crossed an eviction threshold. It adds aNoScheduletaint for that pressure, frees resources, then evicts pods. Node-pressure eviction does not respect PodDisruptionBudgets.FailedSchedulingwith0/N nodes are available: ...: each part counts the nodes refused for one reason, for example1 node(s) had untolerated taint(s),2 Insufficient cpuor1 node(s) were unschedulable. The text afterpreemption:says whether evicting lower-priority pods could help.Insufficient cpuon an idle node: the scheduler adds up the requests of the pods on the node and compares them withAllocatable, not with actual use.Allocated resourcesindescribe nodeshows the requests as a percentage of Allocatable.SchedulingDisabled,Unschedulable: true: the node is cordoned. New pods do not go there; pods already there keep running.
State the cause in one sentence, then the evidence you saw (the condition and its message, the taint, the event, the numbers), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it.
Cordoning, uncordoning, adding or removing a NoSchedule taint and labelling a node are
reversible, so they go under "Ask first" and run only after the user says yes. Draining
or deleting a node, force-deleting pods and anything on the node host are never run by
you: hand them to the user as proposed fixes and say why. Prefer fixing the source the
workload or the node pool is managed from when there is one. Write each undo with the
values you found.
Uncordon a node whose maintenance is over and that is Ready:
Ask first: the agent runs this only after the user approves it.
kubectl uncordon my-node
# Undo: kubectl cordon my-node
Remove a taint left from a maintenance that is over (the key, value and effect exactly
as describe node shows them):
Ask first: the agent runs this only after the user approves it.
kubectl taint nodes my-node example.com/maintenance=true:NoSchedule-
# Undo: kubectl taint nodes my-node example.com/maintenance=true:NoSchedule
Check the kubelet on a node that stopped reporting. This runs on the node, which you cannot reach; a person with access to the machine runs it:
Proposed fix: review before running. The agent does not run this.
systemctl status kubelet
journalctl -u kubelet --since "30 min ago" --no-pager | tail -n 50
sudo systemctl restart kubelet
# Undo: nothing to undo for the first two lines, they only read; a restart cannot be undone, the kubelet just starts again
Move the pods off a node that will not come back soon. Draining evicts every pod that
is not a DaemonSet pod and deletes emptyDir data, so never run it yourself:
Proposed fix: review before running. The agent does not run this.
kubectl drain my-node --ignore-daemonsets --delete-emptydir-data
# Undo: kubectl uncordon my-node (the evicted pods are recreated by their controllers, not moved back)
Sources
Checked against the Kubernetes 1.37 documentation:
- Node Status: conditions, the
50-second
node-monitor-grace-period, Leases inkube-node-leaseand their 10-second renewal. - Nodes: cordon and
spec.unschedulable. - Force Delete StatefulSet Pods: StatefulSet pods on an unreachable node are not replaced on their own.
- Taints and Tolerations
and Well-Known Labels, Annotations and Taints:
taint effects, the
node.kubernetes.io/*taints, taint removal. - Admission Control: DefaultTolerationSeconds and the 300-second default.
- DaemonSet: tolerations DaemonSet pods get.
- Node-pressure Eviction: signals, default thresholds, reclaim order, no PDBs.
- Reserve Compute Resources for System Daemons and Resource Management for Pods and Containers: Allocatable, scheduling by requests.
- kubectl drain, kubectl cordon, kubectl taint and kubectl top node.
- Resource metrics pipeline
and Troubleshooting Clusters:
metrics-server,
describe nodeoutput. - Cluster Autoscaler FAQ: scale-up and scale-down rules, events, the status ConfigMap, annotations.
- The
FailedSchedulingreason texts are from the Kubernetes 1.37 scheduler source (tainttoleration plugin), because the docs show only an older example.