kubernetes-operations
Kubernetes cluster operations on minikube including observability (Grafana, Prometheus, Alertmanager, Loki, Tempo), debugging (kubectl debug, ephemeral containers), and cluster management (ArgoCD). Us
SKILL.md.claude/skills/kubernetes-operations/SKILL.md
GitHub- Access Grafana at
https://grafana.minikube.127.0.0.1.nip.io - Disable ArgoCD selfHeal before manual changes, re-enable after
- Prometheus runs in Agent mode (remote-write only, no query engine) — Mimir is the only PromQL backend
Debugging
| Investigating | Tool | Entry Point |
|---|---|---|
| Traces, metrics, logs, profiles | Grafana | https://grafana.minikube.127.0.0.1.nip.io |
| Pod startup failures, events | kubectl | kubectl get events -n <namespace> |
Pods stuck Pending with FailedCreatePodSandBox: unable to connect to Cilium agent across multiple namespaces |
kubectl | kubectl get node <node> -o jsonpath='{.spec.taints}' for node.cilium.io/agent-not-ready, then kubectl logs -n kube-system <cilium-agent-pod> for quorum check failed - see ## Restart Count Alone Is Not Incident Evidence in Inventory |
| In-container process/network | kubectl debug | Ephemeral container |
kubectl connection refused despite VM running (clusters/users empty in ~/.kube/config) |
minikube | minikube update-context - see Sandbox Connectivity when it fails on the libvirt socket |
ArgoCD Application stuck OutOfSync/Missing |
kubectl | kubectl get application -n argocd <name> -o jsonpath='{.status.operationState.message}' |
ArgoCD Application OutOfSync with the diff confined to a revision pod-template annotation |
kubectl | Not real drift — the repo-server's custom kustomize wrapper (cluster/manifests/argocd/overlays/dev/patches/deployment.yaml) injects spec.template.metadata.annotations.revision from git log -1 --format=%H -- . on every kustomize build, so it changes whenever a later commit touches that Application's own manifest directory; confirm via kubectl diff against the live object rather than trusting Sync Status alone |
ArgoCD Application Degraded from a workload that never becomes ready after an image change |
gh | gh pr list --search "is:open is:pr <app-name>" then gh pr view <number> --json mergeable,mergeStateStatus,reviewDecision - a 00_*.yaml digest-bump PR may already carry the fix and be stuck on reviewDecision: REVIEW_REQUIRED rather than need a fresh manifest edit, since nothing in this repository auto-merges those PRs |
Knative Revision pod stuck Not Ready with its PodAutoscaler already reporting desiredScale: 0/NoTraffic but the Deployment never scales down |
kubectl | Confirm no traffic or tag is pinned to the stale revision (kubectl get ksvc <name> -o jsonpath='{.spec.traffic}{.status.traffic}'), then kubectl delete revision <name> -n <namespace> - cascades the Deployment/Pod/PodAutoscaler/ServerlessService it owns; this is a Knative reconciler gap, not a manifest issue, and can leave the Deployment at its old replica count for days after the PodAutoscaler itself already reports zero |
Observability Investigation (via Grafana)
Use Grafana for all observability signals. Do NOT query backends (Tempo, Mimir, Loki) directly.
- Get query parameters - Check
cluster/manifests/<app>/for namespace, labels,OTEL_SERVICE_NAME(see Queries for parameter locations) - Open Grafana - Access
https://grafana.minikube.127.0.0.1.nip.ioin browser - Select datasource and query - Use appropriate signal:
| Signal | Backend | Datasource | Query Language | Use Case |
|---|---|---|---|---|
| Traces | Tempo | Tempo | TraceQL | Request flow, latency, access destinations |
| Metrics | Mimir (Prometheus) | Mimir | PromQL | Resource usage, HTTP rates, alerting |
| Logs | Loki | Loki | LogQL | Error investigation, audit |
| Profiles | Pyroscope | Pyroscope | Flamegraph UI | CPU/memory hotspots |
| Probes | Blackbox Exporter | Mimir | PromQL | Endpoint reachability |
| Symptom | Signal | Query Approach |
|---|---|---|
| Errors in logs | Loki → Tempo | Extract traceid from logs, trace in Tempo |
| Latency/5xx | Tempo | Search traces with status = error |
| Unknown outbound dependencies | Tempo | Search traces, inspect spans for outbound calls |
| Resource saturation | Mimir | Query CPU/memory metrics |
| High CPU/memory | Pyroscope | Check flamegraphs |
Pod Direct Investigation (via kubectl)
Use kubectl when Grafana cannot answer the question (e.g., pod not starting, container-level inspection).
| Symptom | Action |
|---|---|
| Pod not starting | kubectl get events -n <namespace> |
| CrashLoopBackOff | kubectl logs <pod> -n <namespace> --previous |
| Network connectivity | kubectl debug with ephemeral container |
| Process inspection | kubectl debug with ephemeral container |
| JVM effective flags | kubectl exec → jcmd -l, then jcmd <pid> VM.flags (see below) |
JVM Flag Inspection
Use kubectl exec -c <container>, not kubectl debug — the ephemeral-container image ships no JDK.
PID 1 is often tini, so jcmd 1 fails to attach; find the real PID first.
jcmd -l usually lists it, but returns nothing on images without hsperfdata, so fall back to scanning cmdlines:
for p in /proc/[0-9]*; do tr '\0' ' ' < $p/cmdline | grep -q '^java ' && echo ${p#/proc/}; done
Never run java -XX:+PrintFlagsFinal -version, which starts a second JVM in the same cgroup and can OOM-kill the process being inspected.
Ephemeral Container
kubectl debug <pod-name> -n <namespace> \
--profile=restricted \
--image=ghcr.io/hippocampus-dev/hippocampus/ephemeral-container:main \
--target=<container-name> \
-- <command>
Note: Do not use -it flag when executing commands.
It causes output streaming issues.
Manual Changes
Required when directly modifying ArgoCD-managed resources (e.g., kubectl apply, kubectl patch, kubectl delete) outside of the GitOps workflow.
| Operation | Requires Manual Changes |
|---|---|
kubectl apply/patch/delete on an ArgoCD-managed resource |
Yes |
kubectl delete pod where a ReplicaSet/StatefulSet/DaemonSet recreates it |
No (ArgoCD manages the controller, not the pod) |
| Debugging (read-only: logs, events, debug) | No |
| Grafana queries | No |
| Editing manifests in repo | No (ArgoCD syncs automatically) |
ArgoCD selfHeal Control
ArgoCD reverts manual changes unless selfHeal is disabled first.
Always re-enable after — "after" means once the change is durably reflected in the manifest ArgoCD syncs from (committed and pushed, or already matching git), not merely once the live cluster looks healthy.
If the backing manifest edit cannot yet be committed (e.g. a read-only .git), leave selfHeal disabled until it can be; re-enabling first reverts the live fix back to the stale manifest.
# Disable selfHeal
kubectl patch application <app-name> -n argocd --type=merge \
-p '{"spec":{"syncPolicy":{"automated":{"selfHeal":false}}}}'
# Re-enable selfHeal (after work is complete)
kubectl patch application <app-name> -n argocd --type=merge \
-p '{"spec":{"syncPolicy":{"automated":{"selfHeal":true}}}}'
selfHeal lives under automated; patching it at the syncPolicy root is silently pruned and leaves selfHeal enabled.
Disabling selfHeal on a child Application does not hold if that Application is itself managed by a root app-of-apps Application with its own automated sync (check via kubectl get application <root-app> -n argocd -o jsonpath='{.spec.syncPolicy.automated}').
The root's own reconciliation restores the child Application's spec from git within minutes, undoing the disable before the manual change can be made.
Disabling automated sync on the root Application instead suspends GitOps reconciliation for every Application it manages, not just the one being worked on - treat that as a separate, larger decision rather than a substitute fix.
Re-enabling selfHeal restores the manifest ArgoCD manages, not what an operator derives from it. After reverting an experiment on a CR, check the operator-managed Deployment/StatefulSet directly — operators that merge resource fields rather than replacing them leave removed keys behind.
External Service Access
Services exposed via Istio Gateway follow a tiered host naming convention:
| Host | Authentication | Access |
|---|---|---|
{service}.minikube.127.0.0.1.nip.io |
None | Local development |
{service}.kaidotio.dev |
OAuth2 (ext-authz) | Browser access |
{service}-public.kaidotio.dev |
Custom (JWT, method restriction, etc.) | Programmatic access excluded from OAuth2 |
The -public variant is used when OAuth2 is technically impossible (e.g., W3C Reporting API browser-generated requests) or when the service implements its own AuthorizationPolicy (e.g., GitHub Actions OIDC token validation).
To access a service's data locally, use https://{service}.minikube.127.0.0.1.nip.io.
Observability Stack Manifests
| Component | Path |
|---|---|
| Grafana | cluster/manifests/grafana/ |
| Tempo | cluster/manifests/tempo/ |
| Mimir | cluster/manifests/mimir/ |
| Loki | cluster/manifests/loki/ |
| Pyroscope | cluster/manifests/pyroscope/ |
| Prometheus | cluster/manifests/prometheus/ |
| Fluentd | cluster/manifests/fluentd/ |
| OpenTelemetry | cluster/manifests/otel-agent/, cluster/manifests/otel-collector/ |
Reference
If writing observability queries: See Queries
If an ArgoCD Application failed to sync and never retried:
See ## Sync Wave in .claude/rules/cluster/manifests/argocd-applications.md
If enumerating workloads or containers cluster-wide, reclaiming over-provisioned memory requests, tracing a Pod to its owner, or checking OOMKilled status: See Inventory