k8s-grafana-troubleshooting

Use when Grafana on Kubernetes fails: the Grafana pod crash-loops after a change to its provisioning ("Datasource provisioning error", "Only one datasource per organization can be marked as default",

SKILL.md

GitHub

Grafana troubleshooting

Find out why Grafana does not start, does not show a datasource or dashboard, or shows no data, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

On Kubernetes, Grafana is usually provisioned: datasources and dashboard providers are YAML files under /etc/grafana/provisioning/, and dashboards are JSON files, all mounted from ConfigMaps. With the Grafana Helm chart (also inside kube-prometheus-stack) a sidecar container copies ConfigMaps labelled grafana_dashboard or grafana_datasource into the pod. With the grafana-operator, Grafana, GrafanaDashboard and GrafanaDatasource resources do the same.

The commands use placeholders: monitoring (where Grafana runs), my-grafana (its Deployment and Service), my-grafana-datasources and my-grafana-dashboards (its ConfigMaps) and my-app (a dashboard). Replace them with the real names.

Never read Secrets: not the admin password, not datasource credentials, not grafana.ini values kept in Secrets. The sidecar can also read Secrets with the same labels; look only at ConfigMaps. Do not log in to Grafana's API with the admin password; its /api/health needs no login.

When to use

  • The Grafana pod is in CrashLoopBackOff or not Ready, often right after a change.
  • A provisioned datasource or dashboard is missing in the UI.
  • Panels show "No data", or a datasource test fails.
  • Logins fail, or the sidecar logs 401 Unauthorized.
  • grafana-operator resources do not sync.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • Grafana runs and reaches Prometheus, but Prometheus has no data for the query: k8s-prometheus-troubleshooting.
  • The datasource's Service exists but traffic does not reach it (NetworkPolicy, DNS): k8s-network-troubleshooting.
  • The pod is Pending or its volume does not mount: k8s-workload-troubleshooting, k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the pods. During a rollout the old pod can keep running while the new one crashes; look at each pod, not only the Deployment.

kubectl get deployments,pods -n monitoring -l app=my-grafana -o wide
kubectl get pods -n monitoring -l app=my-grafana -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{"\n"}{range .status.containerStatuses[*]}{"  "}{.name}{" ready="}{.ready}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}{end}'
kubectl events -n monitoring

Avoid kubectl describe pod here: it prints the containers' environment values, and Grafana pods often carry passwords there. If you see a credential anyway, do not repeat it.

The chart labels its pods app.kubernetes.io/name=grafana; use that selector there.

2. Read the logs of the failing pod, the previous run if it restarted. Each pod has the grafana container, and with the chart one container per sidecar (grafana-sc-dashboard, grafana-sc-datasources).

kubectl logs -n monitoring my-grafana-0 -c grafana --previous --tail=50
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=200 | grep -E "level=error|level=warn|Error:"
kubectl logs -n monitoring deployment/my-grafana -c grafana --tail=500 | grep logger=provisioning

Replace my-grafana-0 with the failing pod's name from step 1.

3. Read the provisioning ConfigMaps the pod mounts.

kubectl get deployment my-grafana -n monitoring -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{"\n"}{end}'
kubectl get configmap my-grafana-datasources -n monitoring -o yaml
kubectl get configmaps -n monitoring -l grafana_dashboard
kubectl get configmaps -n monitoring -l grafana_datasource

4. For "No data" or a failing datasource, check the address it uses against the Services that exist.

kubectl get services -A | grep -i -E "prometheus|loki|tempo"
kubectl get endpointslices -n monitoring
kubectl get --raw '/api/v1/namespaces/monitoring/services/my-grafana:http/proxy/api/health'

The last command needs get on services/proxy; if it is forbidden, skip it.

5. Open the matching reference and follow its steps:

What you see Reference
CrashLoopBackOff, Datasource provisioning error, failed to parse, a datasource or dashboard missing, sidecar not picking up ConfigMaps references/provisioning.md
"No data", Bad Gateway, connection refused, wrong datasource UID in a dashboard references/datasources-no-data.md
database is locked, permission errors, plugins, login and 401 Unauthorized, grafana-operator NoMatchingInstance references/runtime-and-operator.md

Read the results

What the common findings mean (Grafana 13.2):

  • The new pod exits with Error: ✗ invalid service state: Failed ... Datasource provisioning error: <reason>: Grafana refuses a provisioned datasource file and stops; the pod crash-loops. datasource.yaml config is invalid. Only one datasource per organization can be marked as default means two datasources have isDefault: true. A YAML error or an unknown field shows up the same way.
  • logger=provisioning.dashboard ... level=error msg="failed to load dashboard from " file=<path> error=<JSON error>: that dashboard file is not valid JSON. Grafana skips it and keeps running, so only that dashboard is missing.
  • the same UID is used more than once or dashboard title is not unique in folder: two files define the same dashboard; only one wins.
  • A datasource whose URL names a Service that does not exist (or the wrong port): Grafana starts normally; every query fails, shown in the UI as "Bad Gateway" or a connection error, and in the log only when a query runs.
  • /api/health returns "database": "ok": Grafana itself is up; failing (HTTP
    1. means its database is not reachable.

State the cause in one sentence, then the evidence you saw (the log line, the ConfigMap entry, the Services), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Provisioning lives in ConfigMaps that come from a Helm chart, kube-prometheus-stack values or a GitOps repository: fix it there. Patching a ConfigMap that only holds provisioning or dashboards, and restarting Grafana to reload it, are reversible, so they go under "Ask first" and run only after the user says yes. Anything with Secrets (admin password, datasource credentials), Grafana's database, its volume and the replica count are never run by you: hand them to the user as proposed fixes. Write each undo with the values you found.

Restart Grafana after its provisioning ConfigMap is fixed (Grafana reads datasource files only at start):

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-grafana -n monitoring
# Undo: kubectl rollout undo deployment/my-grafana -n monitoring

Sources

Checked against the Grafana 13.2 documentation and source (v13.2.4), the Grafana Helm chart 14.0 and grafana-operator 5.25: