k8s-istio-troubleshooting

Use when traffic fails in an Istio service mesh on Kubernetes: HTTP 403 "RBAC: access denied" from an AuthorizationPolicy, 503 "no healthy upstream" or "upstream connect error or disconnect/reset befo

SKILL.md

GitHub

Istio troubleshooting

Find out why requests fail or behave unexpectedly in an Istio mesh, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

In sidecar mode every meshed pod has an Envoy container, istio-proxy, injected when its namespace has istio-injection=enabled (or an istio.io/rev label). The client's sidecar routes by VirtualServices and DestinationRules (subsets, load balancing, TLS); the server's sidecar enforces PeerAuthentication (mTLS) and AuthorizationPolicies. Each sidecar writes an access log line per request with a response flag that says what Envoy did. In ambient mode the node's ztunnel handles L4 and mTLS and an optional waypoint proxy handles L7.

The commands use placeholders: my-namespace, my-app (the server and its Service), my-web (the client) and my-pod. Replace them with the real names. The built-in view role cannot read Istio's resources (Istio adds nothing to it, and its own reader role also grants Secrets); if they are forbidden, say so and ask the user for them. Never read Secrets: not certificates, not istio-ca-secret, not gateway TLS Secrets. For that reason do not run istioctl analyze against the cluster (it reads Secrets), do not use istioctl proxy-config secret, and do not fetch Envoy's full config_dump or /certs.

When to use

  • A request inside the mesh, or through an Istio gateway, returns 403, 404, 503 or a reset, while the app itself is fine.
  • Routing rules (weights, subsets, headers) or policies do not take effect.
  • A pod has no istio-proxy container although its namespace is in the mesh.
  • istiod is down, or proxies do not get configuration.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The Service has no endpoints, or the pods are not Ready: k8s-network-troubleshooting and k8s-workload-troubleshooting.
  • A Kubernetes Gateway API route that Istio does not accept: k8s-gateway-api-troubleshooting (the objects' status) and the reference here.
  • What an Envoy response flag, reset reason or cluster statistic means in general: k8s-envoy-troubleshooting (the proxy itself; Istio writes its config).

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Check the mesh membership of the client and the server: the namespace labels, and whether each pod has an istio-proxy container.

kubectl get namespace my-namespace --show-labels
kubectl get pods -n my-namespace -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].name}{" "}{.spec.initContainers[*].name}{"\n"}{end}'
kubectl get pods -n istio-system

2. Read the client's error and both sidecars' access logs. The response flag after the status code says what went wrong.

kubectl logs -n my-namespace deployment/my-web -c istio-proxy --tail=20
kubectl logs -n my-namespace deployment/my-app -c istio-proxy --tail=20

A log line reads "GET / HTTP/1.1" 503 UH no_healthy_upstream ... outbound|80|v2|my-app...: status, response flag, details, and the Envoy cluster (direction, port, subset, host). Access logs must be on (meshConfig.accessLogFile, or a Telemetry resource); if they are empty, say so.

3. Read the Istio resources for the host and the server.

kubectl get virtualservices,destinationrules,gateways,serviceentries,sidecars -A
kubectl get authorizationpolicies,peerauthentications,requestauthentications -A
kubectl get virtualservice my-app -n my-namespace -o yaml
kubectl get destinationrule my-app -n my-namespace -o yaml
kubectl get pods -n my-namespace --show-labels

4. If the cause is still open, ask the proxy itself through its admin API, with GET only, on the pod that logs the error. This runs a program in the user's pod, so ask first:

Ask first: the agent runs this only after the user approves it.

kubectl exec my-pod -n my-namespace -c istio-proxy -- pilot-agent request GET clusters
kubectl exec my-pod -n my-namespace -c istio-proxy -- pilot-agent request GET stats

clusters lists each Envoy cluster with its endpoints and their health. Never send any method other than GET, and never request config_dump without a resource filter or certs.

5. Open the matching reference and follow its steps:

What you see Reference
503 UH, UF, URX, 404 NR, upstream connect error, routing that does not apply, subsets, ServiceEntries, gateways references/routing.md
403 RBAC: access denied, Connection reset by peer, filter_chain_not_found, mTLS, injection, istiod, ambient references/security-and-injection.md

Read the results

What the common findings mean (Istio 1.31):

  • 403 with the body RBAC: access denied, and the server's sidecar logs 403 - rbac_access_denied_matched_policy[...]: an AuthorizationPolicy denies it. [none] means an ALLOW policy selects the workload and no rule matched this request (a principal, namespace, path or method that is not listed).
  • 503 UH no_healthy_upstream on the client's sidecar, with a subset in the cluster name (outbound|80|v2|...): the route points at a DestinationRule subset whose labels match no ready pod.
  • 404 NR (no route) on the client or a gateway: no VirtualService or route matches the host, path or port.
  • upstream connect error or disconnect/reset before headers. reset reason: ... (with UF or URX): Envoy could not connect to, or keep, the upstream: the app is down, the port is wrong, or the mTLS settings of client and server disagree.
  • The client gets Connection reset by peer, and the server's sidecar logs NR filter_chain_not_found: the server requires mTLS (PeerAuthentication STRICT) and the client sent plaintext, usually because it has no sidecar.

State the cause in one sentence, then the evidence you saw (the access log line and its flag, the resource and the field), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Istio resources change traffic for every client of a host, and policies change who may talk to whom: never create, change or delete them yourself; hand the change to the user as a proposed fix, in the source they are deployed from. Restarting a workload so it gets a sidecar is reversible, so it goes under "Ask first". Write each undo with the values you found.

Restart a workload after its namespace was added to the mesh, so its pods get a sidecar:

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-web -n my-namespace
# Undo: kubectl rollout undo deployment/my-web -n my-namespace

Sources

Checked against the Istio 1.31 documentation and source (1.31.3):