First istioctl tool during an incident
istioctl proxy-status lists every proxy with its
istiod and version; since 1.27 the sync state per xDS type only shows
with -v 1 (or -o json). SYNCED = Envoy ACKed the last push,
NOT SENT = istiod has nothing to send (often normal,
e.g. no routes), STALE = sent but not ACKed →
network istiod↔proxy or an Istio bug: check the istiod logs.
istioctl proxy-status -v 1 istioctl proxy-config routes <pod>.<ns> -o json istioctl proxy-config clusters <pod>.<ns> istioctl proxy-config listeners <pod>.<ns> istioctl proxy-config endpoints <pod>.<ns> istioctl proxy-config secrets <pod>.<ns> # Ambient (also: policy, certificate) istioctl ztunnel-config workload istioctl waypoint status -n prod # Static lint: VirtualService/DR conflicts etc. istioctl analyze -n prod # Bug report (not anonymised; secrets only w/ --full-secrets) istioctl bug-report
Live log of a sidecar
istioctl proxy-config log <pod> --level debug
sets the log level live without a pod restart. Component-specific,
e.g. --level rbac:debug,jwt:debug. After
diagnosis, set it back to warning.
Common failure modes
UF (503): upstream connection failure: wrong
targetPort, or mTLS conflict (DR tls.mode
DISABLE vs. STRICT server).
UC (503): upstream terminated the connection:
keep-alive/idle-timeout race, or pod-IP call without a route at the
server sidecar.
NR: no route configured: VirtualService host/match
order or DR subset (404 when no path matches).
UO (503): circuit breaker,
connectionPool limits of the DR.
UH (503): no healthy upstream host (all ejected by
outlier detection or none ready).
UT (504): upstream request timeout (route
timeout or perTryTimeout).
OUTPUT_CERTS scraping: read: connection reset by peer
App-originated mTLS (Prometheus/Alloy with
proxyMetadata.OUTPUT_CERTS) gets an RST although the
target is PERMISSIVE.
Cause: the client sidecar wraps the app mTLS in a
second ISTIO_MUTUAL tunnel, typically via a DR
host: "*.local" + exportTo: ["*"] (STRICT
migration helper) that also matches direct pod-IP calls.
15006 strips only the outer layer, the app port receives
TLS bytes. PERMISSIVE only decides inbound.
Fix: excludeOutboundPorts (or
includeOutboundIPRanges: "") on the scraper pod, or a
more specific DR with tls.mode: DISABLE for the target.
1.31+ (opt-in): ENVOY_SECURE_METRICS_PORT /
ENVOY_SECURE_MERGED_METRICS_PORT add mTLS scrape
listeners.
# client wraps? sni=outbound_* + alpn istio* = proof
istioctl pc cluster <pod>.<ns> --fqdn <target> -o json \
| jq '.[].transportSocket.typedConfig
|{sni, alpn: .commonTlsContext.alpnProtocols}'