Before You Upgrade: Auditing Your Proxies and Meshes for the kubectl WebSockets Transport

The worst upgrade incidents are the ones where nothing is actually down. The API server is healthy, every workload is running, your dashboards are green, and yet three engineers are on a call because kubectl exec into a production pod hangs and kubectl port-forward dies after a minute. Nobody changed the cluster networking. What changed is the HTTP handshake kubectl uses to open a streaming connection, and something in the path between the laptop and the API server has an opinion about it.

That is the shape of the SPDY to WebSockets transition (KEP-4006). It is not a workload problem and it is not an API server problem. It is a problem with the hops you do not own: the corporate egress proxy, the reverse proxy in front of the API server endpoint, the cloud load balancer, the mesh proxy. This post is the audit I would run before touching a cluster version, plus the per-hop knobs that actually matter.

Who Should Read This?#

This post is for:

• SREs who own Kubernetes cluster upgrades and whose kubectl traffic crosses at least one hop they do not control
• Platform Engineers running a mesh, an ingress controller, or a bastion proxy in front of cluster endpoints
• Anyone whose incident runbooks start with “exec into the pod and check”

What actually changed#

kubectl exec, kubectl attach, kubectl cp (which is built on the exec primitives) and kubectl port-forward all need a long-lived, bi-directional stream. Historically they got one by upgrading an HTTP/1.1 request to SPDY. Per KEP-4006, SPDY “has been deprecated since 2015, and by now many proxies, gateways, and load-balancers do not support SPDY”. WebSockets is the replacement, and it brings two new subprotocol versions:

Both transports start as a plain HTTP request and end with 101 Switching Protocols. That is where the similarity stops, and the difference is exactly what an intermediary sees.

Where this actually stands today#

This is the part most write-ups get wrong, so read the versions carefully.

The KEP’s own metadata (kep.yaml) lists stage: stable, latest-milestone: v1.38 and milestone.stable: v1.38, and the GA promotion PR (kubernetes/enhancements#6364) merged on 2026-09-21 against the v1.38 milestone. But the feature gate reference still lists every gate involved as Beta. So: GA is targeted at v1.38, not shipped.

That sounds like you have time. You do not, and here is why:

Why This Matters: On client and API server v1.31 or newer, kubectl is already attempting WebSockets first for all four commands (v1.30 covers exec, attach and cp; port-forward flipped on both sides in v1.31). Where a hop rejects the upgrade, kubectl quietly falls back to SPDY and the command still works, so a broken hop is invisible to you right now. The thing GA takes away is not the old default, it is that fallback and the escape hatch around it. Per the KEP, at GA kubectl “no longer consults” the two environment variables, prints a deprecation warning if either is set, and always attempts WebSockets first. The four API server gates are “promoted to GA and locked to ON (LockToDefault: true)”.

So the real risk is not that the transport changes on upgrade day. It is that the transport already changed, a silent SPDY fallback has been papering over a broken hop, and GA removes your ability to force the old path while you fix it. Silent fallback is the reason to audit today rather than after 1.38: it means the hop that will break your upgrade is already in your path and already failing, and nothing in your dashboards is telling you.

Why an intermediary cares#

The two handshakes look different on the wire. This is real kubectl --v=8 output captured by a user on kubernetes/kubernetes#126134, not a reconstruction. First, SPDY from a v1.29 client:

round_trippers.go:463] POST https://.../api/v1/namespaces/default/pods/alpine-.../exec?command=sh&container=alpine&stdin=true&stdout=true&tty=true
round_trippers.go:469] Request Headers:
round_trippers.go:473]     User-Agent: kubectl/v1.29.2 (darwin/arm64) kubernetes/4b8e819
round_trippers.go:473]     X-Stream-Protocol-Version: v5.channel.k8s.io
round_trippers.go:473]     X-Stream-Protocol-Version: v4.channel.k8s.io
round_trippers.go:473]     X-Stream-Protocol-Version: v3.channel.k8s.io
round_trippers.go:473]     X-Stream-Protocol-Version: v2.channel.k8s.io
round_trippers.go:473]     X-Stream-Protocol-Version: channel.k8s.io
round_trippers.go:574] Response Status: 101 Switching Protocols in 403 milliseconds
round_trippers.go:577] Response Headers:
round_trippers.go:580]     Upgrade: SPDY/3.1
round_trippers.go:580]     X-Stream-Protocol-Version: v4.channel.k8s.io

Now the same command with a v1.30 client, WebSockets, through the same HTTPS proxy:

round_trippers.go:463] GET https://.../api/v1/namespaces/default/pods/alpine-.../exec?command=sh&container=alpine&stdin=true&stdout=true&tty=true
round_trippers.go:469] Request Headers:
round_trippers.go:473]     Sec-Websocket-Protocol: v5.channel.k8s.io
round_trippers.go:473]     User-Agent: kubectl/v1.30.2 (darwin/arm64) kubernetes/3968350
round_trippers.go:574] Response Status:  in 0 milliseconds
round_trippers.go:577] Response Headers:
error: proxy: unknown scheme: https

Three things changed, and each one is a place an intermediary can say no:

1. The HTTP method. POST became GET. The feature gate reference spells this out: “while SPDY requests utilize HTTP POST (naturally aligning with the create RBAC permission), the WebSocket protocol requires an HTTP GET request for the handshake.”
2. The subprotocol negotiation header. X-Stream-Protocol-Version (a Kubernetes invention) became Sec-Websocket-Protocol (RFC 6455 standard).
3. The upgrade token. Upgrade: SPDY/3.1 became the standard WebSocket upgrade.

Point 1 is the one people miss. Any policy keyed on the HTTP verb, a proxy ACL, a WAF rule, an egress allow-list that permits POST /exec but not GET /exec, now matches the wrong thing.

The one question that scopes the whole audit: who sees plaintext HTTP?#

Before you start grepping proxy configs, narrow the field. kubectl speaks TLS to the API server. A hop that does not terminate that TLS session cannot see the Upgrade header, cannot inspect the method, and cannot reject the handshake. It is moving opaque bytes.

Be precise about the criterion, because it is “parses HTTP”, not “terminates TLS”. An L4 hop that terminates TLS and then forwards the decrypted bytes without ever parsing a request, an AWS NLB TLS listener being the obvious example, still cannot touch your Upgrade header. It goes in the opaque bucket: it can time you out, it cannot reject you.

flowchart TB
    subgraph Client["Client side"]
        K[kubectl]
    end
    subgraph Path["Hops you may not control"]
        P["Corporate proxy<br/>CONNECT tunnel: L4, opaque<br/>TLS-intercepting: L7, sees the upgrade"]
        M["Mesh sidecar / ztunnel<br/>L4 for client-originated TLS"]
        L["Cloud LB<br/>NLB/L4: opaque, even with TLS termination<br/>ALB/L7: parses HTTP"]
        I["Reverse proxy or ingress<br/>L7 only if it terminates TLS and parses HTTP"]
    end
    subgraph Cluster["Kubernetes"]
        A[kube-apiserver]
        N[kubelet]
        C[container runtime]
    end
    K --> P --> M --> L --> I --> A
    A --> N --> C

That diagram is the superset, not a path. Yours has some subset of those hops in some order: kubectl running in a meshed pod hits the sidecar first and has no corporate proxy at all, while a laptop on the VPN may have the proxy and nothing else.

That gives you two buckets, and they fail in completely different ways:

Build this inventory first. For most clusters the list of HTTP-parsing hops in the kubectl path is zero or one, and the audit collapses to a single config file.

Audit step 1: which transport is your client actually negotiating?#

Per the kubectl quick reference, --v=7 gives you “Display HTTP request headers” and --v=8 gives you “Display HTTP request contents”. Level 7 is enough:

# Watch the upgrade on the /exec subresource. Non-interactive, so it exits on its own.
kubectl exec --v=7 deploy/my-app -- true 2>&1 | grep -E "exec\?|Switching Protocols|Sec-Websocket|X-Stream|Upgrade:"

Read the request line for the /exec subresource:

• GET .../exec?... with Sec-Websocket-Protocol: v5.channel.k8s.io means WebSockets
• POST .../exec?... with X-Stream-Protocol-Version: headers means SPDY

Same for port-forward, where you are looking for v2.portforward.k8s.io versus v1.portforward.k8s.io:

kubectl port-forward --v=7 deploy/my-app 8080:8080 2>&1 | grep -E "portforward\?|Switching Protocols|Sec-Websocket|X-Stream"

KEP-4006’s own troubleshooting section uses the same trick for latency, noting that kubectl exec -v=7 produces lines like Response Status: 101 Switching Protocols in 20 milliseconds. That number is your per-hop handshake budget, and it is worth recording before you change anything.

Tip: Run this from every place that execs into pods, not just your laptop. CI runners, bastion hosts, an operator’s pod, the on-call engineer on the VPN. They take different paths and they will not all agree.

The fleet-wide view#

KEP-4006 registers four metrics you can scrape instead of asking every engineer to run --v=7. Check which ones your version actually has before you draw conclusions from them:

Below v1.36 the first two are your only signal, and the KEP is explicit about how to read them: a non-zero count with a 101 status code means clients are streaming over WebSockets. From v1.36 the API server can hand a WebSocket stream straight to the kubelet without translating or tunnelling it, and the KEP’s help text says these two counters only count requests the translator and tunnel proxies handled. So on v1.36 or newer use apiserver_websocket_streaming_requests_total summed over both proxy_type values as the primary signal, and expect the two stream_* counters to fall as your kubelets reach v1.36.

# Successful WebSocket upgrades, exec and attach.
sum(rate(apiserver_stream_translator_requests_total{code="101"}[5m]))

# Successful WebSocket upgrades, port-forward.
sum(rate(apiserver_stream_tunnel_requests_total{code="101"}[5m]))

# v1.36 and newer: WebSocket streams by how they were served.
sum(rate(apiserver_websocket_streaming_requests_total[5m])) by (proxy_type)

On an API server older than v1.36, if both stay at zero while engineers are execing all day, your clients really are still on SPDY, and the GA upgrade will be a step change rather than a no-op. On v1.36 or newer, zero on these two proves nothing on its own: check apiserver_websocket_streaming_requests_total before you draw that conclusion.

Note: Do not run this check against apiserver_websocket_streaming_requests_total unless your API server is v1.36 or newer. The two *_websocket_streaming_* counters shipped in v1.36, so on anything older they do not exist, and an absent counter looks exactly like a counter pinned at zero. That is the fastest way to talk yourself out of an audit you need.

Audit step 2: A/B both transports against the same cluster#

This is the test that actually proves a hop is safe, and it is available only while the gates are still Beta. Run it now.

The client-side switches are environment variables, not flags:

# Force the legacy SPDY path for exec, attach and cp.
KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false kubectl exec --v=7 -it deploy/my-app -- sh

# Force the legacy SPDY path for port-forward.
KUBECTL_PORT_FORWARD_WEBSOCKETS=false kubectl port-forward --v=7 deploy/my-app 8080:8080

# Default behaviour on any client v1.30+ (exec) or v1.31+ (port-forward): WebSockets first.
kubectl exec --v=7 -it deploy/my-app -- sh

Interpret the result honestly:

If you need to take the whole cluster back to SPDY while you fix a hop, the server-side gates do that for every client at once:

# On kube-apiserver, while these gates are still Beta and unlocked.
--feature-gates=TranslateStreamCloseWebsocketRequests=false,PortForwardWebsockets=false

Turning those off makes the API server reject the WebSocket upgrade, which triggers kubectl’s FallbackExecutor to retry over SPDY. The KEP is explicit that this is temporary: at GA these gates are locked on, and removed from the gate list “no earlier than v1.41”.

Note: Treat the environment variables and the server gates as an incident lever with an expiry date, not a configuration choice. Anything in your automation that sets KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false or KUBECTL_PORT_FORWARD_WEBSOCKETS=false is a ticking deprecation warning. Grep your Dockerfiles, CI images and shell profiles for both names now.

Audit step 3: the RBAC change nobody warned you about#

GET instead of POST created a real authorization gap, and AuthorizePodWebsocketUpgradeCreatePermission (Beta, default true, since v1.35) closes it. From the feature gate reference:

clients must be authorized to create Pod subresources even when triggering their creation using a WebSocket. The connection upgrade request occurs for each of the following subresources: pods/exec, pods/attach, and pods/portforward. […] a synthetic RBAC check is now applied to ensure WebSocket upgrades strictly enforce the create permission, matching the existing SPDY security model.

The same doc names the blast radius: “You may want to disable this feature gate if you have existing clients or custom tooling that rely on the previous behavior, specifically, if they connect via WebSockets but do not currently hold the create RBAC permission.”

So audit your Roles. Anything with get but not create on the exec subresources worked over WebSockets on a pre-1.35 cluster and stops working now:

# Find roles that grant the exec/attach/portforward subresources.
kubectl get clusterroles,roles -A -o json \
  | jq -r '.items[] | select(any(.rules[]?.resources[]?; test("pods/(exec|attach|portforward)"))) | "\(.kind)/\(.metadata.namespace // "-")/\(.metadata.name)"'

For each hit, confirm the verbs list contains create, not just get. The usual suspects are hand-rolled Roles for debugging tools, web terminals, and CI service accounts.

# Correct. create is what the WebSocket upgrade is authorized against.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debug-exec
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec", "pods/portforward"]
    verbs: ["create", "get"]

Audit step 4: per-hop notes#

nginx (ingress controller or plain reverse proxy)#

Note: ingress-nginx was retired in March 2026 and the repo is archived: no releases, bugfixes or security patches. The timeout advice below still applies to clusters that run it, but treat it as a reason to plan the move to a Gateway API implementation, not as a long-term fix.

The handshake itself is the easy part. The ingress-nginx docs are blunt about it:

Support for websockets is provided by NGINX out of the box. No special configuration required.

The timeouts are the problem. The same page continues: “The only requirement to avoid the close of connections is the increase of the values of proxy-read-timeout and proxy-send-timeout. The default value of these settings is 60 seconds. A more adequate value to support websockets is a value higher than one hour (3600).”

Sixty seconds is exactly long enough for an engineer to open a shell, read a log, think, and get disconnected mid-sentence.

# Per-Ingress override. Values are unitless seconds.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cluster-api
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
# ...

Result: long-lived exec and port-forward sessions survive an hour of idleness instead of a minute.

One more line from that page matters if the controller sits in front of anything TLS-sensitive: “If the Ingress-Nginx Controller is exposed with a service type=LoadBalancer make sure the protocol between the loadbalancer and NGINX is TCP.”

Envoy, and therefore Istio#

Good news first. Per the Envoy upgrades documentation, upgrade support at the HTTP connection manager is enabled by default (its table shows T (Default) for both the HCM and per-route settings), and “Upgrades pass both the HTTP headers and the upgrade payload through an HTTP filter chain”. A default Envoy hop does not need to be told about WebSockets.

The traps are the non-default configurations:

Envoy also gives you the counters to prove it: downstream_cx_upgrades_total and downstream_cx_upgrades_active. Watch those on the hop while you run the A/B from step 2. If downstream_cx_upgrades_total does not move, the upgrade never reached the filter chain.

For Istio specifically:

• Idle timeout. ConnectionPoolSettings.HTTPSettings.idleTimeout in a DestinationRule is “the idle timeout for upstream connection pool connections. The idle timeout is defined as the period in which there are no active requests. If not set, the default is 1 hour.” One hour is survivable, but it is not infinite, and an interactive shell is idle by definition.
• Sidecar mode. The sidecar is Envoy, so the defaults above apply. Crucially, kubectl originates its own TLS to the API server, so a sidecar in that path is usually looking at an opaque TCP stream, not an HTTP upgrade. It is the timeouts that reach you, not the handshake.
• Ambient mode. Per the Istio data plane modes docs, ambient routes all traffic through “a Layer 4-only node proxy” (ztunnel), and applications “opt in to routing through an Envoy proxy to get Layer 7 features” (a waypoint). An L4-only ztunnel hop cannot break a WebSocket upgrade. A waypoint is a full Envoy and belongs in your HTTP-parsing bucket. I covered the ztunnel and waypoint split in more depth in my Istio Ambient Mesh post.

Cloud load balancers#

If your API server endpoint sits behind a cloud load balancer, the idle timeout is where you will feel this, and the L4 versus L7 choice decides whether the handshake is even visible.

Two details from the AWS NLB documentation worth internalising. First, when the timeout fires you do not get a clean close: “If a client or target sends data after the idle timeout period elapses, the client receives a TCP RST packet to indicate that the connection is no longer valid.” A TCP RST in the middle of an exec session looks like a broken cluster, not a proxy timeout. Second, on a TLS listener the load balancer generates keepalives every 20 seconds once it sees one, but “keepalive packets sent to maintain TLS connections can’t contain data or payload”.

An ALB in this path is the one to scrutinise: it terminates TLS, it parses HTTP, and its default idle timeout is 60 seconds.

That table is AWS only. Every managed load balancer has an equivalent knob under a different name, so if you are on Google Cloud, Azure or anywhere else, look up your own provider’s idle timeout default and whether the product parses HTTP, and slot it into the two buckets above before you start changing configs.

HTTP proxies: proxy-url, HTTPS_PROXY, NO_PROXY#

This hop has a concrete, version-specific trap, and it is the reason the --v=8 output above exists.

The kubeconfig reference documents the client behaviour: proxy-url “is the URL to the proxy to be used for all requests made by this client. URLs with ‘http’, ‘https’, and ‘socks5’ schemes are supported. If this configuration is not provided or the empty string, the client attempts to construct a proxy configuration from http_proxy and https_proxy environment variables.”

Two findings from that:

1. https:// proxy URLs were broken for exec. kubernetes/kubernetes#126134 is titled “Kubectl Versions >= 1.30 Don’t Allow Exec When HTTPS Scheme in proxy-url”. kubectl get worked; kubectl exec returned error: proxy: unknown scheme: https. The first fix, PR #126231 (“Falls back to SPDY for gorilla/websocket https proxy error”, merged into the v1.31 milestone), did not make WebSockets work through an HTTPS proxy: it made kubectl fall back to SPDY. Real support came later. KEP-4006’s GA section states that “WebSocket support for HTTPS proxies shipped in v1.33”.

Why This Matters: If your kubeconfig uses an https:// proxy URL and your clients are between v1.31 and v1.32, exec is working today because of the SPDY fallback. GA removes the environment-variable escape hatch and locks the server gates on. Get those clients to v1.33 or newer before you do anything else in this audit.

2. SOCKS5 users start from a worse place. The same kubeconfig reference notes: “socks5 proxying does not currently support spdy streaming endpoints (exec, attach, port forward).” If you proxy through SOCKS5, exec never worked over SPDY. WebSockets is the path that could unblock you, but the docs only rule out SPDY, they do not promise the new transport works: prove it with the A/B in step 2 before you plan around it.

Finally, the boring but effective option: take the proxy out of the path. If the API server endpoint is reachable directly, add it to NO_PROXY and confirm the A/B test from step 2 passes on both transports. That converts a hop you do not control into no hop at all.

What a failed upgrade actually looks like#

This is the part that costs you the most time on the call, so read KEP-4006’s failure-mode entry carefully:

Failure Mode: Proxy or Gateway that does not support SPDY or WebSockets. […] Detection: The kubectl streaming command will return a connection error. The error returned to the client will be from the proxy or gateway; not from the API Server (since the communication never reaches the API Server). Mitigations: None.

Three practical consequences:

1. The error text belongs to the proxy, not to Kubernetes. You will be searching for a stranger’s error string. error: proxy: unknown scheme: https is not a Kubernetes message.
2. The API server logs are empty. The request never arrived. Do not burn thirty minutes correlating API server audit logs. Go to the proxy, ingress or sidecar access logs and look for the request to the /exec or /portforward subresource, and for whether the response was 101 or something else.
3. “Mitigations: None.” For a hop that supports neither transport there is no flag to flip. That is precisely why this audit happens before the upgrade.

Matched against the hop, the symptoms sort out like this:

What to do now#

In order. None of this needs a maintenance window.

1. Inventory the hops between every kubectl origin (laptops, CI, bastions, operator pods) and the API server. Mark which ones see plaintext HTTP. Only those can break the handshake; the rest can only time you out.
2. Check your client versions. Anything older than v1.33 behind an https:// proxy URL is a known landmine. Pin a client version in CI images so the transport does not change under you mid-audit.
3. Run the A/B from step 2 on every path. KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false and KUBECTL_PORT_FORWARD_WEBSOCKETS=false are your control group, and they expire at GA.
4. Fix the hop, in this order of likelihood: idle timeouts first (60 seconds is the default almost everywhere and the default is wrong for interactive shells), then upgrade passthrough, then buffering, then HTTP/2 hops without Extended CONNECT.
5. Audit RBAC for create on pods/exec, pods/attach and pods/portforward.
6. Grep your repos for KUBECTL_REMOTE_COMMAND_WEBSOCKETS and KUBECTL_PORT_FORWARD_WEBSOCKETS. Every occurrence is work you owe yourself before the gates lock.
7. Then upgrade, and watch apiserver_stream_translator_requests_total{code="101"} and apiserver_stream_tunnel_requests_total{code="101"} alongside it. On v1.36 or newer, make apiserver_websocket_streaming_requests_total and kubelet_websocket_streaming_requests_total the primary pair instead; the two stream_* counters only count the share still translated at the API server, and that share shrinks as kubelets are upgraded. On anything older those two do not exist.

Conclusion#

The version number in this story is the least interesting part of it, and it is the part most likely to be stale by the time you read this. GA is targeted at v1.38 in the KEP metadata; the feature gates are still Beta in the docs; those two facts will converge on their own schedule.

What will not change is the shape of the problem. Kubernetes swapped a bespoke, long-deprecated upgrade handshake for a standard one, and in doing so moved the failure domain from “does my cluster work” to “does every intermediary in the path handle an HTTP upgrade correctly”. The transition is a net win: SPDY was a Kubernetes-only protocol that ordinary L7 infrastructure had every reason not to support, and WebSockets is not. But the win lands on infrastructure you probably do not own, and the automatic SPDY fallback has been hiding the bill.

Two things to take away. First, on client and server v1.31 or newer you are attempting WebSockets first and silently falling back to SPDY wherever a hop rejects the upgrade, so this audit is a check on what is happening today, not a forecast. If your kubeconfig uses an https:// proxy URL, the floor is v1.33, because that is when WebSockets started working through an HTTPS proxy rather than falling back. Second, the single highest-value thing in this entire post is not a feature gate: it is finding out which hops in your path see plaintext HTTP, and raising the idle timeouts above 60 seconds on every hop in the path, opaque or not. Do that this week, and upgrade day stays boring.

If you found this useful, you might also enjoy my related posts:

• Simplifying Kubernetes Service Mesh - A Deep Dive into Istio Ambient Mesh
• Understanding Kubernetes Gateway API
• Ephemeral Containers