Agent Sandbox - Kubernetes-Native Sandboxes for AI Agent Code Execution

A few months ago I built a Kubernetes agent with Google’s ADK for Go. That agent talked to the cluster: it ran kubectl against my kind cluster and explained what it saw. This post is the inverse problem, and it is the one that actually keeps me up at night: when an LLM writes code, where does that code run?

The honest answer for most agent stacks today is “on whatever machine the agent process happens to be on.” The LLM generates a shell command or a Python script, the framework execs it, and everyone hopes the model never emits rm -rf or something that exfiltrates the credentials sitting in the environment. Hosted sandboxes like E2B and Modal exist precisely because of this, but if you already operate Kubernetes, shipping your agents’ code execution to a third-party SaaS feels backwards. You have a scheduler, you have network policy, you have runtimes with kernel isolation. What you were missing was the workload abstraction.

That gap is what agent-sandbox fills. It is a Kubernetes SIG Apps project that ships a Sandbox CRD and controller for isolated, stateful, singleton workloads - the “lightweight single-container VM” shape that neither Deployments nor StatefulSets model well - plus warm pools so an agent gets a sandbox in milliseconds instead of tens of seconds. It hit v0.5.6 yesterday, ships both Go and Python SDKs, and runs on a plain kind cluster. I built a Go agent whose every tool call executes inside a sandbox pod, wired the whole thing with Flux, and hit enough sharp edges along the way to make this write-up worth your time. The demo lives in srekubecraft-demo/agent-sandbox/.

Who Should Read This?#

This post is for:

• Platform Engineers who are being asked to provide “somewhere safe for agents to run code” and want it inside the cluster, not on a SaaS
• SREs who found out the hard way that an agent framework’s subprocess.run() executes on the node with the pod’s service account token mounted
• Go developers building agents with the Anthropic or OpenAI SDKs who need an execution backend with real isolation
• Anyone evaluating E2B, Modal, or Daytona who already operates Kubernetes and wants to know what the self-hosted, CNCF-adjacent option looks like
• Teams running coding agents at scale (CI bots, code interpreters, RL environments) that need hundreds of short-lived, stateful environments with cleanup guarantees

TL;DR#

• agent-sandbox is a Kubernetes SIG Apps project: a Sandbox CRD + controller for long-running, stateful, singleton containers with stable identity, plus extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) for pooling and self-service
• It is a sandbox orchestrator, not a sandbox runtime: kernel-level isolation is delegated to gVisor or Kata Containers via runtimeClassName; on kind you get plain container isolation
• The Go SDK (sigs.k8s.io/agent-sandbox/clients/go/sandbox) gives you CreateSandbox, Run, Read/Write/List, three connection modes, and a Handle interface for mocking
• Warm pools work: claiming a pre-warmed sandbox took 53ms in my kind cluster, versus ~25s to create one from scratch (image pull + schedule + readiness)
• TTL-based garbage collection via spec.lifecycle.shutdownTime means a crashed agent process cannot leak sandboxes - the controller deletes them
• The docs have gaps: the source tree’s k8s/ overlay ships unresolvable ko:// image refs, the install guide and the Go SDK disagree about which namespace the router lives in, and the controller’s auto-created NetworkPolicy will silently 504 you. All three are covered below.

What is Agent Sandbox?#

Agent Sandbox is a CRD and controller developed under Kubernetes SIG Apps, with the code at kubernetes-sigs/agent-sandbox. The core resource is deceptively simple:

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: my-sandbox
spec:
  podTemplate:
    spec:
      containers:
        - name: workspace
          image: python:3.13-slim

One Sandbox, one pod, stable hostname, optional persistent storage, and lifecycle management (scheduled shutdown, pause/resume) handled by the controller. The motivation section of the docs names the use cases directly: AI agent runtimes executing untrusted LLM-generated code, cloud development environments, notebooks, and stateful single-pod services that do not justify a StatefulSet.

The project is explicit about its scope, and this is the most important sentence in the docs:

Agent Sandbox is a sandbox orchestrator. It delegates low-level container isolation to secure “Sandbox Runtimes” (like gVisor or Kata Containers) by managing Pods configured to use these runtimes (via RuntimeClass).

So agent-sandbox gives you the API, the pooling, and the lifecycle. The syscall boundary is whatever your runtimeClassName provides. On a laptop kind cluster that is plain runc - fine for a demo, not for hostile multi-tenancy.

Why Not Just Pods, Jobs, or a StatefulSet?#

You can absolutely hand-roll this with a StatefulSet of size 1, a Service, and a PVC. People do. The comparison explains why a dedicated abstraction won:

Why Not E2B or Modal?#

Bottom line: if you have no Kubernetes footprint, a hosted sandbox is less work. If you already run clusters with GitOps, network policy, and a security review process, agent-sandbox turns “where do agents run code” into a workload type you operate like everything else.

What Agent Sandbox is NOT#

• Not a sandbox runtime - it orchestrates pods onto gVisor/Kata; it does not provide kernel isolation itself
• Not an agent framework - no LLM loop, no tools, no prompts; bring LangGraph, ADK, or (as below) a plain Anthropic SDK tool runner
• Not a code interpreter service - the /execute server in the sandbox image is a reference runtime, not a product
• Not multi-cluster - one controller, one cluster; fleet-level scheduling is on you
• Not GA - v1beta1 API as of v0.5.6, under active development with a real deprecation cycle (v1alpha1 was removed with a migration guide)

Core Concepts#

Four CRDs and one deployment do all the work:

• Sandbox (agents.x-k8s.io/v1beta1): the singleton workload. Holds the pod template, volumeClaimTemplates, operatingMode (Running/Suspended), and the lifecycle fields: shutdownTime (absolute expiry) and shutdownPolicy (Retain keeps the object, Delete garbage-collects it)
• SandboxTemplate (extensions): the admin-owned blueprint - image, resources, runtime class, storage. When a sandbox is provisioned through a template, the controller defaults automountServiceAccountToken to false. Untrusted code gets no API server credentials unless you opt in
• SandboxWarmPool (extensions): keeps replicas sandboxes pre-created from a template, so the expensive parts (scheduling, image pull, readiness) are paid before anyone asks
• SandboxClaim (extensions): the user-facing request. A claim against a pool adopts a warm sandbox instantly; the pool then replaces it in the background
• sandbox-router: a deployment that proxies the SDK’s exec and file operations (/execute, upload, download) to the right sandbox pod

flowchart TB
    subgraph client["Agent process (your machine or in-cluster)"]
        sdk["Go SDK<br/>sigs.k8s.io/agent-sandbox/clients/go/sandbox"]
    end

    subgraph cluster["Kubernetes cluster"]
        api["kube-apiserver"]
        ctrl["agent-sandbox-controller<br/>(agent-sandbox-system)"]
        router["sandbox-router"]

        subgraph pool["SandboxWarmPool"]
            tmpl["SandboxTemplate"]
            warm1["Sandbox (warm)"]
            warm2["Sandbox (warm)"]
        end

        claim["SandboxClaim"]
    end

    sdk -->|"1- create SandboxClaim"| api
    api --> ctrl
    ctrl -->|"2- adopt warm sandbox"| pool
    ctrl -->|"3- claim bound"| claim
    sdk -->|"4- exec / file ops<br/>(port-forward or Gateway)"| router
    router -->|"HTTP :8888"| warm1
    tmpl -.-> warm1
    tmpl -.-> warm2

The SDK talks to the router over one of three modes: Gateway (production, via the Gateway API), port-forward (development - a native SPDY tunnel from client-go, no kubectl needed), or direct URL (in-cluster agents hitting the router Service DNS). The demo uses port-forward.

The Go SDK#

Install:

go get sigs.k8s.io/agent-sandbox/clients/go/sandbox@v0.5.6

The Go SDK is published from the repository’s root module, so repo release tags are SDK versions. It requires Go 1.26+. The core surface is small:

client, err := sandbox.NewClient(ctx, sandbox.Options{Namespace: "default"})
// port-forward mode is the default when GatewayName and APIURL are unset

sb, err := client.CreateSandbox(ctx, "python-sandbox-pool", "default")

res, err := sb.Run(ctx, "python3 --version")   // ExecutionResult{Stdout, Stderr, ExitCode}
err = sb.Write(ctx, "script.py", []byte(src))  // plain filenames only
data, err := sb.Read(ctx, "script.py")
entries, err := sb.List(ctx, ".")

Details that matter in production, straight from the Go client docs and confirmed against the v0.5.6 source:

• File ops retry, Run() does not. Read/Write/List/Exists retry up to 6 times on 5xx and connection errors. Run() defaults to a single attempt because command execution is not idempotent - opt in with sandbox.WithMaxAttempts(6) for commands that are
• Disconnect() vs Close(): Disconnect drops the transport but keeps the SandboxClaim - the sandbox stays warm between user requests. Close deletes the claim
• Port-forward death is detected, not timed out: a background monitor flips the client to ErrNotReady immediately, and Open() reconnects
• Testing without a cluster: the package exports a Handle interface (Open, Close, Run, Read, Write, List, Exists, IsReady) exactly so you can fake the sandbox in unit tests
• EnableAutoCleanup() installs a SIGINT/SIGTERM handler that deletes tracked claims - agents that die at the keyboard do not leak pods

The Demo: A Go Agent with Sandboxed Tools#

The pattern the upstream examples push, and the one I implemented, is: the agentic loop runs outside the sandbox; only tool execution runs inside. The LLM never sees a credential, and the code it writes cannot touch the machine the loop runs on.

My agent is ~250 lines of Go using the Anthropic Go SDK’s tool runner (which drives the request -> tool -> response loop for you) and four tools, each a thin wrapper over the sandbox SDK:

The interesting part is the lifecycle wiring. Every tool call pushes the claim’s shutdownTime forward:

func (l *lifecycle) extend(ctx context.Context) error {
	claim, err := l.helper.ExtensionsClient.SandboxClaims(l.namespace).Get(ctx, l.claim, metav1.GetOptions{})
	if err != nil {
		return err
	}
	shutdownAt := metav1.NewTime(time.Now().Add(l.ttl))
	claim.Spec.Lifecycle = &extensionsv1beta1.Lifecycle{
		ShutdownPolicy: extensionsv1beta1.ShutdownPolicyDelete,
		ShutdownTime:   &shutdownAt,
	}
	_, err = l.helper.ExtensionsClient.SandboxClaims(l.namespace).Update(ctx, claim, metav1.UpdateOptions{})
	return err
}

Why This Matters: cleanup does not depend on the agent process surviving. If the CLI crashes, the laptop sleeps, or the loop hangs, the controller deletes the sandbox 5 minutes after the last tool call. This is the same inactivity-expiry pattern the upstream sandboxed-tools example uses, and it is the difference between “sandboxes as pets” and “sandboxes as garbage-collected sessions.”

Note: wrap that update in retry.RetryOnConflict. The controller writes to the claim too, and my first version lost a race to it mid-session (the object has been modified). Claims are actively reconciled objects, not inert records.

The Infrastructure, GitOps Style#

Everything reconciles through Flux except the two chicken-and-egg installs (Cilium as CNI, Flux itself). The controller comes from a GitRepository pinned to the v0.5.6 tag plus a Kustomization over the upstream k8s/ overlay:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: agent-sandbox
  namespace: flux-system
spec:
  interval: 1h
  sourceRef:
    kind: GitRepository
    name: agent-sandbox
  path: ./k8s
  prune: true
  wait: true
  timeout: 5m
  # the source tree ships ko:// image refs that only resolve in the release
  # pipeline; rewrite to the published controller image, pinned to digest
  images:
    - name: ko://sigs.k8s.io/agent-sandbox/cmd/agent-sandbox-controller
      newName: registry.k8s.io/agent-sandbox/agent-sandbox-controller
      digest: sha256:a502cfdbcf550e77509cc56097978458a1ac3d5b59972f21b7ce0e0a84a5c12e

That images: transformer is not decoration - without it the controller pod lands in InvalidImageName. More on that in the gotchas.

The sandbox side is a SandboxTemplate wrapping the upstream python-runtime image (a FastAPI server on :8888 that the router forwards to), a SandboxWarmPool with 2 replicas, and the router deployment. All images are pinned to digests.

Running It#

git clone https://github.com/nicknikolakakis/srekubecraft-demo
cd srekubecraft-demo/agent-sandbox
task up

export ANTHROPIC_API_KEY="sk-ant-..."
task agent:run -- "write a python script that prints the first 10 fibonacci numbers and run it"

Real output from my kind cluster:

sandbox ready in 44ms: claim=sandbox-claim-ktrtz sandbox=python-sandbox-pool-4mq8r pod=python-sandbox-pool-4mq8r
[tool] write_file: fib.py (157 bytes)
[tool] run_command: python3 /app/fib.py

Created `/app/fib.py` with a generator-based `fibonacci(n)` and ran it.

Output: `[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]`

Two tool calls, one warm claim, done. The better demo is asking the agent to probe its own cage:

task agent:run -- "Check what network access you have from inside your sandbox: \
  try to reach the Kubernetes API server at https://10.96.0.1 with a short timeout, \
  then try fetching https://pypi.org. Report what is reachable and what is blocked."

sandbox ready in 68ms: claim=sandbox-claim-24q62 sandbox=python-sandbox-pool-5qflx pod=python-sandbox-pool-5qflx
[tool] run_command: sh -c "curl -sS -k --max-time 5 -o /dev/null -w 'HTTP:%{http_code} ...' https://10.96.0.1 ..."
[tool] run_command: sh -c "curl -sS --max-time 8 -o /dev/null -w 'HTTP:%{http_code} ...' https://pypi.org ..."
[tool] write_file: netcheck.py (1386 bytes)
[tool] run_command: python3 netcheck.py

Network access results from inside the sandbox:

- Kubernetes API server https://10.96.0.1:443 - BLOCKED. TCP connect times out
  after 5s (no response, no reset). Cluster-internal API access is firewalled off.
- https://pypi.org - REACHABLE. DNS resolves, TCP connects in ~0.1s, HTTPS returns 200.

Notes: curl isn't installed in the sandbox, so I ran the checks with Python
(socket + urllib). Egress to the public internet works; access to the in-cluster
control plane is blocked - the connection silently times out rather than being
refused, which is consistent with a network policy dropping packets to the
cluster service network.

That is the controller’s NetworkPolicy from gotcha #3 doing its job, verified from the inside by the code it constrains. The agent even noticed curl was missing and rewrote its probe in Python without being told - and correctly read “silent timeout, not connection refused” as a policy drop.

The repo also carries a smoke test that exercises the full sandbox path with no LLM in the loop - claim, exec, file round-trip, TTL patch:

cd agent && SANDBOX_SMOKE=1 go test -v -run TestSandboxSmoke

=== RUN   TestSandboxSmoke
"msg"="port-forward established" "localPort"=60997 "pod"="sandbox-router-deployment-576794cd5b-7t54n"
"msg"="API URL discovered" "url"="http://127.0.0.1:60997" "mode"="port-forward"
    smoke_test.go:36: sandbox ready in 53ms: claim=sandbox-claim-xvfmz pod=python-sandbox-pool-7s7mc
    smoke_test.go:50: run: "hello from $(hostname)\n"
--- PASS: TestSandboxSmoke (0.13s)

53 milliseconds from CreateSandbox() to a bound, ready sandbox. For contrast, the warm pool itself took about 25 seconds to bring its sandboxes to Ready when first created - scheduling, image pull, probe. That 25s is exactly the latency your users never see because the pool pre-paid it. This is the whole warm-pool value proposition in two numbers.

And yes, that $(hostname) printed literally is real output - it is gotcha #4.

The Gotchas#

This is the section I wish had existed before I started. Five findings, all reproduced on v0.5.6.

1. The source tree’s k8s/ overlay is not deployable as-is#

The docs say you can render the install “directly from source with kubectl kustomize k8s/”. You can render it - you cannot run it. The deployment references ko://sigs.k8s.io/agent-sandbox/cmd/agent-sandbox-controller, a ko build directive that only resolves inside the release pipeline. Deploy it and the pod sits in InvalidImageName.

If you install from the release asset (sandbox-with-extensions.yaml) you never see this. If you want GitOps against the tagged source - which you do, because the release asset is not a Flux source - add the kustomize images: transformer shown above.

2. Two components disagree about where the router lives#

The installation guide deploys sandbox-router into agent-sandbox-system. The Go SDK’s port-forward mode resolves sandbox-router-svc in the claim’s namespace (tunnel.go lists EndpointSlices in Options.Namespace). Follow the install guide, then run the Go quickstart, and you get:

sandbox: no ready endpoints for sandbox-router-svc in namespace default

Fix: deploy the router into the namespace where your claims live. The Go client docs do say “target namespace” - quietly, in the prerequisites.

3. The controller’s own NetworkPolicy will 504 you after you fix #2#

Fixing #2 walks you straight into the next wall. The controller auto-creates a NetworkPolicy per SandboxTemplate, and its ingress rule only admits router pods from agent-sandbox-system:

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: agent-sandbox-system
        podSelector:
          matchLabels:
            app: sandbox-router

With the router now in default, every exec times out at the CNI and the router returns 504: Timed out waiting for the backend sandbox. Nothing logs a policy drop; with Cilium, cilium endpoint list showing POLICY (ingress): Enabled on the sandbox pods is your tell.

Do not edit the controller-owned policy - it will be reconciled back, and you would be fighting the tool. NetworkPolicies are additive, so grant the extra allow with your own:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-local-router-to-sandboxes
  namespace: default
spec:
  podSelector:
    matchLabels:
      agents.x-k8s.io/created-by: controller
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: sandbox-router
      ports:
        - protocol: TCP
          port: 8888

While you are in there, read the egress side of the controller’s policy. It allows 0.0.0.0/0 except RFC1918, link-local, and their IPv6 equivalents. Sandboxed code can pip install from the internet but cannot reach your cluster services, node metadata, or anything else on private ranges. That is a genuinely good default, and it is worth knowing it exists before you wonder why an agent cannot call an in-cluster API.

4. /execute is not a shell#

The python-runtime’s /execute endpoint execs your command; it does not pass it through a shell. Pipes, &&, redirection, and $VARS arrive as literal argv tokens:

{"command": "echo a | tr a b"}
// {"stdout": "a | tr a b\n", "exit_code": 0}

Quoted arguments are split shlex-style, so the fix is to wrap anything shell-shaped:

{"command": "sh -c \"echo a | tr a b\""}
// {"stdout": "b\n", "exit_code": 0}

For an LLM agent this belongs in the system prompt, not in post-hoc error handling - my agent’s prompt states the working directory (/app), the non-root UID (1000), and the sh -c rule, and Claude complies on the first try instead of burning a tool-call round trip discovering it.

5. Write() takes plain filenames only#

sb.Write(ctx, "dir/script.py", ...) is rejected by the SDK - no directory separators, files land in the server’s working directory. Anything path-shaped goes through run_command. Worth knowing before you design tools that mirror a filesystem.

Bonus: snapshots are Python + GKE only#

The suspend/resume-with-state story (PodSnapshotSandboxClient, freezing a gVisor container’s memory and filesystem) currently requires GKE’s podsnapshot.gke.io CRDs and only exists in the Python SDK. The Go SDK has no equivalent yet. operatingMode: Suspended is portable, but that is a pod teardown with volumes retained, not a memory snapshot.

Pros and Cons#

Pros#

Cons#

When to Use Agent Sandbox#

Use it when:

• Agents (or users) execute LLM-generated code and you need it off the agent host and inside your isolation boundary
• You need many short-lived, stateful environments with cleanup guarantees - code interpreters, CI-adjacent coding agents, RL rollout environments
• Claim latency matters: interactive agents feel the difference between 50ms and 25s
• You already run GitOps and want agent infrastructure managed like every other workload

Skip it when:

• You have no Kubernetes footprint - a hosted sandbox (E2B, Modal) is one API key instead of a controller you operate
• You need hard multi-tenancy on a cluster that cannot run gVisor or Kata - the orchestrator cannot add isolation your runtime lacks
• Your “sandbox” is just batch compute with no state or identity needs - a Job is simpler
• You need cross-region or multi-cluster sandbox placement today

Troubleshooting#

Hands-On Demo Repository#

Everything from this post is in srekubecraft-demo/agent-sandbox/:

• Kind + Cilium 1.20 + Flux bootstrap via a shared Taskfile
• Flux GitRepository/Kustomization for the controller at v0.5.6, digest-pinned
• Router, SandboxTemplate, SandboxWarmPool, and the additive NetworkPolicy
• The Go agent (Anthropic tool runner + 4 sandboxed tools + TTL lifecycle)
• The no-LLM smoke test

git clone https://github.com/nicknikolakakis/srekubecraft-demo
cd srekubecraft-demo/agent-sandbox
task up
task agent:run -- "write a python script that prints the first 10 fibonacci numbers and run it"

Conclusion#

Agent Sandbox answers a question every platform team fielding “we want to run AI agents” requests eventually hits: what is the workload type for code an LLM just wrote? Deployments assume stateless replicas. Jobs assume run-to-completion batch. StatefulSets assume an ordered set. An agent session is none of those - it is a stateful singleton with an identity, a TTL, and a trust level somewhere below “our own code.” Modeling that as a first-class CRD, with warm pools to hide the cold start and controller-side garbage collection to survive client crashes, is the right call, and the 53ms claim latency shows the design earns its complexity.

It is also early. The API is beta, the docs trail the code, and three of my five gotchas were namespace and policy mismatches between components of the same project. None of them were hard to debug with standard tools - and notably, the security posture failed closed every time: the worst default I hit was traffic being blocked too aggressively, never too little.

If you run Kubernetes and agents are on your roadmap, this is the project to watch - it is where the SIG Apps machinery is clearly investing. Start with the warm-pool + claim flow on kind like this demo, put gVisor or Kata under it before any untrusted code shows up, and treat the router as the immature piece it currently is. And keep the agentic loop outside the sandbox: the LLM gets tools, the tools get a pod, and the pod gets a NetworkPolicy that cannot reach your control plane. That layering is the entire point.

If you found this useful, you might also enjoy my related posts:

• Building a Kubernetes Agent with Google’s ADK for Go
• llm-d - Kubernetes-Native Distributed LLM Inference at Scale
• KServe - Production ML Serving on Kubernetes, from sklearn to LLMs