Immutable Kubernetes Nodes as OCI Images - A Close Look at bootc

Every self-managed cluster I have inherited had the same artefact sitting in a corner: a Packer template nobody wanted to touch. It produced the AMI, the AMI produced the node pool, and somewhere between those two steps a dnf update ran against whatever the mirror happened to be serving that morning. Two nodes built three months apart were not the same machine, and the only way to find out how they differed was to SSH into both and start diffing.

The usual answer to that is discipline: pin everything, rebuild often, never touch a running node. The discipline works right up until the 2am kernel CVE, when someone patches in place because rebuilding the bakery takes forty minutes and the maintenance window is now.

bootc takes a different route. Instead of asking you to be disciplined about a package set, it makes the operating system a container image: built with a Containerfile, pushed to a registry, addressed by digest, and swapped atomically at reboot. This post is about what that actually changes for a Kubernetes node, and what it does not.

Who Should Read This?#

This post is for:

• SREs who own node lifecycle on self-managed or hybrid clusters and are tired of drift between nodes built months apart
• Platform Engineers maintaining a Packer/AMI bakery who want the build, promote and rollback story they already have for application images
• Infrastructure teams evaluating immutable node operating systems (Talos, Flatcar, Bottlerocket) and wondering where the bootc lineage fits
• Anyone who has answered the question “what is actually installed on this node?” by running rpm -qa and squinting

If you run EKS, GKE or AKS managed node groups and are happy with them, this is interesting rather than urgent. The cloud provider already owns this problem for you.

What is bootc?#

bootc describes itself as “boot and upgrade via container images”. Concretely, it enables transactional, in-place operating system updates using OCI/Docker container images. The container image includes a Linux kernel (at /usr/lib/modules/$kver/vmlinuz), which is what the machine boots.

It is not a toy. The project has been going since late 2022, sits at 2290 stars, is written in Rust, and was last pushed on 2026-09-25. Underneath it is ostree, which has been powering stable operating system updates for years, so the hard part (atomic deployments, bootloader ordering, rollback) is not new code.

The key reframing: your OS is not a machine that accumulates state. It is an artefact you build, sign, scan and promote with the same tooling you already point at your application images.

What makes an image bootable#

A valid bootc image is an ordinary OCI image with a few requirements:

That last row matters more than it looks. With composefs enabled, “/usr is not different from /; they are part of the same immutable image”, and the entire / is a read-only filesystem. The docs call this “very important for achieving correct semantics”, and they are right: without it, immutability is a convention rather than a property.

Why Not a Packer/AMI Bakery?#

Bottom line: a bakery gives you reproducible nodes if everyone follows the rules. bootc gives you reproducible nodes because /usr is read-only and the only way to change it is to build a new image.

What bootc is NOT#

• Not a container runtime for your OS - the base userspace is not itself running in a container by default. systemd is still pid1, your processes are ordinary processes, and ps looks exactly like it always did
• Not a Kubernetes distribution - bootc boots a machine. What that machine runs is your problem
• Not a fleet manager - there is no controller reconciling your nodes toward a desired image. bootc upgrade runs on the node, and something else has to decide when
• Not a way to avoid understanding your filesystem layout - see the next section, because this is where the real work is

The filesystem contract#

This is the part that decides whether your migration is boring or painful. Three directories, three completely different lifecycles.

flowchart TB
    subgraph Image["OCI image (the artefact)"]
        USR["/usr<br/>kernel, k0s, systemd units<br/>read-only"]
        ETCDEF["/etc defaults<br/>shipped in the image"]
        VARSEED["/var seed content<br/>install-time only"]
    end

    subgraph Machine["The machine (persistent)"]
        ETC["/etc<br/>three-way merged on upgrade"]
        VAR["/var<br/>never touched by an upgrade"]
    end

    USR -->|replaced wholesale| Machine
    ETCDEF -->|merged with local edits| ETC
    VARSEED -->|unpacked at install only| VAR

/usr is the image. Read-only at runtime, replaced wholesale on upgrade. Anything you want to survive a node moving forward goes here.

/etc is the machine, with a merge. By default it holds mutable persistent state, and bootc does a three-way merge on upgrade: the new default /etc is used as a base, then the diff between the current and previous /etc is applied on top. Your local edits survive; image defaults update around them. If you would rather they did not survive, etc.transient = true in /usr/lib/ostree/prepare-root.conf makes /etc transient.

/var is the machine, full stop. Content in /var persists by default. The trap is the other direction: changes made to /var in container images are not automatically applied on subsequent deployments. Only the initial image content unpacks at install. The rationale is sound (application data should not roll back with an OS update), but the failure mode is nasty, and we will come back to it.

Why This Matters: a file you COPY into /var at build time reaches every machine installed from your image and no machine upgraded into it. Your freshly installed test node works perfectly. The fleet you upgraded never sees it, and nothing reports an error. Declare state directories in tmpfiles.d instead and let systemd create them on every boot.

The commands#

The whole surface is small enough to fit in your head.

# What is running, and what is staged for next boot.
sudo bootc status

# Pull a newer build of the image this machine already tracks.
# Stages it for next boot. Does not reboot.
sudo bootc upgrade

# Move this machine to a different image or tag entirely.
# Preserves existing state in /etc and /var.
sudo bootc switch ghcr.io/example/node-image:0.3

# Swap the bootloader ordering back to the previous deployment.
sudo bootc rollback
sudo systemctl reboot

Three properties worth internalising:

1. Nothing reboots by default. bootc upgrade and bootc switch stage a deployment for the next boot, which means the maintenance window stays yours. Add --apply to reboot immediately.
2. You can split fetch from apply. bootc upgrade --download-only pulls now; bootc upgrade --from-downloaded --apply commits later. Useful when bandwidth and downtime have different schedules.
3. Rollback is a bootloader operation. It swaps the boot entry ordering to the previous deployment. Corium’s upgrade guide puts it plainly: rollback “is instant and downloads nothing”, because it reorders boot entries between the deployment you are running and the previous one, and both are already on disk.

Upstream also ships an opinionated bootc-fetch-apply-updates.timer for distributions that want periodic unattended updates. On a Kubernetes node, think hard before enabling that; see the day 2 section.

What this means for a Kubernetes node#

Here is where the pattern stops being an OS curiosity and starts changing your runbooks.

Kubernetes becomes part of the image. The k0s binary lives in the read-only /usr, and it embeds the kubelet, its own containerd and the control plane. It unpacks those supervised binaries under /var/lib/k0s/bin at runtime, but the image is what pins their version. That collapses two version axes into one: there is no longer an “OS version” and a “Kubernetes version” that drift independently, because upgrading either one means booting a different image.

Cluster state lives in /var and survives. The CA, etcd data, container image storage and the kubelet’s own state are all on the writable side of the line. A node that reboots into a new image is the same node: same name, same identity, same certificates.

“Replace the node” and “upgrade the node” become genuinely different operations.

Both start from the same digest, which is the point. You are no longer choosing between “patch it” and “rebuild it” as two different pipelines with two different definitions of correct. They are the same artefact applied two ways.

Incremental transfer is free. Because the OS is an OCI image, upgrades are incremental for the same reason application image pulls are. Corium’s upgrade guide reports the following for a switch between two builds of its node image:

layers already present: 65; layers needed: 11 (214.0 MB)
Deploying...done (15 seconds)

214 MB moved for an image of roughly 2.3 GB. No bespoke delta format, no per-release patch baseline to maintain.

Installing anything on a running node stops working. dnf install fails because /usr is read-only. Dropping a binary into /var with a runcmd works exactly until the next upgrade, when nothing puts it back. If you have a monitoring agent, a CA bundle, a backup client or a driver on your nodes today, all of them become image content. That is the real migration cost, and it is not small.

/opt will break your CNI, and the symptom is not obvious. On an OSTree system /opt is part of the read-only image, and the bootc filesystem docs confirm it stays read-only under composefs. k0s ships its CNI plugin binaries into /opt/cni/bin at runtime, so the CNI DaemonSet fails with mkdir /opt/cni: read-only file system and the node never leaves NotReady. Corium’s Containerfile fixes it the way the bootc docs recommend, by moving the part of /opt that needs to be writable into /var:

RUN rm -rf /opt && ln -s var/opt /opt

CNI configuration is not affected: /etc/cni/net.d sits under /etc, which is writable and merged on upgrade.

A worked example: Corium#

To make this concrete I want a node image that already does the Kubernetes part, and Corium is the most direct example I found: “an immutable Kubernetes node, built as an OCI image. Fedora bootc, k0s baked into a read-only /usr, configured with cloud-init.”

Be clear-eyed about what it is. Corium was created on 2026-09-15, has 8 stars, and its own README opens with a status note that is more honest than most:

Status: 0.x. The architecture is settled and the path works end to end […] Two things are not settled. The configuration surface can still change between minor releases, and the testing is narrower than it looks: everything has been verified on virtual machines, none of it on physical hardware. Pin a version, read the changelog, and do not put anything on it you would miss.

So: a small, very recent project, MIT licensed, built on Fedora bootc and shipping k0s v1.36.4+k0s.0 (Kubernetes 1.36). I am using it as an illustration of the pattern, not as a recommendation to run it in production. If you want the mature options in this space, that is Talos Linux or Kairos, and Corium’s own comparison doc says so plainly.

1. Get a bootable disk#

bootc images are not directly bootable on a hypervisor, so something has to turn the image into a disk. Corium publishes ready-made artefacts per release:

# A qcow2 for Proxmox, KVM or libvirt.
oras pull ghcr.io/corium-os/corium-qcow2:0.3.6

# Verify what you got before you boot it.
cosign verify ghcr.io/corium-os/corium-qcow2:0.3.6 \
  --certificate-identity-regexp 'https://github.com/Corium-OS/Corium/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The 0.3.6 tag above is only an example; Corium’s quickstart points at the release page for the exact artefact coordinates, so check there rather than copying a version out of a blog post.

Building your own goes through image-builder, which produces qcow2 and raw disks; the anaconda-iso type still goes through the older bootc-image-builder entry point. This is the least settled part of the whole ecosystem, and worth checking the current state of before you design a pipeline around it.

2. Write the node configuration#

Corium’s configuration layer is cloud-init, not a bespoke API. A complete single-node cluster:

#cloud-config
corium:
  role: single

users:
  - name: core
    groups: [wheel]
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... you@example.com

A worker joining an existing cluster, pulling its token from a secrets endpoint rather than baking it into the document:

#cloud-config
corium:
  role: worker
  cluster:
    name: edge
  join:
    tokenFrom:
      url: https://secrets.example.com/corium/worker-token

Mind the wrapper. Through cloud-init the configuration has to be a #cloud-config document with a corium: block, as above. The bare, unwrapped schema (role: at the top level) is what the other three sources take: /etc/corium/config.yaml, the kernel command line, and a default baked into a derived image.

The four roles are single, controller, controller+worker and worker. Configuration is resolved from the first source that answers:

One design detail I want to call out, because it is the kind of thing you only add after being burned: a source that fails for any reason other than being absent stops the search. An unreachable config URL does not fall through to a baked-in default, because falling through is how a node silently joins the wrong cluster.

3. Boot and check#

Hand the document to the platform as cloud-init user-data. For a plain NoCloud seed:

printf 'instance-id: node-01\nlocal-hostname: node-01\n' > meta-data
cp node.yaml user-data
genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data

Then verify:

ssh core@192.168.0.190

# What the bootstrap agent did, and why if it failed.
systemctl status corium-bootstrap

sudo k0s kubectl get nodes
sudo k0s kubectl get pods -A

Result: Corium’s quickstart reports a single node reaching Ready in roughly a minute from power-on. Every timing in this post is Corium’s own published measurement, taken on virtual machines, not mine. The API answers well before the node registers, so Ready is the milestone to wait for, not a successful curl against the API server.

4. Derive your own image#

This is the step that decides whether the pattern survives contact with your environment, because every agent you run today has to move into the image. An ordinary Containerfile:

FROM ghcr.io/corium-os/corium:0.3.6

ARG VERSION=1.10.0
ARG SHA256=a1b2c3...   # the checksum you looked up, pinned here

# The binary goes in /usr, which is what an upgrade carries forward.
RUN curl --fail --silent --show-error --location --retry 3 \
        --output /tmp/node_exporter.tar.gz \
        "https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-amd64.tar.gz" \
    && echo "${SHA256}  /tmp/node_exporter.tar.gz" | sha256sum --check \
    && tar -xzf /tmp/node_exporter.tar.gz -C /tmp \
    && install -m 0755 "/tmp/node_exporter-${VERSION}.linux-amd64/node_exporter" \
        /usr/bin/node_exporter \
    && rm -rf /tmp/node_exporter*

# Units belong in /usr/lib/systemd/system, never /etc/systemd/system: that is
# the machine's, and an operator who disables your unit there should stay
# disabled across upgrades.
COPY node-exporter.service /usr/lib/systemd/system/node-exporter.service

RUN systemctl enable node-exporter.service

# Catches the /var mistake, among others, and it is cheap.
RUN bootc container lint --fatal-warnings

Result: every node that boots this digest has node_exporter running, and every node upgraded into it keeps it. Run bootc container lint --fatal-warnings in CI. It is the cheapest guardrail in the whole model.

Day 2: rollout, rollback, and what breaks#

Rolling a cluster#

Nodes are cattle, but the control plane still has a quorum to respect. The manual sequence per node is the one you already know:

# 1. Stop scheduling and move the workloads off.
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# 2. Stage the new image and reboot into it.
ssh node-1 sudo bootc switch ghcr.io/corium-os/corium:0.3.7
ssh node-1 sudo systemctl reboot

# 3. Wait for Ready, then let it take work again.
kubectl wait --for=condition=Ready node/node-1 --timeout=5m
kubectl uncordon node-1

Corium’s upgrade guide reports a drained node coming back in about 45 seconds and rejoining on its own. Corium keeps a marker in /var/lib/corium so the bootstrap agent does not run a second time; since /var survives the image swap, the node does not try to re-bootstrap itself into a new cluster.

Controllers are the part to be careful with, and the arithmetic is the usual etcd arithmetic: with three controllers, one at a time and wait for Ready in between. Never two of three. With five, two at a time is survivable. With two, there is no quorum to lose gracefully and the cluster is unavailable during each reboot.

Note: Corium’s own rollout tool stops at the first node that does not come back on the digest it was sent to. Whatever you build, build that in. A rollout that carries on past a broken machine turns one outage into a cluster-wide one.

Rollback that the node performs itself#

The interesting half of rollback is not bootc rollback. It is a node deciding on its own that the new image is bad. Corium wires greenboot health checks into the boot path, and its docs publish the boot table it measured with an always-failing check:

GREENBOOT_MAX_BOOT_ATTEMPTS in /etc/greenboot/greenboot.conf sets the count, and you add your own checks as executables in /etc/greenboot/check/required.d/ where a non-zero exit fails the boot.

Two design choices in that setup are worth stealing regardless of which image you run:

The health check is deliberately loose. Corium’s shipped check asks whether k0s is running and answering, and explicitly does not require the node to be Ready in Kubernetes. A node can be NotReady for reasons the image cannot fix: no CNI installed yet, a control plane still coming back, a cluster-wide problem. Rolling the OS back in those cases takes a healthy machine out of service at the worst possible moment and fixes nothing. A health check that is too strict is worse than none.

Rollback is gated on an upgrade having staged something. A check that fails on a node nobody upgraded gets you a warning and manual intervention, not a surprise trip to an older image.

The failure modes#

Two things the sources do not settle, and I will not invent answers for:

• What a failed upgrade pull does to a running node. The plausible reading is that nothing happens, because nothing new was staged and the current deployment is untouched. None of the bootc docs I read state it, so confirm it against a live node before a runbook leans on it.
• How this fits cluster-autoscaler and cloud provider node pools. How the “current” digest gets chosen at scale-out, and what a node that boots an older digest than the rest of the fleet does to you, is covered by none of the sources I consulted.

bootc node images: Pros and Cons#

Pros#

Cons#

When not to do this#

Be honest about the baseline. Most clusters run Kubernetes on an ordinary Ubuntu, Debian or RHEL, and that is the option this pattern has to justify itself against. You get complete control, your existing tooling, your package manager, and the operational knowledge your team already has. What you build yourself is atomic updates and rollback, drift prevention, an image release pipeline, and node provisioning. If you do not need those four things, you do not need this.

Skip the bootc node pattern when:

• Your cloud provider already owns node lifecycle. Managed node groups on EKS, GKE or AKS solve the same problem and you are not paying for it in engineering time
• You want the smallest possible attack surface. Talos removes the shell entirely. A Fedora bootc image keeps a normal userland with SSH, which is more surface by construction
• You need commercial support or a compliance story today. Talos Enterprise and RHEL image mode have one. A 0.x project with 8 stars does not
• You need neutral governance for the whole stack. bootc itself is a CNCF Sandbox project (accepted January 2025), but a small project built on it, like Corium, is not. Kairos is also CNCF Sandbox and Flatcar is CNCF-donated. Evaluate the project, not just the pattern
• You are entirely on AWS or in the Rancher ecosystem. Bottlerocket and SUSE SL Micro are better integrated than a general-purpose image will be
• Configuration changes are frequent and ad hoc. If every change goes through a build and a reboot, and your change rate is high, you will hate this

And if you want the pattern without the specific project: the bootc model is what RHEL image mode and Fedora bootc are built on. Corium is one small integration on top of it, not the only way in.

Conclusion#

The thing worth taking from bootc is not immutability, which the industry has been talking about since the first Packer template. It is that the OS stops needing its own supply chain. Build with a Containerfile, store in the registry you already run, scan with the scanner you already run, sign with the keys you already have, promote by moving a tag, and roll back with a bootloader flip. One artefact, one identity, one pipeline.

For a Kubernetes node specifically, the payoff is collapsing two version axes into one. When k0s or the kubelet lives in a read-only /usr, there is no “OS version” and “Kubernetes version” that can drift apart, because there is only one thing to move and one way to move it. That is worth more than the immutability on its own.

The costs are real and front-loaded. Every agent on your nodes today becomes image content. Your rollout tooling has to learn drain, ordering, quorum and stop-on-first-failure, because bootc upgrades exactly one machine. And the disk image build path, the step between “I have an OCI image” and “I have something a hypervisor will boot”, is the least settled part of the ecosystem right now.

If you are choosing today: pick Talos if you want this pattern hardened and supported, Kairos if you want it governed by a foundation, and managed node groups if your cloud provider will do it for you. Pick the raw bootc route when the argument that convinces you is the ergonomic one: that an operating system which is an ordinary container image, handled by the tooling you already have, beats a better mechanism you have to learn separately.

Corium is a useful thing to read even if you never run it. It is small, recent and honest about its limits, and its documentation of the /usr / /etc / /var contract is the clearest statement of the real tradeoff I have found.

If you found this useful, you might also enjoy my related posts on container images and node lifecycle:

• Embracing Minimalist Containerization
• Building Container Images with Buildpacks
• Optimizing Kubernetes Autoscaling with Karpenter
• Why ArgoCD Matters for SREs - A GitOps Approach to Kubernetes Deployments