跳转到内容
nuts.nvirellia.im

Stateless systems (Poettering)

Stateless systems (Poettering)

Section titled “Stateless systems (Poettering)”

Primary: 0pointer.net essay (posted Di 17 Juni 2014; mirror 0pointer.de). Immediate sequel: Revisiting How We Put Together Linux Systems (syndicated 1 Sep 2014). Later personal synthesis: Fitting Everything Together (posted Di 03 Mai 2022).

Core argument

Section titled “Core argument”

The premise is the /usr merge. With it, “most static vendor-supplied OS data is found exclusively in /usr,” and only “a few additional bits in /var and /etc are necessary to make a system boot.” From that split he defines four properties, not three. The fourth is easy to drop and changes the point.

  1. Factory reset. Flush /etc and /var, keep vendor /usr, and the machine is back in “a well-defined, pristine vendor state with no local state or configuration.” He wants this on servers, desktops, and embedded devices.
  2. Stateless systems. Never persist /etc or /var at all. Every reboot is a factory reset (he writes “factor reset”). Aimed at simple containers, network boot, and read-only media, with runtime configuration from packages, DHCP, or hardware discovery.
  3. Reproducible systems. Many machines or containers share one immutable vendor /usr (bind mount, NFS, or a btrfs snapshot of a golden master) and keep private /etc and /var. He names thousands of containers on one OS tree, thin clients on one NFS share, and a dumb installer that unserializes a /usr snapshot, installs a boot loader, and reboots.
  4. Verifiable systems. If storage can cryptographically vouch for the vendor OS as one consistent unit, then /etc and /var must either live inside that image or be unnecessary to boot. Otherwise the verified tree is not what actually runs.

He is explicit that the ideas are not new. Android, ChromeOS, CoreOS, and GNOME OSTree already did pieces of this, “sometimes taking shortcuts that keep only the specific case in mind.” What he claims is new is generic support inside systemd, so ordinary distributions can build on it.

Operationally he narrows to three boot modes:

  • Stateful: populated machine-specific /etc, /usr, and /var.
  • Volatile: configured /etc, empty /var rebuilt at boot.
  • Stateless: neither /etc nor /var populated.

Factory reset is a one-shot of the latter two: boot once without /etc and /var, then the next boot is stateful again. He refuses a mode that wipes /etc but keeps /var. The user-ID problem gets harder, and he “saw no usecase for it worth the trouble.”

Empty /var is the easy half. systemd 214 already rebuilt a basic /var from tmpfiles. Empty /etc is the hard half, because identity and caches live there, above all /etc/passwd and /etc/group. Offline updates of a shared /usr also have to flow into many private /etc and /var trees (rebuild /etc/ld.so.cache, add missing system users) without an installer run on each instance.

Design consequences

Section titled “Design consequences”

/usr is the vendor OS: read-only, versioned, the thing you snapshot, share, and sign. /etc is local configuration. /var is local state. The essay’s claim is that almost nothing required to reach a boot belongs in the last two, and that packages which insist otherwise are the bug.

Why /etc is the problem, in his words and mechanisms:

  • System users cannot stay as imperative useradd -r / groupadd -r in RPM or deb scripts. Those scripts do not run when you replicate a /usr image onto a machine whose /etc is private or empty. systemd-sysusers (shipping with the then-unreleased systemd 215) reads declarative snippets from /usr/lib/sysusers.d/ and creates missing system users and groups in /etc/passwd and /etc/group. Normal users and LDAP are out of scope. He rejects a single static UID map in /usr for general-purpose systems: the system UID/GID space is small (“only containing 998 users and groups on most systems”), so allocation has to be dynamic and match what that image actually needs. The tool can also take UIDs from existing files in /usr, so setuid binaries keep their owners.
  • ConditionNeedsUpdate= runs upgrade helpers only when /usr is newer than /etc or /var, using the mtime of /usr (packaging is supposed to touch the directory). First consumers: sysusers, the udev hwdb, the journal catalog, and ldconfig for /etc/ld.so.cache.
  • An empty /etc at early boot applies vendor unit presets, then continues. A tmpfiles snippet rebuilds the skeleton of /etc. tmpfiles can also copy whole trees into place. The files he names as currently fatal if missing are /etc/pam.d and /etc/dbus-1.
  • Long-term, packages should boot with no /etc file at all: compiled-in defaults, or a fallback to vendor files under /usr. He floats /usr/share/etc as the full original vendor configuration, both as a copy source and so admins can diff /etc against it. He says the name is not settled.
  • Testing hook on systemd-nspawn: --tmpfs=/var --tmpfs=/etc over a read-only tree. He expects a later --mode={stateful|volatile|stateless}. The example command in the post is typed system-nspawn (missing the d). He reports that a then-current Fedora rawhide tree “should mostly work,” except D-Bus and PAM.

What he asks upstreams and packagers to do: create missing /var directories (or ship tmpfiles); fall back when /etc config is absent; ship sysusers files instead of useradd in scriptlets. He notes most distro policies of 2014 did not allow that yet.

The conclusion is a list of products of the split, not a new filesystem layout: factory reset for broken or resold end-user machines (“fresh car smell”); embedded devices whose every boot is a reset; one read-only, cryptographically verified /usr; installers that only deserialize /usr and a boot loader; updaters that swap verified /usr snapshots and fix /etc and /var on the next boot; containers that share a golden /usr; thin clients that NFS-mount it.

What shipped or followed

Section titled “What shipped or followed”

Shipped with the essay, not after it. The systemd 215 announcement (July 2014) points at the post and lists systemd-sysusers, ConditionNeedsUpdate=, systemd-update-done.service, and the services that rebuild the udev database, the journal catalog, and the dynamic loader cache. The announce says a minimal OS can start with /etc empty.

By LinuxCon Europe (Düsseldorf, October 2014) the unsettled directory name had become /usr/share/factory/etc, and systemd-nspawn had the high-level switch the essay only promised: --volatile=no|state|yes. Slides: Stateless Systems, Factory Reset, Golden Master Systems and systemd. Talk notes: Mind’s Embedded Linux Blog, 15 Oct 2014. PAM still needed a tmpfiles copy; D-Bus still did not. Updates of /usr were described as offline, with /etc and /var repaired on the next boot via ConditionNeedsUpdate=.

The September 2014 sequel, Revisiting How We Put Together Linux Systems, cites the essay by name and says the new scheme “relies on the ability to monopolize the vendor OS resources in /usr.” It is a different proposal on top of that split: btrfs subvolumes named usr:, root:, runtime:, framework:, app:, and so on, so vendors ship immutable /usr trees and apps without going through a distro package of the day. It is a design sketch (Kay Sievers, Harald Hoyer, Daniel Mack, Tom Gundersen, David Herrmann, Poettering; Hoyer “has started” Fedora btrfs send/receive images). It is not a claim that this btrfs layout shipped.

Later pieces that implement the same split, with sources that actually tie them:

  • Portable services (systemd v239). Walkthrough, 27 June 2018: a service image (directory tree or GPT disk image) attached with portablectl, unit files copied to the host, execution via RootDirectory= / RootImage=. Not itself a stateless boot. Fitting Everything Together (3 May 2022) is where Poettering places them in this lineage: optional, sandboxed services as their own verity-signed images, beside the host OS image.
  • systemd-sysext extends an immutable /usr and /opt at runtime with overlayfs, without writing the base image. systemd-sysext(8). The same page’s confext counterpart extends /etc instead. The man page says sysext is the wrong vehicle for earliest-boot resources and for systemd-sysusers definitions, and points at portable services for shipping system services. The 2022 essay treats sysext as the way to add optional OS components to a hermetic /usr.
  • systemd-repart grows and adds partitions at boot and builds Discoverable Disk Images. systemd-repart(8) documents a factory-reset mode (--factory-reset=, kernel systemd.factory_reset=yes, or the FactoryResetRequest EFI variable; the switch added in v245): partitions marked FactoryReset= (the boolean itself added in v246) are deleted and recreated empty. That is the essay’s “flush local state, keep the vendor OS” property at partition granularity. The 2022 essay uses repart for a stronger variant: first boot creates and encrypts the root filesystem (TPM2-enrolled), because a hermetic /usr plus tmpfiles and sysusers can populate it, and factory reset must make earlier sensitive data cryptographically inaccessible.
  • OSTree and image-based distros. The 2014 essay lists GNOME OSTree as prior art, not as a systemd deliverable. Later readers connect the essay to Fedora Silverblue: on the systemd 248 thread, HN user Arnavion (31 Mar 2021) calls the essay and the revisiting post Poettering’s long-term goal, “what Fedora SilverBlue and OSTree are trying to achieve,” with sysusers “geared towards this goal.” That is a citation, not a statement that Silverblue implemented this design.

The 2022 essay’s own summary of the destination: image-based rather than package-deployed, immutable reproducible images, a trust chain from firmware through the OS, and “a way to put the system back into a well-defined, guaranteed safe state (‘factory reset’).” Packages remain the build input. Deployment is the image.

Field reaction

Section titled “Field reaction”

Hacker News. The submission itself is thin: story 7906489, “Factory Reset, Stateless Systems, Reproducible Systems and Verifiable Systems,” 16 points, 0 comments, posted by denisw at 2014-06-18 03:36 CST, linking 0pointer.de. The essay is cited later, usually as a justification rather than re-argued:

  • sandGorgon, Debian init GR thread (2014-10-18 02:12 CST): calls “systemd FUD” unfair and describes stateless boot as “very similar to Docker,” useful for a private fleet of pristine machines and for not leaving stale confidential data behind.
  • Arnavion, systemd 248 (2021-03-31 16:49 CST): long-term goal behind Silverblue/OSTree; links the essay, the revisiting post, and sysusers.d.
  • alex-robbins (same era of citations; comment on the essay URL): asks whether Poettering has “discovered NixOS,” and says the post reminds him of it.
  • nwah1, Serenity OS interview thread (2024-02-08 01:37 CST): groups the essay with OSTree, Nix, Guix, and Fedora Silverblue as “reproducibility, statelessness, and transactionality,” and with the wish that a distro behave like git plus bit-identical builds. Earlier, on “Distribution packages considered insecure” (2016-02-22 23:15 CST), the same person points at the essay for “verifiable builds,” which is a looser reading than the essay’s verifiable vendor tree.
  • jcgl, Debian 13 “Trixie” (2025-08-10 13:41 CST): cites the essay and The Case for the /usr Merge against the charge that usr-merge was arbitrary (“reasoning behind it that is over 10 years old”), then still calls the effect “strong-arming distros.”

Reddit. r/linux thread, posted by u/ohet, 45 points, 25 comments, 2014-06-18 02:18 CST, linking the .de URL. Full comment bodies were not retrievable (Reddit returned a block page to a direct fetch); the following is from indexed excerpts of that thread:

  • Praise: “Finally something out of systemd that I can fully embrace.” Live systems with /usr on squashfs and /etc//var on NAS, tmpfs, or USB. A datacenter picture where /usr is versioned centrally per machine class and /etc and /var are owned by Ansible, Chef, or Puppet.
  • Pushback: what happens to /usr/local; a return to reboot-to-update is acceptable on servers and a “laughing stock” on desktop and mobile, so those packages would have to treat /var and /etc as alien and use XDG config; the verifiable-systems paragraph summarized, sarcastically, as “Systemd supports DRM. Film at 11.”

X. search_posts_all on “stateless systems” / “reproducible systems” / 0pointer with Poettering or systemd, then broader stateless systemd, "factory reset" (systemd OR Poettering OR stateless), and from:pid_eins with those terms: no posts about this essay or its follow-ups. The few hits were unrelated (managed Kubernetes, a political use of “factory reset”). Treat X as no signal, not as silence in the 2014 record. Historical search coverage from this connector is not something to over-read.

LWN, same week. LWN pointer and thread (comments 19 June 2014, afternoon CST). The visible argument is whether /etc/fstab blocks the scheme. Chousuke: every mount is a unit, fstab is only a generator input, and a stateless system “by definition” has no local configuration; factory reset wipes it. rwmj: local configuration still has to live in /etc somewhere, and native .mount units do not remove that. zlynx: discover disks with udev, or let Puppet inject state; “a stateless system isn’t very useful until it has grabbed itself some state,” and that state need not be a file in /etc.

A sharper critique is on systemd-devel, not social media. In sysusers and login.defs checks (July 2014), Zbigniew Jędrzejewski-Szmek: “The fallacy here is trying to support stateless systems with users. Users (account numbers, their passwords, privileges) are state, and the most important one at that.” Nuke /etc and local accounts are gone unless they are defined remotely. Fedora’s packaging committee, asked to bless sysusers snippets for exactly this factory-reset / golden-master case (pagure issue 442), answered that the tool was not ready to recommend, including because user creation did not emit the audit events useradd did.

Caveats

Section titled “Caveats”
  • Poettering’s own limit, repeated at the end: systemd groundwork does not make a general-purpose distro stateless. “The majority of Linux packages are simply incompatible.” He does not expect all of Fedora to follow soon. D-Bus and PAM were broken on empty /etc in the post itself.
  • “Reproducible” here means many instances from one vendor snapshot, not bit-identical package builds. Nix-style reproducibility is a later reader’s analogy (HN), not the essay’s definition.
  • “Verifiable” means the vendor OS can be checked as one unit because /etc and /var are outside it. It is not a reproducible-build scheme, and it is not a design for locking users out of their own configuration. The DRM joke on Reddit is a reaction to the cryptographic-vendor-image sentence, not something the essay proposes.
  • He will not flush /etc while keeping /var. Dynamic system UIDs are a requirement of his general-purpose story and a compatibility hazard (login.defs boundaries, existing UID assignments, audit).
  • /usr/share/etc was a provisional name. Within months the public talk used /usr/share/factory/etc.
  • The September 2014 btrfs subvolume taxonomy is a proposal. The 2022 picture (GPT images, verity, UKI, repart, sysext, portable services) is the one he later said he would actually build. Do not treat the 2014 subvolume names as shipped systemd behavior.
  • OSTree, Silverblue, Nix, and Guix are adjacent and often cited next to this post. Only OSTree is in the essay, and there as prior art that systemd should generalize, not as a child project.
  • The essay’s own typos and hedges matter when quoting: “factor reset,” “incomptible,” “we wanto to scale,” and the admission that “the concepts behind this are really not new.”

Sources

Section titled “Sources”