Declared vs. Effective State: the latent-configuration pattern
Survey and formal framing under Latent Configuration Patterns

Table of Contents

Claim

A configuration file and the process it configures are two components of a governance tuple with no invariant connecting them.1 Sync occurs only at restart edges. Between edges, the config file makes an implicit, unfalsified claim: the running system reflects me. Nearly all of operations folklore, half of SRE incident literature, and one sitcom catchphrase are downstream of this single missing invariant. [E]

Refutation: exhibit a mainstream runtime where the declared/effective pair is continuously checked by construction rather than by an external harness. NixOS activation comes closest and still does not close the gap for long-lived process-internal state (see § Eliminating the class).

digraph declared_effective {
    rankdir=LR;
    bgcolor=white;
    node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10];
    edge [color="#888888", fontcolor="#555555", fontname="Helvetica", fontsize=9];

    declared  [label="Declared\n(config on disk)",
               fillcolor="#dcfce7", color="#15803d", fontcolor="#15803d"];
    effective [label="Effective\n(process memory)",
               fillcolor="#dbeafe", color="#1d4ed8", fontcolor="#1d4ed8"];
    divergence [label="Divergence accumulates here.\nNo provenance. No audit edge.",
                shape=note,
                fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];

    declared  -> effective [label="restart / exec\n(only guaranteed sync)",
                            color="#15803d", fontcolor="#15803d"];
    declared  -> effective [label="SIGHUP / reload\n(partial, per-daemon contract)",
                            color="#b45309", fontcolor="#b45309", style=dashed];
    effective -> declared  [label="write-back (rare)\nCisco 'copy run start'",
                            color="#a16207", fontcolor="#a16207", style=dashed];
    effective -> divergence [style=dotted, arrowhead=none,
                             color="#b91c1c"];
}

diagram-declared-effective.png

The pattern, named

Two states, one name. The file is the declared state; the process holds the effective state, loaded at exec time and mutated since. Failure directions:

  1. Stale-effective (the classic): config edited, process never reloaded. Everyone believes the change is live. [E]
  2. Unpersisted-effective (the inverse): runtime mutated directly (debugger, admin CLI, hot patch), never written back. Lost on reboot; worse, the reboot "regression" gets blamed on the reboot.
  3. Delayed detonation (the compound): edited-but-unloaded config is a landmine. A restart is an unreviewed deployment of every config change since the last restart, and it typically fires mid-incident, when someone bounces the service for unrelated reasons.2

Direction 3 is the argument that the divergence is not merely stale but unaudited pending state with no provenance. The change has an author and a timestamp on disk; the activation has neither.

Prior art

Designed-in duality: Cisco IOS

running-config vs startup-config: two named states, explicit sync verbs (copy running-config startup-config), both failure directions representable and routinely realized. The single strongest prior-art citation that treats the divergence as an ontological fact rather than a bug to be papered over. [E] Every wr mem forgotten before a power event is direction 2; every startup edit without a reload is direction

  1. The CLI makes you say which state you are reading. Almost nothing

since has.

Restart as a formal primitive

  • Software rejuvenation: Huang, Kintala, Kolettis, Fulton, FTCS-25 (1995).3 Planned restarts to discard accumulated effective state before it kills you.
  • Crash-only software / microreboots: Candea & Fox, HotOS IX (2003), recovery-oriented computing. If crash-recovery is the only start path, effective state is always derivable from declared state, because there is no other kind of start. [E]
  • Patriot missile, Dhahran (incident 1991-02-25; report GAO/IMTEC-92-26, 1992): the inverse specimen. Clock drift lived in runtime state, not config; periodic reboot was the fix. Restart as drift-reset rather than config-activation. [E]

The reboot is the one guaranteed transition to a state derivable from the declaration. This is why it works and why it teaches nothing.

Delayed detonation in the incident literature

The SRE corpus (Google SRE book configuration chapters; assorted public postmortems) documents direction 3 repeatedly: inert config activated months later by an unrelated restart.4 The operational corollary: configtest before every restart is not paranoia, it is the only refutation harness on the activation edge most stacks have.

Eliminating the class: congruence vs convergence

  • Traugott & Brown, "Why Order Matters: Turing Equivalence in Automated Systems Administration," LISA 2002. Sharpest theoretical statement: convergent tools (CFEngine; Burgess, "Computer Immunology," LISA 1998 [E]; later formalized as promise theory, Burgess 2005 [E]) tolerate divergence and nudge toward the declaration; congruent systems rebuild from the declaration so the question cannot arise. [E]
  • Dolstra, The Purely Functional Software Deployment Model (PhD,

    1. → NixOS; Guix system reconfigure. The declaration is a

    derivation; activation is the refinement check. Endpoint of the congruence line.5

  • Fowler's bliki triad (SnowflakeServer 2012, PhoenixServer, ImmutableServer): the popularized middle.
  • Reconciliation loops (Kubernetes controllers, Crossplane): the invariant checked continuously rather than at edges. Cost already noted elsewhere on this site: the same loop that heals drift widens blast radius, because the system mutates in response to divergence without a plan gate.6

Detection tooling (the refutation harnesses that exist)

Harness Claim it refutes
terraform plan "cloud state matches the declaration"
needrestart / checkrestart "running processes reflect installed libraries"
lsof +L1 same, one layer down (deleted .so still mapped)
nginx -t, sshd -t, named-checkconf "the pending declaration is even loadable"
systemctl daemon-reload hint "systemd's view matches /etc/fstab et al."
/var/run/reboot-required "no kernel/ABI debt is pending"

Each is elenctic: it exists because a specific implicit claim kept turning out false. The gaps in this table are the research program.

Pop culture register

The IT Crowd (2006) owns the phrase, but "have you tried turning it off and on again?" encodes a real epistemic position: the responder cannot cheaply learn the system's effective state, so the rational move is to force effective back to declared via the one guaranteed sync point. The joke works because the reboot is epistemically honest and diagnostically bankrupt — it fixes without explaining, and it destroys the evidence.7 Adjacent artifacts: the Windows "pending reboot" purgatory; the three-finger salute as cultural muscle memory; BOFH's weaponized restarts.

Specimen log (local incidents, this homelab)

Collected before the genus was named. All direction 1 unless noted.

  • /etc/fstab edited on pi; mount fails; systemd holds the old view until systemctl daemon-reload. Textbook: two declared states (file, systemd's parsed copy) and one effective, diverging pairwise.
  • sshtunnel on nexushive: stale PID file plus orphaned daemon; service restart reports failure while the old process keeps the old config live. Direction 1 with a corrupted supervisor view of which effective state even exists.
  • InspIRCd: startup failure leaves no effective state while the declared state looks complete; the divergence detected only by pgrep, i.e., by hand.
  • RB-002 (emcp presence): mDNS TXT ts field stale > 10 min. The broadcast is a declared state about effective state; when the sync timer dies, peers reason from a stale declaration. Same genus, one meta-level up.
  • Ollama on minibos: Docker and local daemon both bound to :11434; "restart" kills one effective state while the other keeps serving. Restart is only a sync point if you restart the process that holds the state.

Mapping to the house frameworks

  • Governance tuple: (declared, effective, activation-actor, activation-policy, audit-edge). Most stacks populate the first two and leave the rest null. The Cisco design populates all five, which is why it is the reference point.
  • Elenctic reading: every config file asserts "I describe the running system." terraform plan, needrestart, configtest are refutation harnesses for that assertion. Where no harness exists, the assertion is not established, merely unrefuted by neglect. [P]
  • CPRR: conjecture = "restart edges are sufficient sync points"; refutations = specimen log above; refinement = either continuous reconciliation (accept the blast-radius cost) or congruence (accept the rebuild cost). The unrefined middle — edit files, hope someone reloads — is where production lives.

Open questions

  1. Is there a useful formalization of the activation edge as a refinement mapping (declared spec ⊑ effective implementation) checkable without restart? Bisimulation is too strong; what is the weakest useful relation? [H]
  2. Provenance on effective state: what would an append-only activation logbook look like at the OS level — every config load recorded with (file hash, actor, timestamp)? Harbormaster's logbook shape applies directly. Next action: sketch a divert(4)-style shim or audit(4) hook on FreeBSD for config-open events by supervised daemons.
  3. The LLM-in-the-loop variant: model-version drift is direction 1 where the "config" is a model identifier and the provider restarts the effective state for you, silently. Contract surface undefined. Feeds the schema-pact line. [I]

Related in-corpus

  • Latent Configuration Patterns — the empirical postmortem this note is the theoretical scaffold for (parent).
  • Annotation Systems — the :VERDICT: drawer + verification ledger address the analogous declared-vs-observed gap for claims rather than configuration.
  • Time-Travel Chat: First-Principles Spec — the same declared/effective distinction (message-as-declared, session-as-effective, transcript-as-observed) for conversational state.
  • CI/CD Pipelines (2020) — deploy step of the operator table; the deploy → activate gap became the standard observability question of the 2020s.

Footnotes:

1

"No invariant" is the load-bearing phrase: there is usually a convention (SIGHUP, reload verbs) but per-daemon, partial, and unverified. Provenance: synthesis of this site's own incident notes plus the survey below; claim original in framing, not in content.

2

Phrase "a restart is an unreviewed deployment" — mine, but the pattern is documented across the public postmortem corpus; specific incident citations not pinned here. Provenance flag: recalled, not independently re-checked against a named postmortem. Downgrade to [P] if a reader needs a citable instance and cannot find one in ten minutes of the usual archives.

3

Huang, Kintala, Kolettis, Fulton, "Software rejuvenation: analysis, module and applications," FTCS-25, 1995. Title verified against Semantic Scholar (lowercase "module" is intentional in the original). Upgrade year/venue and title string to [E].

4

The SRE book treats config as code with a release process precisely because activation is a deploy. Provenance: recalled from reading, chapter number not pinned.

5

"Closest but not closed": NixOS makes the system declaration congruent but a long-running daemon still accumulates process-internal effective state (caches, pools, negotiated protocol versions) that no switch-to-configuration touches without restart. The gap moves; it does not vanish. Refutation of the stronger claim "Nix solves this": any NixOS host where nixos-rebuild switch succeeded and behavior still required a service restart to change. Trivially exhibitable.

6

Cross-reference: the Terraform-abstraction survey note (reconciliation section). Same tradeoff, stated there in IaC terms.

7

Evidence destruction is the real cost and the reason crash-only architecture is not a diagnostic strategy. If the fault recurs, you have bought availability with observability. Concrete next action for crowsnest: capture effective-state snapshot (open FDs, mapped libs, loaded-config hashes) as a pre-restart hook, so the reboot stops being an evidence shredder.