Latent Configuration: When Deploy != Activate
A Case Study in Dormant Misconfigurations

Running state diverges from declared state over time. Manual changes accumulate. A restart reveals the drift by applying the declared state.

Configuration Drift:
  Declared: service_port=8080
  Running:  service_port=9090 (manual change)
  Restart:  Port reverts to 8080, breaks clients

Latent Configuration:
  Declared: new_feature=true
  Running:  new_feature=false (package not installed)
  Upgrade:  Feature activates, breaks dependencies

Intentional pattern: deploy code but don't activate it. Feature flags control activation. Latent configuration is the unintentional version.

Who gets paged when latent configuration activates? Often not the person who wrote it. The activation event (upgrade, failover, restart) is temporally and causally disconnected from the configuration change.

• Long-lived servers accumulate state
• Manual configuration changes persist
• Upgrades reveal latent misconfigurations
• Debugging requires archaeology

# Hydra has 847 days of accumulated uptime (running since early 2024)
$ uptime
 9:03PM  up 847 days, 12:34
# Config changes from 2+ years ago may still be latent

• Servers are ephemeral and replaceable
• All state comes from declared configuration
• No latent config—each boot applies current state
• Failures are remediated by replacement, not repair

# Kubernetes approach: pod restart applies new config immediately
$ kubectl rollout restart deployment/telegraf

If we had proper GitOps reconciliation:

# Desired state in Git
telegraf:
  package: installed
  config: /usr/local/etc/telegraf.conf
  enabled: true
  running: true

# Reconciliation would fail on Apr 4:
# ERROR: telegraf.config declared but telegraf.package not installed
# ERROR: telegraf.enabled but telegraf service not found

1. Partial GitOps: Config in Git, but package state not declared
2. No reconciliation loop: Manual deployment, no drift detection
3. Deferred activation: rc.conf enables a service that doesn't exist yet

Old trading code was left on one of eight servers during deployment. When activated, it executed obsolete trading logic at high speed.

• Pattern: Deploy to 7/8 servers; 8th had latent old config
• Activation: Market open triggered code path
• Damage: $440M loss, company sold

A typo in a manual command removed more servers than intended. The playbook hadn't been tested against current fleet size.

• Pattern: Manual runbook diverged from automation
• Activation: Human execution of stale procedure
• Damage: 4-hour outage, cascading AWS failures

A WAF rule update caused catastrophic backtracking. The rule had been in the repo for weeks but wasn't deployed.

• Pattern: Staged deployment with manual promotion
• Activation: Global deploy of accumulated changes
• Damage: 27-minute global outage

A sysadmin ran the wrong command on the wrong server. Backup procedures had been documented but not tested.

• Pattern: Documented config vs actual config divergence
• Activation: Manual intervention during incident
• Damage: 6 hours data loss, 18 hours to recover

Burgess distinguishes these concepts:

• Idempotence: Running an operation multiple times produces the same result as running it once (f(f(x)) = f(x))
• Convergence: The system moves toward a desired end-state from any initial state, self-repairing broken parts

Our incident violated convergence: the system could not self-repair because the repair mechanism (telegraf) was not installed.

Burgess formalized autonomous system cooperation in "promise theory":

A promise is a declaration of intent by an autonomous agent.

Agent A promises Agent B to maintain property P.

Key properties:
- Voluntary: agents cannot be forced to keep promises
- Local: agents only control their own behavior
- Verifiable: outcomes can be checked independently

In our incident, the telegraf configuration file made a promise ("I will listen on port 30003") that could not be kept because the promising agent (telegraf service) did not exist.

Declarative tools detect drift by comparing desired to actual state. Imperative tools cannot—they only know what steps to run, not what state should exist.

Cattle eliminate latent configuration: each instance boots fresh, applying current declared state. There is no gap for configuration to become latent.

Our hydra server is a pet (847 days uptime), accumulating configuration changes that may not activate until the next reboot or upgrade.

Puppet uses a two-phase model:

1. Compile phase (on master): Generate catalog from manifests
2. Apply phase (on agent): Enforce catalog on target node

# Puppet manifest - telegraf.pp
package { 'telegraf':
  ensure => installed,
}

file { '/usr/local/etc/telegraf.conf':
  ensure  => file,
  source  => 'puppet:///modules/telegraf/telegraf.conf',
  require => Package['telegraf'],  # Explicit dependency
}

service { 'telegraf':
  ensure  => running,
  enable  => true,
  require => [Package['telegraf'], File['/usr/local/etc/telegraf.conf']],
}

How Puppet would have caught our bug:

• The require => Package['telegraf'] dependency means the file resource won't be applied until the package is installed
• The Puppet agent checks every 30 minutes and would report drift
• If the package isn't installed, the catalog application fails loudly

Puppet's limitation:

If the manifest is committed but never applied (no Puppet agent running), the configuration remains latent—exactly our scenario.

Ansible executes tasks sequentially in a playbook:

# playbook.yml
- name: Configure telegraf
  hosts: hydra
  tasks:
    - name: Install telegraf
      ansible.builtin.package:
        name: telegraf
        state: present

    - name: Deploy telegraf config
      ansible.builtin.copy:
        src: telegraf.conf
        dest: /usr/local/etc/telegraf.conf
      notify: restart telegraf

    - name: Ensure telegraf is running
      ansible.builtin.service:
        name: telegraf
        state: started
        enabled: yes

  handlers:
    - name: restart telegraf
      ansible.builtin.service:
        name: telegraf
        state: restarted

How Ansible would have caught our bug:

• Tasks execute in order; config deployment follows package installation
• If package installation fails, subsequent tasks fail
• Running ansible-playbook --check performs dry-run validation

Ansible's limitation:

Ansible is push-based, not pull-based. If you deploy config files manually (as we did with cp) without running the playbook, Ansible has no reconciliation loop to detect drift. The config sits latent until the next playbook run.

"Poor dependency handling is the number one cause of flaky, intermittent failures in large-scale Ansible playbooks." — Ansible best practices documentation

Terraform maintains explicit state and requires plan before apply:

# main.tf
resource "freebsd_pkg" "telegraf" {
  name = "telegraf"
}

resource "local_file" "telegraf_config" {
  filename = "/usr/local/etc/telegraf.conf"
  content  = file("${path.module}/telegraf.conf")

  depends_on = [freebsd_pkg.telegraf]
}

resource "freebsd_service" "telegraf" {
  name    = "telegraf"
  enabled = true
  running = true

  depends_on = [local_file.telegraf_config]
}

How Terraform would have caught our bug:

• terraform plan compares desired state to actual infrastructure
• If telegraf package isn't installed, plan shows it needs to be created
• Running terraform plan -refresh-only detects drift without proposing changes

Terraform's drift detection:

$ terraform plan -refresh-only
Refreshing state...

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform
since the last "terraform apply":

  # freebsd_pkg.telegraf has been deleted
  - resource "freebsd_pkg" "telegraf" {
      - name = "telegraf" -> null
    }

Terraform's limitation:

Terraform only knows about resources it manages. If you create a config file outside Terraform (manual cp or another tool), Terraform's state file doesn't track it. The file is invisible to drift detection.

"Terraform plan only compares the current state file with your configuration and doesn't always check what's really deployed." — HashiCorp Documentation

Chef has a notorious two-phase execution model:

1. Compile phase: Ruby code executes, building resource collection
2. Converge phase: Resources are applied to the system

# recipe/telegraf.rb
package 'telegraf' do
  action :install
end

template '/usr/local/etc/telegraf.conf' do
  source 'telegraf.conf.erb'
  notifies :restart, 'service[telegraf]'
end

service 'telegraf' do
  action [:enable, :start]
end

Classic Chef convergence bug:

# WRONG: File.exist? runs at COMPILE time, before package is installed
package 'telegraf'

if File.exist?('/usr/local/bin/telegraf')
  template '/usr/local/etc/telegraf.conf' do
    source 'telegraf.conf.erb'
  end
end

The File.exist? check runs during compile phase, before the package resource executes. The file doesn't exist yet, so the template is never added to the resource collection.

Correct pattern:

# RIGHT: only_if runs at CONVERGE time
template '/usr/local/etc/telegraf.conf' do
  source 'telegraf.conf.erb'
  only_if { ::File.exist?('/usr/local/bin/telegraf') }
end

How Chef would have caught our bug:

• Chef-client runs periodically (like Puppet agent)
• Resources with dependencies fail if dependencies aren't met
• Chef's why-run mode (like Ansible's --check) shows what would change

Chef's limitation:

If Chef isn't running on the node, or if the cookbook is never applied, configuration remains latent.

All tools share one vulnerability: they must be running to detect drift.

Our incident fell into the first category: telegraf.conf was committed to Git, but telegraf package wasn't installed. No tool was running to detect the discrepancy.

1. Git as single source of truth: All desired state in version control
2. Declarative descriptions: Describe what, not how
3. Automated reconciliation: Agents continuously sync actual to desired
4. Closed loop: Changes only through Git, never direct mutation

In a proper GitOps setup, our incident could not occur:

# GitOps manifest would declare the full stack
telegraf:
  package:
    state: installed
    version: "1.31.0"
  service:
    state: running
    enabled: true
  config:
    source: telegraf.conf
    ports:
      - 8125/udp  # StatsD
      - 8092/udp  # Influx Line Protocol
      # NOTE: Never 30003 - reserved for dump1090 SBS

The reconciliation agent would fail immediately if the package was not installed, rather than silently deploying configuration for a non-existent service.

"An Empirical Study on Configuration Errors in Commercial and Open Source Systems" analyzed 546 real-world configuration errors:

• 70.0%–85.5% are mistakes in setting configuration parameters
• 38.1%–53.7% of parameter mistakes are illegal parameters that violate format or rules
• 12.2%–29.7% are inconsistencies between parameter values
• Configuration errors account for a significant portion of production system failures

Key finding relevant to our incident: many configuration errors are latent until a specific trigger activates the misconfigured code path.

"Do Not Blame Users for Misconfigurations" argues that software developers should take active responsibility for configuration errors:

"Configuration errors are one of the major causes of today's system failures… These issues leave users clueless and forced to report to developers for technical support."

The paper proposes generating misconfigurations based on constraints inferred from source code—essentially fuzzing configuration space.

"Early Detection of Configuration Errors" monitors system metrics to catch misconfigurations before they cause major failures. The approach analyzes metric patterns to flag bad configurations like memory limits and thread pool sizes.