Hot Config Reload

Hot Config Reload: change the server's configuration without dropping a request - a problem every long-running process has, and one with two classic wounds. Wound one: TORN READS. Update the live config hash field by field and a request that starts mid-update sees half old, half new (rate limit from v2, burst from v1 - now your limiter math is nonsense). Wound two: the BAD CONFIG. A reload that applies first and validates never is how a typo'd YAML takes down what the deploy …

source on github

a real captured run

HOT CONFIG RELOAD (in-flight requests keep their world; proposals prove themselves)

  the in-place update (mutate the live hash, field by field):
    120 requests served, 35 saw a TORN config (limit != burst mid-swap)

  the atomic swap (build aside, validate, freeze, assign once):
    120 requests served, 0 torn; config now v2

  the bad proposal (rate_limit 500, burst 100 - a typo'd deploy):
    validation refused it: rate_limit/burst must match (got 500/100)
    live config still v2, still serving - the reload failed, the SERVER didn't

  the whole pattern in four verbs: BUILD the new config as a
  separate object (never edit the one requests are holding);
  VALIDATE the proposal while it's still a proposal - invariants,
  not just parse success, because 'valid YAML' and 'valid config'
  are different claims; FREEZE it so nothing can tear it later;
  SWAP one reference, which Ruby makes atomic for free. requests
  in flight finish in the world they started in - a request is a
  promise, a config file is a proposal, and the server's job is
  to never confuse the two. 35 torn reads say the in-place
  shortcut isn't hypothetical; zero say the cure is complete.

source

# frozen_string_literal: true

# Hot Config Reload: change the server's configuration without
# dropping a request - a problem every long-running process has,
# and one with two classic wounds. Wound one: TORN READS. Update
# the live config hash field by field and a request that starts
# mid-update sees half old, half new (rate limit from v2, burst
# from v1 - now your limiter math is nonsense). Wound two: the BAD
# CONFIG. A reload that applies first and validates never is how a
# typo'd YAML takes down what the deploy didn't. The cure for both
# is the same discipline: build the ENTIRE new config off to the
# side, validate it there, freeze it, and swap ONE reference. In-
# flight requests keep the object they started with; the swap is
# atomic; invalid proposals never touch the living.
#
#   bundle exec ruby examples/hot_config_reload.rb
#
# Runs offline; exits 1 unless tearing is demonstrated, then cured,
# and the bad config is refused with the old one still serving.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

# Config invariant: limit and burst always ship as a matched pair
V1 = {version: 1, rate_limit: 100, burst: 100, motd: class="s">"steady"}.freeze
V2 = {version: 2, rate_limit: 200, burst: 200, motd: class="s">"spicy"}.freeze

# Serve requests concurrently while an updater changes config midway.
# Each request reads the config ONCE and checks the pair invariant.
def serve_through_update(holder, updater)
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 8)
  torn = 0
  served = 0
  add_request = ->(i) {
    task = Agentic:class="y">:Task.new(description: class="s">"req #{i}", agent_spec: {class="s">"name" => class="s">"req", class="s">"instructions" => class="s">"serve"})
    orchestrator.add_task(task, agent: ->(_t) {
      cfg = holder[class="y">:current] # one read; whatever object this is, we keep it
      sleep(0.004)           # the request does some work mid-flight
      torn += 1 if cfg[class="y">:rate_limit] != cfg[class="y">:burst] # the invariant a torn read breaks
      served += 1
      class="y">:ok
    })
  }
  16.times { |i| add_request.call(i) }
  # the reload arrives while traffic is flowing, as reloads do
  swap = Agentic:class="y">:Task.new(description: class="s">"config update", agent_spec: {class="s">"name" => class="s">"cfg", class="s">"instructions" => class="s">"swap"})
  orchestrator.add_task(swap, agent: ->(_t) {
    updater.call(holder)
    class="y">:updated
  })
  (16...120).each { |i| add_request.call(i) }
  orchestrator.execute_plan
  [torn, served]
end

VALIDATE = ->(candidate) {
  problems = []
  problems << class="s">"rate_limit/burst must match (got #{candidate[class="y">:rate_limit]}/#{candidate[class="y">:burst]})" if candidate[class="y">:rate_limit] != candidate[class="y">:burst]
  problems << class="s">"rate_limit must be positive" unless candidate[class="y">:rate_limit].to_i.positive?
  problems
}

puts class="s">"HOT CONFIG RELOAD (in-flight requests keep their world; proposals prove themselves)"
puts

# --- wound one: field-by-field mutation tears requests -------------------------------
holder = {current: V1.dup}
torn, served = serve_through_update(holder, ->(h) {
  h[class="y">:current][class="y">:rate_limit] = 200 # the tempting in-place edit
  sleep(0.02)                    # ...and the gap where requests see half a config
  h[class="y">:current][class="y">:burst] = 200
  h[class="y">:current][class="y">:version] = 2
})
puts class="s">"  the in-place update (mutate the live hash, field by field):"
puts class="s">"    #{served} requests served, #{torn} saw a TORN config (limit != burst mid-swap)"
puts

# --- the cure: build aside, validate, freeze, swap one reference ---------------------
holder = {current: V1}
torn2, served2 = serve_through_update(holder, ->(h) {
  candidate = V2 # built entirely off to the side
  raise class="s">"invalid" unless VALIDATE.call(candidate).empty?
  h[class="y">:current] = candidate.freeze # ONE reference assignment; old requests finish on V1
})
puts class="s">"  the atomic swap (build aside, validate, freeze, assign once):"
puts class="s">"    #{served2} requests served, #{torn2} torn; config now v#{holder[class="y">:current][class="y">:version]}"
puts

# --- wound two: the bad proposal never touches the living -----------------------------
bad = {version: 3, rate_limit: 500, burst: 100, motd: class="s">"oops"}
problems = VALIDATE.call(bad)
applied = problems.empty?
puts class="s">"  the bad proposal (rate_limit 500, burst 100 - a typo'd deploy):"
puts class="s">"    validation refused it: #{problems.first}"
puts class="s">"    live config still v#{holder[class="y">:current][class="y">:version]}, still serving - the reload failed, the SERVER didn't"
puts

failures = []
failures << class="s">"in-place mutation didn't tear (weird scheduling?)" unless torn.positive?
failures << class="s">"atomic swap tore (#{torn2})" unless torn2.zero? && served2 == 120
failures << class="s">"swap didn't take effect" unless holder[class="y">:current][class="y">:version] == 2
failures << class="s">"bad config was applied" if applied

puts class="s">"  the whole pattern in four verbs: BUILD the new config as a"
puts class="s">"  separate object (never edit the one requests are holding);"
puts class="s">"  VALIDATE the proposal while it's still a proposal - invariants,"
puts class="s">"  not just parse success, because 'valid YAML' and 'valid config'"
puts class="s">"  are different claims; FREEZE it so nothing can tear it later;"
puts class="s">"  SWAP one reference, which Ruby makes atomic for free. requests"
puts class="s">"  in flight finish in the world they started in - a request is a"
puts class="s">"  promise, a config file is a proposal, and the server's job is"
puts class="s">"  to never confuse the two. #{torn} torn reads say the in-place"
puts class="s">"  shortcut isn't hypothetical; zero say the cure is complete."
exit(failures.empty? ? 0 : 1)