The Executable Runbook

The Executable Runbook: every ops wiki has a page titled "If the queue gets stuck" with eleven steps, three of which are stale, one of which is dangerous, and none of which have been run since the person who wrote them left. The fix is to make the runbook a PROGRAM: each step declares a check (read-only: is this step even needed?), an action (the mutation), and a verify (did it work?). Dry-run mode executes only the checks and provably touches nothing. Live mode skips steps …

source on github

a real captured run

THE EXECUTABLE RUNBOOK (documentation that cannot rot silently)

  1. dry run against the sick system (the 3am confidence pass):
    [dry] pause intake             WOULD run
    [dry] clear stuck jobs         WOULD run
    [dry] restart workers          WOULD run
    [dry] resume intake            would skip (already satisfied)
    [dry] confirm queue draining   WOULD run
     world untouched: true (byte-compared; a dry run that writes is a lie)

  2. live run:
    [live] pause intake             ran; verified
    [live] clear stuck jobs         ran; verified
    [live] restart workers          ran; verified
    [live] resume intake            ran; verified
    [live] confirm queue draining   ran; verified
     system healthy: true

  3. the shaky-hands re-run (someone ran it twice; someone always does):
    [live] pause intake             skipped (check says already fine)
    [live] clear stuck jobs         skipped (check says already fine)
    [live] restart workers          skipped (check says already fine)
    [live] resume intake            skipped (check says already fine)
    [live] confirm queue draining   ran; verified

  the three properties, proven in order: DRY-RUN IS READ-ONLY
  (world byte-compared before and after - a dry run you can't
  trust is worse than none); EVERY STEP IS GUARDED (check before
  action, verify after - the book skips what's already fine, so
  a half-recovered system doesn't get un-recovered); and the
  RE-RUN IS SAFE (second pass executed 1 action - the
  always-assess step - because idempotency is what makes a
  runbook usable by someone at 3am whose hands are shaking).
  wikis rot because nothing fails when they do. this book runs
  in CI against a simulated sick system; the day a step goes
  stale, a build goes red, and the on-call finds out at 3PM
  instead of 3AM.

source

# frozen_string_literal: true

# The Executable Runbook: every ops wiki has a page titled "If the
# queue gets stuck" with eleven steps, three of which are stale,
# one of which is dangerous, and none of which have been run since
# the person who wrote them left. The fix is to make the runbook a
# PROGRAM: each step declares a check (read-only: is this step even
# needed?), an action (the mutation), and a verify (did it work?).
# Dry-run mode executes only the checks and provably touches
# nothing. Live mode skips steps whose checks say "already fine" -
# so the book is safe to re-run at 3am, twice, by someone whose
# hands are shaking. Documentation that executes cannot rot
# silently; it fails in CI like everything else you trust.
#
#   bundle exec ruby examples/executable_runbook.rb
#
# Runs offline; exits 1 unless dry-run is provably read-only and
# the live book is provably idempotent.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

# The world: a job system having a bad morning
def sick_world
  {intake: class="y">:open, workers: class="y">:wedged, queue: [class="y">:job, class="y">:job, class="y">:stuck_job, class="y">:job, class="y">:stuck_job], processed: 0}
end

RUNBOOK = [
  # The check tests for the PROBLEM, not for the action's applicability -
  # "intake is open" is true on a healthy system too, and a guard that
  # merely asks "could I?" will happily re-break what a re-run should skip
  {step: class="s">"pause intake",
   check: ->(w) { w[class="y">:workers] == class="y">:wedged || w[class="y">:queue].include?(class="y">:stuck_job) },
   action: ->(w) { w[class="y">:intake] = class="y">:paused },
   verify: ->(w) { w[class="y">:intake] == class="y">:paused }},
  {step: class="s">"clear stuck jobs",
   check: ->(w) { w[class="y">:queue].include?(class="y">:stuck_job) },
   action: ->(w) { w[class="y">:queue].reject! { |j| j == class="y">:stuck_job } },
   verify: ->(w) { !w[class="y">:queue].include?(class="y">:stuck_job) }},
  {step: class="s">"restart workers",
   check: ->(w) { w[class="y">:workers] != class="y">:running },
   action: ->(w) { w[class="y">:workers] = class="y">:running },
   verify: ->(w) { w[class="y">:workers] == class="y">:running }},
  {step: class="s">"resume intake",
   check: ->(w) { w[class="y">:intake] == class="y">:paused },
   action: ->(w) { w[class="y">:intake] = class="y">:open },
   verify: ->(w) { w[class="y">:intake] == class="y">:open }},
  {step: class="s">"confirm queue draining",
   check: ->(w) { true }, # always assess
   action: ->(w) { w[class="y">:processed] += w[class="y">:queue].size },
   verify: ->(w) { w[class="y">:processed].positive? }}
].freeze

def run_book(world, mode:)
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 1) # runbooks are read aloud, in order
  log = []
  actions_executed = 0
  previous = nil
  RUNBOOK.each do |entry|
    task = Agentic:class="y">:Task.new(description: entry[class="y">:step], agent_spec: {class="s">"name" => entry[class="y">:step], class="s">"instructions" => class="s">"op"})
    orchestrator.add_task(task, previous ? [previous] : [], agent: ->(_t) {
      needed = entry[class="y">:check].call(world)
      if mode == class="y">:dry_run
        log << class="s">"  [dry] #{entry[class="y">:step].ljust(24)} #{needed ? "WOULD runclass="s">" : "would skip (already satisfied)class="s">"}"
        next class="y">:assessed
      end
      if needed
        entry[class="y">:action].call(world)
        actions_executed += 1
        raise class="s">"VERIFY FAILED after #{entry[class="y">:step]}" unless entry[class="y">:verify].call(world)
        log << class="s">"  [live] #{entry[class="y">:step].ljust(24)} ran; verified"
      else
        log << class="s">"  [live] #{entry[class="y">:step].ljust(24)} skipped (check says already fine)"
      end
      class="y">:done
    })
    previous = task
  end
  status = orchestrator.execute_plan.status
  [log, actions_executed, status]
end

puts class="s">"THE EXECUTABLE RUNBOOK (documentation that cannot rot silently)"
puts

world = sick_world
before = Marshal.dump(world)
dry_log, dry_actions, = run_book(world, mode: class="y">:dry_run)
puts class="s">"  1. dry run against the sick system (the 3am confidence pass):"
dry_log.each { |l| puts class="s">"  #{l}" }
untouched = Marshal.dump(world) == before
puts class="s">"     world untouched: #{untouched} (byte-compared; a dry run that writes is a lie)"
puts

live_log, live_actions, live_status = run_book(world, mode: class="y">:live)
puts class="s">"  2. live run:"
live_log.each { |l| puts class="s">"  #{l}" }
healthy = world[class="y">:workers] == class="y">:running && world[class="y">:intake] == class="y">:open && !world[class="y">:queue].include?(class="y">:stuck_job)
puts class="s">"     system healthy: #{healthy}"
puts

rerun_log, rerun_actions, = run_book(world, mode: class="y">:live)
puts class="s">"  3. the shaky-hands re-run (someone ran it twice; someone always does):"
rerun_log.each { |l| puts class="s">"  #{l}" }
puts

failures = []
failures << class="s">"dry run mutated the world" unless untouched && dry_actions.zero?
failures << class="s">"live run failed (#{live_status})" unless live_status == class="y">:completed && healthy && live_actions == 5
failures << class="s">"re-run was not idempotent (#{rerun_actions} actions)" unless rerun_actions == 1 # only the always-assess step
puts class="s">"  the three properties, proven in order: DRY-RUN IS READ-ONLY"
puts class="s">"  (world byte-compared before and after - a dry run you can't"
puts class="s">"  trust is worse than none); EVERY STEP IS GUARDED (check before"
puts class="s">"  action, verify after - the book skips what's already fine, so"
puts class="s">"  a half-recovered system doesn't get un-recovered); and the"
puts class="s">"  RE-RUN IS SAFE (second pass executed #{rerun_actions} action - the"
puts class="s">"  always-assess step - because idempotency is what makes a"
puts class="s">"  runbook usable by someone at 3am whose hands are shaking)."
puts class="s">"  wikis rot because nothing fails when they do. this book runs"
puts class="s">"  in CI against a simulated sick system; the day a step goes"
puts class="s">"  stale, a build goes red, and the on-call finds out at 3PM"
puts class="s">"  instead of 3AM."
exit(failures.empty? ? 0 : 1)