The Supervision Tree

The Supervision Tree: "let it crash" for plans. The agents in this file contain NO rescue clauses - error handling is not the worker's job. Recovery is a POLICY, and policies live one level up, in a supervisor that knows three strategies from OTP: one_for_one (restart the crashed child, keep everyone else's work), rest_for_one (restart the crashed child and every child started after it - their state may derive from its world), one_for_all (restart everything). And because a …

source on github

a real captured run

THE SUPERVISION TREE (recovery is a policy, and policies live one level up)

  strategy       restarts    runs per child (c/f/h/s)     who re-ran, and why
  one_for_one    1           1/2/1/1                      only fetch - its crash is its own
  rest_for_one   1           1/2/2/1                      fetch AND heartbeat - started after fetch, state suspect
  one_for_all    1           2/2/2/1                      everyone - the world is rebuilt

  the hopeless child: child :flaky_disk reached maximum restart intensity (3); escalating
    (ran 4 times: 1 start + 3 restarts, then UP the tree it goes)

  the agents in this file contain zero rescue clauses - that's the
  design, not an omission. "let it crash" splits every system into
  workers that do the happy path and supervisors that own recovery
  POLICY: whom to restart (the strategies differ exactly in their
  blast radius: 1, downstream, all) and how often (intensity, so a
  permanent failure escalates instead of looping). completed work
  is state the supervisor protects: heartbeat's result survived
  one_for_one, was rebuilt under rest_for_one - both on purpose.

source

# frozen_string_literal: true

# The Supervision Tree: "let it crash" for plans. The agents in this
# file contain NO rescue clauses - error handling is not the worker's
# job. Recovery is a POLICY, and policies live one level up, in a
# supervisor that knows three strategies from OTP: one_for_one
# (restart the crashed child, keep everyone else's work),
# rest_for_one (restart the crashed child and every child started
# after it - their state may derive from its world), one_for_all
# (restart everything). And because a supervisor that restarts
# forever is just a slow crash, restart INTENSITY is bounded: exceed
# it and the failure escalates up the tree, loudly.
#
#   bundle exec ruby examples/supervision_tree.rb
#
# Runs offline; the same crash is supervised three ways, then a
# hopeless child exhausts its restart budget.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

class Supervisor
  def initialize(strategy:, max_restarts: 3)
    @strategy = strategy
    @max_restarts = max_restarts
  end

  # children: [{name:, after: [names], agent: ->(deps_hash) {}}] in START ORDER
  def run(children)
    completed = {}
    restarts = 0

    loop do
      failed = execute_round(children, completed)
      return {status: class="y">:completed, outputs: completed, restarts: restarts} unless failed
      restarts += 1
      if restarts > @max_restarts
        return {status: class="y">:escalated, restarts: restarts - 1,
                reason: class="s">"child #{failed.inspect} reached maximum restart intensity (#{@max_restarts}); escalating"}
      end
      invalidated(children, failed).each { |name| completed.delete(name) }
    end
  end

  private

  # One plan run over the not-yet-completed children; finished
  # dependencies are injected so survivors never re-run just to feed
  # their dependents. Returns the first crashed child, or nil.
  def execute_round(children, completed)
    orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 4, retry_policy: {max_retries: 0, retryable_errors: []})
    tasks = {}
    pending = children.reject { |c| completed.key?(c[class="y">:name]) }
    pending.each do |child|
      task = Agentic:class="y">:Task.new(description: child[class="y">:name].to_s, agent_spec: {class="s">"name" => child[class="y">:name].to_s, class="s">"instructions" => class="s">"work"})
      deps = child[class="y">:after].filter_map { |d| tasks[d] } # only edges into this round
      orchestrator.add_task(task, deps, agent: ->(t) {
        inputs = child[class="y">:after].to_h { |d| [d, completed[d] || t.output_of(tasks[d])] }
        child[class="y">:agent].call(inputs)
      })
      tasks[child[class="y">:name]] = task
    end
    result = orchestrator.execute_plan
    pending.each do |child|
      task_result = result.task_result(tasks[child[class="y">:name]].id)
      completed[child[class="y">:name]] = task_result.output if task_result&.successful?
    end
    pending.find { |c| !completed.key?(c[class="y">:name]) }&.fetch(class="y">:name)
  end

  def invalidated(children, failed)
    names = children.map { |c| c[class="y">:name] }
    case @strategy
    when class="y">:one_for_one then [failed]
    when class="y">:rest_for_one then names.drop(names.index(failed))
    when class="y">:one_for_all then names
    end
  end
end

# The same tree for every scenario: a crash in :fetch on its first
# run, while :heartbeat (started after fetch) has already finished
def tree(runs)
  make = ->(name) {
    ->(_deps) {
      runs[name] += 1
      class="s">"#{name} ok"
    }
  }
  [
    {name: class="y">:connect, after: [], agent: make.call(class="y">:connect)},
    {name: class="y">:fetch, after: [class="y">:connect], agent: ->(_deps) {
      runs[class="y">:fetch] += 1
      raise Agentic:class="y">:Errors:class="y">:LlmRateLimitError, class="s">"upstream flapped" if runs[class="y">:fetch] == 1
      class="s">"fetch ok"
    }},
    {name: class="y">:heartbeat, after: [class="y">:connect], agent: make.call(class="y">:heartbeat)},
    {name: class="y">:serve, after: [class="y">:fetch], agent: make.call(class="y">:serve)}
  ]
end

puts class="s">"THE SUPERVISION TREE (recovery is a policy, and policies live one level up)"
puts
puts format(class="s">"  %-14s %-11s %-28s %s", class="s">"strategy", class="s">"restarts", class="s">"runs per child (c/f/h/s)", class="s">"who re-ran, and why")

expectations = {one_for_one: [1, 2, 1, 1], rest_for_one: [1, 2, 2, 1], one_for_all: [2, 2, 2, 1]}
failures = []
notes = {
  one_for_one: class="s">"only fetch - its crash is its own",
  rest_for_one: class="s">"fetch AND heartbeat - started after fetch, state suspect",
  one_for_all: class="s">"everyone - the world is rebuilt"
}

expectations.each_key do |strategy|
  runs = Hash.new(0)
  outcome = Supervisor.new(strategy: strategy).run(tree(runs))
  counts = [class="y">:connect, class="y">:fetch, class="y">:heartbeat, class="y">:serve].map { |n| runs[n] }
  failures << class="s">"#{strategy} ran #{counts.inspect}, expected #{expectations[strategy].inspect}" unless counts == expectations[strategy] && outcome[class="y">:status] == class="y">:completed
  puts format(class="s">"  %-14s %-11d %-28s %s", strategy, outcome[class="y">:restarts], counts.join(class="s">"/"), notes[strategy])
end

# The hopeless child: restart budgets exist because a supervisor that
# restarts forever is a crash loop with better manners
runs = Hash.new(0)
doomed = [{name: class="y">:flaky_disk, after: [], agent: ->(_d) {
  runs[class="y">:flaky_disk] += 1
  raise Agentic:class="y">:Errors:class="y">:LlmRateLimitError, class="s">"io error"
}}]
outcome = Supervisor.new(strategy: class="y">:one_for_one, max_restarts: 3).run(doomed)
puts
puts class="s">"  the hopeless child: #{outcome[class="y">:reason]}"
puts class="s">"    (ran #{runs[class="y">:flaky_disk]} times: 1 start + 3 restarts, then UP the tree it goes)"
failures << class="s">"escalation broke" unless outcome[class="y">:status] == class="y">:escalated && runs[class="y">:flaky_disk] == 4

puts
puts class="s">"  the agents in this file contain zero rescue clauses - that's the"
puts class="s">"  design, not an omission. \"let it crash\class="s">" splits every system into"
puts class="s">"  workers that do the happy path and supervisors that own recovery"
puts class="s">"  POLICY: whom to restart (the strategies differ exactly in their"
puts class="s">"  blast radius: 1, downstream, all) and how often (intensity, so a"
puts class="s">"  permanent failure escalates instead of looping). completed work"
puts class="s">"  is state the supervisor protects: heartbeat's result survived"
puts class="s">"  one_for_one, was rebuilt under rest_for_one - both on purpose."
exit(failures.empty? ? 0 : 1)