The Circuit Breaker

The Circuit Breaker: when an upstream is down, the cheapest request is the one you don't send. The breaker trips after 3 consecutive retryable failures (or ONE non-retryable - no auth error deserves a second strike), fast-fails while open, and probes with a single request before closing again. Every skipped call is money.

source on github

a real captured run

CIRCUIT BREAKER (trip after 3 strikes, cooldown 4 ticks)

  act one: 503s from tick 4 to 9
    tick   breaker     what happened
    0      closed      ok
    1      closed      ok
    2      closed      ok
    3      closed      ok
    4      closed      failed (retryable: true)
    5      closed      failed (retryable: true)
    6      open        failed (retryable: true) - breaker TRIPS
    7      open        call SKIPPED - not sent, not billed, not waited on
    8      open        call SKIPPED - not sent, not billed, not waited on
    9      open        call SKIPPED - not sent, not billed, not waited on
    10     closed      probe SUCCEEDED - breaker closes
    11     closed      ok
    12     closed      ok
    13     closed      ok
    14     closed      ok
    15     closed      ok

  act two: key revoked at tick 2
    tick   breaker     what happened
    0      closed      ok
    1      closed      ok
    2      open        failed (retryable: false) - breaker TRIPS
    3      open        call SKIPPED - not sent, not billed, not waited on
    4      open        call SKIPPED - not sent, not billed, not waited on
    5      open        call SKIPPED - not sent, not billed, not waited on
    6      open        failed (retryable: false) - breaker TRIPS
    7      open        call SKIPPED - not sent, not billed, not waited on

  act one's outage lasted 6 ticks but only 3 calls felt it: the
  breaker ate the middle as 3 skips (nothing sent, nothing billed,
  no timeout waited out) and spent one probe discovering the recovery.
  act two never reached three strikes - the 401's journaled verdict
  (retryable: false) tripped the breaker on FIRST contact, and the
  probe kept finding the same wall (2 real calls, 4 skipped).
  strike counts are for errors that might pass; testimony that the
  error can never pass deserves an instant trip.

source

# frozen_string_literal: true

# The Circuit Breaker: when an upstream is down, the cheapest request
# is the one you don't send. The breaker trips after 3 consecutive
# retryable failures (or ONE non-retryable - no auth error deserves a
# second strike), fast-fails while open, and probes with a single
# request before closing again. Every skipped call is money.
#
#   bundle exec ruby examples/circuit_breaker.rb
#
# Runs offline; act one is a 503 outage, act two a revoked key.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

# A breaker is a tiny state machine: closed -> open -> half_open -> closed
class Breaker
  TRIP_AFTER = 3
  COOLDOWN_TICKS = 4

  attr_reader class="y">:state, class="y">:skipped

  def initialize
    @state = class="y">:closed
    @strikes = 0
    @cooldown = 0
    @skipped = 0
  end

  def allow?
    case @state
    when class="y">:closed then true
    when class="y">:open
      @cooldown -= 1
      if @cooldown <= 0
        @state = class="y">:half_open
        true # the probe
      else
        @skipped += 1
        false
      end
    when class="y">:half_open then true
    end
  end

  def record_success
    @state = class="y">:closed
    @strikes = 0
  end

  # The framework's nil convention (TaskFailure#hopeless?): only an
  # EXPLICIT false verdict trips instantly. An error that expressed no
  # opinion gets a strike - suspicion, not a death sentence.
  def record_failure(verdict)
    hopeless = verdict == false
    @strikes += hopeless ? TRIP_AFTER : 1
    return unless @strikes >= TRIP_AFTER || @state == class="y">:half_open

    @state = class="y">:open
    @strikes = 0
    @cooldown = COOLDOWN_TICKS
  end
end

def run_scenario(name, ticks, journal_path, &upstream)
  File.delete(journal_path) if File.exist?(journal_path)
  journal = Agentic:class="y">:ExecutionJournal.new(path: journal_path)
  breaker = Breaker.new

  puts class="s">"  #{name}"
  puts format(class="s">"    %-6s %-11s %s", class="s">"tick", class="s">"breaker", class="s">"what happened")

  ticks.times do |tick|
    unless breaker.allow?
      puts format(class="s">"    %-6d %-11s call SKIPPED - not sent, not billed, not waited on", tick, breaker.state)
      next
    end

    probe = breaker.state == class="y">:half_open
    orchestrator = Agentic:class="y">:PlanOrchestrator.new(
      lifecycle_hooks: journal.lifecycle_hooks,
      retry_policy: {max_retries: 0, retryable_errors: []}
    )
    orchestrator.add_task(Agentic:class="y">:Task.new(
      description: class="s">"call:#{tick}", agent_spec: {class="s">"name" => class="s">"caller", class="s">"instructions" => class="s">"call"}
    ), agent: ->(_t) { upstream.call(tick) })
    result = orchestrator.execute_plan

    if result.successful?
      breaker.record_success
      puts format(class="s">"    %-6d %-11s %s", tick, breaker.state, probe ? class="s">"probe SUCCEEDED - breaker closes" : class="s">"ok")
    else
      # The journal's write-time verdict feeds the breaker - the error
      # already testified whether retrying could ever help
      verdict = Agentic:class="y">:ExecutionJournal.replay(path: journal_path).events
        .reverse.find { |e| e[class="y">:event] == class="s">"task_failed" }[class="y">:retryable]
      breaker.record_failure(verdict)
      puts format(class="s">"    %-6d %-11s failed (retryable: %s)%s", tick, breaker.state, verdict,
        (breaker.state == class="y">:open) ? class="s">" - breaker TRIPS" : class="s">"")
    end
  end
  puts
  [Agentic:class="y">:ExecutionJournal.replay(path: journal_path), breaker]
end

puts class="s">"CIRCUIT BREAKER (trip after #{Breaker:class="y">:TRIP_AFTER} strikes, cooldown #{Breaker:class="y">:COOLDOWN_TICKS} ticks)"
puts

# Act one: a transient outage - three strikes, trip, skip, probe, recover
OUTAGE = (4..9)
state, breaker = run_scenario(class="s">"act one: 503s from tick 4 to 9", 16,
  File.join(Dir.tmpdir, class="s">"agentic_breaker_1.jsonl")) do |tick|
  raise Agentic:class="y">:Errors:class="y">:LlmServerError, class="s">"503 (tick #{tick})" if OUTAGE.cover?(tick)

  class="s">"ok"
end

# Act two: a revoked key - ONE strike, because the error testified
# that no retry can ever help
state2, breaker2 = run_scenario(class="s">"act two: key revoked at tick 2", 8,
  File.join(Dir.tmpdir, class="s">"agentic_breaker_2.jsonl")) do |tick|
  raise Agentic:class="y">:Errors:class="y">:LlmAuthenticationError, class="s">"401 key revoked" if tick >= 2

  class="s">"ok"
end

felt = state.events.count { |e| e[class="y">:event] == class="s">"task_failed" }
puts class="s">"  act one's outage lasted #{OUTAGE.size} ticks but only #{felt} calls felt it: the"
puts class="s">"  breaker ate the middle as #{breaker.skipped} skips (nothing sent, nothing billed,"
puts class="s">"  no timeout waited out) and spent one probe discovering the recovery."
puts class="s">"  act two never reached three strikes - the 401's journaled verdict"
puts class="s">"  (retryable: false) tripped the breaker on FIRST contact, and the"
puts class="s">"  probe kept finding the same wall (#{state2.events.count { |e| e[class="y">:event] == "task_failedclass="s">" }} real calls, #{breaker2.skipped} skipped)."
puts class="s">"  strike counts are for errors that might pass; testimony that the"
puts class="s">"  error can never pass deserves an instant trip."