The Retry Budget

The Retry Budget: a retry storm is a self-inflicted DDoS - every job politely retrying 3x turns one outage into four. Retries are a SHARED resource, so give the fleet one wallet: transient failures spend from it, hopeless failures spend nothing (they get no retry at all), and when the wallet is empty the kindest thing left is failing fast - the fleet already knows the upstream is down.

source on github

a real captured run

RETRY BUDGET (12 jobs, upstream dead, max 3 retries each)

  strategy A - every job for itself:
    45 calls fired at a host that was down for all of them.
    11 transient jobs x (1 + 3 retries) + 1 auth job x 1 = 45:
    the outage was 1 incident; the fleet made it 45 requests.

  strategy B - one wallet of 5 retries for the whole fleet:
    17 calls total: 12 first attempts + 5 budgeted retries.
    10 jobs failed FAST once the wallet emptied - no call, no
    timeout, no bill. the auth job spent nothing from the wallet:
    hopeless failures don't get retries, so they can't drain the
    budget the transient ones might still need.

  the fleet cut 28 pointless requests (45 -> 17) and lost nothing -
  every retry was doomed anyway. per-job retry policies answer
  "should I try again?"; the budget answers "should ANYONE?" -
  round 9's breaker asked that per-upstream, this asks it per-
  window, and both read the same journaled verdicts. retries are
  a shared resource. give them a wallet, not a habit. (and the
  wallet is now a real windowed RateLimit - try_acquire says no
  without making anyone wait for it, which is the entire point.)

source

# frozen_string_literal: true

# The Retry Budget: a retry storm is a self-inflicted DDoS - every
# job politely retrying 3x turns one outage into four. Retries are a
# SHARED resource, so give the fleet one wallet: transient failures
# spend from it, hopeless failures spend nothing (they get no retry
# at all), and when the wallet is empty the kindest thing left is
# failing fast - the fleet already knows the upstream is down.
#
#   bundle exec ruby examples/retry_budget.rb
#
# Runs offline; the upstream is dead for the whole run.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

MAX_RETRIES = 3
JOBS = 12

# The wallet is a windowed RateLimit asked politely: try_acquire
# (round 11) answers false RIGHT NOW instead of making the caller
# wait for capacity - and waiting for retry capacity during an
# outage would just be the storm with extra steps. The custom
# budget class this example shipped with has retired.
class RetryBudget
  def initialize(allowance, per: 60)
    @wallet = Agentic:class="y">:RateLimit.new(allowance, per: per)
    @spent = 0
  end

  attr_reader class="y">:spent

  def spend?
    admitted = @wallet.try_acquire
    @spent += 1 if admitted
    admitted
  end
end

def run_job(name, error, journal)
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(
    lifecycle_hooks: journal.lifecycle_hooks,
    retry_policy: {max_retries: 0, retryable_errors: []}
  )
  orchestrator.add_task(Agentic:class="y">:Task.new(
    description: name, agent_spec: {class="s">"name" => class="s">"w", class="s">"instructions" => class="s">"sync"}
  ), agent: ->(_t) { raise error })
  orchestrator.execute_plan
end

def drill(strategy, budget: nil)
  path = File.join(Dir.tmpdir, class="s">"agentic_budget_#{strategy}.jsonl")
  File.delete(path) if File.exist?(path)
  journal = Agentic:class="y">:ExecutionJournal.new(path: path)

  calls = 0
  fast_failed = 0
  JOBS.times do |i|
    # Job 7 hits a revoked key; everyone else hits the dead upstream
    error = (i == 7) ? Agentic:class="y">:Errors:class="y">:LlmAuthenticationError.new(class="s">"401") : Agentic:class="y">:Errors:class="y">:LlmServerError.new(class="s">"503")
    attempts = 0
    loop do
      run_job(class="s">"job#{i}", error, journal)
      calls += 1
      attempts += 1

      verdict = Agentic:class="y">:ExecutionJournal.replay(path: path).events
        .reverse.find { |e| e[class="y">:event] == class="s">"task_failed" }[class="y">:retryable]
      break if verdict == false # hopeless: no retry spends anything, ever
      break if attempts > MAX_RETRIES

      if budget
        unless budget.spend?
          fast_failed += 1
          break
        end
      end
    end
  end
  [calls, fast_failed, budget&.spent]
end

puts class="s">"RETRY BUDGET (#{JOBS} jobs, upstream dead, max #{MAX_RETRIES} retries each)"
puts

calls_a, = drill(class="s">"naive")
puts class="s">"  strategy A - every job for itself:"
puts class="s">"    #{calls_a} calls fired at a host that was down for all of them."
puts class="s">"    11 transient jobs x (1 + #{MAX_RETRIES} retries) + 1 auth job x 1 = #{calls_a}:"
puts class="s">"    the outage was 1 incident; the fleet made it #{calls_a} requests."
puts

budget = RetryBudget.new(5)
calls_b, fast_failed, spent = drill(class="s">"budgeted", budget: budget)
puts class="s">"  strategy B - one wallet of 5 retries for the whole fleet:"
puts class="s">"    #{calls_b} calls total: #{JOBS} first attempts + #{spent} budgeted retries."
puts class="s">"    #{fast_failed} jobs failed FAST once the wallet emptied - no call, no"
puts class="s">"    timeout, no bill. the auth job spent nothing from the wallet:"
puts class="s">"    hopeless failures don't get retries, so they can't drain the"
puts class="s">"    budget the transient ones might still need."
puts
puts class="s">"  the fleet cut #{calls_a - calls_b} pointless requests (#{calls_a} -> #{calls_b}) and lost nothing -"
puts class="s">"  every retry was doomed anyway. per-job retry policies answer"
puts class="s">"  \"should I try again?\class="s">"; the budget answers \"should ANYONE?\class="s">" -"
puts class="s">"  round 9's breaker asked that per-upstream, this asks it per-"
puts class="s">"  window, and both read the same journaled verdicts. retries are"
puts class="s">"  a shared resource. give them a wallet, not a habit. (and the"
puts class="s">"  wallet is now a real windowed RateLimit - try_acquire says no"
puts class="s">"  without making anyone wait for it, which is the entire point.)"