The Stampede Simulator

The Stampede Simulator: twenty workers hit a hiccuping upstream, all fail at once, all retry. With jitter OFF they come back as a single synchronized herd - the second outage. With jitter ON (the default, as of this round) the herd spreads. The histogram is the argument.

source on github

a real captured run

STAMPEDE SIMULATOR: 20 workers, all failing at t=0, 120ms constant backoff

  jitter OFF (retry arrivals per 20ms bucket):
     320- 340ms  #################### 20

  jitter ON - the default (same workers, same failure, same seed):
      80- 100ms  #                    1
     100- 120ms  ###########          11
     120- 140ms  ###                  3
     140- 160ms  #####                5

  peak herd size per bucket:  20 without jitter  ->  11 with

twenty synchronized retries is a second outage wearing a recovery
costume. jitter is on by default now; the upstream sends its thanks.

source

# frozen_string_literal: true

# The Stampede Simulator: twenty workers hit a hiccuping upstream, all
# fail at once, all retry. With jitter OFF they come back as a single
# synchronized herd - the second outage. With jitter ON (the default,
# as of this round) the herd spreads. The histogram is the argument.
#
#   bundle exec ruby examples/stampede_sim.rb [seed]
#
# Runs offline; the upstream is a shared counter with feelings.

require class="s">"bundler/setup"
require class="s">"agentic"

# 40 scripted failures are the point, not news
Agentic.logger.level = class="y">:fatal

WORKERS = 20
BACKOFF = 0.12
BUCKET_MS = 20

def run_stampede(jitter:, seed:)
  srand(seed) # jitter uses Kernel#rand; seed it for a fair comparison
  started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
  retry_arrivals = []
  attempts = Hash.new(0)

  orchestrator = Agentic:class="y">:PlanOrchestrator.new(
    concurrency_limit: WORKERS,
    retry_policy: {
      max_retries: 2,
      retryable_errors: [class="s">"RuntimeError"],
      backoff_strategy: class="y">:constant,
      backoff_constant: BACKOFF,
      backoff_jitter: jitter
    }
  )

  WORKERS.times do |i|
    task = Agentic:class="y">:Task.new(
      description: class="s">"worker-#{i}",
      agent_spec: {class="s">"name" => class="s">"worker", class="s">"instructions" => class="s">"call upstream"}
    )
    orchestrator.add_task(task, agent: ->(t) {
      attempts[t.description] += 1
      if attempts[t.description] == 1
        raise class="s">"upstream hiccup" # everyone fails together at t=0
      end

      retry_arrivals << Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started
      class="y">:ok
    })
  end

  orchestrator.execute_plan
  retry_arrivals
end

def histogram(arrivals)
  buckets = Hash.new(0)
  arrivals.each { |t| buckets[(t * 1000 / BUCKET_MS).floor * BUCKET_MS] += 1 }
  buckets.sort.map { |bucket_ms, count|
    format(class="s">"    %4d-%4dms  %-20s %d", bucket_ms, bucket_ms + BUCKET_MS, class="s">"#" * count, count)
  }.join(class="s">"\n")
end

seed = (ARGV.first || 20260707).to_i

puts class="s">"STAMPEDE SIMULATOR: #{WORKERS} workers, all failing at t=0, " \
  class="s">"#{(BACKOFF * 1000).round}ms constant backoff"
puts

herd = run_stampede(jitter: false, seed: seed)
puts class="s">"  jitter OFF (retry arrivals per #{BUCKET_MS}ms bucket):"
puts histogram(herd)
peak_off = herd.group_by { |t| (t * 1000 / BUCKET_MS).floor }.values.map(&class="y">:size).max

spread = run_stampede(jitter: true, seed: seed)
puts
puts class="s">"  jitter ON - the default (same workers, same failure, same seed):"
puts histogram(spread)
peak_on = spread.group_by { |t| (t * 1000 / BUCKET_MS).floor }.values.map(&class="y">:size).max

puts
puts format(class="s">"  peak herd size per bucket:  %d without jitter  ->  %d with", peak_off, peak_on)
puts
puts class="s">"twenty synchronized retries is a second outage wearing a recovery"
puts class="s">"costume. jitter is on by default now; the upstream sends its thanks."