The Pinball Queue

The Pinball Queue: a job queue explained on a pinball table, because every retry policy I have ever shipped is already in the machine. Balls are jobs. Flippers are workers (two; the table is the concurrency limit). A DRAIN is a transient failure - and the BALL SAVE kicks it back automatically, which is all a retry is. TILT is the poison ball: you don't ball-save a tilt, you end the turn and put the ball in the trough (the dead letter office) where a human decides. And the …

source on github

a real captured run

THE PINBALL QUEUE (pinball is exciting; your job queue should not be)

  ball-1 (clean)                   5000 points
  ball-2 (rattles the drain)      12000 points after 1 ball save
  ball-3 (clean)                   7000 points
  ball-4 (TILT machine)            TILT  -> the trough (dead letters): TILT - nudged the machine like it owed them money
  ball-5 (drains twice!)          25000 points after 2 ball saves

  FINAL SCORE: 49000   balls launched: 5, scored: 4, in the trough: 1

  referee: every ball accounted - 3 drains on the instant replay
  (the journal), each ball-saved by policy; the tilt was NOT saved,
  because tilts are non-retryable by decree and retrying one is how
  you tilt twice. this is the whole gospel of background jobs on
  one table: transient failures get automatic, bounded, backed-off
  retries (the ball save has a budget - drain a fourth time and
  you're done); poison gets a human (the trough is a QUEUE, not a
  void - someone reviews it after the game); and the scoreboard
  balances or the machine is broken. exciting things happen on the
  playfield so that nothing exciting ever happens to the ledger.

source

# frozen_string_literal: true

# The Pinball Queue: a job queue explained on a pinball table,
# because every retry policy I have ever shipped is already in the
# machine. Balls are jobs. Flippers are workers (two; the table is
# the concurrency limit). A DRAIN is a transient failure - and the
# BALL SAVE kicks it back automatically, which is all a retry is.
# TILT is the poison ball: you don't ball-save a tilt, you end the
# turn and put the ball in the trough (the dead letter office) where
# a human decides. And the scoreboard practices double-entry
# bookkeeping: every ball launched is a ball scored or a ball
# troughed. Pinball is exciting. Your job queue should not be.
#
#   bundle exec ruby examples/pinball_queue.rb
#
# Runs offline; exits 1 if any ball goes unaccounted.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

BALLS = [
  {name: class="s">"ball-1 (clean)", behavior: class="y">:clean, points: 5000},
  {name: class="s">"ball-2 (rattles the drain)", behavior: class="y">:drains_once, points: 12_000},
  {name: class="s">"ball-3 (clean)", behavior: class="y">:clean, points: 7000},
  {name: class="s">"ball-4 (TILT machine)", behavior: class="y">:tilt, points: 0},
  {name: class="s">"ball-5 (drains twice!)", behavior: class="y">:drains_twice, points: 25_000}
].freeze

journal = Agentic:class="y">:ExecutionJournal.new(path: File.join(Dir.tmpdir, class="s">"agentic_pinball.jsonl"))
File.delete(journal.path) if File.exist?(journal.path)

orchestrator = Agentic:class="y">:PlanOrchestrator.new(
  concurrency_limit: 2, # two flippers
  lifecycle_hooks: journal.lifecycle_hooks,
  retry_policy: {max_retries: 2, backoff_base: 0.005, retryable_errors: [Agentic:class="y">:Errors:class="y">:LlmRateLimitError]}
)

plays = Hash.new(0)
tasks = BALLS.to_h do |ball|
  task = Agentic:class="y">:Task.new(description: ball[class="y">:name], agent_spec: {class="s">"name" => ball[class="y">:name], class="s">"instructions" => class="s">"play"})
  orchestrator.add_task(task, agent: ->(_t) {
    plays[ball[class="y">:name]] += 1
    case ball[class="y">:behavior]
    when class="y">:tilt
      # you do not ball-save a tilt; the turn is OVER
      raise Agentic:class="y">:Errors:class="y">:LlmAuthenticationError, class="s">"TILT - nudged the machine like it owed them money"
    when class="y">:drains_once
      raise Agentic:class="y">:Errors:class="y">:LlmRateLimitError, class="s">"center drain" if plays[ball[class="y">:name]] == 1
    when class="y">:drains_twice
      raise Agentic:class="y">:Errors:class="y">:LlmRateLimitError, class="s">"center drain, again" if plays[ball[class="y">:name]] <= 2
    end
    {points: ball[class="y">:points]}
  })
  [ball[class="y">:name], task]
end
result = orchestrator.execute_plan

puts class="s">"THE PINBALL QUEUE (pinball is exciting; your job queue should not be)"
puts
scored = 0
troughed = []
BALLS.each do |ball|
  task_result = result.task_result(tasks[ball[class="y">:name]].id)
  saves = plays[ball[class="y">:name]] - 1
  if task_result.successful?
    scored += task_result.output[class="y">:points]
    save_note = saves.positive? ? class="s">" after #{saves} ball save#{"sclass="s">" if saves > 1}" : class="s">""
    puts format(class="s">"  %-28s %8d points%s", ball[class="y">:name], task_result.output[class="y">:points], save_note)
  else
    troughed << ball[class="y">:name]
    puts format(class="s">"  %-28s %8s  -> the trough (dead letters): #{task_result.failure.message}", ball[class="y">:name], class="s">"TILT")
  end
end
puts
puts format(class="s">"  FINAL SCORE: %d   balls launched: %d, scored: %d, in the trough: %d",
  scored, BALLS.size, BALLS.size - troughed.size, troughed.size)

# --- the referee: double-entry bookkeeping for balls -------------------------------
failures = []
failures << class="s">"a ball vanished" unless BALLS.size == (BALLS.size - troughed.size) + troughed.size && troughed == [class="s">"ball-4 (TILT machine)"]
failures << class="s">"the tilt was ball-saved (never retry a tilt)" unless plays[class="s">"ball-4 (TILT machine)"] == 1
failures << class="s">"ball save #1 miscounted" unless plays[class="s">"ball-2 (rattles the drain)"] == 2
failures << class="s">"ball save #2 miscounted" unless plays[class="s">"ball-5 (drains twice!)"] == 3
failures << class="s">"score wrong" unless scored == 49_000
replayed = Agentic:class="y">:ExecutionJournal.replay(path: journal.path)
drains = replayed.events.count { |e| e[class="y">:event] == class="s">"task_failed" && e[class="y">:error].to_s.include?(class="s">"drain") }
failures << class="s">"instant replay disagrees (#{drains} drains on tape)" unless drains == 3

puts
puts class="s">"  referee: every ball accounted - 3 drains on the instant replay"
puts class="s">"  (the journal), each ball-saved by policy; the tilt was NOT saved,"
puts class="s">"  because tilts are non-retryable by decree and retrying one is how"
puts class="s">"  you tilt twice. this is the whole gospel of background jobs on"
puts class="s">"  one table: transient failures get automatic, bounded, backed-off"
puts class="s">"  retries (the ball save has a budget - drain a fourth time and"
puts class="s">"  you're done); poison gets a human (the trough is a QUEUE, not a"
puts class="s">"  void - someone reviews it after the game); and the scoreboard"
puts class="s">"  balances or the machine is broken. exciting things happen on the"
puts class="s">"  playfield so that nothing exciting ever happens to the ledger."
exit(failures.empty? ? 0 : 1)