The Error Taxonomy Drill

The Error Taxonomy Drill: three tasks fail three different ways - a rate limit (retryable, says the error itself), an auth failure (not retryable, says the error itself), and a mystery error (no opinion, so the policy's type list decides). One retry policy, three correct outcomes, because errors now testify.

source on github

a real captured run

ERROR: Task execution failed: 429 slow down
ERROR: Task execution failed: 401 key revoked
ERROR: Task 6e1192e6-74be-44bf-8288-6406ba17ae8c failed: 401 key revoked
ERROR: Task execution failed: something vague
ERROR: Task execution failed: 429 slow down
ERROR TAXONOMY DRILL (max 3 retries for everyone)

  OK  rate-limited sync      3 attempt(s)  synced on attempt 3
  DEAD bad-credentials sync   1 attempt(s)  gave up: 401 key revoked
  OK  mystery-error sync     2 attempt(s)  recovered on attempt 2

plan: partial_failure

why each outcome is right:
  - the rate limit said retryable? -> true: retried until it cleared
  - the auth error said retryable? -> false: ONE attempt, even though
    someone unwisely put it in the retryable_errors list. a revoked
    key does not improve with persistence; the error knew that
  - the mystery RuntimeError had no opinion: the policy's type list
    decided, and it earned its second chance

source

# frozen_string_literal: true

# The Error Taxonomy Drill: three tasks fail three different ways -
# a rate limit (retryable, says the error itself), an auth failure
# (not retryable, says the error itself), and a mystery error (no
# opinion, so the policy's type list decides). One retry policy,
# three correct outcomes, because errors now testify.
#
#   bundle exec ruby examples/error_taxonomy_drill.rb
#
# Runs offline and deterministically.

require class="s">"bundler/setup"
require class="s">"agentic"

attempts = Hash.new(0)

drills = {
  class="s">"rate-limited sync" => lambda { |task|
    attempts[task.description] += 1
    if attempts[task.description] < 3
      raise Agentic:class="y">:Errors:class="y">:LlmRateLimitError.new(class="s">"429 slow down", retry_after: 1)
    end
    class="s">"synced on attempt #{attempts[task.description]}"
  },
  class="s">"bad-credentials sync" => lambda { |task|
    attempts[task.description] += 1
    raise Agentic:class="y">:Errors:class="y">:LlmAuthenticationError.new(class="s">"401 key revoked")
  },
  class="s">"mystery-error sync" => lambda { |task|
    attempts[task.description] += 1
    raise class="s">"something vague" if attempts[task.description] < 2
    class="s">"recovered on attempt #{attempts[task.description]}"
  }
}

orchestrator = Agentic:class="y">:PlanOrchestrator.new(
  concurrency_limit: 3,
  retry_policy: {
    max_retries: 3,
    backoff_strategy: class="y">:constant,
    backoff_constant: 0.02,
    # The type list is the fallback for errors with no opinion.
    # RuntimeError is listed; the auth error's own verdict will overrule
    # any list. That's the point.
    retryable_errors: [class="s">"RuntimeError", class="s">"Agentic:class="y">:Errors:class="y">:LlmAuthenticationError"]
  }
)

tasks = drills.map do |name, drill|
  task = Agentic:class="y">:Task.new(
    description: name,
    agent_spec: {class="s">"name" => name, class="s">"instructions" => class="s">"call the API"},
    payload: drill
  )
  orchestrator.add_task(task, agent: ->(t) { t.payload.call(t) })
  task
end

result = orchestrator.execute_plan

puts class="s">"ERROR TAXONOMY DRILL (max 3 retries for everyone)"
puts
tasks.each do |task|
  task_result = result.results[task.id]
  outcome = task_result.successful? ? task_result.output : class="s">"gave up: #{task_result.failure.message}"
  verdict = task_result.successful? ? class="s">"OK " : class="s">"DEAD"
  puts format(class="s">"  %s %-22s %d attempt(s)  %s", verdict, task.description, attempts[task.description], outcome)
end

puts
puts class="s">"plan: #{result.status}"
puts
puts class="s">"why each outcome is right:"
puts class="s">"  - the rate limit said retryable? -> true: retried until it cleared"
puts class="s">"  - the auth error said retryable? -> false: ONE attempt, even though"
puts class="s">"    someone unwisely put it in the retryable_errors list. a revoked"
puts class="s">"    key does not improve with persistence; the error knew that"
puts class="s">"  - the mystery RuntimeError had no opinion: the policy's type list"
puts class="s">"    decided, and it earned its second chance"