The Flaky API Drill

The Flaky API Drill: a task that times out twice before succeeding, run under a retry policy with exponential backoff and a journal. The timeline shows every attempt, every backoff gap, and the journal proves the whole ordeal - failures included - survived to disk.

source on github

a real captured run

FLAKY API DRILL (max 3 retries, exponential backoff from 100ms)

   82ms  > attempt 1 calling the flaky API...
ERROR: Task execution failed: upstream took too long
   85ms  x attempt failed: TimeoutError (upstream took too long)
  207ms  > attempt 2 calling the flaky API...
ERROR: Task execution failed: upstream took too long
  210ms  x attempt failed: TimeoutError (upstream took too long)
  400ms  > attempt 3 calling the flaky API...
  400ms  + sync:accounts succeeded: {"synced"=>42}
  407ms  + audit:trail succeeded: "audited 42 accounts"

plan: completed in 329ms, 3 attempts for one success

the journal remembers the whole ordeal:
  2 failed attempts and 2 successes on disk
  completed?("sync:accounts") => true  (by name - a rerun tomorrow gets new task ids and still knows)
  completed?("audit:trail")   => true

source

# frozen_string_literal: true

# The Flaky API Drill: a task that times out twice before succeeding,
# run under a retry policy with exponential backoff and a journal.
# The timeline shows every attempt, every backoff gap, and the journal
# proves the whole ordeal - failures included - survived to disk.
#
#   bundle exec ruby examples/flaky_api_drill.rb
#
# Runs offline; the flakiness is scripted so the drill is repeatable.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

# The error class name must match the retry policy's retryable_errors
class TimeoutError < StandardError; end

JOURNAL = File.join(Dir.tmpdir, class="s">"agentic_flaky_drill.journal.jsonl")
File.delete(JOURNAL) if File.exist?(JOURNAL)

journal = Agentic:class="y">:ExecutionJournal.new(path: JOURNAL)
started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
stamp = -> { format(class="s">"%5dms", (Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started) * 1000) }

timeline_hooks = journal.lifecycle_hooks(
  after_task_failure: ->(task_id:, task:, failure:, duration:) {
    puts class="s">"#{stamp.call}  x attempt failed: #{failure.type} (#{failure.message})"
  },
  after_task_success: ->(task_id:, task:, result:, duration:) {
    puts class="s">"#{stamp.call}  + #{task.description} succeeded: #{result.output.inspect}"
  }
)

orchestrator = Agentic:class="y">:PlanOrchestrator.new(
  concurrency_limit: 1,
  lifecycle_hooks: timeline_hooks,
  retry_policy: {
    max_retries: 3,
    retryable_errors: [class="s">"TimeoutError"],
    backoff_strategy: class="y">:exponential,
    backoff_base: 0.1
  }
)

# Fails twice with a retryable timeout, then delivers
attempts = 0
sync = Agentic:class="y">:Task.new(
  description: class="s">"sync:accounts",
  agent_spec: {class="s">"name" => class="s">"AccountSync", class="s">"instructions" => class="s">"sync"},
  payload: nil
)
orchestrator.add_task(sync, agent: ->(_t) {
  attempts += 1
  puts class="s">"#{stamp.call}  > attempt #{attempts} calling the flaky API..."
  raise TimeoutError, class="s">"upstream took too long" if attempts < 3

  {class="s">"synced" => 42}
})

# An innocent bystander task, to show the plan keeps moving
audit = Agentic:class="y">:Task.new(
  description: class="s">"audit:trail",
  agent_spec: {class="s">"name" => class="s">"Auditor", class="s">"instructions" => class="s">"audit"}
)
orchestrator.add_task(audit, [sync], agent: ->(t) {
  class="s">"audited #{t.output_of(sync)["syncedclass="s">"]} accounts"
})

puts class="s">"FLAKY API DRILL (max 3 retries, exponential backoff from 100ms)"
puts
result = orchestrator.execute_plan
puts
puts class="s">"plan: #{result.status} in #{(result.execution_time * 1000).round}ms, " \
  class="s">"#{attempts} attempts for one success"

state = Agentic:class="y">:ExecutionJournal.replay(path: JOURNAL)
failures = state.events.count { |e| e[class="y">:event] == class="s">"task_failed" }
successes = state.events.count { |e| e[class="y">:event] == class="s">"task_succeeded" }
puts
puts class="s">"the journal remembers the whole ordeal:"
puts class="s">"  #{failures} failed attempts and #{successes} successes on disk"
puts class="s">"  completed?(\"sync:accounts\class="s">") => #{state.completed?("sync:accountsclass="s">")}  (by name - " \
  class="s">"a rerun tomorrow gets new task ids and still knows)"
puts class="s">"  completed?(\"audit:trail\class="s">")   => #{state.completed?("audit:trailclass="s">")}"