The Failure Weather Report

The Failure Weather Report: a journal of three days, read as a forecast. Retryable failures are WEATHER - showers that pass on their own or with an umbrella. Non-retryable failures are CLIMATE - no amount of waiting fixes a drought; someone must dig a well. The journal now records which is which at the moment it rains.

Reliability & Recovery Round 9 Matz exit 0

source on github

bundle exec ruby examples/failure_weather.rb

a real captured run

FAILURE WEATHER REPORT (3 journaled days)

  Monday     storm damage (1 structural)  digest, backup, invoice
  Tuesday    storm damage (1 structural)  backup, invoice
  Wednesday  drought continues            invoice

  extended forecast:
    digest: rained earlier this week, clear now - weather does that
    backup: rained earlier this week, clear now - weather does that
    invoice: this is not weather, it is climate - 401 key expired

  6 rainy events this week: 3 were weather (they passed, or will),
  and 3 were the same drought, reported daily.
  no forecast fixes a drought: invoice's 401 has held for three days
  and will hold forever, because keys do not expire back. the journal
  told us which failures to wait out and which to dig a well for.

source

# frozen_string_literal: true

# The Failure Weather Report: a journal of three days, read as a
# forecast. Retryable failures are WEATHER - showers that pass on
# their own or with an umbrella. Non-retryable failures are CLIMATE -
# no amount of waiting fixes a drought; someone must dig a well.
# The journal now records which is which at the moment it rains.
#
#   bundle exec ruby examples/failure_weather.rb
#
# Runs offline; three scripted days of mixed conditions.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

JOURNAL = File.join(Dir.tmpdir, class="s">"agentic_weather.journal.jsonl")
File.delete(JOURNAL) if File.exist?(JOURNAL)
journal = Agentic:class="y">:ExecutionJournal.new(path: JOURNAL)

DAYS = [
  {name: class="s">"Monday",
   jobs: {class="s">"digest" => Agentic:class="y">:Errors:class="y">:LlmRateLimitError.new(class="s">"429"),
          class="s">"backup" => Agentic:class="y">:Errors:class="y">:LlmTimeoutError.new(class="s">"slow disk"),
          class="s">"invoice" => Agentic:class="y">:Errors:class="y">:LlmAuthenticationError.new(class="s">"401 key expired"),
          class="s">"greet" => nil}},
  {name: class="s">"Tuesday",
   jobs: {class="s">"digest" => nil, # the shower passed
          class="s">"backup" => Agentic:class="y">:Errors:class="y">:LlmTimeoutError.new(class="s">"slow disk again"),
          class="s">"invoice" => Agentic:class="y">:Errors:class="y">:LlmAuthenticationError.new(class="s">"401 key expired"),
          class="s">"greet" => nil}},
  {name: class="s">"Wednesday",
   jobs: {class="s">"digest" => nil,
          class="s">"backup" => nil, # cleared overnight
          class="s">"invoice" => Agentic:class="y">:Errors:class="y">:LlmAuthenticationError.new(class="s">"401 key expired"),
          class="s">"greet" => nil}}
].freeze

DAYS.each do |day|
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(
    lifecycle_hooks: journal.lifecycle_hooks,
    retry_policy: {max_retries: 0, retryable_errors: []}
  )
  day[class="y">:jobs].each do |name, error|
    orchestrator.add_task(Agentic:class="y">:Task.new(
      description: name, agent_spec: {class="s">"name" => name, class="s">"instructions" => class="s">"run"},
      payload: error
    ), agent: ->(t) {
      raise t.payload if t.payload

      class="y">:ok
    })
  end
  orchestrator.execute_plan
end

# --- the forecast desk ---------------------------------------------------------
state = Agentic:class="y">:ExecutionJournal.replay(path: JOURNAL)
day_events = state.events.slice_when { |a, b|
  a[class="y">:event] == class="s">"plan_completed" && b[class="y">:event] != class="s">"plan_completed"
}.to_a

def sky(failed)
  weather = failed.count { |e| e[class="y">:retryable] }
  climate = failed.count { |e| e[class="y">:retryable] == false }
  return class="s">"clear skies" if failed.empty?
  return class="s">"storm damage (#{climate} structural)" if climate.positive? && weather.positive?
  return class="s">"drought continues" if climate.positive?

  class="s">"passing showers (#{weather})"
end

puts class="s">"FAILURE WEATHER REPORT (#{DAYS.size} journaled days)"
puts
day_events.each_with_index do |events, index|
  failed = events.select { |e| e[class="y">:event] == class="s">"task_failed" }
  puts format(class="s">"  %-10s %-28s %s", DAYS[index][class="y">:name], sky(failed),
    failed.map { |e| e[class="y">:description] }.join(class="s">", "))
end
puts

# Weather clears; climate persists. The distinction IS the journal's
# retryable verdict, recorded when each drop fell.
latest = {}
state.events.each do |e|
  latest[e[class="y">:description]] = e if %w[task_failed task_succeeded].include?(e[class="y">:event])
end
weather_jobs = latest.values.select { |e| e[class="y">:event] == class="s">"task_failed" && e[class="y">:retryable] }
climate_jobs = latest.values.select { |e| e[class="y">:event] == class="s">"task_failed" && e[class="y">:retryable] == false }
cleared = state.events.select { |e| e[class="y">:event] == class="s">"task_failed" }.map { |e| e[class="y">:description] }.uniq
  .select { |d| latest[d][class="y">:event] == class="s">"task_succeeded" }

puts class="s">"  extended forecast:"
cleared.each { |d| puts class="s">"    #{d}: rained earlier this week, clear now - weather does that" }
weather_jobs.each { |e| puts class="s">"    #{e[class="y">:description]}: still raining, but it is rain - bring retries" }
climate_jobs.each { |e| puts class="s">"    #{e[class="y">:description]}: this is not weather, it is climate - #{e[class="y">:error]}" }
puts
rain = state.events.select { |e| e[class="y">:event] == class="s">"task_failed" }
puts class="s">"  #{rain.size} rainy events this week: #{rain.count { |e| e[class="y">:retryable] }} were weather (they passed, or will),"
puts class="s">"  and #{rain.count { |e| e[class="y">:retryable] == false }} were the same drought, reported daily."
puts class="s">"  no forecast fixes a drought: invoice's 401 has held for three days"
puts class="s">"  and will hold forever, because keys do not expire back. the journal"
puts class="s">"  told us which failures to wait out and which to dig a well for."