The Variance Detective

The Variance Detective: ten journaled runs of the same plan, then a hunt for the task whose p90/p50 ratio betrays it. Averages hide flakiness; percentile spreads name it. Uses duration_percentile (new this round) over the journal's accumulated samples.

source on github

a real captured run

VARIANCE DETECTIVE (20 journaled runs, seed 20260707)

  task                         p50     p90   worst   p90/p50
  fetch:profile               22ms    24ms    24ms      1.1x
  fetch:permissions           16ms    18ms    18ms      1.1x
  render:dashboard            33ms    35ms    35ms      1.1x
  fetch:recommendations       21ms    91ms   100ms      4.3x  <- SUSPECT

  verdict: fetch:recommendations runs fine at the median and
  terribly at the tail - the signature of flakiness (cold caches,
  lock contention, a retried upstream). an AVERAGE would have
  reported ~30ms and told you nothing was wrong.

source

# frozen_string_literal: true

# The Variance Detective: ten journaled runs of the same plan, then a
# hunt for the task whose p90/p50 ratio betrays it. Averages hide
# flakiness; percentile spreads name it. Uses duration_percentile
# (new this round) over the journal's accumulated samples.
#
#   bundle exec ruby examples/variance_detective.rb [seed]
#
# Runs offline; one task is scripted-flaky, the others honest.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

seed = (ARGV.first || 20260707).to_i
rng = Random.new(seed)

JOURNAL = File.join(Dir.tmpdir, class="s">"agentic_variance.journal.jsonl")
File.delete(JOURNAL) if File.exist?(JOURNAL)
journal = Agentic:class="y">:ExecutionJournal.new(path: JOURNAL)

# Steady tasks jitter a little; the flaky one occasionally stalls
PROFILE = {
  class="s">"fetch:profile" => ->(rng) { 0.020 + rng.rand * 0.004 },
  class="s">"fetch:permissions" => ->(rng) { 0.015 + rng.rand * 0.003 },
  class="s">"render:dashboard" => ->(rng) { 0.030 + rng.rand * 0.005 },
  class="s">"fetch:recommendations" => ->(rng) {
    (rng.rand < 0.3) ? 0.080 + rng.rand * 0.02 : 0.018 + rng.rand * 0.004
  }
}.freeze

RUNS = 20
RUNS.times do
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(
    concurrency_limit: 4, lifecycle_hooks: journal.lifecycle_hooks
  )
  PROFILE.each do |name, latency|
    orchestrator.add_task(Agentic:class="y">:Task.new(
      description: name, agent_spec: {class="s">"name" => name, class="s">"instructions" => class="s">"serve"},
      payload: latency
    ), agent: ->(t) { sleep(t.payload.call(rng)) || class="y">:ok })
  end
  orchestrator.execute_plan
end

# --- the investigation --------------------------------------------------------
state = Agentic:class="y">:ExecutionJournal.replay(path: JOURNAL)

puts class="s">"VARIANCE DETECTIVE (#{RUNS} journaled runs, seed #{seed})"
puts
puts format(class="s">"  %-24s %7s %7s %7s %9s", class="s">"task", class="s">"p50", class="s">"p90", class="s">"worst", class="s">"p90/p50")

suspects = []
PROFILE.each_key do |name|
  p50 = state.duration_percentile(name, 50)
  p90 = state.duration_percentile(name, 90)
  worst = state.duration_percentile(name, 100)
  ratio = p90 / p50
  flaky = ratio > 2.0
  suspects << name if flaky
  puts format(class="s">"  %-24s %5.0fms %5.0fms %5.0fms %8.1fx  %s",
    name, p50 * 1000, p90 * 1000, worst * 1000, ratio, flaky ? class="s">"<- SUSPECT" : class="s">"")
end

puts
if suspects.any?
  puts class="s">"  verdict: #{suspects.join(", class="s">")} runs fine at the median and"
  puts class="s">"  terribly at the tail - the signature of flakiness (cold caches,"
  puts class="s">"  lock contention, a retried upstream). an AVERAGE would have"
  puts class="s">"  reported ~#{(state.duration_percentile(suspects.first, 50) * 1000 * 1.4).round}ms and told you nothing was wrong."
else
  puts class="s">"  no suspects. suspiciously well-behaved - check the seed."
end