The Implementation Shootout

The Implementation Shootout: two candidates for the same capability, one eval set, and a verdict computed instead of vibed. v1 is a fast regex; v2 is a slower keyword-weight model. The scoreboard reports quality AND latency, because "which is better" has two axes and every README that hides one is selling something.

source on github

a real captured run

IMPLEMENTATION SHOOTOUT: route_ticket (8 eval cases)

  case (expected)                                v1 regex     v2 weights
  I was charged twice, I want a refund... (billing) pass         pass
  App crashes when I open settings... (bug)      pass         pass
  Can't login, password reset email ne... (account) pass         pass
  I paid but my invoice shows money ow... (billing) pass         pass
  The export fails and I lost my work... (bug)   FAIL(general) pass
  My account is locked after the updat... (account) FAIL(general) pass
  How do I change my plan?... (general)          pass         pass
  Password reset email shows an error ... (account) FAIL(bug)    pass

  scoreboard:
    v1 regex     accuracy  63%   p50 2.1ms
    v2 weights   accuracy 100%   p50 10.1ms

  verdict: v2 wins 8/8 to 5/8 - and costs 5x the latency.
  the deciding cases share a shape: 'password reset email shows an
  error page' has one bug word and five points of account evidence.
  first-match regex answers by clause order - an accident of code
  layout - while weights answer by total evidence. whether that is
  worth 8ms per ticket is YOUR call; the shootout's job is to put
  both axes on one table so the tradeoff is chosen, not discovered.
  and a perfect v2 score means the EVAL SET stopped discriminating,
  not that v2 is done - add cases until your best candidate fails.

source

# frozen_string_literal: true

# The Implementation Shootout: two candidates for the same capability,
# one eval set, and a verdict computed instead of vibed. v1 is a fast
# regex; v2 is a slower keyword-weight model. The scoreboard reports
# quality AND latency, because "which is better" has two axes and
# every README that hides one is selling something.
#
#   bundle exec ruby examples/impl_shootout.rb
#
# Runs offline; the verdict includes the price of the quality.

require class="s">"bundler/setup"
require class="s">"agentic"

SPEC = Agentic:class="y">:CapabilitySpecification.new(
  name: class="s">"route_ticket", description: class="s">"Route a ticket to a queue", version: class="s">"?",
  inputs: {text: {type: class="s">"string", required: true}},
  outputs: {queue: {type: class="s">"string", required: true, enum: %w[billing bug account general]}}
)

# Candidate 1: the regex that shipped in an afternoon
V1 = lambda do |i|
  queue = case i[class="y">:text].downcase
  when /refund|charge|invoice/ then class="s">"billing"
  when /crash|error|broken/ then class="s">"bug"
  when /password|login|email/ then class="s">"account"
  else class="s">"general"
  end
  sleep(0.002)
  {queue: queue}
end

# Candidate 2: stem weights, summed as evidence - slower, subtler
WEIGHTS = {
  class="s">"billing" => {class="s">"refund" => 3, class="s">"charge" => 2, class="s">"invoice" => 3, class="s">"paid" => 2, class="s">"money" => 1},
  class="s">"bug" => {class="s">"crash" => 3, class="s">"error" => 2, class="s">"broken" => 2, class="s">"lost" => 1, class="s">"fail" => 2},
  class="s">"account" => {class="s">"password" => 3, class="s">"login" => 3, class="s">"email" => 2, class="s">"lock" => 2}
}.freeze
V2 = lambda do |i|
  words = i[class="y">:text].downcase.scan(/[a-z]+/)
  scores = WEIGHTS.transform_values { |stems|
    stems.sum { |stem, weight| (words.any? { |w| w.start_with?(stem) }) ? weight : 0 }
  }
  best, score = scores.max_by { |_, s| s }
  sleep(0.01)
  {queue: (score > 0) ? best : class="s">"general"}
end

EVALS = [
  {text: class="s">"I was charged twice, I want a refund", queue: class="s">"billing"},
  {text: class="s">"App crashes when I open settings", queue: class="s">"bug"},
  {text: class="s">"Can't login, password reset email never arrives", queue: class="s">"account"},
  {text: class="s">"I paid but my invoice shows money owed", queue: class="s">"billing"},
  {text: class="s">"The export fails and I lost my work", queue: class="s">"bug"},
  {text: class="s">"My account is locked after the update", queue: class="s">"account"},
  {text: class="s">"How do I change my plan?", queue: class="s">"general"},
  # The decider: one bug word, five points of account evidence
  {text: class="s">"Password reset email shows an error page", queue: class="s">"account"}
].freeze

def run_candidate(impl)
  EVALS.map do |eval_case|
    started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
    output = impl.call(text: eval_case[class="y">:text])
    {
      correct: output[class="y">:queue] == eval_case[class="y">:queue],
      got: output[class="y">:queue],
      latency: Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started
    }
  end
end

results = {class="s">"v1 regex" => run_candidate(V1), class="s">"v2 weights" => run_candidate(V2)}

puts class="s">"IMPLEMENTATION SHOOTOUT: #{SPEC.name} (#{EVALS.size} eval cases)"
puts
puts format(class="s">"  %-46s %-12s %s", class="s">"case (expected)", class="s">"v1 regex", class="s">"v2 weights")
EVALS.each_with_index do |eval_case, index|
  marks = results.values.map { |r|
    r[index][class="y">:correct] ? class="s">"pass" : class="s">"FAIL(#{r[index][class="y">:got]})"
  }
  puts format(class="s">"  %-46s %-12s %s", class="s">"#{eval_case[class="y">:text][0, 36]}... (#{eval_case[class="y">:queue]})", *marks)
end

puts
puts class="s">"  scoreboard:"
results.each do |name, rows|
  accuracy = rows.count { |r| r[class="y">:correct] } / EVALS.size.to_f
  p50 = rows.map { |r| r[class="y">:latency] }.sort[rows.size / 2]
  puts format(class="s">"    %-12s accuracy %3d%%   p50 %.1fms", name, (accuracy * 100).round, p50 * 1000)
end

v1_acc = results[class="s">"v1 regex"].count { |r| r[class="y">:correct] }
v2_acc = results[class="s">"v2 weights"].count { |r| r[class="y">:correct] }
puts
puts class="s">"  verdict: v2 wins #{v2_acc}/#{EVALS.size} to #{v1_acc}/#{EVALS.size} - and costs 5x the latency."
puts class="s">"  the deciding cases share a shape: 'password reset email shows an"
puts class="s">"  error page' has one bug word and five points of account evidence."
puts class="s">"  first-match regex answers by clause order - an accident of code"
puts class="s">"  layout - while weights answer by total evidence. whether that is"
puts class="s">"  worth 8ms per ticket is YOUR call; the shootout's job is to put"
puts class="s">"  both axes on one table so the tradeoff is chosen, not discovered."
puts class="s">"  and a perfect v2 score means the EVAL SET stopped discriminating,"
puts class="s">"  not that v2 is done - add cases until your best candidate fails."