The Plan Heckler

The Plan Heckler: mutation testing for workflows. Your plan has tests. Cute. Do the tests actually FAIL when the plan is wrong, or are they decoration? The heckler finds out the honest way: it breaks the plan on purpose - five sabotages, one at a time - and runs your spec against each mutant. A mutant your spec kills is coverage. A mutant that SURVIVES is a bug your tests would wave through the door. Tests that can't fail aren't tests.

Testing & Verification Round 17 Ryan Davis exit 0

source on github

bundle exec ruby examples/plan_heckler.rb

a real captured run

THE PLAN HECKLER (tests that can't fail aren't tests)

  baseline: unmutated plan passes every assertion (heckling requires green)

  heckling with SPEC v1 - the tests the team wrote (3 assertions):
    price_returns_zero     KILLED by "total is positive"
    discount_never_fires   SURVIVED - your tests shrug
    tax_off_by_10x         SURVIVED - your tests shrug
    tax_stage_bypassed     SURVIVED - your tests shrug
    receipt_truncates      SURVIVED - your tests shrug
    score: 1/5 mutants killed

  heckling with SPEC v2 - plus one golden total and a stage roll-call (5 assertions):
    price_returns_zero     KILLED by "total is positive"
    discount_never_fires   SURVIVED - your tests shrug
    tax_off_by_10x         KILLED by "small order prices to the golden 7884"
    tax_stage_bypassed     KILLED by "small order prices to the golden 7884"
    receipt_truncates      KILLED by "small order prices to the golden 7884"
    score: 4/5 mutants killed

  heckling with SPEC v3 - plus a golden total that CROSSES the discount bar (6 assertions):
    price_returns_zero     KILLED by "total is positive"
    discount_never_fires   KILLED by "BULK order prices to the golden 12247"
    tax_off_by_10x         KILLED by "small order prices to the golden 7884"
    tax_stage_bypassed     KILLED by "small order prices to the golden 7884"
    receipt_truncates      KILLED by "small order prices to the golden 7884"
    score: 5/5 mutants killed

  v1 waved four saboteurs through - "completes, says TOTAL, positive"
  can't see a bypassed tax stage or a 10x rate error. v2's golden
  number killed three more, but discount_never_fires still walked:
  the small order never crosses the discount bar, so a dead discount
  branch is INVISIBLE to any assertion about it. that's the heckler's
  real product - it doesn't just grade your assertions, it audits your
  FIXTURES: every branch your inputs never reach is a mutant sanctuary.
  one bulk order later, 5/5. pin golden numbers you priced by hand,
  roll-call the stages, and make your inputs visit every branch.
  mutation testing doesn't ask whether tests pass; it asks whether
  they can FAIL - the only thing a test is for.

source

# frozen_string_literal: true

# The Plan Heckler: mutation testing for workflows. Your plan has
# tests. Cute. Do the tests actually FAIL when the plan is wrong, or
# are they decoration? The heckler finds out the honest way: it
# breaks the plan on purpose - five sabotages, one at a time - and
# runs your spec against each mutant. A mutant your spec kills is
# coverage. A mutant that SURVIVES is a bug your tests would wave
# through the door. Tests that can't fail aren't tests.
#
#   bundle exec ruby examples/plan_heckler.rb
#
# Runs offline; exits 1 if any mutant survives the final spec.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

SMALL_ORDER = [{sku: class="s">"flog", cents: 4200}, {sku: class="s">"flay", cents: 3100}].freeze # 7300, under the discount bar
BULK_ORDER = [{sku: class="s">"flog", cents: 4200}] * 3                                 # 12600, discount fires
SMALL_GOLDEN = 7884   # 7300 * 1.08, hand-priced
BULK_GOLDEN = 12_247  # (12600 * 0.9) * 1.08 = 12247.2, rounded

# --- the plan under test: a pricing pipeline, optionally sabotaged ----------------
def build_pipeline(order, mutation = nil)
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 1)
  ran = []
  stage = ->(name, deps, fn) {
    task = Agentic:class="y">:Task.new(description: name, agent_spec: {class="s">"name" => name, class="s">"instructions" => class="s">"price"})
    orchestrator.add_task(task, deps, agent: ->(t) {
      ran << name
      fn.call(t.previous_output)
    })
    task
  }

  price_fn = ->(_) { {cents: order.sum { |i| i[class="y">:cents] }} }
  price_fn = ->(_) { {cents: 0} } if mutation == class="y">:price_returns_zero

  discount_fn = ->(o) { {cents: (o[class="y">:cents] >= 10_000) ? (o[class="y">:cents] * 0.9).round : o[class="y">:cents]} }
  discount_fn = ->(o) { o } if mutation == class="y">:discount_never_fires

  tax_rate = (mutation == class="y">:tax_off_by_10x) ? 1.008 : 1.08
  tax_fn = ->(o) { {cents: (o[class="y">:cents] * tax_rate).round} }

  receipt_fn = ->(o) { class="s">"TOTAL: #{o[class="y">:cents]} cents" }
  receipt_fn = ->(o) { class="s">"TOTAL: #{o[class="y">:cents] / 100 * 100} cents" } if mutation == class="y">:receipt_truncates

  price = stage.call(class="s">"price", [], price_fn)
  discount = stage.call(class="s">"discount", [price], discount_fn)
  tax = stage.call(class="s">"tax", [discount], tax_fn)
  receipt_dep = (mutation == class="y">:tax_stage_bypassed) ? discount : tax
  receipt = stage.call(class="s">"receipt", [receipt_dep], receipt_fn)

  [orchestrator, receipt, ran]
end

MUTANTS = [class="y">:price_returns_zero, class="y">:discount_never_fires, class="y">:tax_off_by_10x, class="y">:tax_stage_bypassed, class="y">:receipt_truncates].freeze

# --- three specs: what the team wrote, then what the heckler extorts, twice -------
SPEC_V1 = {
  class="s">"plan completes" => ->(runs) { runs.all? { |r| r[class="y">:result].status == class="y">:completed } },
  class="s">"receipt says TOTAL" => ->(runs) { runs.all? { |r| r[class="y">:out].to_s.include?(class="s">"TOTAL") } },
  class="s">"total is positive" => ->(runs) { runs.all? { |r| r[class="y">:out].to_s[/\d+/].to_i.positive? } }
}.freeze

SPEC_V2 = SPEC_V1.merge(
  class="s">"small order prices to the golden 7884" => ->(runs) { runs.first[class="y">:out].to_s[/\d+/].to_i == SMALL_GOLDEN },
  class="s">"all four stages ran, in order" => ->(runs) { runs.all? { |r| r[class="y">:ran] == %w[price discount tax receipt] } }
).freeze

SPEC_V3 = SPEC_V2.merge(
  class="s">"BULK order prices to the golden 12247" => ->(runs) { runs.last[class="y">:out].to_s[/\d+/].to_i == BULK_GOLDEN }
).freeze

def run_spec(spec, mutation = nil)
  runs = [SMALL_ORDER, BULK_ORDER].map do |order|
    orchestrator, receipt, ran = build_pipeline(order, mutation)
    result = orchestrator.execute_plan
    {result: result, out: result.task_result(receipt.id)&.output, ran: ran}
  end
  spec.reject { |_name, check| check.call(runs) }.keys
end

def heckle(spec_name, spec)
  puts class="s">"  heckling with #{spec_name} (#{spec.size} assertions):"
  survivors = MUTANTS.reject do |mutation|
    failed = run_spec(spec, mutation)
    puts format(class="s">"    %-22s %s", mutation, failed.any? ? class="s">"KILLED by #{failed.first.inspect}" : class="s">"SURVIVED - your tests shrug")
    failed.any?
  end
  puts format(class="s">"    score: %d/%d mutants killed", MUTANTS.size - survivors.size, MUTANTS.size)
  puts
  survivors
end

puts class="s">"THE PLAN HECKLER (tests that can't fail aren't tests)"
puts

baseline = run_spec(SPEC_V3)
abort(class="s">"  baseline is red; heckling a broken plan proves nothing") unless baseline.empty?
puts class="s">"  baseline: unmutated plan passes every assertion (heckling requires green)"
puts

heckle(class="s">"SPEC v1 - the tests the team wrote", SPEC_V1)
survivors_v2 = heckle(class="s">"SPEC v2 - plus one golden total and a stage roll-call", SPEC_V2)
survivors_v3 = heckle(class="s">"SPEC v3 - plus a golden total that CROSSES the discount bar", SPEC_V3)

puts class="s">"  v1 waved four saboteurs through - \"completes, says TOTAL, positive\class="s">""
puts class="s">"  can't see a bypassed tax stage or a 10x rate error. v2's golden"
puts class="s">"  number killed three more, but #{survivors_v2.first} still walked:"
puts class="s">"  the small order never crosses the discount bar, so a dead discount"
puts class="s">"  branch is INVISIBLE to any assertion about it. that's the heckler's"
puts class="s">"  real product - it doesn't just grade your assertions, it audits your"
puts class="s">"  FIXTURES: every branch your inputs never reach is a mutant sanctuary."
puts class="s">"  one bulk order later, 5/5. pin golden numbers you priced by hand,"
puts class="s">"  roll-call the stages, and make your inputs visit every branch."
puts class="s">"  mutation testing doesn't ask whether tests pass; it asks whether"
puts class="s">"  they can FAIL - the only thing a test is for."
exit(survivors_v3.empty? ? 0 : 1)