Discovery Testing

Discovery Testing: most people use test doubles to ISOLATE code that already exists. The better trick is using them to DISCOVER code that doesn't: start at the top with fakes for collaborators you haven't designed yet, let the failing test tell you what interfaces you wish existed, then descend one level and make each wish real. The doubles are scaffolding; the interfaces they discovered are the building.

source on github

a real captured run

DISCOVERY TESTING (the fakes are scaffolding; the interfaces are the building)

  ok   act 1: triager orchestrates two DISCOVERED interfaces
  ok   act 2: real classifier honors the discovered interface
  ok   act 2: triager unchanged as fakes become real (the seam held)
  ok   act 3: router realized as a two-task plan, same interface
  ok   fakes still match reality: #classify
  ok   fakes still match reality: #route

  read the acts as a design session, because that's what they were:
  act 1 wrote fakes for collaborators that DIDN'T EXIST, and the
  messages we wished for (classify(text), route(id, label)) became
  the design - discovered under test pressure, not drawn on a
  whiteboard. act 2 made one wish real without touching the
  triager: the seam held, which is the proof the seam was right.
  act 3's payoff is the plan-shaped one: a discovered interface is
  INDIFFERENT to whether a lambda, a class, or a whole orchestrated
  plan stands behind it - route(id, label) became two tasks and
  nobody upstream knew. and the last checks keep the round-12
  rule: fakes that outlive their realization must show their
  papers, or they quietly start vouching for a design that moved.

source

# frozen_string_literal: true

# Discovery Testing: most people use test doubles to ISOLATE code
# that already exists. The better trick is using them to DISCOVER
# code that doesn't: start at the top with fakes for collaborators
# you haven't designed yet, let the failing test tell you what
# interfaces you wish existed, then descend one level and make each
# wish real. The doubles are scaffolding; the interfaces they
# discovered are the building.
#
#   bundle exec ruby examples/discovery_testing.rb
#
# Runs offline; three acts, each act's checks actually execute.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

CHECKS = []
def check(claim, ok) = CHECKS << [claim, ok]

# --- ACT 1: the top, with everything below it imaginary -------------------------
# We want a TicketTriager. What does it NEED? We don't know yet - so
# we write the fakes we WISH existed, and the wishes become the design.
class TicketTriager
  def initialize(classifier:, router:)
    @classifier = classifier
    @router = router
  end

  def triage(ticket)
    label = @classifier.classify(ticket[class="y">:text]) # <- wish #1: classify(text) -> label
    @router.route(ticket[class="y">:id], label)           # <- wish #2: route(id, label) -> receipt
  end
end

fake_classifier = Class.new {
  def classify(text) = class="s">"billing"
}.new
fake_router = Class.new {
  attr_reader class="y">:routed

  def route(id, label)
    (@routed = [id, label]) && class="s">"receipt-#{id}"
  end
}.new

triager = TicketTriager.new(classifier: fake_classifier, router: fake_router)
receipt = triager.triage({id: 7, text: class="s">"refund please"})
check(class="s">"act 1: triager orchestrates two DISCOVERED interfaces", receipt == class="s">"receipt-7" && fake_router.routed == [7, class="s">"billing"])

# --- ACT 2: descend one level - realize the classifier, router stays fake -------
# The wish list from act 1 is now a spec: classify(text) -> label.
class KeywordClassifier
  def classify(text) = text.match?(/refund|charge/i) ? class="s">"billing" : class="s">"general"
end

check(class="s">"act 2: real classifier honors the discovered interface",
  KeywordClassifier.new.classify(class="s">"refund please") == class="s">"billing" &&
  KeywordClassifier.new.classify(class="s">"hello") == class="s">"general")

triager = TicketTriager.new(classifier: KeywordClassifier.new, router: fake_router)
check(class="s">"act 2: triager unchanged as fakes become real (the seam held)",
  triager.triage({id: 8, text: class="s">"you charged me twice"}) == class="s">"receipt-8")

# --- ACT 3: realize the router AS A PLAN - the seams become tasks ----------------
class PlanRouter
  def route(id, label)
    orchestrator = Agentic:class="y">:PlanOrchestrator.new
    enqueue = Agentic:class="y">:Task.new(description: class="s">"enqueue:#{id}", agent_spec: {class="s">"name" => class="s">"q", class="s">"instructions" => class="s">"w"})
    notify = Agentic:class="y">:Task.new(description: class="s">"notify:#{id}", agent_spec: {class="s">"name" => class="s">"n", class="s">"instructions" => class="s">"w"})
    orchestrator.add_task(enqueue, agent: ->(_t) { class="s">"#{label}-queue" })
    orchestrator.add_task(notify, [enqueue], agent: ->(t) { class="s">"receipt-#{id} (#{t.previous_output})" })
    orchestrator.execute_plan.task_result(notify.id).output
  end
end

triager = TicketTriager.new(classifier: KeywordClassifier.new, router: PlanRouter.new)
check(class="s">"act 3: router realized as a two-task plan, same interface",
  triager.triage({id: 9, text: class="s">"refund"}) == class="s">"receipt-9 (billing-queue)")

# The final honesty pass: every fake must still match the real thing
# it stood in for (round 12's verifier, kept in the toolbox)
[[fake_classifier, KeywordClassifier], [fake_router, PlanRouter]].each do |fake, real|
  real.instance_methods(false).each do |m|
    matches = fake.respond_to?(m) && fake.method(m).parameters.map(&class="y">:first) == real.instance_method(m).parameters.map(&class="y">:first)
    check(class="s">"fakes still match reality: ##{m}", matches)
  end
end

puts class="s">"DISCOVERY TESTING (the fakes are scaffolding; the interfaces are the building)"
puts
CHECKS.each { |claim, ok| puts format(class="s">"  %-4s %s", ok ? class="s">"ok" : class="s">"FAIL", claim) }
failures = CHECKS.count { |_, ok| !ok }
puts
puts class="s">"  read the acts as a design session, because that's what they were:"
puts class="s">"  act 1 wrote fakes for collaborators that DIDN'T EXIST, and the"
puts class="s">"  messages we wished for (classify(text), route(id, label)) became"
puts class="s">"  the design - discovered under test pressure, not drawn on a"
puts class="s">"  whiteboard. act 2 made one wish real without touching the"
puts class="s">"  triager: the seam held, which is the proof the seam was right."
puts class="s">"  act 3's payoff is the plan-shaped one: a discovered interface is"
puts class="s">"  INDIFFERENT to whether a lambda, a class, or a whole orchestrated"
puts class="s">"  plan stands behind it - route(id, label) became two tasks and"
puts class="s">"  nobody upstream knew. and the last checks keep the round-12"
puts class="s">"  rule: fakes that outlive their realization must show their"
puts class="s">"  papers, or they quietly start vouching for a design that moved."
exit(failures.zero? ? 0 : 1)