Shadow Traffic

Shadow Traffic: the safest way to replace a component at scale is to never let the replacement answer. The OLD implementation serves every request; the NEW one runs beside it in the shadow - same inputs, measured and compared, its results thrown away. Users feel nothing; you collect a mismatch report and a latency comparison, and the cutover decision becomes a table instead of a leap.

source on github

a real captured run

SHADOW TRAFFIC (v1 serves; v2 rehearses; users feel nothing)

  6 requests served by v1; 6 shadowed by v2
  agreement: 5/6 (83%)
  latency:   v1 p50 0.01ms, v2 p50 1.10ms

  the mismatch report (the whole reason to shadow):
    "Please resend my invoice"
      served: general | candidate: billing

  1 mismatch(es) journaled - the cutover meeting reads a table,
  not a hunch. and the mismatch is EXACTLY the case that would have
  paged after a blind cutover: v2 casts a broader net ('invoice'),
  which is either the bug fixed or the regression introduced - the
  shadow can't tell you which, but it guarantees a HUMAN decides
  with the example in hand, before a single user was reclassified.
  the discipline that makes this safe at scale: the shadow's output
  feeds NOTHING (asserted every request), shadow failures can't
  fail the plan, and the comparison is journaled where the recovery
  and audit tooling already live. rehearse in production, serve
  from the incumbent, cut over on evidence.

source

# frozen_string_literal: true

# Shadow Traffic: the safest way to replace a component at scale is
# to never let the replacement answer. The OLD implementation serves
# every request; the NEW one runs beside it in the shadow - same
# inputs, measured and compared, its results thrown away. Users feel
# nothing; you collect a mismatch report and a latency comparison,
# and the cutover decision becomes a table instead of a leap.
#
#   bundle exec ruby examples/shadow_traffic.rb
#
# Runs offline; v2 disagrees on exactly the case that would've paged.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

# v1 serves production today
V1 = ->(text) { text.match?(/refund|charge/i) ? class="s">"billing" : class="s">"general" }

# v2 is the candidate: faster on average, and subtly different
V2 = lambda do |text|
  sleep(0.001)
  return class="s">"billing" if text.match?(/refund|charge|invoice/i) # broader net
  class="s">"general"
end

JOURNAL = File.join(Dir.tmpdir, class="s">"agentic_shadow.jsonl")
File.delete(JOURNAL) if File.exist?(JOURNAL)
journal = Agentic:class="y">:ExecutionJournal.new(path: JOURNAL)

REQUESTS = [
  class="s">"I want a refund for order 8",
  class="s">"How do I reset my password?",
  class="s">"You charged me twice",
  class="s">"Please resend my invoice",       # <- the divergence
  class="s">"What are your business hours?",
  class="s">"Refund the duplicate charge"
].freeze

# One plan per request: the serve task answers; the shadow task runs
# the candidate on the same input and RECORDS, never serves
mismatches = []
latencies = {v1: [], v2: []}

REQUESTS.each_with_index do |text, i|
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(lifecycle_hooks: journal.lifecycle_hooks)
  serve = Agentic:class="y">:Task.new(description: class="s">"serve:#{i}", agent_spec: {class="s">"name" => class="s">"v1", class="s">"instructions" => class="s">"serve"})
  shadow = Agentic:class="y">:Task.new(description: class="s">"shadow:#{i}", agent_spec: {class="s">"name" => class="s">"v2", class="s">"instructions" => class="s">"shadow"})

  orchestrator.add_task(serve, agent: ->(_t) {
    t0 = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
    answer = V1.call(text)
    latencies[class="y">:v1] << Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - t0
    answer
  })
  # The shadow depends on serve's output so it can COMPARE - and its
  # own output is deliberately unused by anything downstream
  orchestrator.add_task(shadow, [serve], agent: ->(t) {
    t0 = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
    candidate = V2.call(text)
    latencies[class="y">:v2] << Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - t0
    if candidate != t.previous_output
      mismatches << {request: text, served: t.previous_output, candidate: candidate}
      journal.record(class="y">:shadow_mismatch, description: class="s">"shadow:#{i}", served: t.previous_output, candidate: candidate)
    end
    candidate
  })
  result = orchestrator.execute_plan
  # What the user received came from v1, always:
  raise class="s">"shadow leaked into serving!" unless result.task_result(serve.id).output == V1.call(text)
end

puts class="s">"SHADOW TRAFFIC (v1 serves; v2 rehearses; users feel nothing)"
puts
puts format(class="s">"  %d requests served by v1; %d shadowed by v2", REQUESTS.size, REQUESTS.size)
puts format(class="s">"  agreement: %d/%d (%.0f%%)", REQUESTS.size - mismatches.size, REQUESTS.size,
  (REQUESTS.size - mismatches.size) * 100.0 / REQUESTS.size)
puts format(class="s">"  latency:   v1 p50 %.2fms, v2 p50 %.2fms", latencies[class="y">:v1].sort[2] * 1000, latencies[class="y">:v2].sort[2] * 1000)
puts
puts class="s">"  the mismatch report (the whole reason to shadow):"
mismatches.each do |m|
  puts class="s">"    #{m[class="y">:request].inspect}"
  puts class="s">"      served: #{m[class="y">:served]} | candidate: #{m[class="y">:candidate]}"
end
puts
state = Agentic:class="y">:ExecutionJournal.replay(path: JOURNAL)
puts class="s">"  #{state.events.count { |e| e[class="y">:event] == "shadow_mismatchclass="s">" }} mismatch(es) journaled - the cutover meeting reads a table,"
puts class="s">"  not a hunch. and the mismatch is EXACTLY the case that would have"
puts class="s">"  paged after a blind cutover: v2 casts a broader net ('invoice'),"
puts class="s">"  which is either the bug fixed or the regression introduced - the"
puts class="s">"  shadow can't tell you which, but it guarantees a HUMAN decides"
puts class="s">"  with the example in hand, before a single user was reclassified."
puts class="s">"  the discipline that makes this safe at scale: the shadow's output"
puts class="s">"  feeds NOTHING (asserted every request), shadow failures can't"
puts class="s">"  fail the plan, and the comparison is journaled where the recovery"
puts class="s">"  and audit tooling already live. rehearse in production, serve"
puts class="s">"  from the incumbent, cut over on evidence."