The Contract Overhead Bench

The Contract Overhead Bench: "should we validate every call?" is a performance question, so answer it with a measurement instead of a vibe. Benchmarks the validator across contract sizes and rule counts, then prices it against the thing it protects - because overhead is a fraction, and everyone keeps quoting the numerator.

source on github

a real captured run

CONTRACT OVERHEAD BENCH (2000 validations per row)

  contract                   per call     share of an 800ms LLM call
  3 keys, no rules             0.0754ms   0.0094%  ###################
  10 keys, no rules            0.1578ms   0.0197%  #######################################
  10 keys, 5 relations         0.2114ms   0.0264%  ########################################
  30 keys, 15 relations        0.3932ms   0.0491%  ########################################
  rejection, 5 rules broken    1.6627ms   (the expensive path: exception + 5 rule reports)

  the whole table rounds to zero: the biggest contract here
  (30 keys, 15 relations) costs 0.3932ms per call - 0.04915% of the LLM
  round-trip it guards. even rejection, the slow path, is 1.66ms.
  "we skip validation for performance" saves the price of a
  rounding error (3 keys, no rules: 0.0754ms) to risk shipping a malformed
  prompt to an 800ms call that BILLS you for the mistake.
  validate both doors. the meter says you can afford it.

source

# frozen_string_literal: true

# The Contract Overhead Bench: "should we validate every call?" is a
# performance question, so answer it with a measurement instead of a
# vibe. Benchmarks the validator across contract sizes and rule
# counts, then prices it against the thing it protects - because
# overhead is a fraction, and everyone keeps quoting the numerator.
#
#   bundle exec ruby examples/contract_overhead.rb
#
# Runs offline; times are real, the LLM latency is the industry's.

require class="s">"bundler/setup"
require class="s">"agentic"

ITERATIONS = 2_000
LLM_CALL_MS = 800.0 # a conservative round-trip for a real model call

def bench(iterations)
  # Warm up, then measure - the first call pays dry-schema compilation
  yield
  started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
  iterations.times { yield }
  (Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started) / iterations * 1000
end

def spec_with(keys:, relations:)
  inputs = keys.times.to_h { |i| [:class="s">"field_#{i}", {type: class="s">"number", required: true, min: 0, max: 1_000}] }
  rules = relations.times.to_h { |i|
    [:class="s">"rule_#{i}", {relation: class="y">:sum_lte, fields: [:class="s">"field_#{i}", :class="s">"field_#{(i + 1) % keys}"], limit: 2_000}]
  }
  Agentic:class="y">:CapabilitySpecification.new(
    name: class="s">"bench", description: class="s">"bench", version: class="s">"1.0.0", inputs: inputs, rules: rules
  )
end

puts class="s">"CONTRACT OVERHEAD BENCH (#{ITERATIONS} validations per row)"
puts
puts format(class="s">"  %-26s %-12s %s", class="s">"contract", class="s">"per call", class="s">"share of an #{LLM_CALL_MS.to_i}ms LLM call")

rows = [
  [class="s">"3 keys, no rules", spec_with(keys: 3, relations: 0), {field_0: 1, field_1: 2, field_2: 3}],
  [class="s">"10 keys, no rules", spec_with(keys: 10, relations: 0), 10.times.to_h { |i| [:class="s">"field_#{i}", i] }],
  [class="s">"10 keys, 5 relations", spec_with(keys: 10, relations: 5), 10.times.to_h { |i| [:class="s">"field_#{i}", i] }],
  [class="s">"30 keys, 15 relations", spec_with(keys: 30, relations: 15), 30.times.to_h { |i| [:class="s">"field_#{i}", i] }]
]

results = rows.map do |label, spec, payload|
  validator = Agentic:class="y">:CapabilityValidator.new(spec)
  ms = bench(ITERATIONS) { validator.validate_inputs!(payload) }
  share = ms / LLM_CALL_MS * 100
  puts format(class="s">"  %-26s %8.4fms   %.4f%%  %s", label, ms, share, class="s">"#" * [(share * 2000).round, 40].min)
  [label, ms]
end

# The failure path costs too - measure a rejection (exception + message building)
reject_spec = spec_with(keys: 10, relations: 5)
reject_validator = Agentic:class="y">:CapabilityValidator.new(reject_spec)
bad_payload = 10.times.to_h { |i| [:class="s">"field_#{i}", 1_900] } # breaks every sum_lte
reject_ms = bench(500) {
  begin
    reject_validator.validate_inputs!(bad_payload)
  rescue Agentic:class="y">:Errors:class="y">:ValidationError
    nil
  end
}
puts format(class="s">"  %-26s %8.4fms   (the expensive path: exception + %d rule reports)", class="s">"rejection, 5 rules broken", reject_ms, 5)

puts
fastest, cheapest = results.first
biggest, priciest = results.last
puts class="s">"  the whole table rounds to zero: the biggest contract here"
puts format(class="s">"  (%s) costs %.4fms per call - %.5f%% of the LLM", biggest.downcase, priciest, priciest / LLM_CALL_MS * 100)
puts format(class="s">"  round-trip it guards. even rejection, the slow path, is %.2fms.", reject_ms)
puts class="s">"  \"we skip validation for performance\class="s">" saves the price of a"
puts format(class="s">"  rounding error (%s: %.4fms) to risk shipping a malformed", fastest.downcase, cheapest)
puts class="s">"  prompt to an #{LLM_CALL_MS.to_i}ms call that BILLS you for the mistake."
puts class="s">"  validate both doors. the meter says you can afford it."