The Rule Prober

The Rule Prober: structured rules declare which fields they read - so now that claim can be AUDITED. For each rule, perturb fields it does NOT declare; if the verdict flips, the rule is reading fields off the books. One of the rules below lies. The prober finds it.

source on github

a real captured run

RULE PROBER (seed 20260707, 300 trials per rule)

  affordability: honest - declares [:amount, :income], and no undeclared field ever changed its verdict
  subprime_needs_cosigner: honest - declares [:score, :cosigned], and no undeclared field ever changed its verdict
  jumbo_screening: LYING - declares [:amount] but its verdict
    flipped when :score changed (71 of 300 trials)

why it matters: Piotr's 422 renderer highlights the DECLARED fields.
a rule that secretly reads :score sends users to fix :amount while
the real problem sits unhighlighted. field declarations are now
testable claims - so test them.

source

# frozen_string_literal: true

# The Rule Prober: structured rules declare which fields they read -
# so now that claim can be AUDITED. For each rule, perturb fields it
# does NOT declare; if the verdict flips, the rule is reading fields
# off the books. One of the rules below lies. The prober finds it.
#
#   bundle exec ruby examples/rule_prober.rb [seed]
#
# Runs offline and deterministically.

require class="s">"bundler/setup"
require class="s">"agentic"

seed = (ARGV.first || 20260707).to_i
rng = Random.new(seed)

SPEC = Agentic:class="y">:CapabilitySpecification.new(
  name: class="s">"approve_loan",
  description: class="s">"Approve a loan application",
  version: class="s">"1.0.0",
  inputs: {
    amount: {type: class="s">"number", required: true, min: 1},
    income: {type: class="s">"number", required: true, min: 0},
    score: {type: class="s">"number", required: true, min: 300, max: 850},
    cosigned: {type: class="s">"boolean", required: true}
  },
  rules: {
    affordability: {
      message: class="s">"amount may not exceed 5x income",
      fields: [class="y">:amount, class="y">:income],
      check: ->(i) { i[class="y">:amount] <= i[class="y">:income] * 5 }
    },
    subprime_needs_cosigner: {
      message: class="s">"scores under 600 require a cosigner",
      fields: [class="y">:score, class="y">:cosigned],
      check: ->(i) { i[class="y">:score] >= 600 || i[class="y">:cosigned] }
    },
    # The liar: declares [:amount] but secretly reads :score too
    jumbo_screening: {
      message: class="s">"amounts over 400k get extra screening",
      fields: [class="y">:amount],
      check: ->(i) { i[class="y">:amount] <= 400_000 || i[class="y">:score] > 700 }
    }
  }
)

# Generate a conforming application
def sample_application(rng)
  {
    amount: rng.rand(10_000..900_000),
    income: rng.rand(30_000..250_000),
    score: rng.rand(300..850),
    cosigned: [true, false].sample(random: rng)
  }
end

# Perturb one field to another conforming value
def perturb(application, field, rng)
  fresh = sample_application(rng)
  application.merge(field => fresh[field])
end

TRIALS = 300
findings = Hash.new { |h, k| h[k] = [] }

SPEC.rules.each do |rule_id, definition|
  undeclared = SPEC.inputs.keys - definition[class="y">:fields]

  TRIALS.times do
    application = sample_application(rng)
    verdict = definition[class="y">:check].call(application)

    undeclared.each do |field|
      mutated = perturb(application, field, rng)
      next if mutated == application

      if definition[class="y">:check].call(mutated) != verdict
        findings[rule_id] << field
      end
    end
  end
end

puts class="s">"RULE PROBER (seed #{seed}, #{TRIALS} trials per rule)"
puts
SPEC.rules.each do |rule_id, definition|
  flipped = findings[rule_id].tally
  if flipped.empty?
    puts class="s">"  #{rule_id}: honest - declares #{definition[class="y">:fields].inspect}, " \
      class="s">"and no undeclared field ever changed its verdict"
  else
    puts class="s">"  #{rule_id}: LYING - declares #{definition[class="y">:fields].inspect} but its verdict"
    flipped.each do |field, count|
      puts class="s">"    flipped when :#{field} changed (#{count} of #{TRIALS} trials)"
    end
  end
end

puts
puts class="s">"why it matters: Piotr's 422 renderer highlights the DECLARED fields."
puts class="s">"a rule that secretly reads class="y">:score sends users to fix class="y">:amount while"
puts class="s">"the real problem sits unhighlighted. field declarations are now"
puts class="s">"testable claims - so test them."
exit findings.empty? ? 0 : 1