agentic examples

The Relation Prober

The Relation Prober: relation-typed rules are new, and new predicates deserve hostility. Each relation is probed with edge inputs - zeros, negatives, floats, missing keys, nils - and every verdict is checked against an independent hand-written oracle. The prober also walks off the paved road on purpose: in round 10 a rule referencing an undeclared field met a string and escaped as a raw TypeError; the round-11 release files that edge down, and this prober is the acceptance …

Testing & Verification Round 10 Jeremy Evans exit 0

source on github

bundle exec ruby examples/relation_prober.rb

a real captured run

RELATION PROBER (13 probes against a hand-written oracle)

  sum_lte: both at zero                      oracle: allow   got: allow   ok
  sum_lte: exactly at the limit              oracle: allow   got: allow   ok
  sum_lte: one over, via floats              oracle: reject  got: reject  ok
  sum_lte: negative rescues the sum          oracle: allow   got: allow   ok
  sum_lte: missing field counts as 0         oracle: allow   got: allow   ok
  requires: trigger absent                   oracle: allow   got: allow   ok
  requires: trigger present, need met        oracle: allow   got: allow   ok
  requires: trigger present, need missing    oracle: reject  got: reject  ok
  requires: three-field chain broken         oracle: reject  got: reject  ok
  mutually_exclusive: neither                oracle: allow   got: allow   ok
  mutually_exclusive: one                    oracle: allow   got: allow   ok
  mutually_exclusive: both                   oracle: reject  got: reject  ok
  mutually_exclusive: empty string is present oracle: reject  got: reject  ok

  13 probes, 0 divergence(s) on the paved road.

  off the paved road: rules that must refuse to construct
    sum_lte over an UNDECLARED field           refused at boot: rule :r (sum_lte) references undeclared ...
    sum_lte over a declared STRING             refused at boot: rule :r (sum_lte) can only sum declared ...
    requires with a typo'd field (fail-open)   refused at boot: rule :r (requires) references undeclared...

  the paved road holds and the roadside refuses construction.
  note the third edge: a typo'd field in requires used to fail
  OPEN - the rule just never fired, which no test of valid inputs
  would ever notice. now the typo can't boot. a validator's
  errors must wear its uniform, and its typos must not compile.

source

# frozen_string_literal: true

# The Relation Prober: relation-typed rules are new, and new
# predicates deserve hostility. Each relation is probed with edge
# inputs - zeros, negatives, floats, missing keys, nils - and every
# verdict is checked against an independent hand-written oracle.
# The prober also walks off the paved road on purpose: in round 10
# a rule referencing an undeclared field met a string and escaped as
# a raw TypeError; the round-11 release files that edge down, and
# this prober is the acceptance test that proves it stays down.
#
#   bundle exec ruby examples/relation_prober.rb
#
# Runs offline; exits 1 if any probe draws blood again.

require class="s">"bundler/setup"
require class="s">"agentic"

def spec_for(rules, inputs)
  Agentic:class="y">:CapabilitySpecification.new(
    name: class="s">"probe", description: class="s">"probe", version: class="s">"1.0.0", inputs: inputs, rules: rules
  )
end

def verdict(spec, payload)
  Agentic:class="y">:CapabilityValidator.new(spec).validate_inputs!(payload)
  class="y">:allow
rescue Agentic:class="y">:Errors:class="y">:ValidationError
  class="y">:reject
end

NUMERIC = {a: {type: class="s">"number"}, b: {type: class="s">"number"}}.freeze
STRINGS = {x: {type: class="s">"string"}, y: {type: class="s">"string"}}.freeze

# Each probe: [description, spec, payload, oracle verdict]
PROBES = [
  [class="s">"sum_lte: both at zero", spec_for({r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:b], limit: 0}}, NUMERIC),
    {a: 0, b: 0}, class="y">:allow],
  [class="s">"sum_lte: exactly at the limit", spec_for({r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:b], limit: 10}}, NUMERIC),
    {a: 4, b: 6}, class="y">:allow],
  [class="s">"sum_lte: one over, via floats", spec_for({r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:b], limit: 10}}, NUMERIC),
    {a: 4.5, b: 5.6}, class="y">:reject],
  [class="s">"sum_lte: negative rescues the sum", spec_for({r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:b], limit: 10}}, NUMERIC),
    {a: 15, b: -6}, class="y">:allow],
  [class="s">"sum_lte: missing field counts as 0", spec_for({r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:b], limit: 10}}, NUMERIC),
    {a: 7}, class="y">:allow],
  [class="s">"requires: trigger absent", spec_for({r: {relation: class="y">:requires, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {y: class="s">"alone is fine"}, class="y">:allow],
  [class="s">"requires: trigger present, need met", spec_for({r: {relation: class="y">:requires, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {x: class="s">"t", y: class="s">"met"}, class="y">:allow],
  [class="s">"requires: trigger present, need missing", spec_for({r: {relation: class="y">:requires, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {x: class="s">"t"}, class="y">:reject],
  [class="s">"requires: three-field chain broken", spec_for({r: {relation: class="y">:requires, fields: [class="y">:x, class="y">:y, class="y">:z]}}, STRINGS.merge(z: {type: class="s">"string"})),
    {x: class="s">"t", y: class="s">"met"}, class="y">:reject],
  [class="s">"mutually_exclusive: neither", spec_for({r: {relation: class="y">:mutually_exclusive, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {}, class="y">:allow],
  [class="s">"mutually_exclusive: one", spec_for({r: {relation: class="y">:mutually_exclusive, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {x: class="s">"only"}, class="y">:allow],
  [class="s">"mutually_exclusive: both", spec_for({r: {relation: class="y">:mutually_exclusive, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {x: class="s">"one", y: class="s">"two"}, class="y">:reject],
  [class="s">"mutually_exclusive: empty string is present", spec_for({r: {relation: class="y">:mutually_exclusive, fields: [class="y">:x, class="y">:y]}}, STRINGS),
    {x: class="s">"", y: class="s">"two"}, class="y">:reject]
].freeze

puts class="s">"RELATION PROBER (#{PROBES.size} probes against a hand-written oracle)"
puts
divergences = 0
PROBES.each do |description, spec, payload, oracle|
  actual = verdict(spec, payload)
  divergences += 1 if actual != oracle
  puts format(class="s">"  %-42s oracle: %-7s got: %-7s %s",
    description, oracle, actual, (actual == oracle) ? class="s">"ok" : class="s">"DIVERGED")
end

puts
puts class="s">"  #{PROBES.size} probes, #{divergences} divergence(s) on the paved road."
puts

# --- off the paved road ---------------------------------------------------------
# In round 10, a rule referencing an undeclared field let a string
# reach sum_lte's arithmetic: raw TypeError, a 422 path turned 500
# path. The round-11 fix refuses at CONSTRUCTION - the typo fails at
# boot, where it names itself, before any request can find it.
edges = {
  class="s">"sum_lte over an UNDECLARED field" =>
    [{r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:undeclared], limit: 10}}, {a: {type: class="s">"number"}}],
  class="s">"sum_lte over a declared STRING" =>
    [{r: {relation: class="y">:sum_lte, fields: [class="y">:a, class="y">:b], limit: 10}}, {a: {type: class="s">"number"}, b: {type: class="s">"string"}}],
  class="s">"requires with a typo'd field (fail-open)" =>
    [{r: {relation: class="y">:requires, fields: [class="y">:a, class="y">:customs_kode]}}, {a: {type: class="s">"number"}, customs_code: {type: class="s">"string"}}]
}

blood = 0
puts class="s">"  off the paved road: rules that must refuse to construct"
edges.each do |name, (rules, inputs)|
  Agentic:class="y">:CapabilityValidator.new(spec_for(rules, inputs))
  blood += 1
  puts format(class="s">"    %-42s CONSTRUCTED - the edge is back", name)
rescue ArgumentError => e
  puts format(class="s">"    %-42s refused at boot: %s", name, e.message[0, 40] + class="s">"...")
rescue => e
  blood += 1
  puts format(class="s">"    %-42s wrong uniform: %s", name, e.class)
end

puts
if divergences.zero? && blood.zero?
  puts class="s">"  the paved road holds and the roadside refuses construction."
  puts class="s">"  note the third edge: a typo'd field in requires used to fail"
  puts class="s">"  OPEN - the rule just never fired, which no test of valid inputs"
  puts class="s">"  would ever notice. now the typo can't boot. a validator's"
  puts class="s">"  errors must wear its uniform, and its typos must not compile."
else
  puts class="s">"  BLOOD DRAWN: #{divergences} divergence(s), #{blood} escaped edge(s)."
end
exit((divergences + blood).zero? ? 0 : 1)