The Batch Import

The Batch Import: 500 rows of the kind of data people actually upload - typos, header drift, impossible combinations - run through one contract. Good rows proceed; bad rows land in a REJECT FILE with the field, the reason, and the rule that caught them. An importer that raises on row 37 is a tool for importing 36 rows.

source on github

a real captured run

BATCH IMPORT (500 rows through one contract in 275ms)

  accepted: 382   rejected: 118

  reject file, summarized by cause:
    customs        45  #############################################
    mode           36  ####################################
    weight         26  ##########################
    fits           11  ###########

  first three lines of the reject file (the thing support actually opens):
    line 4: weight: must be less than or equal to 5000
    line 11: weight: must be less than or equal to 5000
    line 12: customs: express requires customs_code

  500 rows cost 275ms of validation - 549.05us a row - and
  every rejection names its line, its field, and its rule, including
  the cross-field ones ("fits", "customs") no per-column check
  catches. two design rules for importers: never raise on row 37
  (collect, don't crash), and never write "invalid row" (a reject
  file without reasons is a support ticket generator). the contract
  supplied both for free.

source

# frozen_string_literal: true

# The Batch Import: 500 rows of the kind of data people actually
# upload - typos, header drift, impossible combinations - run through
# one contract. Good rows proceed; bad rows land in a REJECT FILE
# with the field, the reason, and the rule that caught them. An
# importer that raises on row 37 is a tool for importing 36 rows.
#
#   bundle exec ruby examples/batch_import.rb
#
# Runs offline; the dirty data is seeded and repeatable.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"json"

CONTRACT = Agentic:class="y">:CapabilitySpecification.new(
  name: class="s">"import_shipment", description: class="s">"One row of the shipments upload", version: class="s">"1.0.0",
  inputs: {
    mode: {type: class="s">"string", required: true, enum: %w[air sea road]},
    weight: {type: class="s">"number", required: true, min: 1, max: 5_000},
    volume: {type: class="s">"number", min: 0},
    express: {type: class="s">"boolean"},
    customs_code: {type: class="s">"string"}
  },
  rules: {
    fits: {relation: class="y">:sum_lte, fields: [class="y">:weight, class="y">:volume], limit: 6_000},
    customs: {relation: class="y">:requires, fields: [class="y">:express, class="y">:customs_code]}
  }
)

# 500 rows, seeded: roughly three-quarters clean, the rest wrong in
# the ways uploads actually are
rng = Random.new(20_260_707)
ROWS = 500.times.map do |i|
  row = {
    mode: %w[air sea road].sample(random: rng),
    weight: rng.rand(1..4_000),
    volume: rng.rand(0..1_500)
  }
  row[class="y">:express] = true if rng.rand < 0.25
  row[class="y">:customs_code] = class="s">"HS-#{rng.rand(100)}" if row[class="y">:express] && rng.rand < 0.7
  case rng.rand
  when 0..0.03 then row[class="y">:mode] = class="s">"trian"            # typo
  when 0.03..0.06 then row[class="y">:weight] = 0             # zero weight
  when 0.06..0.09 then row[class="y">:weight] = rng.rand(5_001..9_000) # too heavy
  when 0.09..0.12 then row[class="y">:volume] = rng.rand(4_000..8_000) # breaks the sum rule
  when 0.12..0.14 then row.delete(class="y">:mode)            # header drift
  end
  row
end

validator = Agentic:class="y">:CapabilityValidator.new(CONTRACT)
accepted = []
rejects = []

started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
ROWS.each_with_index do |row, index|
  validator.validate_inputs!(row)
  accepted << row
rescue Agentic:class="y">:Errors:class="y">:ValidationError => e
  reasons = e.violations.except(class="y">:base).map { |field, msgs| class="s">"#{field}: #{msgs.first}" }
  reasons += e.rule_violations.map { |v| class="s">"#{v[class="y">:rule]}: #{v[class="y">:message]}" }
  rejects << {line: index + 2, reasons: reasons} # +2: 1-based plus header row
end
elapsed = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started

puts class="s">"BATCH IMPORT (#{ROWS.size} rows through one contract in #{(elapsed * 1000).round}ms)"
puts
puts class="s">"  accepted: #{accepted.size}   rejected: #{rejects.size}"
puts

by_reason = rejects.flat_map { |r| r[class="y">:reasons].map { |reason| reason[/\A[^:]+/] } }.tally
puts class="s">"  reject file, summarized by cause:"
by_reason.sort_by { |_, count| -count }.each do |cause, count|
  puts format(class="s">"    %-14s %-3d %s", cause, count, class="s">"#" * count)
end
puts
puts class="s">"  first three lines of the reject file (the thing support actually opens):"
rejects.first(3).each do |reject|
  puts class="s">"    line #{reject[class="y">:line]}: #{reject[class="y">:reasons].join("; class="s">")}"
end
puts
puts class="s">"  #{ROWS.size} rows cost #{(elapsed * 1000).round}ms of validation - #{format("%.2fclass="s">", elapsed / ROWS.size * 1_000_000)}us a row - and"
puts class="s">"  every rejection names its line, its field, and its rule, including"
puts class="s">"  the cross-field ones (\"fits\class="s">", \"customs\class="s">") no per-column check"
puts class="s">"  catches. two design rules for importers: never raise on row 37"
puts class="s">"  (collect, don't crash), and never write \"invalid row\class="s">" (a reject"
puts class="s">"  file without reasons is a support ticket generator). the contract"
puts class="s">"  supplied both for free."