The Survey Scrubber

The Survey Scrubber: you asked ten humans "what's blocking your team?" and they answered like humans - with names, emails, phone numbers, and @handles embedded in the grievances. Every downstream system that touches those answers (the category model, the data warehouse, the exec summary) is a system that can leak them. So the pipeline's FIRST stage, before anything is categorized or stored, is the scrubber - and the referee greps everything downstream for every seeded PII …

source on github

a real captured run

THE SURVEY SCRUBBER (data about people is people; the pipeline order is the ethics)

  TEAM BLOCKERS, Q3 (10 responses)
    tooling  #### 4
    infra    ### 3
    process  ## 2
    people   # 1
    sample voices: CI is flaky, ask [NAME] she has the details | deploys blocked on approvals - email [EMAIL] about it

  privacy referee: 6 seeded identifiers (names, emails, a phone,
  a handle) grepped against the report, the warehouse file, and every
  categorized record: ZERO leaks

  the design choices that matter: the scrubber runs FIRST - before
  categorization, before the warehouse write, before anything with
  a disk or a memory - because every stage that sees raw text is a
  stage that can leak it, and you shrink that set to one. the name
  rule is blunt on purpose ([A-Z]\w+ [A-Z]\w+ catches some false
  positives): for PII, RECALL beats precision - redacting 'Le
  Sigh' by mistake costs a chuckle; missing one real name costs a
  person. and the verification is a grep, not a policy document:
  every seeded identifier hunted through every downstream surface.
  the aggregate report kept everything the survey was FOR - counts,
  themes, even sample voices - because anonymized is not the same
  as useless. it's just useful without a body count.

source

# frozen_string_literal: true

# The Survey Scrubber: you asked ten humans "what's blocking your
# team?" and they answered like humans - with names, emails, phone
# numbers, and @handles embedded in the grievances. Every downstream
# system that touches those answers (the category model, the data
# warehouse, the exec summary) is a system that can leak them. So
# the pipeline's FIRST stage, before anything is categorized or
# stored, is the scrubber - and the referee greps everything
# downstream for every seeded PII string, because 'we scrub the
# data' is a claim and grep is a fact. The safest PII is the PII
# you never stored; data about people IS people, and the pipeline's
# order is its ethics.
#
#   bundle exec ruby examples/survey_scrubber.rb
#
# Runs offline; exits 1 if one identifying string survives past
# the scrubber.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

RESPONSES = [
  class="s">"CI is flaky, ask Maria Santos she has the details",
  class="s">"deploys blocked on approvals - email varun.k@corp.example about it",
  class="s">"our staging db is tiny. @davenotdave complains weekly",
  class="s">"hiring! we lost two people and process is drowning us",
  class="s">"the linter wars. also call 555-0142 if you want the real story",
  class="s">"nobody owns the flaky specs so they rot",
  class="s">"process process process. three tickets to change a label",
  class="s">"infra costs review meeting eats every tuesday",
  class="s">"tooling is fine, honestly it's the approvals",
  class="s">"ask Chen Wei or maria.santos@corp.example - migrations block everything"
].freeze
SEEDED_PII = [class="s">"Maria Santos", class="s">"varun.k@corp.example", class="s">"@davenotdave", class="s">"555-0142", class="s">"Chen Wei", class="s">"maria.santos@corp.example"].freeze

SCRUB_RULES = [
  [/[a-z0-9._]+@[a-z0-9.-]+\.[a-z]{2,}/i, class="s">"[EMAIL]"],
  [/\b\d{3}-\d{4}\b/, class="s">"[PHONE]"],
  [/(?<!\w)@\w+/, class="s">"[HANDLE]"],
  [/\b[A-Z][a-z]+ [A-Z][a-z]+\b/, class="s">"[NAME]"] # blunt on purpose; recall beats precision for PII
].freeze

CATEGORIES = {
  class="s">"tooling" => [class="s">"ci", class="s">"flaky", class="s">"linter", class="s">"specs", class="s">"tooling"],
  class="s">"process" => [class="s">"approvals", class="s">"process", class="s">"tickets", class="s">"meeting"],
  class="s">"people" => [class="s">"hiring", class="s">"lost", class="s">"owns"],
  class="s">"infra" => [class="s">"db", class="s">"staging", class="s">"infra", class="s">"costs", class="s">"migrations", class="s">"deploys"]
}.freeze

warehouse = File.join(Dir.mktmpdir(class="s">"survey"), class="s">"responses.jsonl")

orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 4)
scrubbed_tasks = RESPONSES.each_with_index.map do |raw, i|
  scrub = Agentic:class="y">:Task.new(description: class="s">"scrub #{i}", agent_spec: {class="s">"name" => class="s">"scrubber", class="s">"instructions" => class="s">"w"}, payload: raw)
  tag = Agentic:class="y">:Task.new(description: class="s">"categorize #{i}", agent_spec: {class="s">"name" => class="s">"tagger", class="s">"instructions" => class="s">"w"})
  orchestrator.add_task(scrub, agent: ->(t) {
    SCRUB_RULES.reduce(t.payload) { |text, (pattern, replacement)| text.gsub(pattern, replacement) }
  })
  orchestrator.add_task(tag, [scrub], agent: ->(t) {
    text = t.previous_output
    hits = CATEGORIES.transform_values { |words| words.count { |w| text.downcase.include?(w) } }
    category = hits.max_by { |_, v| v }
    record = {text: text, category: (category[1]).zero? ? class="s">"other" : category[0]}
    File.write(warehouse, class="s">"#{record}\n", mode: class="s">"a") # only scrubbed text ever touches disk
    record
  })
  tag
end

report_task = Agentic:class="y">:Task.new(description: class="s">"report", agent_spec: {class="s">"name" => class="s">"analyst", class="s">"instructions" => class="s">"w"})
orchestrator.add_task(report_task, scrubbed_tasks, agent: ->(t) {
  records = scrubbed_tasks.map { |st| t.output_of(st) }
  tally = records.group_by { |r| r[class="y">:category] }.transform_values(&class="y">:size).sort_by { |_, v| -v }
  quotes = records.first(2).map { |r| r[class="y">:text] }
  class="s">"TEAM BLOCKERS, Q3 (#{records.size} responses)\n" +
    tally.map { |cat, n| class="s">"  #{cat.ljust(8)} #{"#class="s">" * n} #{n}" }.join(class="s">"\n") +
    class="s">"\n  sample voices: #{quotes.join(" | class="s">")}"
})
result = orchestrator.execute_plan
report = result.task_result(report_task.id).output

puts class="s">"THE SURVEY SCRUBBER (data about people is people; the pipeline order is the ethics)"
puts
report.lines.each { |l| puts class="s">"  #{l.rstrip}" }
puts

# --- the referee: grep beats 'we scrub the data' --------------------------------------
downstream = [report, File.read(warehouse), scrubbed_tasks.map { |st| result.task_result(st.id).output.to_s }.join]
leaks = SEEDED_PII.flat_map { |pii| downstream.each_index.select { |d| downstream[d].include?(pii) }.map { |d| [pii, [class="y">:report, class="y">:warehouse, class="y">:records][d]] } }
tallied = report[/\((\d+) responses\)/, 1].to_i

puts class="s">"  privacy referee: #{SEEDED_PII.size} seeded identifiers (names, emails, a phone,"
puts class="s">"  a handle) grepped against the report, the warehouse file, and every"
puts class="s">"  categorized record: #{leaks.empty? ? "ZERO leaksclass="s">" : "LEAKED: #{leaks.inspect}class="s">"}"
puts

failures = []
failures << class="s">"PII leaked downstream: #{leaks.inspect}" unless leaks.empty?
failures << class="s">"responses lost in aggregation (#{tallied})" unless tallied == RESPONSES.size
failures << class="s">"categories degenerate" unless report.include?(class="s">"process") && report.include?(class="s">"tooling")
failures << class="s">"warehouse got raw text" if SEEDED_PII.any? { |pii| File.read(warehouse).include?(pii) }

puts class="s">"  the design choices that matter: the scrubber runs FIRST - before"
puts class="s">"  categorization, before the warehouse write, before anything with"
puts class="s">"  a disk or a memory - because every stage that sees raw text is a"
puts class="s">"  stage that can leak it, and you shrink that set to one. the name"
puts class="s">"  rule is blunt on purpose ([A-Z]\\w+ [A-Z]\\w+ catches some false"
puts class="s">"  positives): for PII, RECALL beats precision - redacting 'Le"
puts class="s">"  Sigh' by mistake costs a chuckle; missing one real name costs a"
puts class="s">"  person. and the verification is a grep, not a policy document:"
puts class="s">"  every seeded identifier hunted through every downstream surface."
puts class="s">"  the aggregate report kept everything the survey was FOR - counts,"
puts class="s">"  themes, even sample voices - because anonymized is not the same"
puts class="s">"  as useless. it's just useful without a body count."
exit(failures.empty? ? 0 : 1)