agentic examples

The Document Refinery

The Document Refinery: an HTML-to-digest pipeline where every stage assumes the input is hostile until proven boring - because it is. Real-world markup arrives with script injections, event handlers, javascript: hrefs, unclosed tags, and encoding damage, and "parse then use" is how that hostility reaches your users. The refinery runs sanitize -> extract -> normalize as a plan per document (documents in parallel), fans into one digest, and a referee proves nothing dangerous …

Reliability & Recovery Round 17 Mike Dalessio exit 0

source on github

bundle exec ruby examples/document_refinery.rb

a real captured run

THE DOCUMENT REFINERY (all input is hostile until proven boring)

  refined 3 feeds in parallel:
    - Changelog Weekly — Issue 12 (1 links) - Changelog Weekly — Issue 12 Changelog We
    - Ruby News (1 links) - Ruby News SEO garbage Release 3.4 Patch
    - Café Ruby, la gazette (1 links) - Café Ruby, la gazette Nouveautés: 7 gems

  referee: no script bodies, no javascript: hrefs, no event handlers,
           all output valid UTF-8, relative links resolved, 3/3 feeds present

  the pipeline shape is the security model: DECODE first (the gazette
  arrived as latin-1 bytes wearing a UTF-8 flag, and every regex
  downstream raises on damaged bytes - encoding repair is the price
  of admission, not a finishing touch); sanitize BEFORE extract (so
  extraction can't be fooled by what it finds); resolve links last;
  and a referee that greps the refined product for everything the
  fixtures smuggled in. the first draft ran encoding repair LAST and
  the plan itself refused: sanitize crashed on the gazette. the
  pipeline knew more about encoding order than its author did.

source

# frozen_string_literal: true

# The Document Refinery: an HTML-to-digest pipeline where every
# stage assumes the input is hostile until proven boring - because
# it is. Real-world markup arrives with script injections, event
# handlers, javascript: hrefs, unclosed tags, and encoding damage,
# and "parse then use" is how that hostility reaches your users.
# The refinery runs sanitize -> extract -> normalize as a plan per
# document (documents in parallel), fans into one digest, and a
# referee proves nothing dangerous survived refinement.
#
#   bundle exec ruby examples/document_refinery.rb
#
# Runs offline against embedded fixtures; exits 1 if anything
# hostile leaks through. (Stdlib-only parsing here for offline
# honesty - in production you'd put Nokogiri at stage 2 and I'd
# thank you for it.)

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

FEEDS = {
  class="s">"changelog weekly" => class="s">"<html><head><title>Changelog Weekly — Issue 12</title></head>" \
    class="s">"<body><script>track(document.cookie)</script><h1>Changelog Weekly</h1>" \
    class="s">"<a href='https://changelog.example/12'>Read issue</a>" \
    class="s">"<a href='javascript:alert(1)'>totally safe link</a><p>Gems shipped this week: 41</p>",
  class="s">"ruby news" => class="s">"<html><title>Ruby News</title><body onload='pwn()'>" \
    class="s">"<p style='display:none'>SEO garbage</p><a href='/34' onclick='steal()'>Release 3.4</a>" \
    class="s">"<img src='https://tracker.example/pixel.gif' width='1'><p>Patch tuesday: 3 CVEs fixed</p>",
  class="s">"mojibake gazette" => class="s">"<html><title>Caf\xE9 Ruby, la gazette</title><body>" \
    class="s">"<p>Nouveaut\xE9s: 7 gems</p><a href='https://cafe.example/fr'>Lire</a>"
}.freeze

# --- the refinery stages, each one paranoid on purpose ------------------------------
# DECODE runs FIRST, not last: every regex downstream assumes valid
# UTF-8 and raises on damaged bytes, so encoding repair is the price
# of admission, not a finishing touch. Damage is data, not a crash -
# transcode the legacy bytes instead of moving the outage downstream.
DECODE = ->(raw) {
  utf8 = raw.dup.force_encoding(Encoding:class="y">:UTF_8)
  utf8.valid_encoding? ? utf8 : raw.dup.force_encoding(Encoding:class="y">:ISO_8859_1).encode(Encoding:class="y">:UTF_8)
}

SANITIZE = ->(html) {
  html.gsub(%r{<script.*?</script>}mi, class="s">"")             # no executable content
    .gsub(/\son\w+\s*=\s*(['class="s">"]).*?\1/i, "")            # no event handlers
    .gsub(/href\s*=\s*(['class="s">"])\s*javascript:.*?\1/i, "href='#neutralized'")
}

EXTRACT = ->(html) {
  {title: html[%r{<title>(.*?)</title>}mi, 1].to_s.strip,
   links: html.scan(/href\s*=\s*['class="s">"]([^'"]+)['"]/i).flatten,
   text: html.gsub(%r{<[^>]*>}, class="s">" ").squeeze(class="s">" ").strip}
}

RESOLVE = ->(doc, base_url) {
  {title: doc[class="y">:title],
   links: doc[class="y">:links].map { |l| l.start_with?(class="s">"/") ? base_url + l : l }.reject { |l| l == class="s">"#neutralized" },
   text: doc[class="y">:text]}
}

journal = Agentic:class="y">:ExecutionJournal.new(path: File.join(Dir.tmpdir, class="s">"agentic_refinery.jsonl"))
File.delete(journal.path) if File.exist?(journal.path)
orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 3, lifecycle_hooks: journal.lifecycle_hooks)

refined_tasks = FEEDS.map do |name, raw|
  decode = Agentic:class="y">:Task.new(description: class="s">"decode: #{name}", agent_spec: {class="s">"name" => class="s">"d", class="s">"instructions" => class="s">"w"}, payload: raw)
  sanitize = Agentic:class="y">:Task.new(description: class="s">"sanitize: #{name}", agent_spec: {class="s">"name" => class="s">"s", class="s">"instructions" => class="s">"w"})
  extract = Agentic:class="y">:Task.new(description: class="s">"extract: #{name}", agent_spec: {class="s">"name" => class="s">"e", class="s">"instructions" => class="s">"w"})
  resolve = Agentic:class="y">:Task.new(description: class="s">"resolve: #{name}", agent_spec: {class="s">"name" => class="s">"r", class="s">"instructions" => class="s">"w"})
  orchestrator.add_task(decode, agent: ->(t) { DECODE.call(t.payload) })
  orchestrator.add_task(sanitize, [decode], agent: ->(t) { SANITIZE.call(t.previous_output) })
  orchestrator.add_task(extract, [sanitize], agent: ->(t) { EXTRACT.call(t.previous_output) })
  orchestrator.add_task(resolve, [extract], agent: ->(t) { RESOLVE.call(t.previous_output, class="s">"https://#{name.delete(" class="s">")}.example") })
  resolve
end

digest = Agentic:class="y">:Task.new(description: class="s">"digest", agent_spec: {class="s">"name" => class="s">"d", class="s">"instructions" => class="s">"w"})
orchestrator.add_task(digest, refined_tasks, agent: ->(t) {
  refined_tasks.map { |rt| t.output_of(rt) }.map { |d| class="s">"#{d[class="y">:title]} (#{d[class="y">:links].size} links) - #{d[class="y">:text][0, 40]}" }
})

result = orchestrator.execute_plan
entries = result.task_result(digest.id).output

puts class="s">"THE DOCUMENT REFINERY (all input is hostile until proven boring)"
puts
puts class="s">"  refined #{FEEDS.size} feeds in parallel:"
entries.each { |e| puts class="s">"    - #{e}" }
puts

# --- the referee: prove the hostility died in the refinery --------------------------
outputs = refined_tasks.map { |rt| result.task_result(rt.id).output }
violations = []
outputs.each do |doc|
  blob = [doc[class="y">:title], doc[class="y">:text], *doc[class="y">:links]].join(class="s">" ")
  violations << class="s">"script content survived" if blob.match?(/track\(|pwn\(|steal\(|<script/i)
  violations << class="s">"javascript: href survived" if doc[class="y">:links].any? { |l| l.start_with?(class="s">"javascript:") }
  violations << class="s">"invalid encoding in output" unless blob.valid_encoding? && blob.encoding == Encoding:class="y">:UTF_8
  violations << class="s">"relative link escaped unresolved" if doc[class="y">:links].any? { |l| l.start_with?(class="s">"/") }
end
violations << class="s">"a feed went missing from the digest" unless entries.size == FEEDS.size

puts class="s">"  referee: #{violations.empty? ? "no script bodies, no javascript: hrefs, no event handlers,class="s">" : violations.uniq.join("; class="s">")}"
puts class="s">"           all output valid UTF-8, relative links resolved, #{entries.size}/#{FEEDS.size} feeds present" if violations.empty?
puts
puts class="s">"  the pipeline shape is the security model: DECODE first (the gazette"
puts class="s">"  arrived as latin-1 bytes wearing a UTF-8 flag, and every regex"
puts class="s">"  downstream raises on damaged bytes - encoding repair is the price"
puts class="s">"  of admission, not a finishing touch); sanitize BEFORE extract (so"
puts class="s">"  extraction can't be fooled by what it finds); resolve links last;"
puts class="s">"  and a referee that greps the refined product for everything the"
puts class="s">"  fixtures smuggled in. the first draft ran encoding repair LAST and"
puts class="s">"  the plan itself refused: sanitize crashed on the gazette. the"
puts class="s">"  pipeline knew more about encoding order than its author did."
exit(violations.empty? ? 0 : 1)