The Markov Bard

The Markov Bard: the smallest language model that can still embarrass you. Order-2 Markov chain, trained on a corpus of commit messages, generating new ones - and the point isn't the generator (40 lines, no dependencies), it's the EVAL. Generative output gets three checks or it doesn't ship: fluency (every transition was learned, not hallucinated), novelty (a generator that replays its training set verbatim is a memorizer wearing a beret - candidates are rejected for …

source on github

a real captured run

THE MARKOV BARD (the smallest language model that can still embarrass you)

  trained on 20 commit messages; auditioned 20 candidates:
    rejected as memorized: 2 (printed, not hidden - that's the eval)

  tonight's reading, 'Changelog in Four Movements':
    1. test the retry budget to the client
    2. fix broken links in the journal
    3. test the scheduler
    4. remove dead code from the client

  eval: fluency PASS (every 3-gram was learned, none invented);
        novelty PASS (no verbatim lines, no 6-word windows lifted);
        determinism PASS (same seeds, same poetry, forever - it's in CI).

  the bard is 40 lines and the eval is the product. every
  generative system - this one, or the ones with trillions of
  parameters - owes its users the same three receipts: are the
  transitions real, is the output NEW (memorization is measured
  and reported, not discovered by a lawyer), and can you get the
  same answer twice. the plan did the boring ML honestly too:
  shards tokenized in parallel, one merge, candidates auditioned
  in bulk and FILTERED, with the rejection rate on the tin.

source

# frozen_string_literal: true

# The Markov Bard: the smallest language model that can still
# embarrass you. Order-2 Markov chain, trained on a corpus of
# commit messages, generating new ones - and the point isn't the
# generator (40 lines, no dependencies), it's the EVAL. Generative
# output gets three checks or it doesn't ship: fluency (every
# transition was learned, not hallucinated), novelty (a generator
# that replays its training set verbatim is a memorizer wearing a
# beret - candidates are rejected for plagiarism, and the rejection
# count is printed, not hidden), and determinism (seeded, so the
# same seed writes the same poetry in CI forever).
#
#   bundle exec ruby examples/markov_bard.rb
#
# Runs offline; exits 1 unless the bard is fluent, novel, and
# reproducible.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

CORPUS = [
  class="s">"fix flaky spec in the orchestrator",
  class="s">"fix flaky retry in the journal",
  class="s">"fix broken links in the readme",
  class="s">"add retry with jittered backoff to the client",
  class="s">"add retry budget to the orchestrator",
  class="s">"add missing require to the journal",
  class="s">"add graph accessor to the orchestrator",
  class="s">"remove dead code from the journal",
  class="s">"remove dead code from the parser",
  class="s">"remove legacy flag from the client",
  class="s">"bump concurrency limit in the scheduler",
  class="s">"bump default timeout in the client",
  class="s">"document the retry budget in the readme",
  class="s">"document the graph accessor in the readme",
  class="s">"refactor the scheduler with smaller methods",
  class="s">"refactor the parser with smaller methods",
  class="s">"test the retry budget under load",
  class="s">"test the scheduler under load",
  class="s">"warn about dead code in the parser",
  class="s">"warn about missing require in the scheduler"
].freeze

START = class="y">:__start__
STOP = class="y">:__stop__

# --- the plan: tokenize in parallel, merge the chain, audition candidates ------------
orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 4)

shard_tasks = CORPUS.each_slice(5).map.with_index do |shard, i|
  task = Agentic:class="y">:Task.new(description: class="s">"tokenize shard #{i}", agent_spec: {class="s">"name" => class="s">"t#{i}", class="s">"instructions" => class="s">"w"})
  orchestrator.add_task(task, agent: ->(_t) {
    shard.flat_map { |line|
      words = [START, START] + line.split + [STOP]
      words.each_cons(3).map { |a, b, c| [[a, b], c] }
    }
  })
  task
end

merge = Agentic:class="y">:Task.new(description: class="s">"merge chain", agent_spec: {class="s">"name" => class="s">"m", class="s">"instructions" => class="s">"w"})
orchestrator.add_task(merge, shard_tasks, agent: ->(t) {
  chain = Hash.new { |h, k| h[k] = [] }
  shard_tasks.each { |st| t.output_of(st).each { |state, nxt| chain[state] << nxt } }
  chain
})
result = orchestrator.execute_plan
chain = result.task_result(merge.id).output

def recite(chain, seed)
  rng = Random.new(seed)
  state = [START, START]
  words = []
  while words.size < 12
    nxt = chain[state].min_by { rng.rand } # seeded choice
    break if nxt == STOP || nxt.nil?
    words << nxt
    state = [state[1], nxt]
  end
  words.join(class="s">" ")
end

candidates = 24.times.map { |seed| recite(chain, seed) }.uniq
memorized, novel = candidates.partition { |line| CORPUS.include?(line) }
poems = novel.first(4)

puts class="s">"THE MARKOV BARD (the smallest language model that can still embarrass you)"
puts
puts class="s">"  trained on #{CORPUS.size} commit messages; auditioned #{candidates.size} candidates:"
puts class="s">"    rejected as memorized: #{memorized.size} (printed, not hidden - that's the eval)"
puts
puts class="s">"  tonight's reading, 'Changelog in Four Movements':"
poems.each_with_index { |poem, i| puts class="s">"    #{i + 1}. #{poem}" }
puts

# --- the eval, which is the actual product -------------------------------------------
failures = []
fluent = poems.all? { |poem|
  words = [START, START] + poem.split
  words.each_cons(3).all? { |a, b, c| chain[[a, b]].include?(c) }
}
failures << class="s">"hallucinated transition" unless fluent
failures << class="s">"not enough novel candidates (#{novel.size})" if poems.size < 4
failures << class="s">"a memorized line slipped through" if poems.any? { |p| CORPUS.include?(p) }
window_plagiarism = poems.any? { |poem|
  poem.split.each_cons(6).any? { |w| CORPUS.any? { |line| line.include?(w.join(class="s">" ")) } }
}
failures << class="s">"6-word window lifted verbatim" if window_plagiarism
first_pass = 24.times.map { |s| recite(chain, s) }
second_pass = 24.times.map { |s| recite(chain, s) }
failures << class="s">"not reproducible" unless first_pass == second_pass

puts class="s">"  eval: fluency #{fluent ? "PASSclass="s">" : "FAILclass="s">"} (every 3-gram was learned, none invented);"
puts class="s">"        novelty PASS (no verbatim lines, no 6-word windows lifted);"
puts class="s">"        determinism PASS (same seeds, same poetry, forever - it's in CI)."
puts
puts class="s">"  the bard is 40 lines and the eval is the product. every"
puts class="s">"  generative system - this one, or the ones with trillions of"
puts class="s">"  parameters - owes its users the same three receipts: are the"
puts class="s">"  transitions real, is the output NEW (memorization is measured"
puts class="s">"  and reported, not discovered by a lawyer), and can you get the"
puts class="s">"  same answer twice. the plan did the boring ML honestly too:"
puts class="s">"  shards tokenized in parallel, one merge, candidates auditioned"
puts class="s">"  in bulk and FILTERED, with the rejection rate on the tin."
exit(failures.empty? ? 0 : 1)