The Journal Tail Pager

The Journal Tail Pager: production journals grow like production tables, and the question asked of both is always the same one - "what happened RECENTLY?" Answering it by replaying the whole file is SELECT * wearing a filesystem costume. This pager reads pages from the END, backwards, in fixed-size chunks: page 1 costs kilobytes no matter how many megabytes the journal holds.

source on github

a real captured run

THE JOURNAL TAIL PAGER (2542KB journal, 20,000 events)

  last page (50 events):     0.3ms,    16KB read  (t19950 .. t19999)
  full replay (control):  4834.1ms,  2542KB read

  paging backwards through history:
    page 1: t19950 .. t19999  (50 events)
    page 2: t19900 .. t19949  (50 events)
    page 3: t19850 .. t19899  (50 events)

  the arithmetic: page 1 cost 16KB of a 2542KB file (0.6%) and
  ran 16094x faster than full replay. the cursor is a BYTE OFFSET,
  not a page number - kaminari taught everyone what OFFSET 19950
  costs on a growing table, and the same lesson holds for growing
  files: numbered pages shift when rows append; cursors don't.
  full replay remains the right tool for RESUME (you need all
  completions); the pager is the right tool for LOOKING (the
  incident was ten minutes ago, not ten thousand events ago).

source

# frozen_string_literal: true

# The Journal Tail Pager: production journals grow like production
# tables, and the question asked of both is always the same one -
# "what happened RECENTLY?" Answering it by replaying the whole file
# is SELECT * wearing a filesystem costume. This pager reads pages
# from the END, backwards, in fixed-size chunks: page 1 costs
# kilobytes no matter how many megabytes the journal holds.
#
#   bundle exec ruby examples/journal_tail.rb
#
# Runs offline; a 20,000-event journal is built, then barely read.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"
require class="s">"json"

# Kaminari's lesson, ported: a page knows its items AND how to reach
# the one before it (the cursor is a byte offset, not a page number -
# offsets don't shift when new events append)
class JournalTailPager
  CHUNK = 16 * 1024

  Page = Struct.new(class="y">:events, class="y">:prev_cursor, class="y">:bytes_read, keyword_init: true)

  def initialize(path)
    @path = path
  end

  def last_page(per: 50) = page_before(File.size(@path), per: per)

  def page_before(cursor, per: 50)
    lines = []
    bytes_read = 0
    position = cursor
    buffer = +class="s">""

    while position.positive? && lines.size <= per
      step = [CHUNK, position].min
      position -= step
      chunk = File.open(@path, class="s">"rb") { |f|
        f.seek(position)
        f.read(step)
      }
      bytes_read += step
      buffer = chunk + buffer
      lines = buffer.split(class="s">"\n", -1)
    end

    # First fragment may be a partial line unless we hit file start
    complete = position.zero? ? lines : lines.drop(1)
    complete = complete.reject(&class="y">:empty?)
    page_lines = complete.last(per)
    consumed = page_lines.sum(&class="y">:bytesize) + page_lines.size
    events = page_lines.filter_map do |l|
      JSON.parse(l, symbolize_names: true)
    rescue JSON:class="y">:ParserError
      nil
    end
    Page.new(events: events, prev_cursor: cursor - consumed, bytes_read: bytes_read)
  end
end

# --- build a big journal ---------------------------------------------------------
path = File.join(Dir.tmpdir, class="s">"agentic_tail.journal.jsonl")
File.delete(path) if File.exist?(path)
journal = Agentic:class="y">:ExecutionJournal.new(path: path, fsync_every: 500)
20_000.times do |i|
  journal.record(class="y">:task_succeeded, task_id: class="s">"t#{i}", description: class="s">"job:#{i % 40}", duration: 0.01, output: nil)
end
journal.sync
total_size = File.size(path)

puts class="s">"THE JOURNAL TAIL PAGER (#{total_size / 1024}KB journal, 20,000 events)"
puts

pager = JournalTailPager.new(path)
started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
page = pager.last_page(per: 50)
page_ms = (Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started) * 1000

started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
Agentic:class="y">:ExecutionJournal.replay(path: path)
full_ms = (Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started) * 1000

puts format(class="s">"  last page (50 events):  %6.1fms, %5dKB read  (%s .. %s)",
  page_ms, page.bytes_read / 1024, page.events.first[class="y">:task_id], page.events.last[class="y">:task_id])
puts format(class="s">"  full replay (control):  %6.1fms, %5dKB read", full_ms, total_size / 1024)
puts

# Walk two more pages backwards - the cursor is a byte offset
page2 = pager.page_before(page.prev_cursor, per: 50)
page3 = pager.page_before(page2.prev_cursor, per: 50)
puts class="s">"  paging backwards through history:"
[page, page2, page3].each_with_index do |p, i|
  puts format(class="s">"    page %d: %s .. %s  (%d events)", i + 1, p.events.first[class="y">:task_id], p.events.last[class="y">:task_id], p.events.size)
end
puts
puts format(class="s">"  the arithmetic: page 1 cost %dKB of a %dKB file (%.1f%%) and", page.bytes_read / 1024, total_size / 1024, page.bytes_read * 100.0 / total_size)
puts format(class="s">"  ran %.0fx faster than full replay. the cursor is a BYTE OFFSET,", full_ms / page_ms)
puts class="s">"  not a page number - kaminari taught everyone what OFFSET 19950"
puts class="s">"  costs on a growing table, and the same lesson holds for growing"
puts class="s">"  files: numbered pages shift when rows append; cursors don't."
puts class="s">"  full replay remains the right tool for RESUME (you need all"
puts class="s">"  completions); the pager is the right tool for LOOKING (the"
puts class="s">"  incident was ten minutes ago, not ten thousand events ago)."