Connection Pool Care

Connection Pool Care: the production incident with the most misleading symptoms - the database is fine, the app is fine, and nothing works, because the connection pool drained one leaked checkout at a time. The leak is always the same code: a checkout without an ensure, an exception on the unhappy path, a connection that never comes home. Two disciplines fix it forever: the BLOCK FORM is the API (checkout/checkin as separate calls is rope, and not the climbing kind), and the …

source on github

a real captured run

CONNECTION POOL CARE (the block form is the API; the pool keeps receipts)

  the leaky version (checkout with no ensure, unhappy path every 4th job):
    served: 19; pool now holds 0/5 connections
    6 job(s) hit exhaustion, and the error came with RECEIPTS:
      job 5 waited past timeout; pool of 5 exhausted. current holders:
      conn-1 held by job 3 for 54ms

  the block-form version (same jobs, same failure rate):
    served: 23; failed cleanly: 7; exhausted: 0
    pool restored to 5/5 - every connection came home, including from the failures

  the two disciplines, priced: the unhappy path leaked exactly
  5 connections (one per early failure until the pool was dry),
  and from then on INNOCENT jobs paid - exhaustion punishes
  whoever arrives after the leak, which is why pool incidents
  always look like someone else's fault. the block form ends the
  species: ensure runs on the unhappy path too, so every failure
  returned its connection. and the receipts matter as much as the
  fix - an exhaustion error that names its holders and their
  hold-times turns 'the database is slow??' into 'job 3 has held
  conn-4 for 40ms and never checks in' - attribution at the
  moment of pain, not archaeology after it.

source

# frozen_string_literal: true

# Connection Pool Care: the production incident with the most
# misleading symptoms - the database is fine, the app is fine, and
# nothing works, because the connection pool drained one leaked
# checkout at a time. The leak is always the same code: a checkout
# without an ensure, an exception on the unhappy path, a connection
# that never comes home. Two disciplines fix it forever: the BLOCK
# FORM is the API (checkout/checkin as separate calls is rope, and
# not the climbing kind), and the pool itself keeps RECEIPTS - when
# exhaustion hits, the timeout error names every holder and how
# long they've squatted, so the leak is attributed at the moment it
# hurts instead of guessed at from graphs three hours later.
#
#   bundle exec ruby examples/connection_pool_care.rb
#
# Runs offline; exits 1 unless the leak is caught with names AND
# the block form provably ends it.

require class="s">"bundler/setup"
require class="s">"agentic"

Agentic.logger.level = class="y">:fatal

def mono = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)

class PoolExhausted < StandardError; end

class Pool
  attr_reader class="y">:size

  def initialize(size)
    @size = size
    @available = Array.new(size) { |i| class="s">"conn-#{i}" }
    @holders = {}
  end

  def checkout(who, timeout: 0.05)
    deadline = mono + timeout
    while @available.empty?
      raise PoolExhausted, exhaustion_report(who) if mono > deadline
      sleep(0.005)
    end
    conn = @available.pop
    @holders[conn] = {who: who, since: mono}
    conn
  end

  def checkin(conn)
    @holders.delete(conn)
    @available.push(conn)
  end

  # The block form IS the api; everything else is rope
  def with(who, &block)
    conn = checkout(who)
    begin
      yield conn
    ensure
      checkin(conn)
    end
  end

  def available_count = @available.size

  def exhaustion_report(who)
    lines = @holders.map { |conn, h| class="s">"#{conn} held by #{h[class="y">:who]} for #{((mono - h[class="y">:since]) * 1000).round}ms" }
    class="s">"#{who} waited past timeout; pool of #{@size} exhausted. current holders:\n      " + lines.join(class="s">"\n      ")
  end
end

# Thirty jobs; every fourth hits the unhappy path and raises
def run_jobs(pool, style:)
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(concurrency_limit: 8, retry_policy: {max_retries: 0, retryable_errors: []})
  outcomes = {served: 0, failed_cleanly: 0, exhausted: []}
  30.times do |i|
    task = Agentic:class="y">:Task.new(description: class="s">"job #{i}", agent_spec: {class="s">"name" => class="s">"job #{i}", class="s">"instructions" => class="s">"query"})
    orchestrator.add_task(task, agent: ->(_t) {
      work = -> {
        sleep(0.005)
        raise class="s">"unhappy path in job #{i}" if i % 4 == 3
        outcomes[class="y">:served] += 1
      }
      begin
        if style == class="y">:leaky
          conn = pool.checkout(class="s">"job #{i}") # no ensure. the author was in a hurry.
          work.call
          pool.checkin(conn) # unreachable on the unhappy path
        else
          pool.with(class="s">"job #{i}") { work.call }
        end
        class="y">:done
      rescue PoolExhausted => e
        outcomes[class="y">:exhausted] << e.message
        raise
      rescue RuntimeError
        outcomes[class="y">:failed_cleanly] += 1
        raise
      end
    })
  end
  orchestrator.execute_plan
  outcomes
end

puts class="s">"CONNECTION POOL CARE (the block form is the API; the pool keeps receipts)"
puts

leaky_pool = Pool.new(5)
leaky = run_jobs(leaky_pool, style: class="y">:leaky)
puts class="s">"  the leaky version (checkout with no ensure, unhappy path every 4th job):"
puts class="s">"    served: #{leaky[class="y">:served]}; pool now holds #{leaky_pool.available_count}/5 connections"
puts class="s">"    #{leaky[class="y">:exhausted].size} job(s) hit exhaustion, and the error came with RECEIPTS:"
puts class="s">"      #{leaky[class="y">:exhausted].first&.lines&.first}#{leaky[class="y">:exhausted].first&.lines&.[](1)}"
puts

healthy_pool = Pool.new(5)
healthy = run_jobs(healthy_pool, style: class="y">:block)
puts class="s">"  the block-form version (same jobs, same failure rate):"
puts class="s">"    served: #{healthy[class="y">:served]}; failed cleanly: #{healthy[class="y">:failed_cleanly]}; exhausted: #{healthy[class="y">:exhausted].size}"
puts class="s">"    pool restored to #{healthy_pool.available_count}/5 - every connection came home, including from the failures"
puts

failures = []
failures << class="s">"leak didn't drain the pool" unless leaky_pool.available_count.zero? && leaky[class="y">:exhausted].any?
failures << class="s">"exhaustion report lacks receipts" unless leaky[class="y">:exhausted].first.to_s.include?(class="s">"held by job")
failures << class="s">"block form leaked (#{healthy_pool.available_count}/5)" unless healthy_pool.available_count == 5
failures << class="s">"block form dropped work" unless healthy[class="y">:served] == 23 && healthy[class="y">:failed_cleanly] == 7

puts class="s">"  the two disciplines, priced: the unhappy path leaked exactly"
puts class="s">"  #{5 - leaky_pool.available_count} connections (one per early failure until the pool was dry),"
puts class="s">"  and from then on INNOCENT jobs paid - exhaustion punishes"
puts class="s">"  whoever arrives after the leak, which is why pool incidents"
puts class="s">"  always look like someone else's fault. the block form ends the"
puts class="s">"  species: ensure runs on the unhappy path too, so every failure"
puts class="s">"  returned its connection. and the receipts matter as much as the"
puts class="s">"  fix - an exhaustion error that names its holders and their"
puts class="s">"  hold-times turns 'the database is slow??' into 'job 3 has held"
puts class="s">"  conn-4 for 40ms and never checks in' - attribution at the"
puts class="s">"  moment of pain, not archaeology after it."
exit(failures.empty? ? 0 : 1)