The Throughput Knee

The Throughput Knee: sweep one limiter's ceiling from 1 to 8 against an upstream that quietly serializes above 4, and measure TWO clocks - service time (inside the upstream) and total time (including the wait for a slot). The knee is where they diverge: past it, you're not going faster, you're just queueing somewhere you can't see.

source on github

a real captured run

THROUGHPUT KNEE (24 jobs per ceiling; upstream parallelism undisclosed)

  ceiling   jobs/sec     service p50    total p50
  1             48.6         20.2ms        265.5ms   #####
  2             97.3         20.1ms        136.5ms   ##########
  3            147.8         20.1ms         97.2ms   ###############
  4            196.1         20.1ms         79.3ms   ####################
  5            132.8         40.1ms         99.5ms   #############
  6             99.7         60.1ms        140.2ms   ##########
  7             92.1         80.1ms        140.1ms   #########
  8             79.8        100.1ms        140.1ms   ########

  the knee is at ceiling 4. below it, more lanes bought more
  jobs/sec. above it, throughput didn't plateau - it FELL, because
  overload slows everyone, not just the excess. and SERVICE time rose:
  the upstream only runs 4 at once, so lanes 5-8 didn't add
  parallelism, they just moved the queue from your limiter (where
  total p50 measures it) onto the server (where service p50 hides
  it, and where you usually can't see it at all). watch both clocks:
  when raising your ceiling raises the SERVER's latency, you've
  found their ceiling, and the polite move is to stop pushing.

source

# frozen_string_literal: true

# The Throughput Knee: sweep one limiter's ceiling from 1 to 8 against
# an upstream that quietly serializes above 4, and measure TWO clocks -
# service time (inside the upstream) and total time (including the wait
# for a slot). The knee is where they diverge: past it, you're not
# going faster, you're just queueing somewhere you can't see.
#
#   bundle exec ruby examples/throughput_knee.rb
#
# Runs offline; the upstream's true parallelism is 4.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"async"

TRUE_PARALLELISM = 4
SERVICE_TIME = 0.02
JOBS = 24

# The upstream: work beyond its parallelism doesn't fail, it queues -
# invisibly, on the server's side of the wire
server_in_flight = 0
upstream = lambda do
  server_in_flight += 1
  queued = [server_in_flight - TRUE_PARALLELISM, 0].max
  sleep(SERVICE_TIME * (1 + queued))
  server_in_flight -= 1
end

limiter = Agentic:class="y">:RateLimit.new(1)
rows = []

Sync do
  (1..8).each do |ceiling|
    limiter.resize(ceiling)
    batch_started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
    service_times = []
    total_times = []

    JOBS.times.map {
      Async do
        submitted = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
        limiter.acquire do
          admitted = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
          upstream.call
          finished = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
          service_times << finished - admitted
          total_times << finished - submitted
        end
      end
    }.each(&class="y">:wait)

    elapsed = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - batch_started
    rows << {
      ceiling: ceiling,
      throughput: JOBS / elapsed,
      service_p50: service_times.sort[service_times.size / 2],
      total_p50: total_times.sort[total_times.size / 2]
    }
  end
end

puts class="s">"THROUGHPUT KNEE (#{JOBS} jobs per ceiling; upstream parallelism undisclosed)"
puts
puts format(class="s">"  %-9s %-12s %-14s %-14s %s", class="s">"ceiling", class="s">"jobs/sec", class="s">"service p50", class="s">"total p50", class="s">"")
rows.each do |row|
  bar = class="s">"#" * (row[class="y">:throughput] / 10).round
  puts format(class="s">"  %-9d %8.1f     %8.1fms     %8.1fms   %s",
    row[class="y">:ceiling], row[class="y">:throughput], row[class="y">:service_p50] * 1000, row[class="y">:total_p50] * 1000, bar)
end

# The knee: the last ceiling where throughput still grew meaningfully
knee = rows.each_cons(2).find { |a, b| b[class="y">:throughput] < a[class="y">:throughput] * 1.08 }&.first || rows.last
puts
puts class="s">"  the knee is at ceiling #{knee[class="y">:ceiling]}. below it, more lanes bought more"
puts class="s">"  jobs/sec. above it, throughput didn't plateau - it FELL, because"
puts class="s">"  overload slows everyone, not just the excess. and SERVICE time rose:"
puts class="s">"  the upstream only runs #{TRUE_PARALLELISM} at once, so lanes 5-8 didn't add"
puts class="s">"  parallelism, they just moved the queue from your limiter (where"
puts class="s">"  total p50 measures it) onto the server (where service p50 hides"
puts class="s">"  it, and where you usually can't see it at all). watch both clocks:"
puts class="s">"  when raising your ceiling raises the SERVER's latency, you've"
puts class="s">"  found their ceiling, and the polite move is to stop pushing."