The Adaptive Throttle

The Adaptive Throttle: nobody TELLS you an upstream's capacity - you discover it. An AIMD controller (TCP's algorithm) probes upward one lane at a time and halves on congestion, converging on the capacity the provider never documented. Watch the target find the truth.

source on github

a real captured run

ADAPTIVE THROTTLE (upstream capacity: undisclosed; AIMD will find it)

  batch   target   p50                                 action
  1       1          20.2ms   ###                      healthy -> probe up to 2
  2       2          20.2ms   ######                   healthy -> probe up to 3
  3       3          20.1ms   #########                healthy -> probe up to 4
  4       4          50.1ms   ############             congested -> halve to 2
  5       2          22.2ms   ######                   healthy -> probe up to 3
  6       3          20.1ms   #########                healthy -> probe up to 4
  7       4          50.1ms   ############             congested -> halve to 2
  8       2          20.2ms   ######                   healthy -> probe up to 3
  9       3          22.1ms   #########                healthy -> probe up to 4
  10      4          50.1ms   ############             congested -> halve to 2
  11      2          20.2ms   ######                   healthy -> probe up to 3
  12      3          24.8ms   #########                healthy -> probe up to 4

  the controller oscillates around 3.0 lanes - the upstream's
  secret capacity is 3. AIMD never saw that constant; it derived
  it from latency alone, and it will re-derive it when the upstream
  changes. static concurrency limits are a guess frozen at deploy
  time; adaptive ones are a measurement that never stops. and the
  controller steers ONE live RateLimit via resize - every client
  sharing that limiter inherits each correction, mid-flight.

source

# frozen_string_literal: true

# The Adaptive Throttle: nobody TELLS you an upstream's capacity - you
# discover it. An AIMD controller (TCP's algorithm) probes upward one
# lane at a time and halves on congestion, converging on the capacity
# the provider never documented. Watch the target find the truth.
#
#   bundle exec ruby examples/adaptive_throttle.rb
#
# Runs offline; the upstream secretly handles 3 concurrent calls well.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"async"

SECRET_CAPACITY = 3
BASE_LATENCY = 0.02
BATCHES = 12
BATCH_SIZE = 6

# The upstream: fast until you exceed its capacity, then it degrades
in_flight = 0
upstream = lambda do
  in_flight += 1
  overload = [in_flight - SECRET_CAPACITY, 0].max
  sleep(BASE_LATENCY * (1 + overload * 1.5))
  in_flight -= 1
end

# AIMD: additive increase, multiplicative decrease - steering ONE live
# limiter via resize, the same object the clients would share
target = 1
limiter = Agentic:class="y">:RateLimit.new(target)
history = []
congestion_threshold = BASE_LATENCY * 1.6

puts class="s">"ADAPTIVE THROTTLE (upstream capacity: undisclosed; AIMD will find it)"
puts
puts format(class="s">"  %-7s %-8s %-10s %-24s %s", class="s">"batch", class="s">"target", class="s">"p50", class="s">"", class="s">"action")

Sync do
  BATCHES.times do |batch|
    limiter.resize(target)
    latencies = []

    BATCH_SIZE.times.map {
      Async do
        limiter.acquire do
          started = Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC)
          upstream.call
          latencies << Process.clock_gettime(Process:class="y">:CLOCK_MONOTONIC) - started
        end
      end
    }.each(&class="y">:wait)

    p50 = latencies.sort[latencies.size / 2]
    congested = p50 > congestion_threshold
    history << target

    action = if congested
      new_target = [target / 2, 1].max
      verdict = class="s">"congested -> halve to #{new_target}"
      target = new_target
      verdict
    else
      target += 1
      class="s">"healthy -> probe up to #{target}"
    end

    puts format(class="s">"  %-7d %-8d %6.1fms   %-24s %s",
      batch + 1, history.last, p50 * 1000, class="s">"#" * (history.last * 3), action)
  end
end

puts
settled = history.last(6)
puts format(class="s">"  the controller oscillates around %.1f lanes - the upstream's", settled.sum.to_f / settled.size)
puts class="s">"  secret capacity is #{SECRET_CAPACITY}. AIMD never saw that constant; it derived"
puts class="s">"  it from latency alone, and it will re-derive it when the upstream"
puts class="s">"  changes. static concurrency limits are a guess frozen at deploy"
puts class="s">"  time; adaptive ones are a measurement that never stops. and the"
puts class="s">"  controller steers ONE live RateLimit via resize - every client"
puts class="s">"  sharing that limiter inherits each correction, mid-flight."