The Kill Switch

The Kill Switch: feature flags answer "who should get this?"; kill switches answer a grimmer question - "how fast can a human make this STOP?" Every capability that talks to money, email, or an external API needs a big red button: instant, global, requiring no deploy, leaving an audit trail of who pressed it and why. Two minutes of incident is a story; twenty is a postmortem.

source on github

a real captured run

THE KILL SWITCH (how fast can a human make it stop?)

  monday, all switches closed:
    digest ran: "emailed: 42 tickets summarized"

  tuesday 09:14, email:send KILLED mid-incident:
    digest status: partial_failure
    email:send is KILLED (by oncall-dana: provider duplicating sends, INC-2291)
    summarize still ran (only the risky capability is dark);
    verdict journaled retryable: false - the dead letter office
    will PARK these, not hammer a bleeding provider with retries.

  tuesday 11:40, provider fixed, switch restored:
    digest ran: "emailed: 42 tickets summarized"

  the audit trail (same journal as the work):
    email:send     killed    by oncall-dana  provider duplicating sends, INC-2291
    email:send     restored  by oncall-dana

  design notes written in pager ink: the switch is checked at USE
  time (no deploy, no restart - the next task sees it); killing is
  per-CAPABILITY, not global (summarize kept working; dark the
  organ, not the patient); killed calls fail with a NON-RETRYABLE
  verdict because a human said stop and the retry machinery must
  not out-vote her; and every flip records who and why, because
  the switch nobody remembers pressing is the outage nobody can
  end. flags ask who should get a feature. switches answer how
  fast you can take one away. build both; press calmly.

source

# frozen_string_literal: true

# The Kill Switch: feature flags answer "who should get this?";
# kill switches answer a grimmer question - "how fast can a human
# make this STOP?" Every capability that talks to money, email, or
# an external API needs a big red button: instant, global, requiring
# no deploy, leaving an audit trail of who pressed it and why. Two
# minutes of incident is a story; twenty is a postmortem.
#
#   bundle exec ruby examples/kill_switch.rb
#
# Runs offline; an incident is simulated, the button is pressed.

require class="s">"bundler/setup"
require class="s">"agentic"
require class="s">"tmpdir"

Agentic.logger.level = class="y">:fatal

class KillSwitches
  def initialize(journal:)
    @journal = journal
    @killed = {}
    @lock = Mutex.new
  end

  # Killing takes WHO and WHY - a red button with no audit trail
  # becomes a mystery outage six months later
  def kill!(capability, by:, reason:)
    @lock.synchronize { @killed[capability] = {by: by, reason: reason} }
    @journal.record(class="y">:kill_switch, description: capability, actor: by, reason: reason, state: class="s">"killed")
  end

  def restore!(capability, by:)
    @lock.synchronize { @killed.delete(capability) }
    @journal.record(class="y">:kill_switch, description: capability, actor: by, state: class="s">"restored")
  end

  def killed?(capability) = @lock.synchronize { @killed.key?(capability) }

  # The guard wraps an agent: killed capabilities fail fast with a
  # HOPELESS verdict - retrying a kill switch is defying the human
  def guard(capability, agent)
    lambda do |task|
      if killed?(capability)
        info = @lock.synchronize { @killed[capability] }
        raise Agentic:class="y">:Errors:class="y">:LlmAuthenticationError, # non-retryable: a human said stop
          class="s">"#{capability} is KILLED (by #{info[class="y">:by]}: #{info[class="y">:reason]})"
      end
      agent.call(task)
    end
  end
end

journal = Agentic:class="y">:ExecutionJournal.new(path: File.join(Dir.tmpdir, class="s">"agentic_kill.jsonl"))
File.delete(journal.path) if File.exist?(journal.path)
switches = KillSwitches.new(journal: journal)

def run_digest(switches, journal)
  orchestrator = Agentic:class="y">:PlanOrchestrator.new(
    lifecycle_hooks: journal.lifecycle_hooks, retry_policy: {max_retries: 0, retryable_errors: []}
  )
  summarize = Agentic:class="y">:Task.new(description: class="s">"summarize", agent_spec: {class="s">"name" => class="s">"s", class="s">"instructions" => class="s">"w"})
  email = Agentic:class="y">:Task.new(description: class="s">"email:digest", agent_spec: {class="s">"name" => class="s">"e", class="s">"instructions" => class="s">"w"})
  orchestrator.add_task(summarize, agent: switches.guard(class="s">"llm:summarize", ->(_t) { class="s">"42 tickets summarized" }))
  orchestrator.add_task(email, [summarize], agent: switches.guard(class="s">"email:send", ->(t) { class="s">"emailed: #{t.previous_output}" }))
  orchestrator.execute_plan
end

puts class="s">"THE KILL SWITCH (how fast can a human make it stop?)"
puts

result = run_digest(switches, journal)
puts class="s">"  monday, all switches closed:"
puts class="s">"    digest ran: #{result.results.values.map(&class="y">:output).last.inspect}"
puts

# TUESDAY, 09:14 - the email provider is duplicating sends. INCIDENT.
switches.kill!(class="s">"email:send", by: class="s">"oncall-dana", reason: class="s">"provider duplicating sends, INC-2291")
result = run_digest(switches, journal)
failed = result.results.values.find { |r| !r.successful? }
puts class="s">"  tuesday 09:14, email:send KILLED mid-incident:"
puts class="s">"    digest status: #{result.status}"
puts class="s">"    #{failed.failure.message}"
puts class="s">"    summarize still ran (only the risky capability is dark);"
puts class="s">"    verdict journaled retryable: #{failed.failure.retryable?.inspect} - the dead letter office"
puts class="s">"    will PARK these, not hammer a bleeding provider with retries."
puts

switches.restore!(class="s">"email:send", by: class="s">"oncall-dana")
result = run_digest(switches, journal)
puts class="s">"  tuesday 11:40, provider fixed, switch restored:"
puts class="s">"    digest ran: #{result.results.values.map(&class="y">:output).last.inspect}"
puts

state = Agentic:class="y">:ExecutionJournal.replay(path: journal.path)
flips = state.events.select { |e| e[class="y">:event] == class="s">"kill_switch" }
puts class="s">"  the audit trail (same journal as the work):"
flips.each { |f| puts format(class="s">"    %-14s %-9s by %-12s %s", f[class="y">:description], f[class="y">:state], f[class="y">:actor], f[class="y">:reason]) }
puts
puts class="s">"  design notes written in pager ink: the switch is checked at USE"
puts class="s">"  time (no deploy, no restart - the next task sees it); killing is"
puts class="s">"  per-CAPABILITY, not global (summarize kept working; dark the"
puts class="s">"  organ, not the patient); killed calls fail with a NON-RETRYABLE"
puts class="s">"  verdict because a human said stop and the retry machinery must"
puts class="s">"  not out-vote her; and every flip records who and why, because"
puts class="s">"  the switch nobody remembers pressing is the outage nobody can"
puts class="s">"  end. flags ask who should get a feature. switches answer how"
puts class="s">"  fast you can take one away. build both; press calmly."