trail-boss/docs/plan/plan.md

# Trail Boss — design plan

The complete design: the problem, operating principles, the session/delivery model, the
architecture, what's been empirically confirmed, the phases, and the open questions. See
[`../research/claude-code-mechanics.md`](../research/claude-code-mechanics.md) for the Claude
Code primitives and [`../research/related-work.md`](../research/related-work.md) for prior art.

---

## Problem

You run long-form, human-in-the-loop agentic coding across many concurrent agent sessions,
one per tmux window. Each session periodically stalls waiting on **you**: a permission prompt,
a clarifying question, or a finished turn awaiting the next instruction. Today you find those
stalls by **manually cycling windows**. That polling is the bottleneck: most of your time goes
to *finding* the session that needs you, not *answering* it, and blocked sessions burn
wall-clock while otherwise-parallel work waits.

---

## Operating principles

### Human *on* the loop, not *in* it

Classic agentic HITL wires you into the inner cycle — approving each step, answering each
prompt — so you are the bottleneck on every iteration. Trail Boss flips it: agents run
autonomously by default and you supervise from above, **engaged only by exception**. When an
agent can't proceed on its own, it falls through to you.

**The human is the failure mode.** Trail Boss is a **dead-letter queue for a fleet of agents**:
the happy path never touches you; only stalled work routes to you, you process the exception
(reply or skip), and it goes back on the wire.

### The "stuck = needs attention" axiom — and stuck is stuck

A session that has stopped *or* is waiting at a permission prompt cannot progress toward its
goal until the human responds — therefore it needs intervention, by definition. This collapses
two fuzzy questions at once:

- **No "idle vs. done" distinction.** There is no separate "finished but fine" state; if it
  stopped and you haven't responded, it's waiting on you. Every stop is a queue item.
- **No "permission vs. stopped" distinction.** It doesn't matter *why* a session is stuck —
  both mean "blocked until the human acts." The two are detected by different hooks (see below)
  but are **treated identically** in the queue. `reason` is display-only metadata, never a
  priority input.

So the queue is a flat **dead-letter queue**: stuck sessions accumulate and the operator
depletes them. (The deeper fix — making the interactive CLIs longer-running so they stop less
often — is a separate workstream, not Trail Boss's concern.)

### Navigator, not relay

Trail Boss does **not** inject answers into sessions. It routes *your attention* to the live
session and you interact with the real CLI directly. It is an attention router + tmux
navigator, not an input relay or an autonomous responder. (Rationale and the rejected
alternatives are in [`../notes/decisions.md`](../notes/decisions.md).)

---

## Non-goals

- **Not a fleet spawner / supervisor.** Trail Boss does not launch, kill, or cost-optimize
  agents. It assumes sessions already exist and surfaces the stuck ones. (Spawning is the job
  of separate fleet tooling.)
- **Not an autonomous responder.** It never synthesizes or auto-sends a reply. It only routes
  you to a session; all input is human-authored, typed into the real CLI.
- **Not multi-operator / multi-tenant.** Single operator, single host, single tmux server.
- **Not a remote web product.** It is a same-host, tmux-native tool whose sole job is to
  eliminate manual tab-switching. Remote access is out of scope for v1.
- **Not dependent on plan mode.** Vanilla Claude Code plan mode is assumed disabled (the
  operator uses their own); `reason` collapses to **permission** and **stopped/needs-next**.

---

## Session & delivery model

There are two mutually exclusive ways to run agent sessions; Trail Boss commits to one.

- **Model A — live panes (CHOSEN).** Long-running interactive CLIs stay resident in tmux
  panes. Delivery happens by *interacting with the live process* — Trail Boss navigates you to
  it. Survives disconnect via tmux (see Architecture).
- **Model B — transcript sessions (rejected for v1).** No resident process; a session exists
  only as its transcript, and a fresh `claude --resume <id>` is spawned per turn with Trail
  Boss rendering output itself.

**Why not Model B / resume-to-deliver:** a session's durable state is its transcript JSONL, but
a *live* interactive CLI holds its own in-memory copy and does not re-read that file for
outside changes. Running a second `claude --resume <id>` while the original is alive yields two
processes over one transcript: the resuming process produces the response *in itself*, the
original pane never reflects it, and concurrent writes risk divergence. The existence of
`--fork-session` ("create a new session ID instead of reusing the original") confirms plain
`--resume` reuses the session and is not designed for a concurrent second attach. So a reply
delivered via resume **does not** reach the original interactive pane. Model A + direct
interaction sidesteps this entirely.

---

## What is needed (capabilities)

1. **A blocked-state signal from every session** — Claude Code hooks (`Stop`,
   `PermissionRequest`), POSTing their stdin JSON to a local collector.
2. **A central collector + live state store** — tracks every session's status, holds the
   `session_id → pane` registry, broadcasts the queue.
3. **Context extraction** — *what* each session is asking. Largely free from the hook payload
   (see below); transcript tail for deeper/permission context.
4. **The Trail Boss queue** — a FIFO depletion surface (oldest-stuck first), keyboard-driven.
5. **Delivery by navigation** — route the operator to the live pane (tmux), no relay.

---

## Detection model

Two enqueue triggers, treated identically. **Both are required** — they catch *different*
stuck conditions: a session waiting at a permission prompt is mid-turn and does **not** emit
`Stop`, so without `PermissionRequest` it would never be detected. `Notification` is dropped.

| Hook | Why it's needed | Status |
|------|----------------------|--------|
| `Stop` | Turn finished; session waiting for the next instruction. | Confirmed firing in interactive and `-p` (probe 2026-05-25); payload carries `last_assistant_message` |
| `PermissionRequest` | Session blocked mid-turn on approval — emits **no** `Stop`, so this is the only signal for the permission case. | Exists; firing/payload **not yet probed** (phase 1) |
| `UserPromptSubmit` | Input submitted → session unstuck → **dequeue**. | Confirmed primitive |
| `SessionStart` / `SessionEnd` | Register / retire the session (and re-assert `session_id → pane`). | Confirmed firing (probe) |
| ~~`Notification`~~ | Dropped — `Stop` + `PermissionRequest` cover every stuck case; the dead-letter queue just fills and drains. | Not used |

Both `Stop` and `PermissionRequest` enqueue a plain stuck item with no priority difference.

---

## Correlation & the self-healing registry

**Confirmed by probe (2026-05-25):** hook commands inherit the full ambient environment,
including `$TMUX_PANE`. So the `session_id → pane` mapping does not require a special hook — it
is rebuilt continuously.

- **Capture `$TMUX_PANE` on *every* emit** (not just `SessionStart`). Each event re-asserts
  `session_id → pane`, so the registry self-heals across resume, pane reuse, and window moves:
  a reused pane is corrected by the next event from its new session.
- Identity is available **both** as env vars and in the stdin payload (belt-and-suspenders):
  - env: `CLAUDE_CODE_SESSION_ID`, `CLAUDE_PROJECT_DIR`, `TMUX_PANE`, `TMUX`,
    `CLAUDE_CODE_ENTRYPOINT` (`cli` interactive vs `sdk-cli` for `-p`), `CLAUDECODE=1`,
    `TERM_PROGRAM=tmux`, `CLAUDE_ENV_FILE` (per-session state dir).
  - payload: `session_id`, `transcript_path`, `cwd`, `hook_event_name`, plus event-specific
    fields (below).
- Pane ids (`%446`) are **tmux-server-global** — addressable by any `tmux` command from outside
  tmux, which is what makes navigation-from-a-daemon possible.

---

## Context — what the session is asking

- **From the `Stop` payload directly (no transcript needed for the basic case):**
  `last_assistant_message` contains what the agent just said — render it straight in the queue
  card. Confirmed present in both modes. Stop also carries `permission_mode`, `effort`,
  `stop_hook_active`, `background_tasks`, `session_crons`.
- **From the transcript JSONL** (`transcript_path`, append-only, tailable): deeper context and,
  for `PermissionRequest`, the proposed tool/command. An enhancement, not a requirement.
- **From `tmux capture-pane -p <pane>`:** the literal on-screen prompt verbatim, as a fallback.

---

## Delivery — navigation via tmux (no relay)

In Model A you interact with the real CLI, so there is no fidelity/relay problem. tmux 3.5a on
the host provides every needed primitive (all confirmed present):
`switch-client`, `select-window`, `select-pane`, `link-window`/`unlink-window`,
`join-pane`/`break-pane`, `pipe-pane`, `capture-pane`, `display-popup`.

- **Minimal (recommended start):** route the operator's client to the most-stuck pane —
  `switch-client` + `select-window`/`select-pane %id`. This *is* "eliminate manual
  tab-switching," with zero relay.
- **Embedded (optional polish):** `link-window` the target session's window into a Trail Boss
  view so the queue and live pane are co-visible, then `unlink-window`. Non-destructive (tmux
  windows are shared objects). **Avoid `join-pane` as the primary** — it relocates the pane out
  of its home window and does not cleanly round-trip.
- **`send-keys`** remains available for plain text (basic submission confirmed working in the
  probe) but is a secondary path; native interaction is preferred.
- **Rejected:** `claude --remote-control` routes a session to the claude.ai / desktop / mobile
  surface, not a local control channel — useless for a same-host tool.

Because you interact with the real prompt, **"edit before allow" is native** — you just type in
the live pane. No special edit affordance or `canUseTool` round-trip is required in Model A.

---

## State & reliability

**The transcript JSONL is the ground truth; hooks are only the low-latency notification.**

- **Reconcile loop:** after a `Stop` for a session, watch its transcript; if new lines appear (a
  user message, a new assistant turn) it progressed → **dequeue**. A periodic sweep of all known
  transcripts rebuilds "is this session currently stopped?" purely from file state.
- This self-corrects **dropped hook POSTs** (hooks are fire-and-forget and must exit 0, so a
  POST to a down/slow collector is silently lost) and **collector restarts** (in-flight status
  is rebuilt from transcripts, not from missed transition events).
- It also handles **"you answered directly in the pane"** for free: a new user entry in the
  transcript → the item dequeues without a UI action.

---

## Architecture

```
   tmux server (host) ─ survives client disconnect
   ┌──────────────────────────────────────────────────────────┐
   │  agent pane 1 ─┐ Claude Code (hooks → curl localhost:4000) │
   │  agent pane 2 ─┤   Stop / PermissionRequest / SessionStart │
   │  agent pane N ─┘   (each emit carries $TMUX_PANE)          │
   │                          │                                 │
   │  ┌───────────────────────▼──────────────────┐             │
   │  │  Trail Boss daemon  (its own tmux window) │             │
   │  │  • ingest hooks, upsert state (SQLite)    │             │
   │  │  • session_id → pane registry (self-heal) │             │
   │  │  • transcript reconcile loop (ground truth)│            │
   │  │  • FIFO depletion queue (oldest-stuck 1st)│             │
   │  └───────────────────────┬──────────────────┘             │
   └──────────────────────────┼─────────────────────────────────┘
                              │ presentation (on reattach or keybinding)
                              ▼
           display-popup (queue overlay)  ──select──▶  switch-client /
           + optional status-line "N stuck"            select-window/pane
                                                        → you land on the
                                                          live most-stuck pane
```

### Daemon vs. presentation split

- **Control plane — the daemon.** Always-on; ingests hooks, holds state, runs the reconcile
  loop, orders the queue FIFO, and issues `tmux` commands to navigate. It drives tmux "from outside" the agent
  panes — it does not need to occupy an agent pane to do so.
- **Presentation plane — transient & tmux-native.** A keybinding fires `tmux display-popup -E
  trailboss` to overlay the queue on your client; selecting an item runs `switch-client` +
  `select-window`/`select-pane` to drop you on the live pane. An optional status-line segment
  shows ambient "N stuck."

### Durability (the disconnect requirement)

Two things must survive an SSH disconnect:

1. **Agent sessions** — already durable: the tmux server is host-side; your terminal is just a
   client. Detach/drop leaves panes running; `tmux attach` restores them.
2. **The Trail Boss daemon** — a process started in your login shell would die on SIGHUP. So run
   it **in its own tmux window** (simplest; one server then holds agents *and* Trail Boss) or
   under **`systemd --user`** (if you also want it to survive host *reboots* — tmux does not).

**Backlog accumulation (a feature of this design):** because the daemon and the hook POSTs keep
running on the host while you're disconnected, the queue accumulates whatever got stuck in your
absence. On reattach, Trail Boss shows exactly what piled up — disconnecting becomes a
non-event instead of context loss.

---

## Layering: harness-coupled detection vs. harness-agnostic core

The most consequential open architecture question (and a deliberate seam): **at what layer does
Trail Boss operate?** The two halves want different answers.

- **Switching is already tmux-level and harness-agnostic.** Navigating to a stuck session is
  `switch-client`/`select-window`/`select-pane %id` — it works for *any* program in a pane,
  Claude Code or a future coding harness. Nothing about routing is Claude-specific.
- **Detection is currently Claude-Code-coupled.** The stuck/unstuck signal comes from Claude
  Code hooks (`Stop`, `PermissionRequest`, `UserPromptSubmit`). That is the reliable signal, but
  it binds detection to one harness.

To keep the door open for future harnesses without coupling the core, put detection behind an
**adapter interface**. The daemon consumes a normalized event — *"session S at pane P became
stuck / unstuck"* — and everything downstream (queue, FIFO depletion, navigation) is
harness-agnostic. Adapters produce that normalized event however they can:

- **Claude Code adapter (v1):** hooks → normalized event. Reliable, confirmed.
- **Future harness adapters:** their own hooks if they have them; else log/transcript tailing;
  else a tmux-level heuristic (e.g. pane output gone quiet at a prompt). Less reliable, but the
  core doesn't change.

**Decision for v1:** build the Claude Code adapter (hooks), but define the daemon's input as the
normalized stuck/unstuck event — *not* raw hook payloads — so the harness coupling stays
isolated to the adapter. The reliability of detection is the adapter's problem; switching is
always tmux. **Open:** the exact normalized event contract, and whether a purely tmux-level
detector (no hooks) is viable as a universal fallback.

---

## Queue & interaction loop (depletion)

The queue is a flat FIFO dead-letter queue, and the interaction model is **auto-advance
depletion**: you are always looking at one stuck session; when you finish with it, the next one
loads.

- **Membership:** every stuck session (from `Stop` or `PermissionRequest`). No priority between
  reasons; `reason` is display-only. Reconcile removes any that have progressed.
- **Order:** oldest-stuck first (FIFO). The head of the queue is "the current session."
- **Auto-advance:** the operator's focus is navigated to the current session. When that session
  resolves — `UserPromptSubmit` fires (you responded) or you `skip` — Trail Boss **loads the
  next** stuck session into focus. The operator drains the queue one session at a time without
  ever choosing "which next."
- **Skip:** advances to the next without acting; the skipped session stays in the queue and
  re-surfaces later in the cycle.
- **Dequeue:** transcript advances past the last stuck point, or `UserPromptSubmit` fires, or
  `SessionEnd`.
- **Empty queue:** nothing stuck → no auto-advance; the operator is free. New stuck sessions
  re-arm the loop.

Saturation is a non-issue by construction: the queue can be arbitrarily long; the operator just
keeps depleting it, and the next-in-line always loads. There is no ceiling logic.

---

## Confirmed mechanics (empirical, 2026-05-25)

Probe: a `--settings`-loaded hook dumping env + stdin payload, run both via `claude -p` and a
driven interactive session in a throwaway tmux pane.

- **`$TMUX_PANE` is present in the hook environment** (`%445` / `%446`, matching the launch
  pane) — in *both* interactive (`CLAUDE_CODE_ENTRYPOINT=cli`) and headless (`sdk-cli`) modes.
  This is the load-bearing fact for the self-healing registry.
- **Both `SessionStart` and `Stop` fire**; interactive `Stop` fired ~4s after a human-prompted
  turn.
- **`Stop` payload includes `last_assistant_message`** → queue context for free.
- **Identity also exposed as env vars** (`CLAUDE_CODE_SESSION_ID`, `CLAUDE_PROJECT_DIR`).
- **send-keys plain-text submission works** (typed prompt + `Enter` submitted cleanly).
- **tmux 3.5a** has all required commands; **`claude` 2.1.150** offers `-r/--resume`,
  `-c/--continue`, `--fork-session`, `--session-id`, `--settings`, `--remote-control`.

Still **unverified** (probe before depending on): `PermissionRequest` firing + payload shape
(esp. how the proposed command is represented and which gate types trigger it).

---

## Implementation phases

1. **Probe `PermissionRequest`** — confirm it fires for the gate types you hit and what its
   payload carries (the one remaining unknown). The `Stop`/`SessionStart`/`$TMUX_PANE` path is
   already confirmed.
2. **Emitter** — `trailboss-emit.sh` (carries `$TMUX_PANE` on every event) + the `settings.json`
   hook wiring.
3. **Daemon (control plane)** — ingest endpoint behind the normalized stuck/unstuck adapter
   contract, SQLite state, self-healing registry, the transcript reconcile loop, FIFO queue.
   Runs in its own tmux window.
4. **Navigation** — `switch-client`/`select-window`/`select-pane` to route to a pane by id, and
   auto-advance to the next stuck session on resolve/skip.
5. **Presentation** — `display-popup` queue overlay + keybinding; optional status-line segment.
6. **Close the loop (walking skeleton)** — stuck pane → loaded into focus → interact → reconcile
   dequeues it → next stuck session auto-loads. This end-to-end depletion path is the first
   milestone.
7. Iterate: auto-advance trigger tuning, skip/re-surface behavior, embedded `link-window` view,
   and a second harness adapter to validate the abstraction.

---

## Failure modes & invariants

**Invariants**
- *Human-authored only:* the daemon never sends synthesized input to a session.
- *The queue never lies:* a displayed item reflects current transcript state (reconcile is
  authoritative over hook events).

**Failure modes**
- *Hook POST dropped (collector down/slow):* hook exits 0, event lost → reconcile sweep
  recovers it from transcripts.
- *Daemon restart:* SQLite persists rows; current blocked-status is rebuilt from transcripts.
- *Pane reused / session resumed:* next event re-asserts `session_id → pane`; navigation always
  targets the pane in the latest event.
- *Host reboot:* tmux server (and thus everything) is lost. **v1: the operator re-invokes Trail
  Boss and relaunches sessions manually after a restart** — no auto-resurrection.
- *Stale navigation target:* worst case you land on a pane that already moved on; reconcile
  would have dequeued it, so the popup shouldn't have offered it — acceptable, non-destructive.

---

## Open questions

**Open**

1. **Harness layering / adapter contract** *(the main one)* — define the normalized
   stuck/unstuck event the daemon consumes, so detection stays isolated to a per-harness
   adapter. Is a purely tmux-level detector (no hooks) viable as a universal fallback for future
   harnesses? See "Layering" above.
2. **`PermissionRequest` specifics** — confirm it fires for the gate types you hit and what its
   payload carries (the proposed command, for display). Detection coverage depends on it; phase 1.
3. **Auto-advance trigger** — exactly what counts as "done with the current session" →
   load next: `UserPromptSubmit` and explicit `skip` are clear; should manually navigating away
   also advance? And is the jump immediate or on a keypress?
4. **Presentation UX** — `display-popup` queue + jump vs. a dedicated always-visible window;
   decide after the walking skeleton.

**Resolved this round (recorded so they don't get re-litigated)**

- *Permission vs. stopped priority* → none. Stuck is stuck; `reason` is display-only, queue is
  FIFO.
- *`Notification`* → dropped; `Stop` + `PermissionRequest` cover every stuck case.
- *Multiple tmux clients* → not a real scenario; one active focus, auto-advanced through the
  queue (single-operator non-goal).
- *Reboot durability* → out; operator re-invokes after restart.
- *Concurrency ceiling* → non-issue; the depletion loop just loads the next one, no ceiling
  logic.