From 5c6a6924c1ab96e4838be315e10f8ace74e89dac Mon Sep 17 00:00:00 2001 From: jedarden Date: Tue, 7 Jul 2026 06:19:29 -0400 Subject: [PATCH] docs(bf-34lwt): root-cause tokenless-sim REJECT in WS hello path Definitive root-cause for the bf-4iewr split's first child. Confirms against current source (with file:line citations) that: - The token validator is wired UNCONDITIONALLY (main.go:4494) - It reads hello.Token only (server.go:513); the sim never populates it - The sim sends its token only as the X-Spaxel-Token header (cmd/sim main.go:633/scenario.go:318), and the mothership never reads that header (HandleNodeWS upgrades unconditionally at server.go:469) -> the token-supply path is currently DEAD - The default 24h migration window (config.go:139) is the only thing accepting sim nodes -> current 'no reject' is a window MASK, not a fix - REJECT fires only when the window closes (SPAXEL_MIGRATION_WINDOW_HOURS=0 or uptime > 24h) Corroborated by the e2e harness comment (tests/e2e/e2e_test.go:86-94) which pins the window open and documents exactly this. Co-Authored-By: Claude --- docs/notes/token-reject-root-cause.md | 227 ++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 docs/notes/token-reject-root-cause.md diff --git a/docs/notes/token-reject-root-cause.md b/docs/notes/token-reject-root-cause.md new file mode 100644 index 0000000..7680aa4 --- /dev/null +++ b/docs/notes/token-reject-root-cause.md @@ -0,0 +1,227 @@ +# Root Cause: tokenless-sim REJECT in the WS hello path + +**Tracking:** bead `bf-34lwt` (first link of the `bf-4iewr` split). +**Date:** 2026-07-07. +**Scope:** explain exactly *why* a tokenless `spaxel-sim` node is currently +NOT rejected on a default boot, and exactly *when* it breaks. This is the +precise target the remaining `bf-4iewr` children build on. + +> **Path convention.** Citations below are full repo-relative paths. The +> `bf-4iewr` task body drops the `mothership/` module prefix, so e.g. its +> `cmd/sim/main.go:633` is `mothership/cmd/sim/main.go:633` here, and its +> `cmd/mothership/main.go:4494` is `mothership/cmd/mothership/main.go:4494`. +> +> **Two sim copies — both behave identically for this finding.** There are two +> `spaxel-sim` source trees and both send the token only as a header with no +> `token` in the hello body: +> - `cmd/sim/main.go` — the module `go.work` lists (`use ./cmd/sim`); the dev/test +> path. Header set at `cmd/sim/main.go:313`; hello body `cmd/sim/main.go:348-359` +> (no `token` key). This copy has no `scenario.go`. +> - `mothership/cmd/sim/` — the copy the **Dockerfile actually builds** (it does +> `COPY mothership/ ./` then `go build ./cmd/sim`, so `./cmd/sim` resolves here). +> Header set at `mothership/cmd/sim/main.go:633` and +> `mothership/cmd/sim/scenario.go:318`; hello bodies +> `mothership/cmd/sim/main.go:652-665` and `mothership/cmd/sim/scenario.go:327-340` +> (neither has a `token` key). +> +> A `grep '"token"'` over all four hello bodies returns only the `--token` flag +> definition, never a JSON body key — so `hello.Token` deserializes to `""` from +> either binary. + +## TL;DR + +The current "no reject" is a **24-hour migration-window MASK, not a real +fix.** The token validator is wired unconditionally, but the sim presents its +token only as an HTTP header the mothership never reads, so `hello.Token` is +always empty and `tokenOK` is always `false`. The only thing keeping sim nodes +connected is the default 24h migration window, which accepts any tokenless +node as `Unpaired`. The moment that window closes (`SPAXEL_MIGRATION_WINDOW_HOURS=0` +or uptime > 24h), every sim node is `sendReject`-ed with `invalid_token` and +disconnected. + +## Confirmed facts (with citations) + +### 1. The validator is wired UNCONDITIONALLY + +`mothership/cmd/mothership/main.go:4494`: + +```go +ingestSrv.SetTokenValidator(provSrv.ValidateToken) +``` + +There is **no config flag and no build gate** around this call. "Validator +configured?" is therefore **always YES** at runtime. The only thing the next +block gates on is the *deadline*, not the validator: + +```go +// mothership/cmd/mothership/main.go:4495-4499 +if cfg.MigrationWindowHours > 0 { + deadline := time.Now().Add(time.Duration(cfg.MigrationWindowHours) * time.Hour) + ingestSrv.SetMigrationDeadline(deadline) + ... +} +``` + +So `s.tokenValidator` is always non-nil; only `s.migrationDeadline` is +conditional. + +### 2. The validator reads `hello.Token` only + +`mothership/internal/ingestion/server.go:513`: + +```go +tokenOK := hello.Token != "" && validator(hello.MAC, hello.Token) +``` + +`validator` is `provSrv.ValidateToken`, signature `func(mac, token string) bool` +(see `SetTokenValidator` at `mothership/internal/ingestion/server.go:321-327`). +Its only inputs are `hello.MAC` and `hello.Token`. `hello.Token` maps to the +JSON field `token,omitempty` (`mothership/internal/ingestion/message.go:22`). + +There is no other path that can set `tokenOK = true`: it requires a non-empty +`hello.Token` that the validator accepts. + +### 3. The sim sends its token ONLY as the `X-Spaxel-Token` HTTP header + +- `mothership/cmd/sim/main.go:633`: `headers.Set("X-Spaxel-Token", token)` +- `mothership/cmd/sim/scenario.go:318`: `headers.Set("X-Spaxel-Token", token)` + +The header is attached to the WebSocket *dial* request +(`websocket.DefaultDialer.DialContext(ctx, url, headers)`). + +The hello **JSON body** does **NOT** include a `token` field. The hello map +built at `mothership/cmd/sim/main.go:652-665` contains only: + +``` +type, mac, firmware_version, capabilities, chip, flash_mb, +uptime_ms, wifi_rssi, ip, pos_x, pos_y, pos_z +``` + +(identical field set in `mothership/cmd/sim/scenario.go:327-340`). No `token` +key. So `hello.Token` always deserializes to `""`. + +### 4. The mothership NEVER reads that header → the token-supply path is DEAD + +A search of the entire mothership tree for the literal header name returns +**zero** hits in any non-sim, non-test source: + +``` +$ grep -rn 'X-Spaxel-Token' mothership/ # only sim files +mothership/cmd/sim/main.go:633: headers.Set("X-Spaxel-Token", token) +mothership/cmd/sim/scenario.go:318: headers.Set("X-Spaxel-Token", token) +``` + +The WS upgrade handler never inspects the request either. +`mothership/internal/ingestion/server.go:469`: + +```go +conn, err := s.upgrader.Upgrade(w, r, nil) +``` + +`HandleNodeWS` (`mothership/internal/ingestion/server.go:455`) reads nothing +off `r` before upgrading — no `r.Header.Get(...)`. (All `Header.Get` / `Header.Set` +hits under `mothership/internal/` are on the *outbound* webhook/notify path or +in their tests, e.g. `webhook`, `notify`, `notifications/ntfy.go`; none touch +the node WS path.) + +**Consequence:** for every sim node, `hello.Token == ""`, so +`hello.Token != ""` short-circuits to `false` at `server.go:513`, making +`tokenOK` always `false` regardless of what token the sim provisioned. The +parent bead's "have the sim supply valid tokens" option is therefore +**currently NON-FUNCTIONAL** — the token is supplied to a header nobody reads. + +### 5. The only thing accepting sim nodes is the migration window + +`mothership/internal/ingestion/server.go:514-519`: + +```go +if !tokenOK { + if !deadline.IsZero() && time.Now().Before(deadline) { + // accepted as Unpaired + nc.Unpaired = true + } else { + ... reject ... + } +} +``` + +`MigrationWindowHours` **defaults to 24** (`mothership/internal/config/config.go:139`, +`cfg.MigrationWindowHours = 24`). On a fresh boot with the default, +`mothership/cmd/mothership/main.go:4496` sets +`deadline = time.Now().Add(24h)`, so `!deadline.IsZero() && Now().Before(deadline)` +is true → every tokenless sim node is accepted with `nc.Unpaired = true` +rather than rejected. This matches the "no reject" observed in `PROGRESS.md` +under bead `bf-3hji`. + +**Independent corroboration in the test harness.** The e2e harness pins this +exact behavior open and documents it inline — `mothership/tests/e2e/e2e_test.go:86-94` +sets `SPAXEL_MIGRATION_WINDOW_HOURS=24` with the comment: *"spaxel-sim nodes +present no token in their hello message (only the X-Spaxel-Token header, which +the ingestion server does not read), so they are effectively tokenless … a +tokenless node is only rejected when the migration window is closed … Pin the +window open here so the harness never rejects tokenless sim nodes regardless of +the production default (bf-4iewr)."* The harness itself therefore asserts that +the production default (24h) is the only thing preventing reject, and that the +header is unread — exactly points 4–6. + +### 6. REJECT fires ONLY when the window is closed + +The reject branch (`server.go:519-528`) is reached in exactly two cases: + +**(a) `SPAXEL_MIGRATION_WINDOW_HOURS=0`.** Then +`mothership/cmd/mothership/main.go:4495` (`if cfg.MigrationWindowHours > 0`) +is false, `SetMigrationDeadline` is **never called**, and +`s.migrationDeadline` stays at its zero value (`time.Time{}`). +`server.go:515`'s `!deadline.IsZero()` is then false → reject branch. + +**(b) Uptime > 24h** (default window). `time.Now().Before(deadline)` becomes +false → reject branch. + +In both cases, because `hello.Token == ""` (point 4), `server.go:520-521` +logs `Node rejected: missing token`, then `server.go:525` +`s.sendReject(conn, "invalid_token")` and `server.go:526` `conn.Close()` +disconnect the node. (`sendReject` is defined at +`mothership/internal/ingestion/server.go:841-846`; it writes a +`{"type":"reject","reason":"invalid_token"}` frame and the sim exits non-zero +on receipt, per the plan's simulator contract.) + +## Summary statements (for the `bf-4iewr` acceptance criteria) + +1. **Validator configured? — always YES.** + `mothership/cmd/mothership/main.go:4494` wires it unconditionally. +2. **Migration window closed? — not on a default boot.** + Default 24h (`config.go:139`) opens a window that masks the missing token. +3. **Current "no reject" is a 24h-window MASK, not a real fix.** + It vanishes under `SPAXEL_MIGRATION_WINDOW_HOURS=0` or after 24h uptime. +4. **The token-supply path is currently DEAD.** + The sim sends `X-Spaxel-Token` as a header (`cmd/sim/main.go:633`, + `cmd/sim/scenario.go:318`) that the mothership never reads; `hello.Token` + is empty in the body (`cmd/sim/main.go:652-665`), so `tokenOK` is always + `false` (`server.go:513`). + +## Implications for the remaining `bf-4iewr` children + +There are three independent, non-overlapping fixes; the children should pick +exactly one and the others become moot: + +- **(A) Accept tokenless sim nodes on purpose** — leave the validator on but + guarantee the migration window is open for sim/test boots (e.g. a test-only + `SetMigrationDeadline` far in the future, or `SPAXEL_MIGRATION_WINDOW_HOURS` + sized to the test). Cheapest, but leaves the token path dead and relies on + the very mask documented here. +- **(B) Make the token path real** — have the mothership read `X-Spaxel-Token` + off the WS upgrade request in `HandleNodeWS` (`server.go:455-469`) and feed + it into validation, OR have the sim include `token` in the hello body so + `server.go:513` sees it. This makes "have the sim supply valid tokens" a + genuine option and is the correct long-term fix; it also requires the sim's + provisioned token to actually pass `ValidateToken` (HMAC over the install + secret). +- **(C) Disable the validator for the hardware-free build** — add a config/build + gate around the `SetTokenValidator` call at `main.go:4494` (e.g. an env flag + defaulting to off only in the sim/test path). Simplest for CI but disables a + real security control and must not leak into production. + +The two statements that must hold in the final state regardless of choice: +the "no reject" must no longer depend on the 24h mask, and the chosen token +contract (header vs. body vs. none) must be honored end-to-end by both the +sim and the mothership.