# Root Cause: tokenless-sim REJECT in the WS hello path **Tracking:** bead `bf-34lwt` (first link of the `bf-4iewr` split). **Date:** 2026-07-07. **Scope:** explain exactly *why* a tokenless `spaxel-sim` node is currently NOT rejected on a default boot, and exactly *when* it breaks. This is the precise target the remaining `bf-4iewr` children build on. > **Path convention.** Citations below are full repo-relative paths. The > `bf-4iewr` task body drops the `mothership/` module prefix, so e.g. its > `cmd/sim/main.go:633` is `mothership/cmd/sim/main.go:633` here, and its > `cmd/mothership/main.go:4494` is `mothership/cmd/mothership/main.go:4494`. > > **Two sim copies — both behave identically for this finding.** There are two > `spaxel-sim` source trees and both send the token only as a header with no > `token` in the hello body: > - `cmd/sim/main.go` — the module `go.work` lists (`use ./cmd/sim`); the dev/test > path. Header set at `cmd/sim/main.go:313`; hello body `cmd/sim/main.go:348-359` > (no `token` key). This copy has no `scenario.go`. > - `mothership/cmd/sim/` — the copy the **Dockerfile actually builds** (it does > `COPY mothership/ ./` then `go build ./cmd/sim`, so `./cmd/sim` resolves here). > Header set at `mothership/cmd/sim/main.go:633` and > `mothership/cmd/sim/scenario.go:318`; hello bodies > `mothership/cmd/sim/main.go:652-665` and `mothership/cmd/sim/scenario.go:327-340` > (neither has a `token` key). > > A `grep '"token"'` over all four hello bodies returns only the `--token` flag > definition, never a JSON body key — so `hello.Token` deserializes to `""` from > either binary. ## TL;DR The current "no reject" is a **24-hour migration-window MASK, not a real fix.** The token validator is wired unconditionally, but the sim presents its token only as an HTTP header the mothership never reads, so `hello.Token` is always empty and `tokenOK` is always `false`. The only thing keeping sim nodes connected is the default 24h migration window, which accepts any tokenless node as `Unpaired`. The moment that window closes (`SPAXEL_MIGRATION_WINDOW_HOURS=0` or uptime > 24h), every sim node is `sendReject`-ed with `invalid_token` and disconnected. ## Confirmed facts (with citations) ### 1. The validator is wired UNCONDITIONALLY `mothership/cmd/mothership/main.go:4494`: ```go ingestSrv.SetTokenValidator(provSrv.ValidateToken) ``` There is **no config flag and no build gate** around this call. "Validator configured?" is therefore **always YES** at runtime. The only thing the next block gates on is the *deadline*, not the validator: ```go // mothership/cmd/mothership/main.go:4495-4499 if cfg.MigrationWindowHours > 0 { deadline := time.Now().Add(time.Duration(cfg.MigrationWindowHours) * time.Hour) ingestSrv.SetMigrationDeadline(deadline) ... } ``` So `s.tokenValidator` is always non-nil; only `s.migrationDeadline` is conditional. ### 2. The validator reads `hello.Token` only `mothership/internal/ingestion/server.go:513`: ```go tokenOK := hello.Token != "" && validator(hello.MAC, hello.Token) ``` `validator` is `provSrv.ValidateToken`, signature `func(mac, token string) bool` (see `SetTokenValidator` at `mothership/internal/ingestion/server.go:321-327`). Its only inputs are `hello.MAC` and `hello.Token`. `hello.Token` maps to the JSON field `token,omitempty` (`mothership/internal/ingestion/message.go:22`). There is no other path that can set `tokenOK = true`: it requires a non-empty `hello.Token` that the validator accepts. ### 3. The sim sends its token ONLY as the `X-Spaxel-Token` HTTP header - `mothership/cmd/sim/main.go:633`: `headers.Set("X-Spaxel-Token", token)` - `mothership/cmd/sim/scenario.go:318`: `headers.Set("X-Spaxel-Token", token)` The header is attached to the WebSocket *dial* request (`websocket.DefaultDialer.DialContext(ctx, url, headers)`). The hello **JSON body** does **NOT** include a `token` field. The hello map built at `mothership/cmd/sim/main.go:652-665` contains only: ``` type, mac, firmware_version, capabilities, chip, flash_mb, uptime_ms, wifi_rssi, ip, pos_x, pos_y, pos_z ``` (identical field set in `mothership/cmd/sim/scenario.go:327-340`). No `token` key. So `hello.Token` always deserializes to `""`. ### 4. The mothership NEVER reads that header → the token-supply path is DEAD A search of the entire mothership tree for the literal header name returns **zero** hits in any non-sim, non-test source: ``` $ grep -rn 'X-Spaxel-Token' mothership/ # only sim files mothership/cmd/sim/main.go:633: headers.Set("X-Spaxel-Token", token) mothership/cmd/sim/scenario.go:318: headers.Set("X-Spaxel-Token", token) ``` The WS upgrade handler never inspects the request either. `mothership/internal/ingestion/server.go:469`: ```go conn, err := s.upgrader.Upgrade(w, r, nil) ``` `HandleNodeWS` (`mothership/internal/ingestion/server.go:455`) reads nothing off `r` before upgrading — no `r.Header.Get(...)`. (All `Header.Get` / `Header.Set` hits under `mothership/internal/` are on the *outbound* webhook/notify path or in their tests, e.g. `webhook`, `notify`, `notifications/ntfy.go`; none touch the node WS path.) **Consequence:** for every sim node, `hello.Token == ""`, so `hello.Token != ""` short-circuits to `false` at `server.go:513`, making `tokenOK` always `false` regardless of what token the sim provisioned. The parent bead's "have the sim supply valid tokens" option is therefore **currently NON-FUNCTIONAL** — the token is supplied to a header nobody reads. ### 5. The only thing accepting sim nodes is the migration window `mothership/internal/ingestion/server.go:514-519`: ```go if !tokenOK { if !deadline.IsZero() && time.Now().Before(deadline) { // accepted as Unpaired nc.Unpaired = true } else { ... reject ... } } ``` `MigrationWindowHours` **defaults to 24** (`mothership/internal/config/config.go:139`, `cfg.MigrationWindowHours = 24`). On a fresh boot with the default, `mothership/cmd/mothership/main.go:4496` sets `deadline = time.Now().Add(24h)`, so `!deadline.IsZero() && Now().Before(deadline)` is true → every tokenless sim node is accepted with `nc.Unpaired = true` rather than rejected. This matches the "no reject" observed in `PROGRESS.md` under bead `bf-3hji`. **Independent corroboration in the test harness.** The e2e harness pins this exact behavior open and documents it inline — `mothership/tests/e2e/e2e_test.go:86-94` sets `SPAXEL_MIGRATION_WINDOW_HOURS=24` with the comment: *"spaxel-sim nodes present no token in their hello message (only the X-Spaxel-Token header, which the ingestion server does not read), so they are effectively tokenless … a tokenless node is only rejected when the migration window is closed … Pin the window open here so the harness never rejects tokenless sim nodes regardless of the production default (bf-4iewr)."* The harness itself therefore asserts that the production default (24h) is the only thing preventing reject, and that the header is unread — exactly points 4–6. ### 6. REJECT fires ONLY when the window is closed The reject branch (`server.go:519-528`) is reached in exactly two cases: **(a) `SPAXEL_MIGRATION_WINDOW_HOURS=0`.** Then `mothership/cmd/mothership/main.go:4495` (`if cfg.MigrationWindowHours > 0`) is false, `SetMigrationDeadline` is **never called**, and `s.migrationDeadline` stays at its zero value (`time.Time{}`). `server.go:515`'s `!deadline.IsZero()` is then false → reject branch. **(b) Uptime > 24h** (default window). `time.Now().Before(deadline)` becomes false → reject branch. In both cases, because `hello.Token == ""` (point 4), `server.go:520-521` logs `Node rejected: missing token`, then `server.go:525` `s.sendReject(conn, "invalid_token")` and `server.go:526` `conn.Close()` disconnect the node. (`sendReject` is defined at `mothership/internal/ingestion/server.go:841-846`; it writes a `{"type":"reject","reason":"invalid_token"}` frame and the sim exits non-zero on receipt, per the plan's simulator contract.) ## Summary statements (for the `bf-4iewr` acceptance criteria) 1. **Validator configured? — always YES.** `mothership/cmd/mothership/main.go:4494` wires it unconditionally. 2. **Migration window closed? — not on a default boot.** Default 24h (`config.go:139`) opens a window that masks the missing token. 3. **Current "no reject" is a 24h-window MASK, not a real fix.** It vanishes under `SPAXEL_MIGRATION_WINDOW_HOURS=0` or after 24h uptime. 4. **The token-supply path is currently DEAD.** The sim sends `X-Spaxel-Token` as a header (`cmd/sim/main.go:633`, `cmd/sim/scenario.go:318`) that the mothership never reads; `hello.Token` is empty in the body (`cmd/sim/main.go:652-665`), so `tokenOK` is always `false` (`server.go:513`). ## Implications for the remaining `bf-4iewr` children There are three independent, non-overlapping fixes; the children should pick exactly one and the others become moot: - **(A) Accept tokenless sim nodes on purpose** — leave the validator on but guarantee the migration window is open for sim/test boots (e.g. a test-only `SetMigrationDeadline` far in the future, or `SPAXEL_MIGRATION_WINDOW_HOURS` sized to the test). Cheapest, but leaves the token path dead and relies on the very mask documented here. - **(B) Make the token path real** — have the mothership read `X-Spaxel-Token` off the WS upgrade request in `HandleNodeWS` (`server.go:455-469`) and feed it into validation, OR have the sim include `token` in the hello body so `server.go:513` sees it. This makes "have the sim supply valid tokens" a genuine option and is the correct long-term fix; it also requires the sim's provisioned token to actually pass `ValidateToken` (HMAC over the install secret). - **(C) Disable the validator for the hardware-free build** — add a config/build gate around the `SetTokenValidator` call at `main.go:4494` (e.g. an env flag defaulting to off only in the sim/test path). Simplest for CI but disables a real security control and must not leak into production. The two statements that must hold in the final state regardless of choice: the "no reject" must no longer depend on the 24h mask, and the chosen token contract (header vs. body vs. none) must be honored end-to-end by both the sim and the mothership.