docs(bf-34lwt): root-cause tokenless-sim REJECT in WS hello path
Definitive root-cause for the bf-4iewr split's first child. Confirms against current source (with file:line citations) that: - The token validator is wired UNCONDITIONALLY (main.go:4494) - It reads hello.Token only (server.go:513); the sim never populates it - The sim sends its token only as the X-Spaxel-Token header (cmd/sim main.go:633/scenario.go:318), and the mothership never reads that header (HandleNodeWS upgrades unconditionally at server.go:469) -> the token-supply path is currently DEAD - The default 24h migration window (config.go:139) is the only thing accepting sim nodes -> current 'no reject' is a window MASK, not a fix - REJECT fires only when the window closes (SPAXEL_MIGRATION_WINDOW_HOURS=0 or uptime > 24h) Corroborated by the e2e harness comment (tests/e2e/e2e_test.go:86-94) which pins the window open and documents exactly this. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jedarden
parent
8014ad1768
commit
5c6a6924c1
1 changed files with 227 additions and 0 deletions
227
docs/notes/token-reject-root-cause.md
Normal file
227
docs/notes/token-reject-root-cause.md
Normal file
|
|
@ -0,0 +1,227 @@
|
|||
# Root Cause: tokenless-sim REJECT in the WS hello path
|
||||
|
||||
**Tracking:** bead `bf-34lwt` (first link of the `bf-4iewr` split).
|
||||
**Date:** 2026-07-07.
|
||||
**Scope:** explain exactly *why* a tokenless `spaxel-sim` node is currently
|
||||
NOT rejected on a default boot, and exactly *when* it breaks. This is the
|
||||
precise target the remaining `bf-4iewr` children build on.
|
||||
|
||||
> **Path convention.** Citations below are full repo-relative paths. The
|
||||
> `bf-4iewr` task body drops the `mothership/` module prefix, so e.g. its
|
||||
> `cmd/sim/main.go:633` is `mothership/cmd/sim/main.go:633` here, and its
|
||||
> `cmd/mothership/main.go:4494` is `mothership/cmd/mothership/main.go:4494`.
|
||||
>
|
||||
> **Two sim copies — both behave identically for this finding.** There are two
|
||||
> `spaxel-sim` source trees and both send the token only as a header with no
|
||||
> `token` in the hello body:
|
||||
> - `cmd/sim/main.go` — the module `go.work` lists (`use ./cmd/sim`); the dev/test
|
||||
> path. Header set at `cmd/sim/main.go:313`; hello body `cmd/sim/main.go:348-359`
|
||||
> (no `token` key). This copy has no `scenario.go`.
|
||||
> - `mothership/cmd/sim/` — the copy the **Dockerfile actually builds** (it does
|
||||
> `COPY mothership/ ./` then `go build ./cmd/sim`, so `./cmd/sim` resolves here).
|
||||
> Header set at `mothership/cmd/sim/main.go:633` and
|
||||
> `mothership/cmd/sim/scenario.go:318`; hello bodies
|
||||
> `mothership/cmd/sim/main.go:652-665` and `mothership/cmd/sim/scenario.go:327-340`
|
||||
> (neither has a `token` key).
|
||||
>
|
||||
> A `grep '"token"'` over all four hello bodies returns only the `--token` flag
|
||||
> definition, never a JSON body key — so `hello.Token` deserializes to `""` from
|
||||
> either binary.
|
||||
|
||||
## TL;DR
|
||||
|
||||
The current "no reject" is a **24-hour migration-window MASK, not a real
|
||||
fix.** The token validator is wired unconditionally, but the sim presents its
|
||||
token only as an HTTP header the mothership never reads, so `hello.Token` is
|
||||
always empty and `tokenOK` is always `false`. The only thing keeping sim nodes
|
||||
connected is the default 24h migration window, which accepts any tokenless
|
||||
node as `Unpaired`. The moment that window closes (`SPAXEL_MIGRATION_WINDOW_HOURS=0`
|
||||
or uptime > 24h), every sim node is `sendReject`-ed with `invalid_token` and
|
||||
disconnected.
|
||||
|
||||
## Confirmed facts (with citations)
|
||||
|
||||
### 1. The validator is wired UNCONDITIONALLY
|
||||
|
||||
`mothership/cmd/mothership/main.go:4494`:
|
||||
|
||||
```go
|
||||
ingestSrv.SetTokenValidator(provSrv.ValidateToken)
|
||||
```
|
||||
|
||||
There is **no config flag and no build gate** around this call. "Validator
|
||||
configured?" is therefore **always YES** at runtime. The only thing the next
|
||||
block gates on is the *deadline*, not the validator:
|
||||
|
||||
```go
|
||||
// mothership/cmd/mothership/main.go:4495-4499
|
||||
if cfg.MigrationWindowHours > 0 {
|
||||
deadline := time.Now().Add(time.Duration(cfg.MigrationWindowHours) * time.Hour)
|
||||
ingestSrv.SetMigrationDeadline(deadline)
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
So `s.tokenValidator` is always non-nil; only `s.migrationDeadline` is
|
||||
conditional.
|
||||
|
||||
### 2. The validator reads `hello.Token` only
|
||||
|
||||
`mothership/internal/ingestion/server.go:513`:
|
||||
|
||||
```go
|
||||
tokenOK := hello.Token != "" && validator(hello.MAC, hello.Token)
|
||||
```
|
||||
|
||||
`validator` is `provSrv.ValidateToken`, signature `func(mac, token string) bool`
|
||||
(see `SetTokenValidator` at `mothership/internal/ingestion/server.go:321-327`).
|
||||
Its only inputs are `hello.MAC` and `hello.Token`. `hello.Token` maps to the
|
||||
JSON field `token,omitempty` (`mothership/internal/ingestion/message.go:22`).
|
||||
|
||||
There is no other path that can set `tokenOK = true`: it requires a non-empty
|
||||
`hello.Token` that the validator accepts.
|
||||
|
||||
### 3. The sim sends its token ONLY as the `X-Spaxel-Token` HTTP header
|
||||
|
||||
- `mothership/cmd/sim/main.go:633`: `headers.Set("X-Spaxel-Token", token)`
|
||||
- `mothership/cmd/sim/scenario.go:318`: `headers.Set("X-Spaxel-Token", token)`
|
||||
|
||||
The header is attached to the WebSocket *dial* request
|
||||
(`websocket.DefaultDialer.DialContext(ctx, url, headers)`).
|
||||
|
||||
The hello **JSON body** does **NOT** include a `token` field. The hello map
|
||||
built at `mothership/cmd/sim/main.go:652-665` contains only:
|
||||
|
||||
```
|
||||
type, mac, firmware_version, capabilities, chip, flash_mb,
|
||||
uptime_ms, wifi_rssi, ip, pos_x, pos_y, pos_z
|
||||
```
|
||||
|
||||
(identical field set in `mothership/cmd/sim/scenario.go:327-340`). No `token`
|
||||
key. So `hello.Token` always deserializes to `""`.
|
||||
|
||||
### 4. The mothership NEVER reads that header → the token-supply path is DEAD
|
||||
|
||||
A search of the entire mothership tree for the literal header name returns
|
||||
**zero** hits in any non-sim, non-test source:
|
||||
|
||||
```
|
||||
$ grep -rn 'X-Spaxel-Token' mothership/ # only sim files
|
||||
mothership/cmd/sim/main.go:633: headers.Set("X-Spaxel-Token", token)
|
||||
mothership/cmd/sim/scenario.go:318: headers.Set("X-Spaxel-Token", token)
|
||||
```
|
||||
|
||||
The WS upgrade handler never inspects the request either.
|
||||
`mothership/internal/ingestion/server.go:469`:
|
||||
|
||||
```go
|
||||
conn, err := s.upgrader.Upgrade(w, r, nil)
|
||||
```
|
||||
|
||||
`HandleNodeWS` (`mothership/internal/ingestion/server.go:455`) reads nothing
|
||||
off `r` before upgrading — no `r.Header.Get(...)`. (All `Header.Get` / `Header.Set`
|
||||
hits under `mothership/internal/` are on the *outbound* webhook/notify path or
|
||||
in their tests, e.g. `webhook`, `notify`, `notifications/ntfy.go`; none touch
|
||||
the node WS path.)
|
||||
|
||||
**Consequence:** for every sim node, `hello.Token == ""`, so
|
||||
`hello.Token != ""` short-circuits to `false` at `server.go:513`, making
|
||||
`tokenOK` always `false` regardless of what token the sim provisioned. The
|
||||
parent bead's "have the sim supply valid tokens" option is therefore
|
||||
**currently NON-FUNCTIONAL** — the token is supplied to a header nobody reads.
|
||||
|
||||
### 5. The only thing accepting sim nodes is the migration window
|
||||
|
||||
`mothership/internal/ingestion/server.go:514-519`:
|
||||
|
||||
```go
|
||||
if !tokenOK {
|
||||
if !deadline.IsZero() && time.Now().Before(deadline) {
|
||||
// accepted as Unpaired
|
||||
nc.Unpaired = true
|
||||
} else {
|
||||
... reject ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`MigrationWindowHours` **defaults to 24** (`mothership/internal/config/config.go:139`,
|
||||
`cfg.MigrationWindowHours = 24`). On a fresh boot with the default,
|
||||
`mothership/cmd/mothership/main.go:4496` sets
|
||||
`deadline = time.Now().Add(24h)`, so `!deadline.IsZero() && Now().Before(deadline)`
|
||||
is true → every tokenless sim node is accepted with `nc.Unpaired = true`
|
||||
rather than rejected. This matches the "no reject" observed in `PROGRESS.md`
|
||||
under bead `bf-3hji`.
|
||||
|
||||
**Independent corroboration in the test harness.** The e2e harness pins this
|
||||
exact behavior open and documents it inline — `mothership/tests/e2e/e2e_test.go:86-94`
|
||||
sets `SPAXEL_MIGRATION_WINDOW_HOURS=24` with the comment: *"spaxel-sim nodes
|
||||
present no token in their hello message (only the X-Spaxel-Token header, which
|
||||
the ingestion server does not read), so they are effectively tokenless … a
|
||||
tokenless node is only rejected when the migration window is closed … Pin the
|
||||
window open here so the harness never rejects tokenless sim nodes regardless of
|
||||
the production default (bf-4iewr)."* The harness itself therefore asserts that
|
||||
the production default (24h) is the only thing preventing reject, and that the
|
||||
header is unread — exactly points 4–6.
|
||||
|
||||
### 6. REJECT fires ONLY when the window is closed
|
||||
|
||||
The reject branch (`server.go:519-528`) is reached in exactly two cases:
|
||||
|
||||
**(a) `SPAXEL_MIGRATION_WINDOW_HOURS=0`.** Then
|
||||
`mothership/cmd/mothership/main.go:4495` (`if cfg.MigrationWindowHours > 0`)
|
||||
is false, `SetMigrationDeadline` is **never called**, and
|
||||
`s.migrationDeadline` stays at its zero value (`time.Time{}`).
|
||||
`server.go:515`'s `!deadline.IsZero()` is then false → reject branch.
|
||||
|
||||
**(b) Uptime > 24h** (default window). `time.Now().Before(deadline)` becomes
|
||||
false → reject branch.
|
||||
|
||||
In both cases, because `hello.Token == ""` (point 4), `server.go:520-521`
|
||||
logs `Node <MAC> rejected: missing token`, then `server.go:525`
|
||||
`s.sendReject(conn, "invalid_token")` and `server.go:526` `conn.Close()`
|
||||
disconnect the node. (`sendReject` is defined at
|
||||
`mothership/internal/ingestion/server.go:841-846`; it writes a
|
||||
`{"type":"reject","reason":"invalid_token"}` frame and the sim exits non-zero
|
||||
on receipt, per the plan's simulator contract.)
|
||||
|
||||
## Summary statements (for the `bf-4iewr` acceptance criteria)
|
||||
|
||||
1. **Validator configured? — always YES.**
|
||||
`mothership/cmd/mothership/main.go:4494` wires it unconditionally.
|
||||
2. **Migration window closed? — not on a default boot.**
|
||||
Default 24h (`config.go:139`) opens a window that masks the missing token.
|
||||
3. **Current "no reject" is a 24h-window MASK, not a real fix.**
|
||||
It vanishes under `SPAXEL_MIGRATION_WINDOW_HOURS=0` or after 24h uptime.
|
||||
4. **The token-supply path is currently DEAD.**
|
||||
The sim sends `X-Spaxel-Token` as a header (`cmd/sim/main.go:633`,
|
||||
`cmd/sim/scenario.go:318`) that the mothership never reads; `hello.Token`
|
||||
is empty in the body (`cmd/sim/main.go:652-665`), so `tokenOK` is always
|
||||
`false` (`server.go:513`).
|
||||
|
||||
## Implications for the remaining `bf-4iewr` children
|
||||
|
||||
There are three independent, non-overlapping fixes; the children should pick
|
||||
exactly one and the others become moot:
|
||||
|
||||
- **(A) Accept tokenless sim nodes on purpose** — leave the validator on but
|
||||
guarantee the migration window is open for sim/test boots (e.g. a test-only
|
||||
`SetMigrationDeadline` far in the future, or `SPAXEL_MIGRATION_WINDOW_HOURS`
|
||||
sized to the test). Cheapest, but leaves the token path dead and relies on
|
||||
the very mask documented here.
|
||||
- **(B) Make the token path real** — have the mothership read `X-Spaxel-Token`
|
||||
off the WS upgrade request in `HandleNodeWS` (`server.go:455-469`) and feed
|
||||
it into validation, OR have the sim include `token` in the hello body so
|
||||
`server.go:513` sees it. This makes "have the sim supply valid tokens" a
|
||||
genuine option and is the correct long-term fix; it also requires the sim's
|
||||
provisioned token to actually pass `ValidateToken` (HMAC over the install
|
||||
secret).
|
||||
- **(C) Disable the validator for the hardware-free build** — add a config/build
|
||||
gate around the `SetTokenValidator` call at `main.go:4494` (e.g. an env flag
|
||||
defaulting to off only in the sim/test path). Simplest for CI but disables a
|
||||
real security control and must not leak into production.
|
||||
|
||||
The two statements that must hold in the final state regardless of choice:
|
||||
the "no reject" must no longer depend on the 24h mask, and the chosen token
|
||||
contract (header vs. body vs. none) must be honored end-to-end by both the
|
||||
sim and the mothership.
|
||||
Loading…
Add table
Reference in a new issue