spaxel/docs/notes/token-reject-root-cause.md

Root Cause: tokenless-sim REJECT in the WS hello path

Tracking: bead bf-34lwt (first link of the bf-4iewr split). Date: 2026-07-07. Scope: explain exactly why a tokenless spaxel-sim node is currently NOT rejected on a default boot, and exactly when it breaks. This is the precise target the remaining bf-4iewr children build on.

Path convention. Citations below are full repo-relative paths. The bf-4iewr task body drops the mothership/ module prefix, so e.g. its cmd/sim/main.go:633 is mothership/cmd/sim/main.go:633 here, and its cmd/mothership/main.go:4494 is mothership/cmd/mothership/main.go:4494.

Two sim copies — both behave identically for this finding. There are two spaxel-sim source trees and both send the token only as a header with no token in the hello body:

• cmd/sim/main.go — the module go.work lists (use ./cmd/sim); the dev/test path. Header set at cmd/sim/main.go:313; hello body cmd/sim/main.go:348-359 (no token key). This copy has no scenario.go.
• mothership/cmd/sim/ — the copy the Dockerfile actually builds (it does COPY mothership/ ./ then go build ./cmd/sim, so ./cmd/sim resolves here). Header set at mothership/cmd/sim/main.go:633 and mothership/cmd/sim/scenario.go:318; hello bodies mothership/cmd/sim/main.go:652-665 and mothership/cmd/sim/scenario.go:327-340 (neither has a token key).

A grep '"token"' over all four hello bodies returns only the --token flag definition, never a JSON body key — so hello.Token deserializes to "" from either binary.

TL;DR

The current "no reject" is a 24-hour migration-window MASK, not a real fix. The token validator is wired unconditionally, but the sim presents its token only as an HTTP header the mothership never reads, so hello.Token is always empty and tokenOK is always false. The only thing keeping sim nodes connected is the default 24h migration window, which accepts any tokenless node as Unpaired. The moment that window closes (SPAXEL_MIGRATION_WINDOW_HOURS=0 or uptime > 24h), every sim node is sendReject-ed with invalid_token and disconnected.

Confirmed facts (with citations)

1. The validator is wired UNCONDITIONALLY

mothership/cmd/mothership/main.go:4494:

ingestSrv.SetTokenValidator(provSrv.ValidateToken)

There is no config flag and no build gate around this call. "Validator configured?" is therefore always YES at runtime. The only thing the next block gates on is the deadline, not the validator:

// mothership/cmd/mothership/main.go:4495-4499
if cfg.MigrationWindowHours > 0 {
    deadline := time.Now().Add(time.Duration(cfg.MigrationWindowHours) * time.Hour)
    ingestSrv.SetMigrationDeadline(deadline)
    ...
}

So s.tokenValidator is always non-nil; only s.migrationDeadline is conditional.

2. The validator reads hello.Token only

mothership/internal/ingestion/server.go:513:

tokenOK := hello.Token != "" && validator(hello.MAC, hello.Token)

validator is provSrv.ValidateToken, signature func(mac, token string) bool (see SetTokenValidator at mothership/internal/ingestion/server.go:321-327). Its only inputs are hello.MAC and hello.Token. hello.Token maps to the JSON field token,omitempty (mothership/internal/ingestion/message.go:22).

There is no other path that can set tokenOK = true: it requires a non-empty hello.Token that the validator accepts.

3. The sim sends its token ONLY as the X-Spaxel-Token HTTP header

• mothership/cmd/sim/main.go:633: headers.Set("X-Spaxel-Token", token)
• mothership/cmd/sim/scenario.go:318: headers.Set("X-Spaxel-Token", token)

The header is attached to the WebSocket dial request (websocket.DefaultDialer.DialContext(ctx, url, headers)).

The hello JSON body does NOT include a token field. The hello map built at mothership/cmd/sim/main.go:652-665 contains only:

type, mac, firmware_version, capabilities, chip, flash_mb,
uptime_ms, wifi_rssi, ip, pos_x, pos_y, pos_z

(identical field set in mothership/cmd/sim/scenario.go:327-340). No token key. So hello.Token always deserializes to "".

4. The mothership NEVER reads that header → the token-supply path is DEAD

A search of the entire mothership tree for the literal header name returns zero hits in any non-sim, non-test source:

$ grep -rn 'X-Spaxel-Token' mothership/   # only sim files
mothership/cmd/sim/main.go:633:        headers.Set("X-Spaxel-Token", token)
mothership/cmd/sim/scenario.go:318:    headers.Set("X-Spaxel-Token", token)

The WS upgrade handler never inspects the request either. mothership/internal/ingestion/server.go:469:

conn, err := s.upgrader.Upgrade(w, r, nil)

HandleNodeWS (mothership/internal/ingestion/server.go:455) reads nothing off r before upgrading — no r.Header.Get(...). (All Header.Get / Header.Set hits under mothership/internal/ are on the outbound webhook/notify path or in their tests, e.g. webhook, notify, notifications/ntfy.go; none touch the node WS path.)

Consequence: for every sim node, hello.Token == "", so hello.Token != "" short-circuits to false at server.go:513, making tokenOK always false regardless of what token the sim provisioned. The parent bead's "have the sim supply valid tokens" option is therefore currently NON-FUNCTIONAL — the token is supplied to a header nobody reads.

5. The only thing accepting sim nodes is the migration window

mothership/internal/ingestion/server.go:514-519:

if !tokenOK {
    if !deadline.IsZero() && time.Now().Before(deadline) {
        // accepted as Unpaired
        nc.Unpaired = true
    } else {
        ... reject ...
    }
}

MigrationWindowHours defaults to 24 (mothership/internal/config/config.go:139, cfg.MigrationWindowHours = 24). On a fresh boot with the default, mothership/cmd/mothership/main.go:4496 sets deadline = time.Now().Add(24h), so !deadline.IsZero() && Now().Before(deadline) is true → every tokenless sim node is accepted with nc.Unpaired = true rather than rejected. This matches the "no reject" observed in PROGRESS.md under bead bf-3hji.

Independent corroboration in the test harness. The e2e harness pins this exact behavior open and documents it inline — mothership/tests/e2e/e2e_test.go:86-94 sets SPAXEL_MIGRATION_WINDOW_HOURS=24 with the comment: "spaxel-sim nodes present no token in their hello message (only the X-Spaxel-Token header, which the ingestion server does not read), so they are effectively tokenless … a tokenless node is only rejected when the migration window is closed … Pin the window open here so the harness never rejects tokenless sim nodes regardless of the production default (bf-4iewr)." The harness itself therefore asserts that the production default (24h) is the only thing preventing reject, and that the header is unread — exactly points 4–6.

6. REJECT fires ONLY when the window is closed

The reject branch (server.go:519-528) is reached in exactly two cases:

(a) SPAXEL_MIGRATION_WINDOW_HOURS=0. Then mothership/cmd/mothership/main.go:4495 (if cfg.MigrationWindowHours > 0) is false, SetMigrationDeadline is never called, and s.migrationDeadline stays at its zero value (time.Time{}). server.go:515's !deadline.IsZero() is then false → reject branch.

(b) Uptime > 24h (default window). time.Now().Before(deadline) becomes false → reject branch.

In both cases, because hello.Token == "" (point 4), server.go:520-521 logs Node <MAC> rejected: missing token, then server.go:525 s.sendReject(conn, "invalid_token") and server.go:526 conn.Close() disconnect the node. (sendReject is defined at mothership/internal/ingestion/server.go:841-846; it writes a {"type":"reject","reason":"invalid_token"} frame and the sim exits non-zero on receipt, per the plan's simulator contract.)

Summary statements (for the bf-4iewr acceptance criteria)

1. Validator configured? — always YES. mothership/cmd/mothership/main.go:4494 wires it unconditionally.
2. Migration window closed? — not on a default boot. Default 24h (config.go:139) opens a window that masks the missing token.
3. Current "no reject" is a 24h-window MASK, not a real fix. It vanishes under SPAXEL_MIGRATION_WINDOW_HOURS=0 or after 24h uptime.
4. The token-supply path is currently DEAD. The sim sends X-Spaxel-Token as a header (cmd/sim/main.go:633, cmd/sim/scenario.go:318) that the mothership never reads; hello.Token is empty in the body (cmd/sim/main.go:652-665), so tokenOK is always false (server.go:513).

Implications for the remaining bf-4iewr children

There are three independent, non-overlapping fixes; the children should pick exactly one and the others become moot:

• (A) Accept tokenless sim nodes on purpose — leave the validator on but guarantee the migration window is open for sim/test boots (e.g. a test-only SetMigrationDeadline far in the future, or SPAXEL_MIGRATION_WINDOW_HOURS sized to the test). Cheapest, but leaves the token path dead and relies on the very mask documented here.
• (B) Make the token path real — have the mothership read X-Spaxel-Token off the WS upgrade request in HandleNodeWS (server.go:455-469) and feed it into validation, OR have the sim include token in the hello body so server.go:513 sees it. This makes "have the sim supply valid tokens" a genuine option and is the correct long-term fix; it also requires the sim's provisioned token to actually pass ValidateToken (HMAC over the install secret).
• (C) Disable the validator for the hardware-free build — add a config/build gate around the SetTokenValidator call at main.go:4494 (e.g. an env flag defaulting to off only in the sim/test path). Simplest for CI but disables a real security control and must not leak into production.

The two statements that must hold in the final state regardless of choice: the "no reject" must no longer depend on the 24h mask, and the chosen token contract (header vs. body vs. none) must be honored end-to-end by both the sim and the mothership.