spaxel/PROGRESS.md

# Spaxel Implementation Progress

## Phase 1 — Foundation

Goal: Bare-minimum loop from ESP32 to browser. Zero-config with passive radar and mDNS from day one.

### Status: COMPLETE

| Item | Status | Notes |
|------|--------|-------|
| ESP32 firmware skeleton | **Done** | See iteration 2 below |
| Passive radar support | **Done** | BSSID filter in csi.c |
| BLE scanning | **Done** | Core 0 concurrent with WiFi |
| Mothership WebSocket ingestion | **Done** | See iteration 1 below |
| Dashboard skeleton | **Done** | See iteration 3 below |
| Docker packaging | **Done** | See iteration 4 below |

## Phase 2 — Signal Processing & Detection

Goal: Detect presence on a single link.

### Status

| Item | Status | Notes |
|------|--------|-------|
| Phase sanitisation | **Done** | See iteration 5 below |
| Baseline system | **Done** | EMA with motion-gated updates |
| Motion detection | **Done** | deltaRMS, NBVI selection |
| Dashboard presence indicator | **Pending** | |
| CSI recording buffer | **Pending** | |
| Adaptive sensing rate | **Pending** | |

### Iteration Log

#### Iteration 5 — 2026-03-26

**Completed:** Signal processing package (phase sanitisation, baseline, motion detection)

Implemented the core signal processing pipeline in Go with:

- **Phase sanitization (`signal/phase.go`):**
  - Complex CSI computation: int8 I/Q → float64 complex → amplitude/phase
  - RSSI normalization (AGC compensation): `norm = 10^((rssi_ref - rssi)/20)`
  - Spatial phase unwrapping: correct 2π jumps between adjacent subcarriers
  - Linear regression (OLS): fit `phase = a*k + b` over data subcarriers
  - STO/CFO removal: residual phase = unwrapped - (slope × k + intercept)
  - HT20 subcarrier map: 64 total, 46 data (excluding null, guard, pilot)

- **Baseline system (`signal/baseline.go`):**
  - EMA baseline per link per subcarrier: `baseline = α×amplitude + (1-α)×baseline`
  - α = dt / (τ + dt) ≈ 0.0033 for dt=0.1s, τ=30s
  - Motion-gated updates: only update when smoothDeltaRMS < 0.05
  - Confidence scoring: 0.3 for stale baselines (>7 days), asymptotically → 1.0
  - Snapshot/restore for SQLite persistence
  - BaselineManager for multi-link coordination

- **Motion detection (`signal/features.go`):**
  - NBVI (Normalized Bandwidth Variance Index): `Var(amp) / Mean(amp)²`
  - Welford's online algorithm for numerically stable variance
  - Top-16 subcarrier selection by NBVI score
  - deltaRMS: `sqrt(mean((amp - baseline)²))` over selected subcarriers
  - Exponential smoothing: `smooth = 0.3×raw + 0.7×prev`
  - Motion threshold: smoothDeltaRMS > 0.02

- **Link processor (`signal/processor.go`):**
  - LinkProcessor: ties together phase sanitization, baseline, motion detection
  - ProcessorManager: manages per-link processors with thread-safe access
  - GetAllMotionStates(): returns motion state for all links
  - GetAllBaselines()/RestoreBaseline(): for SQLite persistence

**Constants used (from plan):**
- RSSIRefdBm = -30.0
- DefaultMotionThreshold = 0.05
- DefaultDeltaRMSThreshold = 0.02
- NBVITopCount = 16
- NBVIMinThreshold = 0.001
- DeltaRMSSmoothingAlpha = 0.3

**Tests:** 37 tests covering phase sanitization, baseline, NBVI, motion detection. All pass.

**Files created:**
```
mothership/internal/signal/
├── phase.go          — Phase sanitization algorithms
├── phase_test.go     — 15 tests for phase processing
├── baseline.go       — EMA baseline management
├── baseline_test.go  — 9 tests for baseline
├── features.go       — NBVI selection, deltaRMS, motion detection
├── features_test.go  — 13 tests for features
└── processor.go      — LinkProcessor, ProcessorManager
```

**Remaining for Phase 2:**
- Dashboard presence indicator
- CSI recording buffer
- Adaptive sensing rate

#### Iteration 4 — 2026-03-26

**Completed:** Docker packaging

Implemented container deployment with:

- **Dockerfile:** Multi-stage build for minimal production image
  - Stage 1: `golang:1.23-bookworm` builder with module caching
  - Stage 2: `distroless/static-debian12:nonroot` runtime (no shell, runs as UID 65532)
  - CGO_ENABLED=0 build (pure-Go SQLite compatible)
  - Dashboard static files copied to `/dashboard/`
  - Firmware directory as volume mount point (`/firmware/`)
  - Built-in healthcheck via wget

- **docker-compose.yml:** Production-ready orchestration
  - `network_mode: host` for mDNS multicast discovery (required for ESP32 nodes)
  - Volume mounts: `spaxel-data` for persistence, `./firmware` for OTA binaries
  - Environment variables: TZ, mDNS config, optional MQTT
  - Resource limits: 512m RAM, 2 CPUs (scalable to 1g for 16+ nodes)
  - 35s stop_grace_period for graceful shutdown
  - Ulimits: 4096/8192 file descriptors for node connections
  - Traefik labels (disabled by default, enable for TLS/proxy)

- **Supporting files:**
  - `VERSION` — 0.1.0 for image tagging
  - `.dockerignore` — Excludes docs, build artifacts, IDE files

**Key design decisions:**
- Host networking is REQUIRED for mDNS — Docker bridge blocks multicast 224.0.0.251
- distroless image minimizes attack surface (no shell, no package manager)
- Firmware binaries not bundled — users mount their own `/firmware` volume
- Traefik labels included but disabled — enable for production TLS

**Files created:**
```
Dockerfile
docker-compose.yml
VERSION
.dockerignore
```

**Phase 1 Status:** COMPLETE

All Phase 1 items implemented:
- ✅ ESP32 firmware skeleton
- ✅ Passive radar support
- ✅ BLE scanning
- ✅ Mothership WebSocket ingestion
- ✅ Dashboard skeleton
- ✅ Docker packaging

**Next:** Phase 2 — Signal Processing (baseline, deltaRMS, Fresnel zones)

#### Iteration 3 — 2026-03-26

**Completed:** Dashboard skeleton with 3D visualization

Implemented the web dashboard in vanilla JS + Three.js with:

- **Static file structure:** `dashboard/` directory with HTML/JS
  - `index.html` — Dark theme UI with status bar, node/link panels, amplitude chart
  - `js/app.js` — Main application (~350 lines)

- **3D Scene (Three.js):**
  - Ground grid (10×10m, 20 divisions)
  - OrbitControls for pan/zoom/rotate with damping
  - Axes helper for orientation
  - Responsive canvas with devicePixelRatio support
  - FPS counter in status bar

- **WebSocket connection (`/ws/dashboard`):**
  - Auto-reconnect with 3s backoff
  - JSON message handling for state, node events, link events
  - Binary CSI frame parsing (24-byte header + I/Q payload)
  - Connection status indicator (green/red dot)

- **Node/Link panels:**
  - Live list of connected nodes with MAC, firmware, chip
  - Active links with node:peer MAC pairs
  - Click-to-select for amplitude chart display

- **Amplitude chart (Canvas 2D):**
  - 64-bar visualization of subcarrier amplitudes
  - Real-time update from selected link's CSI frames
  - Color gradient based on amplitude intensity
  - Channel and RSSI display overlay

- **Dashboard package (Go):**
  - `internal/dashboard/hub.go` — Hub for client management and broadcasting
  - `internal/dashboard/server.go` — WebSocket handler with ping/pong keepalive
  - `internal/dashboard/hub_test.go` — 5 tests for hub operations
  - IngestionState interface for querying node/link state
  - CSIBroadcaster interface implementation

- **Main.go updates:**
  - Static file serving with SPA fallback
  - Dashboard WebSocket endpoint at `/ws/dashboard`
  - Auto-discovery of dashboard directory
  - Wiring between ingestion and dashboard packages

**Dependencies used (frontend):**
- Three.js r128 (CDN)
- OrbitControls (CDN)

**Tests:** 5 new tests for dashboard hub. All 27 tests pass.

**Files created:**
```
dashboard/
├── index.html
└── js/
    └── app.js

mothership/internal/dashboard/
├── hub.go
├── hub_test.go
└── server.go
```

**Remaining for Phase 1:**
- Docker packaging

#### Iteration 2 — 2026-03-26

**Completed:** ESP32-S3 firmware skeleton

Implemented the full ESP32 firmware in ESP-IDF C with:

- **Project structure:** Standard ESP-IDF layout with `firmware/` root
  - Top-level `CMakeLists.txt`, `sdkconfig.defaults`, `partitions.csv` (factory + OTA slots)
  - `main/` component with 5 source modules

- **State machine (main.c):** 7-state node lifecycle
  - BOOT → WIFI_CONNECTING → MOTHERSHIP_DISCOVERY → CONNECTED
  - Degraded states: WIFI_LOST, MOTHERSHIP_UNAVAILABLE, CAPTIVE_PORTAL
  - Exponential backoff on WiFi failures (1s → 30s max)
  - 10-failure threshold before captive portal

- **WiFi module (wifi.c/h):**
  - STA connection with exponential backoff
  - mDNS discovery for `_spaxel._tcp.local` with fallback to cached IP
  - Captive portal AP mode (`spaxel-XXXX`) with HTTP config page
  - Event-driven connection state via FreeRTOS event group

- **WebSocket client (websocket.c/h):**
  - Bidirectional communication on single connection
  - Binary CSI frame transmission (24-byte header + I/Q payload)
  - JSON message handling: hello, health, ble, ota_status upstream
  - Downstream command parsing: role, config, ota, reboot, identify, reject
  - OTA download task with progress reporting and automatic reboot

- **CSI capture (csi.c/h):**
  - WiFi promiscuous mode with CSI callback
  - Queue-based processing (32-frame buffer)
  - Passive mode BSSID filtering for radar
  - On-device amplitude variance tracking for motion hints (Welford's algorithm)
  - TX task for active probing

- **BLE scanner (ble.c/h):**
  - Passive BLE scanning on Core 0 (concurrent with WiFi)
  - Device cache (60 entries) with name and manufacturer data parsing
  - 5-second reporting interval via WebSocket
  - GAP event handling for advertisement processing

- **NVS persistence:** Full schema with 15 keys
  - WiFi credentials, node ID/token, mothership config
  - Role/rate persistence for degraded mode operation
  - Schema versioning for migration support

**Files created:**
```
firmware/
├── CMakeLists.txt
├── sdkconfig.defaults
├── partitions.csv
└── main/
    ├── CMakeLists.txt
    ├── spaxel.h
    ├── main.c
    ├── wifi.h / wifi.c
    ├── websocket.h / websocket.c
    ├── csi.h / csi.c
    └── ble.h / ble.c
```

**Remaining for Phase 1:**
- Dashboard skeleton (HTML/JS + Three.js)
- Docker packaging

#### Iteration 1 — 2026-03-26

**Completed:** Mothership WebSocket ingestion server

Implemented the core ingestion server in Go with:

- **Module structure:** `mothership/` with `cmd/mothership/` entrypoint and `internal/ingestion/` package
- **WebSocket endpoint:** `/ws/node` accepts bidirectional connections from ESP32 nodes
- **Binary frame parsing:** 24-byte header + variable payload, per spec in plan.md
  - Validation: min/max length, payload size match, channel validity (1-14), subcarrier limit (128)
  - Malformed frame tracking with warn/close thresholds (100/1000 per minute)
- **JSON message handling:** Parses hello, health, ble, motion_hint, ota_status
- **Per-link ring buffers:** 256-sample circular buffers keyed by `nodeMAC:peerMAC`
- **Connection lifecycle:** Node registration via hello, ping/pong keepalive (30s/60s), graceful shutdown
- **mDNS advertisement:** `_spaxel._tcp.local` via github.com/hashicorp/mdns
- **Role/config push:** Sends initial `rx` role and 20 Hz config on connect
- **Health endpoint:** `GET /healthz` returns `{"status":"ok","version":"..."}`

**Dependencies used:**
- `github.com/go-chi/chi` — HTTP routing
- `github.com/gorilla/websocket` — WebSocket server
- `github.com/hashicorp/mdns` — mDNS advertisement

**Tests:** 22 tests covering frame parsing, JSON messages, and ring buffer operations. All pass.

**Files created:**
```
mothership/
├── cmd/mothership/main.go
├── go.mod
├── go.sum
└── internal/ingestion/
    ├── frame.go
    ├── frame_test.go
    ├── message.go
    ├── message_test.go
    ├── ring.go
    ├── ring_test.go
    └── server.go
```