docs: refocus FABRIC on worker visualization via logging/telemetry
- Shift focus from bead tracking to NEEDLE worker visibility - Emphasize TUI and HTML output formats as primary deliverables - Add CLI interface examples (fabric tui, fabric html, fabric logs) - Define worker event data model and log format contract - Update README to reflect worker-centric purpose Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jeda
parent
727661e0f0
commit
f164e7e253
2 changed files with 187 additions and 111 deletions
42
README.md
42
README.md
|
|
@ -2,32 +2,40 @@
|
|||
|
||||
**Flow Analysis & Bead Reporting Interface Console**
|
||||
|
||||
A visualization and dashboard system for monitoring [NEEDLE](../claude-config)'s bead orchestration output.
|
||||
A visualization system for surfacing NEEDLE worker activity through TUI and HTML dashboards.
|
||||
|
||||
## Purpose
|
||||
|
||||
FABRIC provides real-time and historical insights into bead processing workflows orchestrated by NEEDLE. It transforms raw orchestration data into actionable visualizations, enabling:
|
||||
FABRIC consumes logging and telemetry output from NEEDLE workers, transforming raw execution data into reviewable visualizations:
|
||||
|
||||
- **Flow Analysis**: Track bead lifecycle from creation through completion
|
||||
- **Bead Reporting**: Aggregate metrics on processing times, success rates, and bottlenecks
|
||||
- **Interface Console**: Interactive dashboard for monitoring active and historical workflows
|
||||
- **Flow Analysis**: Visualize worker execution timelines and patterns
|
||||
- **Bead Reporting**: Surface what workers are doing and how they're performing
|
||||
- **Interface Console**: Both TUI (terminal) and HTML dashboards for review
|
||||
|
||||
## Architecture
|
||||
## Output Formats
|
||||
|
||||
FABRIC integrates with NEEDLE's output streams to create a comprehensive view of:
|
||||
- Bead state transitions
|
||||
- Worker assignment and execution patterns
|
||||
- Dependency graphs and blocking relationships
|
||||
- Processing throughput and latency metrics
|
||||
### TUI Dashboard
|
||||
Real-time terminal interface showing:
|
||||
- Active worker status grid
|
||||
- Live log streaming with filtering
|
||||
- Worker detail views and session history
|
||||
- Keyboard-driven navigation
|
||||
|
||||
### HTML Reports
|
||||
Static and interactive browser-based views:
|
||||
- Session timeline visualizations (Gantt-style)
|
||||
- Metrics charts (API calls, tokens, duration)
|
||||
- Searchable log explorer
|
||||
- Shareable, self-contained reports
|
||||
|
||||
## Relationship to NEEDLE
|
||||
|
||||
While NEEDLE orchestrates bead processing, FABRIC answers questions like:
|
||||
- Which beads are currently blocked and why?
|
||||
- What's the average time-to-completion for different bead types?
|
||||
- How many beads are in each state (pending, active, completed)?
|
||||
- Which workers are processing which beads?
|
||||
- Where are the bottlenecks in the workflow?
|
||||
NEEDLE orchestrates workers; FABRIC surfaces their activity:
|
||||
- What is each worker currently doing?
|
||||
- How long are tasks taking?
|
||||
- What errors are occurring?
|
||||
- What's the API/token usage?
|
||||
- What does the execution timeline look like?
|
||||
|
||||
## Status
|
||||
|
||||
|
|
|
|||
256
docs/plan.md
256
docs/plan.md
|
|
@ -4,146 +4,214 @@
|
|||
|
||||
## Overview
|
||||
|
||||
FABRIC will provide visualization and analytics for NEEDLE's bead orchestration system. This document outlines the implementation roadmap.
|
||||
FABRIC provides visualization of NEEDLE worker activity by consuming logging and telemetry output. It generates both TUI (terminal) and HTML visualizations for reviewing worker execution patterns, performance, and output.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Real-time Monitoring**: Live dashboard showing active bead processing
|
||||
2. **Historical Analysis**: Query and visualize past workflow executions
|
||||
3. **Performance Metrics**: Identify bottlenecks and optimization opportunities
|
||||
4. **Debugging Support**: Trace individual bead lifecycles and dependencies
|
||||
1. **Worker Visibility**: Surface what NEEDLE workers are doing in real-time
|
||||
2. **Log Aggregation**: Collect and present worker logging output in digestible formats
|
||||
3. **Telemetry Analysis**: Visualize performance metrics, execution timelines, and resource usage
|
||||
4. **Dual Output**: Generate both TUI dashboards for terminal users and HTML reports for browser review
|
||||
|
||||
## Data Sources
|
||||
|
||||
FABRIC will consume data from:
|
||||
FABRIC consumes NEEDLE's logging and telemetry output:
|
||||
|
||||
### Primary Sources
|
||||
- **Bead JSONL files**: Read `.beads/*.jsonl` for current state
|
||||
- **Worker logs**: Parse execution logs for timing and error data
|
||||
- **NEEDLE orchestration events**: Real-time event stream (if available)
|
||||
- **Worker stdout/stderr**: Captured execution output
|
||||
- **Structured logs**: JSON-formatted log events from workers
|
||||
- **Telemetry streams**: Timing, resource usage, API call metrics
|
||||
- **Session transcripts**: Worker conversation/execution history
|
||||
|
||||
### Data Model
|
||||
```typescript
|
||||
interface BeadRecord {
|
||||
id: string;
|
||||
type: string;
|
||||
status: 'pending' | 'active' | 'completed' | 'failed' | 'blocked';
|
||||
priority: string;
|
||||
created_at: timestamp;
|
||||
started_at?: timestamp;
|
||||
completed_at?: timestamp;
|
||||
assigned_worker?: string;
|
||||
dependencies?: string[];
|
||||
blocking?: string[];
|
||||
interface WorkerEvent {
|
||||
worker_id: string;
|
||||
session_id: string;
|
||||
timestamp: number;
|
||||
event_type: 'log' | 'metric' | 'state_change' | 'tool_call' | 'error';
|
||||
level?: 'debug' | 'info' | 'warn' | 'error';
|
||||
message?: string;
|
||||
metadata?: Record<string, any>;
|
||||
}
|
||||
|
||||
interface WorkerSession {
|
||||
worker_id: string;
|
||||
session_id: string;
|
||||
started_at: number;
|
||||
ended_at?: number;
|
||||
status: 'running' | 'completed' | 'failed' | 'idle';
|
||||
task_description?: string;
|
||||
events: WorkerEvent[];
|
||||
metrics: WorkerMetrics;
|
||||
}
|
||||
|
||||
interface WorkerMetrics {
|
||||
api_calls: number;
|
||||
tool_invocations: number;
|
||||
tokens_in: number;
|
||||
tokens_out: number;
|
||||
duration_ms: number;
|
||||
errors: number;
|
||||
}
|
||||
```
|
||||
|
||||
## Architecture Components
|
||||
|
||||
### 1. Data Ingestion Layer
|
||||
- **Watcher**: Monitor `.beads/*.jsonl` for changes
|
||||
- **Parser**: Extract and normalize bead records
|
||||
- **Event Stream**: Convert file changes to real-time events
|
||||
### 1. Log Collector
|
||||
- **Stream Reader**: Consume NEEDLE worker output streams
|
||||
- **Log Parser**: Extract structured data from log lines
|
||||
- **Event Normalizer**: Convert various log formats to unified schema
|
||||
|
||||
### 2. Processing Layer
|
||||
- **State Aggregator**: Maintain current view of all beads
|
||||
- **Metrics Calculator**: Compute throughput, latency, success rates
|
||||
- **Dependency Resolver**: Build bead relationship graphs
|
||||
### 2. Telemetry Aggregator
|
||||
- **Metrics Accumulator**: Track counters, gauges, histograms
|
||||
- **Timeline Builder**: Construct execution timelines per worker
|
||||
- **Session Tracker**: Group events by worker session
|
||||
|
||||
### 3. Storage Layer
|
||||
- **Time-series DB**: Store metrics for historical analysis (InfluxDB/TimescaleDB)
|
||||
- **Graph Store**: Bead dependency relationships (in-memory or Neo4j)
|
||||
- **Cache**: Fast access to current state (Redis/Valkey)
|
||||
### 3. Visualization Renderers
|
||||
|
||||
### 4. API Layer
|
||||
- **REST API**: Query beads, metrics, and relationships
|
||||
- **WebSocket**: Real-time updates for dashboard
|
||||
- **GraphQL** (optional): Flexible querying for complex visualizations
|
||||
#### TUI Renderer
|
||||
- **Live Dashboard**: Real-time terminal display using blessed/ink/textual
|
||||
- **Log Viewer**: Scrollable, filterable log output
|
||||
- **Worker Grid**: At-a-glance status of all workers
|
||||
- **Detail Pane**: Deep-dive into specific worker sessions
|
||||
|
||||
### 5. Visualization Layer
|
||||
- **Web Dashboard**: React/Vue-based UI
|
||||
- **CLI Tool**: Terminal-based monitoring (`fabric status`, `fabric trace <bead-id>`)
|
||||
- **Metrics Export**: Prometheus-compatible endpoint
|
||||
#### HTML Renderer
|
||||
- **Static Reports**: Self-contained HTML files for sharing/archiving
|
||||
- **Interactive Dashboard**: Browser-based live view
|
||||
- **Timeline Visualization**: Gantt-style execution timelines
|
||||
- **Log Explorer**: Searchable, syntax-highlighted log viewer
|
||||
|
||||
### 4. Output Formats
|
||||
- **TUI**: Direct terminal rendering (ncurses/blessed style)
|
||||
- **HTML**: Static files or served via local HTTP
|
||||
- **JSON**: Raw data export for external tools
|
||||
- **Markdown**: Summary reports for documentation
|
||||
|
||||
## Key Features
|
||||
|
||||
### Phase 1: Foundation (MVP)
|
||||
- [ ] Read and parse bead JSONL files
|
||||
- [ ] Display current bead counts by status
|
||||
- [ ] List active beads with basic details
|
||||
- [ ] Simple CLI for querying state
|
||||
- [ ] Consume NEEDLE worker log streams
|
||||
- [ ] Parse structured JSON log events
|
||||
- [ ] Simple TUI: list active workers with status
|
||||
- [ ] Basic HTML: render session logs as static page
|
||||
|
||||
### Phase 2: Visualization
|
||||
- [ ] Web dashboard with real-time updates
|
||||
- [ ] Bead state distribution charts (pie/bar)
|
||||
- [ ] Timeline view of bead processing
|
||||
- [ ] Dependency graph visualization
|
||||
### Phase 2: TUI Dashboard
|
||||
- [ ] Real-time worker status grid
|
||||
- [ ] Live log streaming with filtering
|
||||
- [ ] Worker detail view (select worker, see history)
|
||||
- [ ] Keyboard navigation and search
|
||||
|
||||
### Phase 3: Analytics
|
||||
- [ ] Historical trend analysis
|
||||
- [ ] Performance metrics (avg completion time, throughput)
|
||||
- [ ] Bottleneck detection (most-blocked beads)
|
||||
- [ ] Worker utilization statistics
|
||||
### Phase 3: HTML Visualizations
|
||||
- [ ] Session timeline visualization (Gantt-style)
|
||||
- [ ] Metrics charts (API calls, tokens, duration)
|
||||
- [ ] Searchable log explorer with syntax highlighting
|
||||
- [ ] Export/share capabilities
|
||||
|
||||
### Phase 4: Advanced Features
|
||||
- [ ] Alerting on stalled workflows
|
||||
- [ ] Predictive completion estimates
|
||||
- [ ] Custom dashboard widgets
|
||||
- [ ] Export capabilities (CSV, JSON, reports)
|
||||
### Phase 4: Advanced Analytics
|
||||
- [ ] Cross-session analysis (patterns, trends)
|
||||
- [ ] Error clustering and root cause hints
|
||||
- [ ] Performance regression detection
|
||||
- [ ] Custom dashboard layouts
|
||||
|
||||
## Technology Stack
|
||||
|
||||
### Backend Options
|
||||
- **Node.js + TypeScript**: Fast development, good JSONL parsing
|
||||
- **Python + FastAPI**: Rich data analysis libraries (pandas, plotly)
|
||||
- **Go**: High performance, good for file watching and streaming
|
||||
### Log Processing
|
||||
- **Node.js streams**: Efficient log consumption
|
||||
- **pino/winston parsers**: Structured log parsing
|
||||
- **RxJS**: Reactive event stream processing
|
||||
|
||||
### Frontend Options
|
||||
- **React + Recharts**: Component-based, good charting library
|
||||
- **Vue + Chart.js**: Lightweight, reactive
|
||||
- **Svelte + D3.js**: Minimal bundle size, powerful visualizations
|
||||
### TUI Framework Options
|
||||
- **blessed/blessed-contrib**: Feature-rich terminal UI (Node.js)
|
||||
- **ink**: React for CLI (Node.js)
|
||||
- **textual**: Modern TUI framework (Python)
|
||||
- **bubbletea**: Elegant TUI framework (Go)
|
||||
|
||||
### Storage
|
||||
- **SQLite**: Simple, file-based, good for MVP
|
||||
- **PostgreSQL + TimescaleDB**: Production-grade time-series
|
||||
- **Redis/Valkey**: Caching and pub/sub for real-time updates
|
||||
### HTML Generation
|
||||
- **Static**: Generate self-contained HTML files with embedded CSS/JS
|
||||
- **Templates**: Handlebars/EJS for report generation
|
||||
- **Charts**: Chart.js, Recharts, or Plotly for visualizations
|
||||
- **Timeline**: vis-timeline or custom D3.js
|
||||
|
||||
## Deployment Model
|
||||
### Serving (Optional)
|
||||
- **Local HTTP server**: Serve HTML dashboard on localhost
|
||||
- **WebSocket**: Real-time updates to browser
|
||||
|
||||
## CLI Interface
|
||||
|
||||
```bash
|
||||
# TUI mode - live dashboard
|
||||
fabric tui
|
||||
|
||||
# Watch specific worker
|
||||
fabric tui --worker <worker-id>
|
||||
|
||||
# Generate HTML report for session
|
||||
fabric html --session <session-id> --output report.html
|
||||
|
||||
# Generate HTML report for all recent sessions
|
||||
fabric html --since 1h --output dashboard.html
|
||||
|
||||
# Stream logs in terminal
|
||||
fabric logs --follow
|
||||
fabric logs --worker <worker-id> --level error
|
||||
|
||||
# Export raw data
|
||||
fabric export --format json --output data.json
|
||||
```
|
||||
|
||||
## Integration with NEEDLE
|
||||
|
||||
FABRIC reads from NEEDLE's output, requiring:
|
||||
|
||||
1. **Log Format Agreement**: NEEDLE outputs structured JSON logs
|
||||
2. **Telemetry Events**: NEEDLE emits timing/metric events
|
||||
3. **Session Boundaries**: Clear start/end markers for worker sessions
|
||||
|
||||
### Expected Log Format
|
||||
```json
|
||||
{"ts":1709337600,"worker":"w-abc123","level":"info","msg":"Starting task","task":"Process bead bd-xyz"}
|
||||
{"ts":1709337601,"worker":"w-abc123","level":"debug","msg":"Tool call","tool":"Read","args":{"path":"/src/main.ts"}}
|
||||
{"ts":1709337605,"worker":"w-abc123","level":"info","msg":"Task complete","duration_ms":5000}
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Development
|
||||
- Run locally alongside NEEDLE
|
||||
- Watch local `.beads/` directory
|
||||
- Serve dashboard on `localhost:3000`
|
||||
```bash
|
||||
# Run TUI alongside NEEDLE
|
||||
fabric tui --source ~/.needle/logs/
|
||||
|
||||
### Production (Kubernetes)
|
||||
- Deploy as sidecar to NEEDLE workers
|
||||
- Aggregate data from multiple workers
|
||||
- Expose dashboard via Ingress
|
||||
- Store metrics in centralized DB
|
||||
# Generate HTML report
|
||||
fabric html --source ~/.needle/logs/ --output ./reports/
|
||||
```
|
||||
|
||||
### Production
|
||||
- Sidecar container reading NEEDLE worker logs
|
||||
- Periodic HTML report generation to shared storage
|
||||
- Optional: hosted dashboard with real-time WebSocket updates
|
||||
|
||||
## Success Metrics
|
||||
|
||||
FABRIC will be successful if it:
|
||||
1. **Reduces debugging time**: Find problematic beads in <30 seconds
|
||||
2. **Improves visibility**: All stakeholders can see workflow status
|
||||
3. **Enables optimization**: Identify and fix bottlenecks based on data
|
||||
4. **Scales efficiently**: Handle 1000+ beads without performance degradation
|
||||
1. **Immediate insight**: See worker status within 1 second of running `fabric tui`
|
||||
2. **Log accessibility**: Find relevant log entries in <10 seconds
|
||||
3. **Shareable reports**: Generate HTML that works offline, no dependencies
|
||||
4. **Low overhead**: <5% CPU impact when monitoring NEEDLE workers
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Prototype CLI tool**: Validate JSONL parsing and basic queries
|
||||
2. **Design data schema**: Finalize storage format for metrics
|
||||
3. **Build API**: Implement core endpoints for querying state
|
||||
4. **Create dashboard mockups**: Define UX before implementation
|
||||
5. **Implement MVP**: Phase 1 features with simple web UI
|
||||
1. **Define log format contract**: Specify what NEEDLE must output
|
||||
2. **Prototype TUI**: Basic worker list with status
|
||||
3. **Prototype HTML**: Static page from sample logs
|
||||
4. **Integrate with NEEDLE**: Connect to actual worker output
|
||||
5. **Iterate on UX**: Refine based on real usage
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should FABRIC store historical data indefinitely or have retention policies?
|
||||
- How should we handle multi-workspace scenarios (multiple NEEDLE instances)?
|
||||
- What's the desired latency for real-time updates (1s, 5s, 10s)?
|
||||
- Should FABRIC be read-only or allow workflow control (pause, retry, cancel)?
|
||||
- Where does NEEDLE write logs? (stdout, files, both?)
|
||||
- What telemetry does NEEDLE currently emit?
|
||||
- Should FABRIC support multiple NEEDLE instances?
|
||||
- Retention: how much history should FABRIC keep accessible?
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue