# zai-proxy

Production-grade LLM reverse proxy with token counting, adaptive rate limiting, and a real-time metrics dashboard.

## What it is

zai-proxy sits in front of the Z.AI Anthropic-compatible API and adds the observability and reliability layer that bare API access lacks:

- **Token tracking** — counts input and output tokens on every request, using API-reported usage when available and tiktoken estimation as a fallback
- **Adaptive rate limiting** — tracks upstream 429 responses with an EWMA ceiling estimator and automatically holds just below the limit, probing periodically to detect ceiling increases
- **Prometheus metrics** — request counts, latency histograms, token rates, error rates, and rate-limit wait times, all labelled by model and deployment variant
- **Blue/green and canary support** — tag requests by `DEPLOYMENT_VARIANT` to compare proxy versions or model configurations side-by-side in the same metrics namespace
- **SSE streaming** — passes through chunked streaming responses without buffering

## Components

### `proxy/`

Go HTTP reverse proxy. Listens on `:8080`, forwards to the upstream LLM API, and exports Prometheus metrics at `/metrics`.

See [proxy/README.md](proxy/README.md) for full configuration and deployment instructions.

### `dashboard/`

Go backend + React frontend for live monitoring. Scrapes the proxy's Prometheus endpoint every 5 seconds, stores dual-resolution snapshots in SQLite, and streams updates to the browser via SSE.

```
proxy :8080/metrics  →  Collector  →  SQLite  →  SSE Hub  →  React UI
                                       metrics_5s (24h)
                                       metrics_1m (7d)
```

Features: request throughput, latency percentiles, token usage totals, error rates, and per-variant comparison panels.

## Quick start

```bash
# 1. Configure the upstream API key and (optionally) override the target URL
export ZAI_API_KEY="your-api-key"
export ZAI_TARGET_URL="https://your-llm-provider.example.com/v1"  # optional

# 2. Start the proxy
cd proxy
go run .

# Proxy → :8080   Metrics → :8080/metrics

# 3. (Optional) start the dashboard in another terminal
cd dashboard
go run .
# Dashboard → :8080
```

## Environment variables

### Proxy

| Variable | Default | Description |
|---|---|---|
| `ZAI_API_KEY` | (required) | API key forwarded to the upstream provider |
| `ZAI_TARGET_URL` | `https://api.z.ai/api/anthropic` | Override the upstream base URL |
| `TOKEN_COUNTING_ENABLED` | `true` | Enable/disable token counting |
| `TOKENIZER_MODEL` | `glm-4` | Tokenizer model name for estimation |
| `MAX_WORKERS` | `10` | Max concurrent upstream requests |
| `DEPLOYMENT_VARIANT` | `production` | Label for blue/green or canary tracking |
| `RATE_LIMIT_INITIAL` | `10.0` | Initial rate (req/s) |
| `RATE_LIMIT_MIN` | `1.0` | Minimum rate floor |
| `RATE_LIMIT_MAX` | `50.0` | Maximum rate ceiling |
| `RATE_LIMIT_CEILING_ALPHA` | `0.3` | EWMA smoothing factor for ceiling estimation |
| `RATE_LIMIT_HOLD_MARGIN` | `0.02` | Hold margin below ceiling (as fraction, e.g., 0.02 = 2%) |
| `RATE_LIMIT_PROBE_INTERVAL` | `10` | Probe ceiling every N clean windows |
| `MAX_RETRIES` | `3` | Retry count on transient errors |

### Dashboard

See [`docs/notes/`](docs/notes/) for dashboard configuration and deployment.

## Metrics

The proxy exposes standard Prometheus metrics at `/metrics`:

- `zai_proxy_requests_total` — request counts by method, path, status, and variant
- `zai_proxy_request_duration_seconds` — latency histogram
- `zai_proxy_request_size_bytes` — request size histogram
- `zai_proxy_response_size_bytes` — response size histogram
- `zai_proxy_concurrent_requests` — concurrent request gauge
- `zai_proxy_upstream_errors_total` — upstream error counts by type and variant
- `zai_proxy_tokens_total` — token counters with labels: direction (input/output/cache_read/cache_write), model, variant, pricing_tier
- `zai_proxy_rate_limit_requests_per_second` — current rate limit gauge
- `zai_proxy_rate_limit_wait_seconds` — time spent waiting on the rate limiter
- `zai_proxy_rate_limit_adjustments_total` — rate limit adjustments by direction and variant
- `zai_proxy_rate_limit_rejections_total` — requests rejected due to rate limiting
- `zai_proxy_retry_attempts_total` — retry attempts by reason and variant
- `zai_proxy_worker_utilization_ratio` — worker utilization gauge
- `zai_proxy_max_workers` — maximum concurrent workers gauge
- `zai_proxy_token_rate_seconds` — token processing time histogram
- `zai_proxy_token_rate` — token throughput histogram (tokens per second)
- `zai_proxy_token_count_duration_seconds` — token counting operation duration
- `zai_proxy_build_info` — build information (version, variant, commit, build time)

## Docs

- [`docs/plan/`](docs/plan/) — architecture decisions and roadmaps
- [`docs/notes/`](docs/notes/) — deployment, operations, monitoring, canary procedures
- [`docs/research/`](docs/research/) — tokenizer research, metrics references
- [`DEVELOPMENT.md`](DEVELOPMENT.md) — contributor guide
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — contribution guidelines

## License

See [LICENSE](LICENSE) if present, or contact the maintainer.