jedarden/zai-proxy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: improve README for clarity and discoverability

Browse source 
Rewrite root README as a provider-agnostic description of the proxy
and dashboard. Remove internal git remote reference. Add quick-start,
environment variable table, and metrics inventory.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jedarden 2026-06-24 07:07:50 -04:00
parent 36727ac66d
commit 4dcafd8a52
1 changed files with 76 additions and 17 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

93
README.md
View file

@ -1,37 +1,96 @@
# zai-proxy
LLM reverse proxy and metrics dashboard for the [Z.AI](https://z.ai) API.
Production-grade LLM reverse proxy with token counting, adaptive rate limiting, and a real-time metrics dashboard.
## What it is
zai-proxy sits in front of an OpenAI-compatible LLM API and adds the observability and reliability layer that bare API access lacks:
- **Token tracking** — counts input and output tokens on every request, using API-reported usage when available and tiktoken estimation as a fallback
- **Adaptive rate limiting** — tracks upstream 429 responses with an EWMA ceiling estimator and automatically holds just below the limit, probing periodically to detect ceiling increases
- **Prometheus metrics** — request counts, latency histograms, token rates, error rates, and rate-limit wait times, all labelled by model and deployment variant
- **Blue/green and canary support** — tag requests by `DEPLOYMENT_VARIANT` to compare proxy versions or model configurations side-by-side in the same metrics namespace
- **SSE streaming** — passes through chunked streaming responses without buffering
## Components
### proxy/
### `proxy/`
OpenAI-compatible reverse proxy that fronts the Z.AI API. Features:
Go HTTP reverse proxy. Listens on `:8080`, forwards to the upstream LLM API, and exports Prometheus metrics at `/metrics`.
- Request/response body parsing and token counting (tiktoken + GLM tokenizers)
- Rate limiting with configurable burst and steady-state limits
- Prometheus metrics export
- Blue/green and canary deployment support
- Translation layer for provider-specific request/response formats
See [proxy/README.md](proxy/README.md) for full configuration and deployment instructions.
See [proxy/README.md](proxy/README.md) for setup and configuration.
### `dashboard/`
### dashboard/
Go backend + React frontend for live monitoring. Scrapes the proxy's Prometheus endpoint every 5 seconds, stores dual-resolution snapshots in SQLite, and streams updates to the browser via SSE.
Go backend + React frontend for visualizing proxy metrics, token usage, and request history.
```
proxy :8080/metrics → Collector → SQLite → SSE Hub → React UI
metrics_5s (24h)
metrics_1m (7d)
```
- Collects metrics from the proxy's Prometheus endpoint
- Stores aggregated data in SQLite
- Serves a Tailwind/Vite frontend via SSE for live updates
Features: request throughput, latency percentiles, token usage totals, error rates, and per-variant comparison panels.
See [dashboard docs](docs/notes/) for deployment and monitoring.
## Quick start
```bash
# 1. Configure the upstream API key and (optionally) override the target URL
export ZAI_API_KEY="your-api-key"
export ZAI_TARGET_URL="https://your-llm-provider.example.com/v1" # optional
# 2. Start the proxy
cd proxy
go run .
# Proxy → :8080 Metrics → :8080/metrics
# 3. (Optional) start the dashboard in another terminal
cd dashboard
go run .
# Dashboard → :3000
```
## Environment variables
### Proxy
| Variable | Default | Description |
|---|---|---|
| `ZAI_API_KEY` | (required) | API key forwarded to the upstream provider |
| `ZAI_TARGET_URL` | provider default | Override the upstream base URL |
| `TOKEN_COUNTING_ENABLED` | `true` | Enable/disable token counting |
| `TOKENIZER_MODEL` | `glm-4` | Tokenizer model name for estimation |
| `MAX_WORKERS` | `10` | Max concurrent upstream requests |
| `DEPLOYMENT_VARIANT` | `production` | Label for blue/green or canary tracking |
| `RATE_LIMIT_INITIAL` | | Initial rate (req/s) |
| `RATE_LIMIT_MIN` | | Minimum rate floor |
| `RATE_LIMIT_MAX` | | Maximum rate ceiling |
| `MAX_RETRIES` | | Retry count on transient errors |
### Dashboard
See [`docs/notes/`](docs/notes/) for dashboard configuration and deployment.
## Metrics
The proxy exposes standard Prometheus metrics at `/metrics`:
- `proxy_requests_total` — request counts by method, path, status, and variant
- `proxy_request_duration_seconds` — latency histogram
- `proxy_response_size_bytes` — response size histogram
- `proxy_upstream_errors_total` — upstream error counts by type and variant
- `proxy_token_input_total` / `proxy_token_output_total` — token counters by model and variant
- `proxy_rate_limit_wait_seconds` — time spent waiting on the rate limiter
## Docs
- [`docs/plan/`](docs/plan/) — architecture decisions and roadmaps
- [`docs/notes/`](docs/notes/) — deployment, operations, monitoring, canary procedures
- [`docs/research/`](docs/research/) — tokenizer research, metrics references
- [`DEVELOPMENT.md`](DEVELOPMENT.md) — contributor guide
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — contribution guidelines
## Git remotes
## License
- Forgejo (primary): `https://git.ardenone.com/jedarden/zai-proxy.git`
See [LICENSE](LICENSE) if present, or contact the maintainer.