jedarden/zai-proxy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(readme): fix metric names, dashboard port, and env var defaults

Browse source 
- Fix metric names to use zai_proxy_ prefix (matching proxy/metrics.go)
- Fix dashboard port from :3000 to :8080 (matching DefaultConfig)
- Fix env var defaults: RATE_LIMIT_INITIAL=10.0, RATE_LIMIT_MIN=1.0, RATE_LIMIT_MAX=50.0, MAX_RETRIES=3
- Add missing RATE_LIMIT_* tunables: CEILING_ALPHA=0.3, HOLD_MARGIN=0.02, PROBE_INTERVAL=10
- Fix ZAI_TARGET_URL default to https://api.z.ai/api/anthropic
- Update description to reflect Z.AI-specific implementation (not OpenAI-compatible)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jedarden 2026-07-02 07:47:43 -04:00
parent 4dcafd8a52
commit a106abd09d
1 changed files with 28 additions and 13 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

41
README.md
View file

@ -4,7 +4,7 @@ Production-grade LLM reverse proxy with token counting, adaptive rate limiting,
## What it is
zai-proxy sits in front of an OpenAI-compatible LLM API and adds the observability and reliability layer that bare API access lacks:
zai-proxy sits in front of the Z.AI Anthropic-compatible API and adds the observability and reliability layer that bare API access lacks:
- **Token tracking** — counts input and output tokens on every request, using API-reported usage when available and tiktoken estimation as a fallback
- **Adaptive rate limiting** — tracks upstream 429 responses with an EWMA ceiling estimator and automatically holds just below the limit, probing periodically to detect ceiling increases
@ -48,7 +48,7 @@ go run .
# 3. (Optional) start the dashboard in another terminal
cd dashboard
go run .
# Dashboard → :3000
# Dashboard → :8080
```
## Environment variables
@ -58,15 +58,18 @@ go run .
| Variable | Default | Description |
|---|---|---|
| `ZAI_API_KEY` | (required) | API key forwarded to the upstream provider |
| `ZAI_TARGET_URL` | provider default | Override the upstream base URL |
| `ZAI_TARGET_URL` | `https://api.z.ai/api/anthropic` | Override the upstream base URL |
| `TOKEN_COUNTING_ENABLED` | `true` | Enable/disable token counting |
| `TOKENIZER_MODEL` | `glm-4` | Tokenizer model name for estimation |
| `MAX_WORKERS` | `10` | Max concurrent upstream requests |
| `DEPLOYMENT_VARIANT` | `production` | Label for blue/green or canary tracking |
| `RATE_LIMIT_INITIAL` | | Initial rate (req/s) |
| `RATE_LIMIT_MIN` | | Minimum rate floor |
| `RATE_LIMIT_MAX` | | Maximum rate ceiling |
| `MAX_RETRIES` | | Retry count on transient errors |
| `RATE_LIMIT_INITIAL` | `10.0` | Initial rate (req/s) |
| `RATE_LIMIT_MIN` | `1.0` | Minimum rate floor |
| `RATE_LIMIT_MAX` | `50.0` | Maximum rate ceiling |
| `RATE_LIMIT_CEILING_ALPHA` | `0.3` | EWMA smoothing factor for ceiling estimation |
| `RATE_LIMIT_HOLD_MARGIN` | `0.02` | Hold margin below ceiling (as fraction, e.g., 0.02 = 2%) |
| `RATE_LIMIT_PROBE_INTERVAL` | `10` | Probe ceiling every N clean windows |
| `MAX_RETRIES` | `3` | Retry count on transient errors |
### Dashboard
@ -76,12 +79,24 @@ See [`docs/notes/`](docs/notes/) for dashboard configuration and deployment.
The proxy exposes standard Prometheus metrics at `/metrics`:
- `proxy_requests_total` — request counts by method, path, status, and variant
- `proxy_request_duration_seconds` — latency histogram
- `proxy_response_size_bytes` — response size histogram
- `proxy_upstream_errors_total` — upstream error counts by type and variant
- `proxy_token_input_total` / `proxy_token_output_total` — token counters by model and variant
- `proxy_rate_limit_wait_seconds` — time spent waiting on the rate limiter
- `zai_proxy_requests_total` — request counts by method, path, status, and variant
- `zai_proxy_request_duration_seconds` — latency histogram
- `zai_proxy_request_size_bytes` — request size histogram
- `zai_proxy_response_size_bytes` — response size histogram
- `zai_proxy_concurrent_requests` — concurrent request gauge
- `zai_proxy_upstream_errors_total` — upstream error counts by type and variant
- `zai_proxy_tokens_total` — token counters with labels: direction (input/output/cache_read/cache_write), model, variant, pricing_tier
- `zai_proxy_rate_limit_requests_per_second` — current rate limit gauge
- `zai_proxy_rate_limit_wait_seconds` — time spent waiting on the rate limiter
- `zai_proxy_rate_limit_adjustments_total` — rate limit adjustments by direction and variant
- `zai_proxy_rate_limit_rejections_total` — requests rejected due to rate limiting
- `zai_proxy_retry_attempts_total` — retry attempts by reason and variant
- `zai_proxy_worker_utilization_ratio` — worker utilization gauge
- `zai_proxy_max_workers` — maximum concurrent workers gauge
- `zai_proxy_token_rate_seconds` — token processing time histogram
- `zai_proxy_token_rate` — token throughput histogram (tokens per second)
- `zai_proxy_token_count_duration_seconds` — token counting operation duration
- `zai_proxy_build_info` — build information (version, variant, commit, build time)
## Docs