docs(bf-57ko1): add Prometheus alerting rules for serve deployments

- Add 7 alerting rules for pdftract serve monitoring
- Required alerts per plan Monitoring section:
  - PdftractExtractionLatencyHigh: p99 > 2× 150ms target for >5m
  - PdftractErrorRateHigh: error rate > 1% over 5m
  - PdftractCacheHitRateLow: cache hit rate < 50%
  - PdftractWorkerSaturated: worker pool > 80% busy
  - PdftractMemoryHigh: RSS > 80% of max_decompress_bytes
- Additional alerts from plan recommendations:
  - PdftractCacheSizeGrowingUnchecked
  - PdftractDiagnosticFlood
- Runbook URLs reference plan.md monitoring section

Acceptance criteria:
- ✓ Valid Prometheus YAML structure
- ✓ Contains 7 alerting rules (5 required + 2 additional)
- ✓ Runbook URLs point to docs/plan/plan.md monitoring section

Closes bf-57ko1
This commit is contained in:
jedarden 2026-07-05 12:49:49 -04:00
parent ef3f4929e3
commit f716d27aac

View file

@ -0,0 +1,114 @@
groups:
- name: pdftract_serve
interval: 30s
rules:
- alert: PdftractExtractionLatencyHigh
expr: |
histogram_quantile(0.99,
sum(rate(pdftract_extraction_duration_seconds_bucket[5m])) by (le, instance)
) > 0.3
for: 5m
labels:
severity: warning
service: pdftract
annotations:
summary: "Extraction latency p99 above 300ms threshold (2× target)"
description: >
The 99th percentile extraction duration is {{ $value }}s, exceeding 300ms
(2× the 150ms target) for 5 minutes. This indicates degraded performance.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"
- alert: PdftractErrorRateHigh
expr: |
sum(rate(pdftract_http_requests_total{status=~"5.."}[5m])) /
sum(rate(pdftract_http_requests_total[5m])) > 0.01
for: 5m
labels:
severity: critical
service: pdftract
annotations:
summary: "HTTP error rate above 1% threshold"
description: >
Error rate is {{ $value | humanizePercentage }}, exceeding 1% threshold
for 5 minutes. This indicates a significant service degradation or failure.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"
- alert: PdftractCacheHitRateLow
expr: |
(
sum(rate(pdftract_cache_hits_total[5m])) /
(sum(rate(pdftract_cache_hits_total[5m])) + sum(rate(pdftract_cache_misses_total[5m])))
) < 0.5
for: 10m
labels:
severity: info
service: pdftract
annotations:
summary: "Cache hit rate below 50% threshold"
description: >
Cache hit rate is {{ $value | humanizePercentage }}, below 50% threshold
for 10 minutes. This may indicate cache configuration issues or
insufficient cache capacity.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"
- alert: PdftractWorkerSaturated
expr: |
pdftract_rayon_pool_utilization > 0.8
for: 5m
labels:
severity: warning
service: pdftract
annotations:
summary: "Worker pool utilization above 80% threshold"
description: >
Rayon worker pool utilization is {{ $value | humanizePercentage }},
above 80% threshold for 5 minutes. The service is at risk of becoming
unresponsive under load.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"
- alert: PdftractMemoryHigh
expr: |
(
process_resident_memory_bytes{job="pdftract-serve"} /
(pdftract_max_decompress_bytes * 0.8)
) > 1
for: 5m
labels:
severity: warning
service: pdftract
annotations:
summary: "Memory usage above 80% of max_decompress_bytes ceiling"
description: >
RSS memory is {{ $value | humanize }}, exceeding 80% of the configured
max_decompress_bytes ceiling. This may lead to OOM under load.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"
# Additional alert from plan recommendations
- alert: PdftractCacheSizeGrowingUnchecked
expr: |
abs(deriv(pdftract_cache_size_bytes[1h])) > 1e9
for: 6h
labels:
severity: warning
service: pdftract
annotations:
summary: "Cache size growing faster than 1 GB/hour"
description: >
Cache size is changing by more than 1 GB/hour (current rate:
{{ $value | humanize }}B/h) for 6 hours. This may indicate a
cache eviction or configuration problem.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"
- alert: PdftractDiagnosticFlood
expr: |
sum(rate(pdftract_diagnostic_emitted_total{severity="error"}[5m])) > 10
for: 5m
labels:
severity: warning
service: pdftract
annotations:
summary: "Error diagnostic rate above 10/sec"
description: >
Error diagnostics are being emitted at {{ $value }}/sec, indicating
a potential software bug or infrastructure issue.
runbook: "https://github.com/jedarden/pdftract/blob/main/docs/plan/plan.md#L3832"