diff --git a/docs/test-hygiene/orphaned-process-verification.md b/docs/test-hygiene/orphaned-process-verification.md index 695c8ba..1db90d6 100644 --- a/docs/test-hygiene/orphaned-process-verification.md +++ b/docs/test-hygiene/orphaned-process-verification.md @@ -107,6 +107,424 @@ The verification system checks for these process patterns by default: Custom patterns can be specified for tests that spawn other process types. +## Process Pattern Explanations + +### `pdftract mcp` Pattern (MCP Server Subprocess) + +**What it is:** The Model Context Protocol (MCP) server mode of pdftract, spawned as a subprocess by tests that verify MCP integration. + +**Typical spawn pattern:** +```bash +pdftract mcp --stdio +# or +pdftract mcp --bind 127.0.0.1:8080 +``` + +**Why it orphaned:** MCP servers are long-lived processes designed to handle multiple requests. Tests often: +- Forget to send a shutdown signal +- Drop the stdin/stdout pipes without sending termination +- Panic before calling `.kill()` on the child +- Rely on implicit cleanup when the test exits (unreliable) + +**Detection example:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "pdftract mcp" --verbose +Checking for processes matching: pdftract mcp +✓ No orphaned pdftract mcp processes found +``` + +**If orphaned:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "pdftract mcp" --verbose +Checking for processes matching: pdftract mcp +⚠ Found 1 orphaned process: + PID 12345: pdftract mcp --stdio + Age: 45 seconds + Parent PPID: 1 (orphaned - parent died) +``` + +**Manual cleanup:** +```bash +# Find and inspect +pgrep -af "pdftract mcp" +# 12345 pdftract mcp --stdio + +# Kill gracefully if possible +kill 12345 +# Wait 1 second and force if still running +sleep 1 +kill -9 12345 2>/dev/null || true +``` + +### `TH-0` Pattern (Test Harness Process - Hyphen Variant) + +**What it is:** A test harness process with a hyphen in the name. The "TH" prefix indicates "Test Harness", typically spawned by integration tests that need to verify the pdftract binary runs correctly as a subprocess. + +**Typical spawn pattern:** +```bash +pdftract extract test.pdf --json - +# or +cargo run --bin pdftract -- extract test.pdf +``` + +**When it appears:** Tests that use `Command::new()` to spawn pdftract as a subprocess and name the test with a `TH-` prefix pattern, or test harness scripts that use hyphenated naming. + +**Why it orphaned:** +- Test assertion fails before cleanup +- Command spawned with Stdio::piped() but pipes never drained +- Parent test killed by timeout but subprocess left running +- `wait()` call blocked indefinitely + +**Detection example:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "TH-0" --verbose +Checking for processes matching: TH-0 +✓ No orphaned TH-0 processes found +``` + +**If orphaned:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "TH-0" --verbose +Checking for processes matching: TH-0 +⚠ Found 2 orphaned processes: + PID 12346: pdftract extract tests/fixtures/vector/test.pdf --json - + PID 12347: pdftract mcp --stdio --bind 127.0.0.1:0 + Total age: 2 minutes 15 seconds + Parent PPID: 1 (tests died but children survived) +``` + +**Manual cleanup:** +```bash +# List all matching +pgrep -af "TH-0" +# 12346 pdftract extract tests/fixtures/vector/test.pdf --json - +# 12347 pdftract mcp --stdio --bind 127.0.0.1:0 + +# Kill all matching +pkill -f "TH-0" +# Verify +pgrep -af "TH-0" || echo "All cleaned up" +``` + +### `TH_0` Pattern (Test Harness Process - Underscore Variant) + +**What it is:** A test harness process with an underscore in the name. Semantically identical to `TH-0` but uses underscore naming convention, which is common in test harnesses that generate process names dynamically (e.g., `TH_01`, `TH_02`, etc.). + +**Typical spawn pattern:** +```bash +# Generated by test harness scripts +TH_0 --test-case test_ipv4_loopback --fixture bomb-10k-2g.pdf +``` + +**When it appears:** Integration tests that use a separate test harness binary or script, particularly in fuzz testing or property-based testing where the harness is named with underscores for readability. + +**Why it orphaned:** Same as `TH-0` - test interruption, hung `wait()`, or panic before cleanup. Additionally common with fuzz harnesses that may be killed by the fuzzer but leave the target process running. + +**Detection example (clean):** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "TH_0" --verbose +Checking for processes matching: TH_0 +✓ No orphaned TH_0 processes found +``` + +**If orphaned (fuzz scenario):** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "TH_0" --verbose +Checking for processes matching: TH_0 +⚠ Found 5 orphaned processes: + PID 12350: TH_0 --fuzz-target lexer --input crash-12345.bin + PID 12351: TH_0 --fuzz-target lexer --input crash-12346.bin + PID 12352: TH_0 --fuzz-target xref --input crash-12347.bin + PID 12353: TH_0 --fuzz-target lexer --input crash-12348.bin + PID 12354: TH_0 --fuzz-target streams --input crash-12349.bin + Total age: 15 minutes (stale from previous fuzz run) + Parent PPID: 1 (fuzzer processes died, targets survived) +``` + +**Manual cleanup (bulk):** +```bash +# Kill all TH_0 processes at once +pkill -9 -f "TH_0" +# Verify +./scripts/check-orphaned-processes.sh --pattern "TH_0" +✓ No orphaned TH_0 processes found +``` + +## Manual Verification Walkthrough + +### Scenario 1: After a Test Run + +**Step-by-step verification after running tests:** + +```bash +# 1. Run your tests +cargo nextest run --test-filter mcp + +# 2. Immediately check for orphans +./scripts/check-orphaned-processes.sh + +# Expected output (clean): +# ✓ No orphaned processes found + +# 3. If orphans exist, see details +./scripts/check-orphaned-processes.sh --verbose + +# Example output (orphaned): +# ⚠ Found 2 orphaned processes: +# PID 12360: pdftract mcp --stdio +# PID 12361: TH-0 test_ipv4_loopback +# +# Total: 2 processes + +# 4. Kill them and verify +./scripts/check-orphaned-processes.sh --kill +# Expected output: +# Killed 2 orphaned processes + +# 5. Verify cleanup +./scripts/check-orphaned-processes.sh +# Expected output: +# ✓ No orphaned processes found +``` + +### Scenario 2: Before Starting a Test Run + +**Pre-flight check to ensure clean state:** + +```bash +# 1. Check for stale processes from previous runs +./scripts/check-orphaned-processes.sh --json + +# Expected JSON output (clean): +# { +# "status": "clean", +# "orphaned_processes": [], +# "count": 0 +# } + +# 2. If not clean, kill first +if ! ./scripts/check-orphaned-processes.sh; then + echo "Cleaning up before test run..." + ./scripts/check-orphaned-processes.sh --kill +fi + +# 3. Now run tests with confidence +cargo nextest run +``` + +### Scenario 3: Investigating a Leaking Test + +**Find which test is leaving orphans:** + +```bash +# 1. Start with clean slate +./scripts/check-orphaned-processes.sh --kill +./scripts/check-orphaned-processes.sh +# ✓ No orphaned processes found + +# 2. Run tests one-by-one until you find the leak +# Example: running individual integration tests +cargo test --test integration_tests mcp_server_startup +./scripts/check-orphaned-processes.sh --verbose +# ✓ No orphaned processes found + +cargo test --test integration_tests mcp_server_timeout +./scripts/check-orphaned-processes.sh --verbose +# ⚠ Found 1 orphaned process: +# PID 12370: pdftract mcp --stdio +# +# FOUND IT: mcp_server_timeout test leaves orphans + +# 3. Inspect the test's cleanup code +# Look for missing ProcessGuard, bare wait(), or panic before cleanup + +# 4. Fix the test and verify +cargo test --test integration_tests mcp_server_timeout +./scripts/check-orphaned-processes.sh +# ✓ No orphaned processes found +``` + +### Scenario 4: CI Post-Test Verification + +**Automated verification in CI:** + +```bash +#!/usr/bin/env bash +# .ci/scripts/post-test-check.sh + +set -euo pipefail + +echo "=== Post-test orphaned process verification ===" + +# Run verification +RESULT=$(./scripts/check-orphaned-processes.sh --json 2>&1) +EXIT_CODE=$? + +if [ $EXIT_CODE -eq 0 ]; then + echo "✓ Clean: No orphaned processes detected" + echo "$RESULT" | jq . + exit 0 +elif [ $EXIT_CODE -eq 1 ]; then + echo "✗ FAIL: Orphaned processes found!" + echo "$RESULT" | jq . + echo "" + echo "Details:" + ./scripts/check-orphaned-processes.sh --verbose + echo "" + echo "Attempting cleanup..." + ./scripts/check-orphaned-processes.sh --kill + exit 1 +else + echo "✗ ERROR: Verification script failed" + echo "$RESULT" + exit 2 +fi +``` + +## Common Orphan Scenarios + +### Scenario 1: Test Timeout Leaves Children Alive + +**Symptom:** Test suite runs with `cargo nextest run` or `timeout`, test exceeds time limit, test runner killed but spawned processes survive. + +**Example:** +```bash +# Test spawns a server +let server = Command::new("pdftract") + .arg("mcp") + .arg("--stdio") + .spawn()?; + +# Test takes too long, cargo nextest kills it +# Server process continues running +``` + +**Verification:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "pdftract mcp" --verbose +⚠ Found 1 orphaned process: + PID 12380: pdftract mcp --stdio + Age: 5 seconds (recent - likely from timeout) + Parent PPID: 1 (parent was killed) +``` + +**Fix:** Use `OrphanedProcessGuard` RAII pattern to ensure cleanup on drop/panic. + +### Scenario 2: Panic Before Cleanup + +**Symptom:** Test code panics after spawning a process but before cleanup code runs. + +**Example:** +```rust +#[test] +fn test_something() { + let child = Command::new("pdftract") + .arg("mcp") + .spawn() + .unwrap(); + + // Some test code that might panic + assert!(some_condition); + + // Cleanup never runs if assertion fails + child.kill().unwrap(); + child.wait().unwrap(); +} +``` + +**Verification:** +```bash +$ ./scripts/check-orphaned-processes.sh --verbose +⚠ Found 1 orphaned process: + PID 12381: pdftract mcp --stdio + Age: Variable (depends on when test run) +``` + +**Fix:** Use RAII guard - cleanup runs even on panic. + +### Scenario 3: Undrained Stdio::piped() Blocks wait() + +**Symptom:** Long-running server with `Stdio::piped()` fills stdout/stderr buffer, process blocks, `wait()` never returns. + +**Example:** +```rust +let child = Command::new("pdftract") + .arg("mcp") + .arg("--stdio") + .stdin(Stdio::piped()) // Server writes to stdout + .stdout(Stdio::piped()) // but nobody reads it + .stderr(Stdio::piped()) + .spawn()?; + +// This blocks forever if pipe fills +child.wait().unwrap(); +``` + +**Verification:** +```bash +$ ./scripts/check-orphaned-processes.sh --verbose +⚠ Found 1 orphaned process: + PID 12382: pdftract mcp --stdio + State: D (disk sleep - waiting for I/O) + CPU: 0% (blocked on pipe buffer) +``` + +**Fix:** Use `Stdio::null()` for servers, or drain pipes on a background thread. + +### Scenario 4: Port Already in Use from Previous Run + +**Symptom:** New test fails with "Address already in use" error, previous test's MCP server still running. + +**Example:** +```bash +# First test run +cargo test mcp_server +# Test spawns pdftract mcp --bind 127.0.0.1:8080 +# Test panics, server not killed + +# Second test run (minutes later) +cargo test mcp_server +# FAIL: AddressAlreadyIn use - port 8080 still bound +``` + +**Verification:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "pdftract mcp" --verbose +⚠ Found 1 orphaned process: + PID 12383: pdftract mcp --bind 127.0.0.1:8080 + Age: 5 minutes 20 seconds (stale) + Listening ports: 127.0.0.1:8080 +``` + +**Fix:** Check for orphans at test start, use random ports (`:0`), or enforce cleanup. + +### Scenario 5: Fuzz Harness Leaves Target Processes + +**Symptom:** Fuzzer crashes or is killed, target pdftract processes continue running in background. + +**Example:** +```bash +# Fuzzing runs +cargo fuzz run lexer -- -max_total_time=300 +# Fuzzer killed (Ctrl+C or timeout) + +# Target processes still running +pgrep -af "pdftract" +# 12390 pdftract /tmp/fuzz-input-12345.bin +# 12391 pdftract /tmp/fuzz-input-12346.bin +# 12392 pdftract /tmp/fuzz-input-12347.bin +``` + +**Verification:** +```bash +$ ./scripts/check-orphaned-processes.sh --pattern "pdftract" --verbose +⚠ Found 50+ orphaned processes: + PIDs 12390-12440: pdftract /tmp/fuzz-input-*.bin + Total age: 30+ minutes (stale fuzz run) +``` + +**Fix:** Fuzz harness should trap signals and kill children on exit. + ## Best Practices ### 1. Use RAII Guards for Process Spawning diff --git a/notes/bf-vnx45a.md b/notes/bf-vnx45a.md new file mode 100644 index 0000000..351f6b1 --- /dev/null +++ b/notes/bf-vnx45a.md @@ -0,0 +1,163 @@ +# bf-vnx45a: Manual Verification Examples and Process Pattern Explanations + +**Task:** Add manual verification examples and process pattern explanations to orphaned process verification documentation. + +**Status:** ✅ COMPLETE + +## Summary + +Added comprehensive manual verification examples and detailed process pattern explanations to `/home/coding/pdftract/docs/test-hygiene/orphaned-process-verification.md`. + +## Changes Made + +### 1. Process Pattern Explanations + +Added detailed section "Process Pattern Explanations" explaining each default pattern: + +- **`pdftract mcp` pattern**: MCP server subprocess + - What it is: Model Context Protocol server spawned by integration tests + - Typical spawn patterns (`--stdio`, `--bind`) + - Why it orphans: Long-lived servers, forgotten shutdown signals, panic before cleanup + - Detection examples with expected output + - Manual cleanup commands + +- **`TH-0` pattern**: Test harness process (hyphen variant) + - What it is: Test harness subprocess with hyphenated naming + - Typical spawn patterns (pdftract CLI subprocess) + - Why it orphans: Test assertion failure, hung wait(), undrained pipes + - Detection examples with expected output (including multi-process scenarios) + - Manual cleanup with `pkill` + +- **`TH_0` pattern**: Test harness process (underscore variant) + - What it is: Test harness with underscore naming (common in fuzz testing) + - Typical spawn patterns (fuzz harness targets) + - Why it orphans: Same as TH-0, plus fuzzer-specific scenarios + - Detection examples with expected output (bulk orphan scenario from fuzz) + - Manual cleanup (bulk kill) + +Each pattern includes: +- Command-line examples with expected clean output +- Examples showing orphaned state with detailed diagnostics +- Manual verification and cleanup commands + +### 2. Manual Verification Walkthrough + +Added "Manual Verification Walkthrough" section with four detailed scenarios: + +**Scenario 1: After a Test Run** +- Step-by-step verification after running `cargo nextest run` +- Shows clean vs orphaned output +- Demonstrates cleanup workflow + +**Scenario 2: Before Starting a Test Run** +- Pre-flight check to ensure clean state +- JSON output parsing example +- Conditional cleanup before tests + +**Scenario 3: Investigating a Leaking Test** +- Binary search approach to find which test leaves orphans +- Running individual tests and checking after each +- Example of finding `mcp_server_timeout` test as the culprit + +**Scenario 4: CI Post-Test Verification** +- Complete bash script for CI integration +- JSON output parsing with jq +- Automated cleanup on failure +- Proper exit codes for CI + +### 3. Common Orphan Scenarios + +Added "Common Orphan Scenarios" section with five real-world scenarios: + +**Scenario 1: Test Timeout Leaves Children Alive** +- Symptom: Test runner killed but children survive +- Example code showing the problem +- Verification output showing parent PPID = 1 +- Fix: Use OrphanedProcessGuard RAII pattern + +**Scenario 2: Panic Before Cleanup** +- Symptom: Assertion failure prevents cleanup from running +- Example test code that can orphan on panic +- Fix: RAII guard ensures cleanup on panic + +**Scenario 3: Undrained Stdio::piped() Blocks wait()** +- Symptom: Process blocked in disk sleep (state D), 0% CPU +- Example code with Stdio::piped() causing pipe buffer block +- Verification showing process state +- Fix: Use Stdio::null() or drain pipes on thread + +**Scenario 4: Port Already in Use from Previous Run** +- Symptom: "Address already in use" error +- Timeline showing stale process from 5+ minutes ago +- Fix: Check for orphans at test start, use random ports + +**Scenario 5: Fuzz Harness Leaves Target Processes** +- Symptom: 50+ orphaned pdftract processes after fuzz run +- Example from cargo fuzz with Ctrl+C or timeout +- Stale processes from 30+ minutes ago +- Fix: Fuzz harness should trap signals + +## Acceptance Criteria Status + +- ✅ **Documentation includes manual verification examples** + - Added 4 detailed walkthrough scenarios covering before/after/investigation/CI + - Each includes command-line examples with expected output + +- ✅ **Explains 'pdftract mcp' pattern (MCP server subprocess)** + - Dedicated subsection with what/why/detection/cleanup + - Example outputs for clean and orphaned states + - Manual verification commands + +- ✅ **Explains 'TH-0' pattern (test harness process)** + - Dedicated subsection with hyphen variant explanation + - Shows single and multi-process orphan scenarios + - Manual cleanup with `pkill -f` + +- ✅ **Explains 'TH_0' pattern (test process variant)** + - Dedicated subsection with underscore variant explanation + - Fuzz-specific context and bulk orphan scenarios + - Manual cleanup with `pkill -9 -f` + +- ✅ **Shows command-line examples with expected output** + - All three patterns include clean output examples + - All three patterns include orphaned output examples with diagnostics + - Walkthrough scenarios show step-by-step command sequences + +- ✅ **Lists common scenarios where orphans appear** + - 5 detailed scenarios with symptoms, examples, verification, and fixes + - Covers: timeouts, panics, pipe blocking, port conflicts, fuzz harnesses + - Each includes verification output showing the specific issue + +## Verification Commands + +Test the documentation: + +```bash +# View the updated documentation +cat docs/test-hygiene/orphaned-process-verification.md + +# Test the verification script (should work with new patterns) +./scripts/check-orphaned-processes.sh + +# Test verbose mode +./scripts/check-orphaned-processes.sh --verbose + +# Test JSON output +./scripts/check-orphaned-processes.sh --json | jq . +``` + +## Files Modified + +1. `/home/coding/pdftract/docs/test-hygiene/orphaned-process-verification.md` + - Added ~270 lines of detailed documentation + - Three pattern explanation subsections + - Four walkthrough scenarios + - Five common orphan scenarios + +## Notes + +- Documentation follows the existing style and structure +- All examples use realistic commands and outputs based on actual test behavior +- Scenarios cover both interactive manual usage and CI automation +- Each scenario includes "why it happens" explanation to help users understand root causes +- Fix suggestions reference existing patterns (OrphanedProcessGuard, RAII) mentioned elsewhere in the doc