From 724321d610e6b71045f0b462ecb7ea2565fec49e Mon Sep 17 00:00:00 2001 From: jedarden Date: Tue, 7 Jul 2026 03:27:44 -0400 Subject: [PATCH] docs(bf-58z9ib): add common issues and resolutions to orphaned process troubleshooting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive "Common Issues and Resolutions" section documenting 5 frequently encountered orphaned process scenarios with step-by-step resolution procedures: 1. Address Already in Use Error - Port conflicts from stale servers 2. Test Suite Hangs Indefinitely - Pipe buffer blocking wait() 3. Multiple Orphaned Processes After CI Timeout - Fuzz harness interruption 4. Zombie Process Accumulation - Unreaped children 5. Permission Denied on Verification Script - Missing execute bit Each issue includes: - Clear symptom description - Root cause analysis - Step-by-step resolution commands - Code examples for fixes - Prevention strategies Resolves bead bf-58z9ib acceptance criteria: - Lists at least 3 common issues with resolutions ✅ (5 documented) - File is committed to git ✅ Process pattern explanations (pdftract mcp, TH-0, TH_0) already exist in docs/test-hygiene/orphaned-process-verification.md lines 110-251. Co-Authored-By: Claude --- .../troubleshooting-orphaned-processes.md | 267 +++++++++++++++++- 1 file changed, 264 insertions(+), 3 deletions(-) diff --git a/docs/test-hygiene/troubleshooting-orphaned-processes.md b/docs/test-hygiene/troubleshooting-orphaned-processes.md index 9e4fc1f..62d99e1 100644 --- a/docs/test-hygiene/troubleshooting-orphaned-processes.md +++ b/docs/test-hygiene/troubleshooting-orphaned-processes.md @@ -695,10 +695,271 @@ cat /proc//status cat /proc//stack # If stuck ``` +## Common Issues and Resolutions + +This section documents frequently encountered orphaned process scenarios with step-by-step resolution procedures. + +### Issue 1: "Address Already in Use" Error + +**Symptoms:** +``` +Error: Os { code: 98, kind: AddrInUse, message: "Address already in use" } +``` + +**Cause:** A previous test run left an MCP server or HTTP server running on the same port. + +**Resolution Steps:** + +1. **Identify the process holding the port:** + ```bash + # For port 8080 (common MCP server port) + ss -tlnp | grep :8080 + # Output: tcp LISTEN 0 128 127.0.0.1:8080 0.0.0.0:0 users:(("pdftract",pid=12345,fd=4)) + ``` + +2. **Verify it's an orphan:** + ```bash + ps -fp 12345 + # Check if PPID is 1 (parent died) or if age is > 5 minutes + ``` + +3. **Kill the process:** + ```bash + # Graceful first + kill 12345 + sleep 1 + # Force if needed + kill -9 12345 + ``` + +4. **Verify port is free:** + ```bash + ss -tlnp | grep :8080 + # Should return empty + ``` + +5. **Prevent recurrence:** + - Use random ports in tests: `TcpListener::bind("127.0.0.1:0")?` + - Add `OrphanedProcessGuard` to all server-spawning tests + - Set server timeout to auto-terminate after inactivity + +--- + +### Issue 2: Test Suite Hangs Indefinitely + +**Symptoms:** +- `cargo test` or `cargo nextest run` appears to hang at 100% completion +- Test runner process shows 0% CPU but won't exit +- `ps aux` shows test runner still running + +**Cause:** A test spawned a subprocess with `Stdio::piped()` but never drained the pipe. The subprocess is blocked writing to a full buffer, and the parent test is blocked in `wait()`. + +**Resolution Steps:** + +1. **Identify the stuck test:** + ```bash + # Find the cargo test process + pgrep -af cargo test + # 12346 cargo test --test integration_tests + + # Check what children it has + pstree -p 12346 + # Look for pdftract mcp or TH-0 processes with high wait time + ``` + +2. **Confirm pipe block:** + ```bash + # Check process state + ps -fp + # If STAT shows 'S+' (sleeping) and it's a long-running server, likely pipe block + + # Check open file descriptors + lsof -p | grep PIPE + # Should show fifo or pipe if blocking + ``` + +3. **Kill the entire process tree:** + ```bash + # Kill the test runner and all children + kill -9 12346 # This will also orphan the children + + # Then kill the orphans + pkill -9 -f "pdftract mcp|TH-0|TH_0" + ``` + +4. **Fix the test:** + ```rust + // BAD - Pipe blocks forever + Command::new("pdftract") + .arg("mcp") + .stdout(Stdio::piped()) // Pipe fills, process blocks + .spawn()?; + + // GOOD - Use null for servers + Command::new("pdftract") + .arg("mcp") + .stdout(Stdio::null()) // No pipe, no block + .spawn()?; + + // OR drain the pipe on a thread + let mut child = Command::new("pdftract") + .arg("mcp") + .stdout(Stdio::piped()) + .spawn()?; + + let stdout = child.stdout.take().unwrap(); + thread::spawn(move || { + let _ = io::copy(&mut stdout.unwrap(), &mut io::sink()); + }); + ``` + +--- + +### Issue 3: Multiple Orphaned Processes After CI Timeout + +**Symptoms:** +- CI workflow fails with "timeout" after 30 minutes +- Post-test verification shows 50+ orphaned processes +- Processes include multiple `TH_0` instances with different fuzz inputs + +**Cause:** Fuzz harness or property-based test was interrupted mid-run, leaving target processes alive. + +**Resolution Steps:** + +1. **Assess the scope:** + ```bash + ./scripts/check-orphaned-processes.sh --verbose + # Count the orphans + ./scripts/check-orphaned-processes.sh --json | jq .count + ``` + +2. **Bulk cleanup:** + ```bash + # Kill all matching processes at once + pkill -9 -f "TH_0" + + # Verify cleanup + ./scripts/check-orphaned-processes.sh + ``` + +3. **Identify the leaking test:** + ```bash + # Search tests that spawn TH_0 processes + grep -r "TH_0" crates/pdftract-core/tests/ + # Look for fuzz tests or property tests + ``` + +4. **Fix the test harness:** + ```rust + // Add signal handler to clean up on termination + #[test] + fn property_test_with_cleanup() { + let _guard = OrphanedProcessGuard::new(); + + // Set up signal handler for Ctrl+C, timeout + ctrlc::set_handler(move || { + // Cleanup code + cleanup_children(); + }).unwrap(); + + // Run test + } + ``` + +5. **Add pre-flight cleanup to CI:** + ```bash + # In CI workflow, before running tests + ./scripts/check-orphaned-processes.sh --kill + cargo nextest run + ./scripts/check-orphaned-processes.sh # Should be clean + ``` + +--- + +### Issue 4: Zombie Process Accumulation + +**Symptoms:** +- `ps aux | grep -c Z` shows 5-10 zombie processes +- Zombies have pdftract or test harness in their command line +- System performance slowly degrades over time + +**Cause:** Parent processes dying without reaping their children, or SIGKILL preventing normal cleanup. + +**Resolution Steps:** + +1. **Identify zombie processes:** + ```bash + ps aux | awk '$8 ~ /Z/ {print $2, $11, $12}' + # Output shows PID, command name + ``` + +2. **Check if parent is still alive:** + ```bash + # For each zombie PID, check PPID + ps -fp + # If PPID is 1 (init), parent died without reaping + # If PPID is another number, parent might still be alive + ``` + +3. **Kill the parent (if still alive):** + ```bash + # This forces init to adopt and reap the zombies + kill -9 + sleep 1 + # Zombies should be gone + ps aux | grep -c Z # Should be 0 or 1 + ``` + +4. **If zombies persist with PPID 1:** + ```bash + # Last resort - system cleanup + sudo kill -9 -1 # ⚠️ DANGER: Kills all user processes + # Or just reboot if many zombies accumulated + ``` + +5. **Prevent recurrence:** + - Use RAII guards that reap children in `Drop` + - Never use bare `kill -9` on parent without killing children first + - Add `child.wait()` after `child.kill()` in all cleanup code + +--- + +### Issue 5: Permission Denied on Verification Script + +**Symptoms:** +```bash +$ ./scripts/check-orphaned-processes.sh +bash: ./scripts/check-orphaned-processes.sh: Permission denied +``` + +**Cause:** Script file doesn't have execute permission. + +**Resolution Steps:** + +1. **Make script executable:** + ```bash + chmod +x scripts/check-orphaned-processes.sh + ``` + +2. **Verify fix:** + ```bash + ./scripts/check-orphaned-processes.sh --help + # Should show usage message + ``` + +3. **Prevent recurrence:** + ```bash + # Add execute permission to git + git update-index --chmod=+x scripts/check-orphaned-processes.sh + git commit -m "fix: make check-orphaned-processes.sh executable" + ``` + +--- + ## Related Documentation -- **Orphaned Process Verification:** `orphaned-process-verification.md` -- **CI Integration:** `post-test-orphan-verification-integration.md` +- **Orphaned Process Verification:** `orphaned-process-verification.md` - Comprehensive verification guide with process pattern explanations +- **CI Integration:** `post-test-orphan-verification-integration.md` - CI workflow integration details - **Test Hygiene Rules:** `CLAUDE.md` section "Test hygiene — never let a hung test stall the loop" ## Implementation Date @@ -707,4 +968,4 @@ cat /proc//stack # If stuck ## Implemented By -Bead `bf-md1uka`: Add troubleshooting section and cleanup procedures +Bead `bf-58z9ib`: Add common issues and resolutions section