test(bf-3m477): capture timing flag outputs
Test each cargo nextest timing flag from bf-4wz6v documentation: - --message-format=libtest-json-plus (2.0M, JSON with exec_time) - --message-format=libtest-json (2.0M, JSON format) - --message-format-version=0.1 (2.0M, versioned JSON) - --message-format-version=1.0 (error, invalid version range) - --message-format-version=1 (error, invalid format) - --status-level=slow (112K, slow test detection) - --final-status-level=slow (412K, slow test summary) Acceptance criteria: - PASS: Output file exists for each tested flag (8 files) - PASS: Each file named after the flag it tests - PASS: At least 3 flags tested (7 unique flags) Closes bf-3m477
This commit is contained in:
parent
05967a51ea
commit
d0e9375cd1
8 changed files with 60720 additions and 0 deletions
72
notes/bf-3m477.md
Normal file
72
notes/bf-3m477.md
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
# bf-3m477: Test Each Timing Flag and Capture Output
|
||||
|
||||
## Task
|
||||
Test each timing flag and capture output for analysis.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Flags Tested
|
||||
|
||||
All 5 timing flags from bf-4wz6v documentation were tested:
|
||||
|
||||
1. **`--message-format=libtest-json-plus`** ✅
|
||||
- Command: `NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1 cargo nextest run -p pdftract-core --lib --message-format=libtest-json-plus`
|
||||
- Output: `notes/bf-mge0o-message-format-libtest-json-plus.log` (2.0M)
|
||||
- Result: SUCCESS - JSON output with `exec_time` fields per test
|
||||
|
||||
2. **`--message-format-version=0.1`** ✅
|
||||
- Command: `NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1 cargo nextest run -p pdftract-core --lib --message-format=libtest-json-plus --message-format-version=0.1`
|
||||
- Output: `notes/bf-mge0o-message-format-version-0.1.log` (2.0M)
|
||||
- Result: SUCCESS - JSON output with versioned schema (version must be in `major.minor` format, major must be 0)
|
||||
|
||||
3. **`--status-level=slow`** ✅
|
||||
- Command: `cargo nextest run -p pdftract-core --lib --status-level=slow`
|
||||
- Output: `notes/bf-mge0o-status-level-slow.log` (112K)
|
||||
- Result: SUCCESS - Standard output format, marks slow tests during execution
|
||||
|
||||
4. **`--final-status-level=slow`** ✅
|
||||
- Command: `cargo nextest run -p pdftract-core --lib --final-status-level=slow`
|
||||
- Output: `notes/bf-mge0o-final-status-level-slow.log` (412K)
|
||||
- Result: SUCCESS - Standard output format with slow test summary at end
|
||||
|
||||
5. **`--message-format=libtest-json`** ✅
|
||||
- Command: `NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1 cargo nextest run -p pdftract-core --lib --message-format=libtest-json`
|
||||
- Output: `notes/bf-mge0o-message-format-libtest-json.log` (2.0M)
|
||||
- Result: SUCCESS - JSON output without nextest-specific extensions
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Experimental flag requirement**: Both `libtest-json` and `libtest-json-plus` require `NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1` environment variable
|
||||
2. **Version format**: `--message-format-version` requires `major.minor` format (e.g., `0.1`), not just `1`
|
||||
3. **Version constraints**: Major version must be 0 (range 0..1)
|
||||
4. **JSON format**: Both JSON formats produce per-test `exec_time` fields for timing data
|
||||
5. **Slow test flags**: `--status-level=slow` and `--final-status-level=slow` work without experimental flags
|
||||
|
||||
### Test Suite
|
||||
- 2,902 lib tests from `pdftract-core`
|
||||
- Runtime: ~4.8s per run
|
||||
- 2857 passed, 45 failed, 2 skipped
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- ✅ **PASS**: Output file exists for each tested flag (5 files created)
|
||||
- ✅ **PASS**: Each file is named after the flag it tests
|
||||
- ✅ **PASS**: At least 3 flags are tested (5 tested, exceeds requirement)
|
||||
|
||||
## Output Files Created
|
||||
|
||||
1. `notes/bf-mge0o-message-format-libtest-json-plus.log` (2.0M)
|
||||
2. `notes/bf-mge0o-message-format-version-0.1.log` (2.0M)
|
||||
3. `notes/bf-mge0o-status-level-slow.log` (112K)
|
||||
4. `notes/bf-mge0o-final-status-level-slow.log` (412K)
|
||||
5. `notes/bf-mge0o-message-format-libtest-json.log` (2.0M)
|
||||
|
||||
## References
|
||||
|
||||
- Third child of split bf-mge0o
|
||||
- Depends on bf-2n36a (baseline comparison)
|
||||
- Documentation from bf-4wz6v timing flag research
|
||||
|
||||
## Verification
|
||||
|
||||
All timing flags successfully executed and output captured. The JSON formats (libtest-json and libtest-json-plus) both provide per-test timing data via the `exec_time` field, making them suitable for programmatic consumption of test execution times.
|
||||
5751
notes/bf-mge0o-final-status-level-slow.log
Normal file
5751
notes/bf-mge0o-final-status-level-slow.log
Normal file
File diff suppressed because it is too large
Load diff
17329
notes/bf-mge0o-message-format-libtest-json-plus.log
Normal file
17329
notes/bf-mge0o-message-format-libtest-json-plus.log
Normal file
File diff suppressed because it is too large
Load diff
17330
notes/bf-mge0o-message-format-libtest-json.log
Normal file
17330
notes/bf-mge0o-message-format-libtest-json.log
Normal file
File diff suppressed because it is too large
Load diff
17330
notes/bf-mge0o-message-format-version-0.1.log
Normal file
17330
notes/bf-mge0o-message-format-version-0.1.log
Normal file
File diff suppressed because it is too large
Load diff
7
notes/bf-mge0o-message-format-version-1.0.log
Normal file
7
notes/bf-mge0o-message-format-version-1.0.log
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
error: error parsing message format version
|
||||
|
||||
Caused by:
|
||||
invalid format version: 1.0
|
||||
|
||||
Caused by:
|
||||
version component `major` value 1 is out of range 0..1
|
||||
7
notes/bf-mge0o-message-format-version-1.log
Normal file
7
notes/bf-mge0o-message-format-version-1.log
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
error: error parsing message format version
|
||||
|
||||
Caused by:
|
||||
invalid format version: 1
|
||||
|
||||
Caused by:
|
||||
expected format version in form of `<major>.<minor>`
|
||||
2894
notes/bf-mge0o-status-level-slow.log
Normal file
2894
notes/bf-mge0o-status-level-slow.log
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Reference in a new issue