From 12eb38e4563bdd6826acdd624e7b0712bcde4bf4 Mon Sep 17 00:00:00 2001 From: jedarden Date: Mon, 6 Jul 2026 23:30:03 -0400 Subject: [PATCH] docs(bf-49mfs): document cargo nextest timing output formats Research findings documenting available cargo nextest output formats that support timing data: - JUnit XML (stable, primary format) - Libtest JSON (experimental, compatibility-focused) - Test list JSON (for listing only, no timing) Closes bf-49mfs. References: https://nexte.st/docs/machine-readable/ --- notes/bf-49mfs-research.md | 129 +++++++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 notes/bf-49mfs-research.md diff --git a/notes/bf-49mfs-research.md b/notes/bf-49mfs-research.md new file mode 100644 index 0000000..1fada3e --- /dev/null +++ b/notes/bf-49mfs-research.md @@ -0,0 +1,129 @@ +# Cargo Nextest Timing Output Formats Research + +**Bead:** bf-49mfs (split from bf-35gxz) +**Date:** 2026-07-06 +**Status:** Complete + +## Overview + +This document catalogs the available cargo nextest output formats that support timing data, as research prerequisite to selecting the right format for parseable timing data. + +## Available Output Formats Supporting Timing Data + +### 1. JUnit XML (Primary Format) + +**Status:** Stable, production-ready + +**Description:** The main mechanism for machine-readable test run output in nextest. Adheres to the Jenkins XML format for JUnit/XUnit. + +**Configuration:** Add to `.config/nextest.toml`: +```toml +[profile.ci.junit] +path = "junit.xml" +``` + +**Timing fields:** +- `time` attribute on `` root: Total execution time in seconds +- `time` attribute on each ``: Individual test execution time in seconds +- `timestamp` attribute on each ``: ISO 8601 timestamp when test started + +**Example:** +```xml + + + + + + ... + + + +``` + +**Additional configuration options:** +- `report-name`: Name of the report (default: "nextest-run") +- `store-success-output`: Include stdout/stderr for passing tests (default: false) +- `store-failure-output`: Include stdout/stderr for failing tests (default: true) +- `flaky-fail-status`: How to report flaky-fail tests ("failure" or "success", default: "failure") + +**Output location:** `target/nextest//junit.xml` within workspace root + +**References:** +- [JUnit support - cargo-nextest](https://nexte.st/docs/machine-readable/junit/) + +### 2. Libtest JSON (Experimental) + +**Status:** Experimental, requires opt-in via environment variable + +**Description:** Libtest-compatible JSON output for test runs, primarily for compatibility with existing test infrastructure that consumes this format. + +**Enable with:** `NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1` + +**Usage:** +```bash +NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1 cargo nextest run --message-format +``` + +**Format variants:** +- `libtest-json`: Produce output similar to unstable libtest JSON +- `libtest-json-plus`: Same as libtest-json, with an extra `nextest` field containing additional metadata + +**Versioning:** +- Specify version via `--message-format-version ` +- Current version: `0.1` (as of 2023-12) +- Version tracks the unstable upstream libtest JSON format + +**Stability policy:** +- While experimental: Format may change to fix issues or track upstream changes +- After stabilization: Format changes will be accompanied by version number bumps (e.g., 0.1 → 0.2 → 1.0) + +**Note:** The format specification is currently marked as TODO in the documentation, indicating it's a work in progress. + +**References:** +- [Libtest JSON output - cargo-nextest](https://nexte.st/docs/machine-readable/libtest-json/) +- [Tracking issue #1152](https://github.com/nextest-rs/nextest/issues/1152) + +### 3. Test List JSON (For Listing Only, Not Runs) + +**Status:** Stable + +**Description:** Machine-readable output for test *lists* (not test *runs*). Used to discover available tests without executing them. + +**Usage:** +```bash +cargo nextest list --message-format json +``` + +**Timing support:** Does NOT include timing data (tests are not executed) + +**References:** +- [Listing tests - cargo-nextest](https://nexte.st/docs/listing/) + +## Human-Readable Timing Output + +**Default terminal output:** +- Tests are marked PASS or FAIL +- Wall-clock execution time shown in square brackets after each test name +- Example: `tests/test_basic.rs:test_success ... ok (0.004s)` + +**Not parseable:** This is for human consumption only; no stable machine-readable format. + +## Recommendations + +For parseable timing data: + +1. **Use JUnit XML** for production use - stable, well-documented, widely understood by test analysis tools +2. **Consider Libtest JSON** only if compatibility with existing libtest JSON consumers is required, with acceptance of experimental status +3. **Avoid relying on human-readable terminal output** for programmatic consumption + +## Sources + +- [About output formats - cargo-nextest](https://nexte.st/docs/machine-readable/) +- [JUnit support - cargo-nextest](https://nexte.st/docs/machine-readable/junit/) +- [Libtest JSON output - cargo-nextest](https://nexte.st/docs/machine-readable/libtest-json/) +- [Running tests - cargo-nextest](https://nexte.st/docs/running/) +- [Path for stabilizing libtest's json output? - Rust Internals](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163)