Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Running Tests

Every #[ktstr_test] boots a fresh KVM microVM with the topology the test declares, on the exact kernel you target. cargo ktstr test resolves that kernel (building and caching it when needed) and wraps cargo nextest run, so nextest’s filtering, retries, and parallelism all apply.

A run failed

Read the assertion, timeline, scheduler log, monitor verdict, dumps, and replay command.

Need topology coverage?

Expand one test across presets, then select variants with nextest filters or a CI budget.

Need regression gates?

Use run sidecars and perf-delta instead of comparing terminal output by eye.

Quick reference

# Run all tests
cargo ktstr test --kernel ../linux

# Run a specific test
cargo ktstr test --kernel ../linux -- -E 'test(sched_basic_proportional)'

# Run all ktstr-managed tests, skipping non-ktstr tests in the same crate
cargo ktstr test --kernel ../linux -- -E 'test(/^ktstr/)'

# Run ignored gauntlet variants
cargo ktstr test --kernel ../linux -- --run-ignored ignored-only -E 'test(gauntlet/)'

More patterns:

GoalCommand
Re-run one exact ktstr casecargo ktstr test --kernel 7.0 -- -E 'test(=ktstr/my_test)'
Compare two kernelscargo ktstr test --kernel 6.14 --kernel 7.0
Stay inside a CI time budgetKTSTR_BUDGET_SECS=300 cargo ktstr test --kernel 7.0
Open the same VM shape manuallycargo ktstr shell --kernel 7.0 --test my_test

What’s in this chapter

  • cargo ktstr — the host-side command: kernel resolution, test dispatch, replay, coverage, export, plus the interactive shell, topo, ctprof, and locks debugging tools (and the standalone ktstr binary that carries them without cargo).
  • Gauntlet — run every test across a matrix of topology presets.
  • BPF Verifier Sweep — verify, attach, and dispatch every declared scheduler across topologies.
  • Reading Failure Output — what a failed test prints, section by section, and how to investigate.
  • Auto-Repro — the second VM that replays a scheduler crash with probes attached.
  • Runs and Regression Gates — result sidecars, stats, and perf-delta.

Test names and variants

Tests registered through #[ktstr_test] show up in nextest output under these name shapes:

  • ktstr/{name} — single-kernel VM run.
  • host/{name} — a host_only test. It never boots a VM and never multiplies across kernels.
  • ktstr/{name}/{kernel} — one case per (test × kernel) when --kernel resolves to two or more kernels.
  • gauntlet/{name}/{preset} — one case per topology preset (see Gauntlet).
  • gauntlet/{name}/{preset}/{kernel} — the full (test × preset × kernel) expansion under a multi-kernel run.

This is what those names look like in a real run:

nextest case name: base ktstr variant
 Nextest run ID 98581174-246f-4824-a170-50992df166d7 with nextest profile: default
    Starting 1 test across 121 binaries (12531 tests skipped)
        PASS [  34.459s] (1/1) ktstr::failure_dump_e2e ktstr/failure_dump_renders_bss_fields
nextest list: gauntlet variants
...
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/4cpu-2llc-nosmt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/8cpu-2llc-smt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/9cpu-3llc-nosmt

Filter by prefix with -E 'test(/^ktstr/)', -E 'test(/^host/)', or -E 'test(/^gauntlet/)'.

The distinct host/ prefix is also an admission identity. cargo ktstr keeps ordinary host tests CPU-bounded while allowing all cheap VM-resource waiters to reach ktstr’s topology scheduler. Older ktstr releases listed host-only entries under the VM namespace; full-name filters for those entries should use host/ now.

Tip

test(NAME) is a substring match; the exact-match form test(=NAME) matches the full nextest name, prefix included. Use test(=ktstr/sched_basic_proportional), not the bare function name — test(=sched_basic_proportional) matches nothing.

The {kernel} suffix is a sanitized kernel label: kernel_ prefix, lowercase, non-alphanumeric characters collapsed to _ — 6.16.1 becomes kernel_6_16_1, and a path spec becomes kernel_path_{basename}_{hash6} (with _dirty appended when the source tree has uncommitted changes). The 6-character hash disambiguates two source paths that share a basename.

RUST_BACKTRACE=1 controls panic backtraces and verbose failure output, not guest console streaming — see Reading Failure Output for the investigation knobs.

Budget-based test selection

Set KTSTR_BUDGET_SECS to select the subset of tests that maximizes configuration coverage within a time budget — useful for CI pipelines and quick smoke tests:

KTSTR_BUDGET_SECS=300 cargo ktstr test --kernel ../linux

The selector encodes each test as a bitset of properties (scheduler, topology class, SMT, workload characteristics) and greedily picks the tests with the highest marginal coverage per estimated second, with duration estimates accounting for VM boot overhead by vCPU count. A summary is printed to stderr during budget-mode listing:

budget-mode selection summary
ktstr budget: 42/1200 tests, 295/300s used, 38/38 configurations covered

Testing your own scheduler

Declare it with declare_scheduler! and reference it from #[ktstr_test(scheduler = ...)] — see Scheduler Definitions and the Test a New Scheduler recipe.