cargo ktstr

cargo ktstr is the host-side command for the whole workflow: it resolves (and if needed builds and caches) the kernel, then drives cargo nextest run so every #[ktstr_test] boots its VM against exactly the kernel you asked for.

Install

cargo install --locked ktstr   # installs both `ktstr` and `cargo-ktstr`

The test-fixture binaries (jemalloc probes, schbench/taobench validators) are behind the non-default integration feature and are not installed by default. To build from a workspace checkout instead: cargo build --bin cargo-ktstr.

Task map

Common flags

These flags mean the same thing on every subcommand that takes them.

--kernel ID — one grammar everywhere:

cargo ktstr test --kernel ../linux                    # local source tree (builds + caches)
cargo ktstr test --kernel 6.14.2                      # version (auto-downloads on miss)
cargo ktstr test --kernel 6.14                        # major.minor prefix → latest patch release
cargo ktstr test --kernel 6.14.2-tarball-x86_64-kc... # cache key (from `kernel list`)
cargo ktstr test --kernel 6.12..6.14                  # range: every stable+longterm release inside
cargo ktstr test --kernel git+https://example.com/r.git#tag=v6.14   # git tag (#branch= / #sha= too)
cargo ktstr test --kernel 6.14.2 --kernel 7.0         # repeatable → multi-kernel matrix

Ranges expand against kernel.org’s releases.json; both endpoints are series-inclusive (6.11..6.14 covers every 6.14.N; spell 6.14.2 for an exact bound), and EOL series silently drop out unless you pass --include-eol. Git sources are fetched at the ref, built, and cached; a moved branch tip rebuilds.

When --kernel resolves to two or more kernels, the kernel becomes another gauntlet dimension: each (test × preset × kernel) tuple is a distinct nextest case with a kernel-label suffix (see test name shapes), and result sidecars partition per kernel under target/ktstr/{kernel}-{project_commit}/. Kernel resolution finishes for every requested kernel before any test runs — a missing kernel aborts up front rather than mid-matrix. host_only tests run once regardless of kernel count.

--no-perf-mode — disable all performance mode features (flock, pinning, RT scheduling, hugepages, NUMA mbind, KVM exit suppression). Also via KTSTR_NO_PERF_MODE.

--no-skip-mode — convert resource-contention and host-topology-insufficient skips into hard failures (exit 1 instead of 0). The default skips so a contended runner does not fail tests that simply could not start.

The three profiles — independent of each other:

• --release — the harness build profile. Release mode applies stricter assertion thresholds (gap_threshold_ms 2000 vs debug’s 3000, spread_threshold_pct 15% vs 35%), so tests that barely pass in debug may fail under --release.
• --profile NAME — the cargo build profile for the scheduler-under-test (a discovered scheduler package). Defaults to release; pass --profile dev for a fast unoptimized scheduler build.
• --nextest-profile NAME — the nextest test profile from .config/nextest.toml (retries, timeouts, output settings).

--relevant / --base / --base-ref / --default-branch — narrow the run to only the tests whose scheduler your working-tree change touches, against a baseline commit (merge-base with main by default). A broad or unattributable change fails safe and runs everything; a docs-only change runs nothing. The CI-matrix counterpart is affected; both are documented in CI.

test

Build the kernel (if needed) and run tests via cargo nextest run. Also available as cargo ktstr nextest (a clap alias). Arguments after -- are passed through to nextest:

cargo ktstr test --kernel ../linux                         # everything
cargo ktstr test --kernel 7.0 -- -E 'test(my_test)'        # nextest filter
cargo ktstr test --kernel 7.0 -- --retries 2               # nextest retries
cargo ktstr test --kernel 7.0 -- --features integration    # cargo features
cargo ktstr test --relevant                                # only tests my edits affect

A real single-test run (the 7.0 series was already built and cached, so resolution maps 7.0 to the latest patch release and reuses the cached image):

$ cargo ktstr test --kernel 7.0 -- --features integration -E 'test(=ktstr/failure_dump_renders_bss_fields)'
cargo ktstr: fetching latest 7.0.x kernel version
cargo ktstr: latest 7.0.x kernel: 7.0.14
cargo ktstr: resolved kernel "7.0"
...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.22s
────────────
 Nextest run ID 98581174-246f-4824-a170-50992df166d7 with nextest profile: default
    Starting 1 test across 121 binaries (12531 tests skipped)
        PASS [  34.459s] (1/1) ktstr::failure_dump_e2e ktstr/failure_dump_renders_bss_fields
────────────
     Summary [  34.498s] 1 test run: 1 passed, 12531 skipped

cargo ktstr: test outputs
  ~/ktstr/target/ktstr/7.0.14-73730e0-dirty
    (1 stats sidecar(s), 0 wprof trace(s) written this run)

An immediate re-run of the same command took the same ~34 s — with the kernel cached, wall time is the test itself, not infrastructure. For a --kernel <path> source tree, a cache hit is announced on stderr as cargo ktstr: cache hit for {path} ({cache_key}, built {age} ago) and skips the build entirely.

Per-test exit codes

Each #[ktstr_test] process exits with one of three codes, so CI gates and dashboards can triage runs:

Exit code 2 is the silent-pass guard: a Pass at a ≤ threshold gate run against a 0 / 0 ratio that synthesized to 0.0 would have shipped a false-green CI run. The harness records Inconclusive instead (see Checking) and projects it to a distinct exit code so external tooling can route the run separately from real regressions. The values are exported as EXIT_PASS / EXIT_FAIL / EXIT_INCONCLUSIVE in the prelude.

replay

Re-run only the tests that failed in the last session, from the result sidecars under target/ktstr/:

cargo ktstr replay              # print the nextest filter (dry-run)
cargo ktstr replay --exec       # actually run it
cargo ktstr replay -E starve    # narrow by test-name substring
cargo ktstr replay --dir PATH   # source sidecars from an archived tree

Dry-run is the default: the filter prints to stdout so you can inspect it (or paste it into CI) before committing to the re-run. --profile / --nextest-profile apply with --exec. Distinct from auto-repro, which fires inside the failing test process; replay is post-hoc, across a whole session.

coverage

Run tests with coverage via cargo llvm-cov nextest. Same kernel resolution, multi-kernel semantics, and common flags as test; arguments after -- pass through to cargo llvm-cov nextest:

cargo ktstr coverage --kernel ../linux
cargo ktstr coverage -- --workspace --lcov --output-path lcov.info

Requires cargo-llvm-cov and the llvm-tools-preview rustup component. Guest-side coverage is flushed over shared memory at VM exit and merged into the report automatically; multi-kernel runs merge every variant’s profraw into a single report.

Profraw files accumulate across runs — including host-side files that plain cargo ktstr test writes next to the cargo-ktstr binary (an LLVM_PROFILE_FILE injection that keeps default.profraw out of your kernel source tree; export LLVM_PROFILE_FILE yourself to opt out). To clean up:

cargo ktstr llvm-cov clean --profraw-only          # only *.profraw under target/llvm-cov-target/
rm -f target/debug/llvm-cov-target/default-*.profraw
rm -f target/release/llvm-cov-target/default-*.profraw

Avoid bare cargo ktstr llvm-cov clean (wipes reports too) and --workspace (also runs cargo clean).

llvm-cov

Raw passthrough to cargo llvm-cov for subcommands that don’t fit the coverage flow (report, clean, show-env), with the same kernel-resolution plumbing:

cargo ktstr llvm-cov report --lcov --output-path lcov.info

Always pass a subcommand: a bare cargo ktstr llvm-cov falls through to cargo test, which skips gauntlet variants and verifier cells entirely (they exist only under the nextest harness).

kernel

Manage cached kernel images: list, build, clean. The standalone ktstr kernel subcommands are identical.

How kernels are discovered

There are two flows, and they intentionally differ:

Explicit (--kernel ...) — the full pipeline. For a source-tree path: validate it is a kernel tree, look up the cache (clean git trees only — a cache hit short-circuits straight to tests), auto-configure (make defconfig if needed, append ktstr’s kconfig fragment, make olddefconfig), build with make -j$(nproc) KCFLAGS=-Wno-error, validate that the critical config options survived (CONFIG_SCHED_CLASS_EXT, CONFIG_DEBUG_INFO_BTF, CONFIG_BPF_SYSCALL, tracing options — the kernel build system silently drops options with unmet dependencies, and each failure gets a remediation hint), generate compile_commands.json, store the result in the cache, and run nextest with KTSTR_KERNEL (and KTSTR_KERNEL_LIST for multi-kernel) exported. Dirty or non-git trees build in place, get a _dirty kernel label, and never cache. Version, range, and git identifiers download/fetch + build + cache on miss.

Implicit (no --kernel) — discovery only, no builds. The test framework checks KTSTR_TEST_KERNEL, then KTSTR_KERNEL, then the newest valid cache entry, then local build trees (./linux, ../linux, /lib/modules/$(uname -r)/build), then host images (/boot/vmlinuz*). Whatever pre-built image it finds is used as-is — nothing is compiled and nothing new lands in the cache.

Tip

If you iterate on a local kernel tree, pass --kernel ../linux (or run cargo ktstr kernel build --kernel ../linux once) so the build is cached and reused. Implicit discovery will find the tree but never populate the cache for it.

kernel list

List cached kernels, newest first:

$ cargo ktstr kernel list
cache: ~/.cache/ktstr/kernels
  KEY                                              VERSION      SOURCE   ARCH    BUILT
  7.0.14-tarball-x86_64-kcabd40422                 7.0.14       tarball  x86_64  2026-07-04T23:06:12Z
  local-b4dc42d-x86_64-kcabd40422                  7.1.0        local    x86_64  2026-07-04T23:00:04Z
  local-5123e5a-x86_64-cfg1982ad42-kcabd40422      7.1.0        local    x86_64  2026-07-04T22:20:13Z
...
  local-c5d2724-x86_64-kcabd40422                  7.0.0        local    x86_64  2026-07-04T22:05:35Z
...
  7.0.13-tarball-x86_64-kc5f2631f0                 7.0.13       tarball  x86_64  2026-06-22T21:54:56Z (stale kconfig)
  7.1.1-tarball-x86_64-kc5f2631f0                  7.1.1        tarball  x86_64  2026-06-22T21:46:19Z (stale kconfig)
warning: entries marked (stale kconfig) were built against a different ktstr.kconfig. Rebuild with: kernel build --force --kernel <entry version> (add --extra-kconfig PATH if the entry also carries the (extra kconfig) tag).

(stale kconfig) marks entries built against a different ktstr kconfig fragment (they are rebuilt automatically when requested); (EOL) marks entries whose series has left kernel.org’s active releases list. --json emits the same data for scripting. With --kernel START..END, list switches to preview mode: it prints the versions the range expands to without downloading or building anything — the cheap answer to “what does 6.12..6.16 actually cover?”.

kernel build

Download (or use a source tree / git ref), build, and cache one kernel or a range:

cargo ktstr kernel build                        # latest stable from kernel.org
cargo ktstr kernel build --kernel 6.14.2        # specific version
cargo ktstr kernel build --kernel 6.12          # latest 6.12.x patch release
cargo ktstr kernel build --kernel 6.11..6.14    # every release in the range
cargo ktstr kernel build --kernel ../linux      # local source tree
cargo ktstr kernel build --force --kernel 6.14.2

With no --kernel, it builds the latest stable series that has had at least 8 maintenance releases — keeping CI off brand-new majors whose early builds are more likely to break. Already-cached entries are skipped unless --force.

A bare relative name is read as a cache key — prefix a relative source directory with ./.

kernel clean

cargo ktstr kernel clean --keep 3                 # keep the 3 most recent valid entries
cargo ktstr kernel clean --corrupt-only --force   # drop only broken entries

Corrupt entries (missing metadata or image) never consume a --keep slot. --force skips the confirmation prompt (required non-interactively).

verifier

Sweep every declare_scheduler!-registered scheduler through the real kernel’s BPF verifier across topologies, checking verify + attach + dispatch per cell:

cargo ktstr verifier --kernel 7.0
cargo ktstr verifier --scheduler scx-ktstr     # one scheduler
cargo ktstr verifier --raw                     # no cycle collapse

See BPF Verifier Sweep for the cell model, real output, and the kernels-filter contract.

shell

Shares the VM boot flow and flags with ktstr shell, plus two additions: --kernel also accepts raw image files (bzImage, Image), and --test NAME derives topology, memory, and include files from a registered #[ktstr_test] (mutually exclusive with --topology / --memory-mib; -i is additive):

cargo ktstr shell --test my_failing_test
cargo ktstr shell --kernel ./arch/x86/boot/bzImage

export

Export a registered test as a self-extracting .run file that reproduces the scenario on bare metal, no VM: the ktstr binary, the scheduler binary, and every declared include file, embedded in a bash preamble that validates root, sched_ext support, cgroup2, the no-other-scheduler-attached invariant, and topology compatibility before launching.

cargo ktstr export my_test -o /tmp/my_test.run

wrote /tmp/sched_basic_proportional.run (90074903 bytes archive, 0 include files)

The generated script opens with the frozen test specification and the preflight checks:

#!/bin/bash
# Generated by `cargo ktstr export`. Do not edit; regenerate to update.
set -euo pipefail

# --- frozen test specification ---
KTSTR_TEST_NAME=sched_basic_proportional
KTSTR_SCHED_NAME=ktstr_sched
KTSTR_GIT_HASH=73730e0
NEED_LLCS=1
NEED_CORES_PER_LLC=2
NEED_THREADS_PER_CORE=1
NEED_NUMA_NODES=1
TEST_DURATION_SECS=12
TEST_WATCHDOG_SECS=15

Scheduler choice, scheduler args, and topology are frozen at export time; --duration, --watchdog-timeout, and --quiet can be overridden at .run invocation. --package NAME scopes the workspace search (and disambiguates duplicate test names — otherwise the first matching binary in path order wins); --release builds the embedded binaries with the release profile to match how you will run them.

Not exportable (rejected with actionable errors): host_only tests, tests using bpf_map_write (they need the host-side probe surface), and KernelBuiltin schedulers.

completions

cargo ktstr completions bash >> ~/.local/share/bash-completion/completions/cargo
cargo ktstr completions zsh > ~/.zfunc/_cargo-ktstr
cargo ktstr completions fish > ~/.config/fish/completions/cargo-ktstr.fish

SHELL is one of bash, zsh, fish, elvish, powershell; --binary overrides the completion target (default cargo).

stats

Sidecar analysis for past runs: stats (analysis of the newest run), stats list (run table), stats list-metrics (the regression metric registry), stats list-values (distinct filter values in the pool), stats show-host --run ID (archived host context), and stats explain-sidecar --run ID (why optional fields are absent). cargo ktstr show-host prints the same host context live, and cargo ktstr show-thresholds TEST prints the merged assertion thresholds a test will run with. See Runs and Regression Gates.

perf-delta

Compare performance_mode test metrics between HEAD and a baseline commit, exiting non-zero when enough metrics regress to trip the gate:

cargo ktstr perf-delta --noise-adjust 5 --kernel 6.14

Baseline resolution, the --noise-adjust statistics, and the full flag set are documented in Runs and Regression Gates; the worked walkthrough is A/B Compare Branches.

affected

Emit the scheduler packages a base..HEAD diff affects, as a JSON array for a GitHub Actions dynamic matrix (["scx_lavd","scx_rusty"]). Attribution is fail-safe: any uncertainty widens to the full set, and only a strictly docs-only change emits []. --relevant is the same engine applied locally to narrow test / coverage / perf-delta runs. Both are documented with the complete CI workflow in CI.

locks

Enumerate every ktstr flock held on this host, read-only, naming holder PIDs and cmdlines — the troubleshooting companion when a run is stalled behind a peer’s reservation. Identical to ktstr locks, where the lock roots and real output are documented.