Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

cargo ktstr

cargo ktstr is the host-side command for the whole workflow: it resolves (and if needed builds and caches) the kernel, then drives cargo nextest run so every #[ktstr_test] boots its VM against exactly the kernel you asked for.

Run

test, replay, and common flags for ordinary ktstr suites.

Prepare

kernel, shell, export, and cache-management flows.

Analyze

verifier, stats, perf-delta, ctprof, and CI targeting.

Install

cargo install --locked ktstr   # installs both `ktstr` and `cargo-ktstr`

The test-fixture binaries (jemalloc probes, schbench/taobench validators) are behind the non-default integration feature and are not installed by default. To build from a workspace checkout instead: cargo build --bin cargo-ktstr.

Task map

I want to…CommandDepth
Run tests on a kernelcargo ktstr testthis page
Re-run last session’s failurescargo ktstr replaythis page
Manage cached kernelscargo ktstr kernelthis page
Sweep schedulers through the BPF verifiercargo ktstr verifierVerifier
Analyze results, gate regressionscargo ktstr stats / perf-deltaRuns
Narrow CI to affected schedulerscargo ktstr affectedCI
Reproduce a test on bare metalcargo ktstr exportthis page
Debug interactively in a VMcargo ktstr shellthis page

Common flags

These flags mean the same thing on every subcommand that takes them.

--kernel ID — one grammar everywhere:

cargo ktstr test --kernel ../linux                    # local source tree (builds + caches)
cargo ktstr test --kernel 6.14.2                      # version (auto-downloads on miss)
cargo ktstr test --kernel 6.14                        # major.minor prefix → latest patch release
cargo ktstr test --kernel 6.14.2-tarball-x86_64-kc... # cache key (from `kernel list`)
cargo ktstr test --kernel 6.12..6.14                  # range: every stable+longterm release inside
cargo ktstr test --kernel git+https://example.com/r.git#tag=v6.14   # git tag (#branch= / #sha= too)
cargo ktstr test --kernel fedora                      # prebuilt distro kernel (latest; fedora-44/f44 pin)
cargo ktstr test --kernel ubuntu                      # latest LTS HWE kernel (ubuntu-24.04 pins the LTS)
cargo ktstr test --kernel amazonlinux                 # AL2023's newest kernel stream (al2023 works too)
cargo ktstr test --kernel steamos                     # Valve's linux-neptune (x86_64 only; steamos-3.8 pins)
cargo ktstr test --kernel gke                         # latest GKE-promoted COS kernel (x86_64)
cargo ktstr test --kernel gke-129                     # newest promoted revision within COS milestone 129
cargo ktstr test --kernel ./kernel-core.rpm           # local package (.rpm / .deb / .pkg.tar.zst)
cargo ktstr test --kernel 6.14.2 --kernel 7.0         # repeatable → multi-kernel matrix

Ranges expand against kernel.org’s releases.json; both endpoints are series-inclusive (6.11..6.14 covers every 6.14.N; spell 6.14.2 for an exact bound), and EOL series silently drop out unless you pass --include-eol. Git sources are fetched at the ref, built, and cached; a moved branch tip rebuilds.

Distro specs skip the build entirely: the kernel is resolved from the distro’s official repo metadata (so a bare name always means the latest published kernel — re-resolved every run), downloaded with sha256 verification alongside its debuginfo, and cached like any built kernel. Prebuilt kernels ship virtio as modules; ktstr extracts the needed drivers from the package and loads them in the guest before any device is touched, so shell, test, coverage, and verifier all work unchanged. The package’s config is checked on acquire: anything the kernel can’t boot without fails fast, and missing ktstr-feature options (e.g. CONFIG_SCHED_CLASS_EXT on an older kernel) warn up front instead of failing mid-run. Local .rpm/.deb/.pkg.tar.zst files take the same path — note Fedora and Ubuntu split the kernel image and its modules across packages, so for those use the distro spec (which fetches the full set) rather than a single file. GKE is the deliberate exception to the package shape: ktstr downloads Google’s exact GKE-promoted bootable vmlinux, matching prepared headers and source from cos-tools, then builds only the missing virtio/Btrfs loadable modules against those exact artifacts. The Google kernel is not rebuilt or modified; all downloads are generation-pinned and MD5-verified before the normal atomic cache install.

Bare gke is deliberately rolling: every resolution scans Google’s current GKE release notes and chooses the numerically newest complete cos-MILESTONE-BUILD-PATCH-REVISION image named there. This means the newest GKE-promoted COS kernel, not specifically the Kubernetes Stable release channel. gke-129 constrains the search to COS milestone 129 but still follows that milestone’s newest promoted build, patch, and revision. For an immutable rerun, resolve once and use the resulting generation-specific cache key shown by cargo ktstr kernel list. Whichever form is used, ktstr refuses an image that lacks its required sched_ext, BPF, debug, virtio, or module capabilities rather than silently falling back to a reduced-capability kernel.

When --kernel resolves to two or more kernels, the kernel becomes another gauntlet dimension: each (test × preset × kernel) tuple is a distinct nextest case with a kernel-label suffix (see test name shapes), and result sidecars partition per kernel under target/ktstr/{kernel}-{project_commit}/. Kernel resolution finishes for every requested kernel before any test runs — a missing kernel aborts up front rather than mid-matrix. host_only tests run once regardless of kernel count.

--no-perf-mode — disable all performance mode features (flock, pinning, RT scheduling, hugepages, NUMA mbind, KVM exit suppression). Also via KTSTR_NO_PERF_MODE.

--no-skip-mode — convert resource-contention and host-topology-insufficient skips into hard failures (exit 1 instead of 0). The default skips so a contended runner does not fail tests that simply could not start.

The three profiles — independent of each other:

  • --release — the harness build profile. Release mode applies stricter assertion thresholds (gap_threshold_ms 2000 vs debug’s 3000, spread_threshold_pct 15% vs 35%), so tests that barely pass in debug may fail under --release.
  • --profile NAME — the cargo build profile for the scheduler-under-test (a discovered scheduler package). Defaults to release; pass --profile dev for a fast unoptimized scheduler build.
  • --nextest-profile NAME — the nextest test profile from .config/nextest.toml (retries, timeouts, output settings).

For every nextest-backed route (test, coverage, verifier, replay --exec, and raw llvm-cov nextest), cargo-ktstr supplies a low-priority nextest tool config. It admits VM-resource waiters immediately so ktstr’s cross-process topology queue—not a second CPU-sized nextest queue—decides which fitting VM starts next, while ordinary host tests remain bounded to the host CPU count. For orchestrated nextest runs, cargo-ktstr normalizes every valid -j / --test-threads spelling to its effectively unbounded admission budget; otherwise a caller-supplied CPU-sized value would reintroduce a second scheduler ahead of ktstr. Repository policy unrelated to the run-slot budget still applies. Raw llvm-cov modes such as report, clean, and show-env are unchanged.

--relevant / --base / --base-ref / --default-branch — narrow the run to only the tests whose scheduler your working-tree change touches, against a baseline commit (merge-base with main by default). A broad or unattributable change fails safe and runs everything; a docs-only change runs nothing. The CI-matrix counterpart is affected; both are documented in CI.

test

Build the kernel (if needed) and run tests via cargo nextest run. Also available as cargo ktstr nextest (a clap alias). Arguments after -- are passed through to nextest:

cargo ktstr test --kernel ../linux                         # everything
cargo ktstr test --kernel 7.0 -- -E 'test(my_test)'        # nextest filter
cargo ktstr test --kernel 7.0 -- --retries 2               # nextest retries
cargo ktstr test --kernel 7.0 -- --features integration    # cargo features
cargo ktstr test --relevant                                # only tests my edits affect

Before a supported command runs nextest or builds/probes downstream ktstr test registries, cargo-ktstr uses a Cargo metadata preflight to find direct optional ktstr dependencies and their ktstr-only feature chains. It automatically adds package-qualified selectors such as --features scx_lavd/ktstr-tests. Ordinary commands restrict those selectors to Cargo’s selected/default workspace members. Existing --features are preserved; an explicit Cargo-side --all-features remains authoritative.

This applies to test/nextest, coverage, llvm-cov nextest, replay --exec, verifier, and the shared test-binary build used by export, shell --test, and cargo ktstr affected. Commands that neither run nextest nor build/probe workspace test binaries do no feature inference.

Cargo metadata cannot identify arbitrary source-level cfg expressions. Only ktstr-specific feature chains are inferred; composite gates and transitive optional-helper arrangements remain explicit. Target-specific optional ktstr dependencies are inferred only when they match the effective Cargo target: cargo-ktstr asks rustc for that target’s complete cfg set and evaluates the manifest platform expression before enabling the gate.

A real single-test run against a local kernel tree looks like this. The source tree was already built, so ktstr resolved it to a cache key and reused the image:

$ cargo ktstr test --kernel ../linux --no-perf-mode -- --features integration -E 'test(=ktstr/scx_empty_run_exits_under_watchdog)'
cargo ktstr: resolved kernel "path_linux_b5562c"
cargo ktstr: BTF type anchor at ~/ktstr/target/ktstr_btf_anchor.h
...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 27.24s
────────────
 Nextest run ID d79da4e4-df49-4b40-a415-c1dff2739ce6 with nextest profile: default
    Starting 1 test across 120 binaries (13597 tests skipped)
        PASS [   8.601s] (1/1) ktstr::scx_cleanup_test ktstr/scx_empty_run_exits_under_watchdog
────────────
     Summary [   8.642s] 1 test run: 1 passed, 13597 skipped

cargo ktstr: test outputs
  ~/ktstr/target/ktstr/7.1.0-486fe68-dirty
    (1 stats sidecar(s), 0 wprof trace(s) written this run) — run `cargo ktstr stats last-run` for the gauntlet analysis

The gauntlet analysis (outlier scan, verifier stats, callback profile, KVM stats) is no longer dumped after every run; the footer points at stats last-run, which prints it on demand.

The VM part of this run took about nine seconds; the longer compile line was Rust rebuilding the test harness. For a --kernel <path> source tree, a cache hit is announced on stderr as cargo ktstr: cache hit for {path} ({cache_key}, built {age} ago) and skips the kernel build entirely.

Per-test exit codes

Each #[ktstr_test] process exits with one of three codes, so CI gates and dashboards can triage runs:

CodeVerdictMeaning
0Pass / SkipAssertions passed, or the test never ran (host too small, perf mode unavailable). Skips degrade to pass unless --no-skip-mode.
1FailAn assertion failed; a retryable transient host-resource failure; an operator --cpu-cap the host cannot satisfy; a skip under --no-skip-mode; or expect_err = true and the test passed.
2InconclusiveA zero-denominator ratio gate could not evaluate — the workload produced no signal to ratio against.

Exit code 2 is the silent-pass guard: a Pass at a ≤ threshold gate run against a 0 / 0 ratio that synthesized to 0.0 would have shipped a false-green CI run. The harness records Inconclusive instead (see Checking) and projects it to a distinct exit code so external tooling can route the run separately from real regressions. The values are exported as EXIT_PASS / EXIT_FAIL / EXIT_INCONCLUSIVE in the prelude.

replay

Re-run only the tests that failed in the last session, from the result sidecars under target/ktstr/:

cargo ktstr replay              # print the nextest filter (dry-run)
cargo ktstr replay --exec       # actually run it
cargo ktstr replay -E crash     # narrow by test-name substring
cargo ktstr replay --dir PATH   # source sidecars from an archived tree

Dry-run is the default: the filter prints to stdout so you can inspect it (or paste it into CI) before committing to the re-run. Because sidecars record the registered bare test name, replay selects every currently listed ordinary, host/, ktstr/, and gauntlet/ variant of that name; absent variants simply match nothing. --profile / --nextest-profile apply with --exec. Distinct from auto-repro, which fires inside the failing test process; replay is post-hoc, across a whole session.

coverage

Run tests with coverage via cargo llvm-cov nextest. Same kernel resolution, multi-kernel semantics, and common flags as test; arguments after -- pass through to cargo llvm-cov nextest:

cargo ktstr coverage --kernel ../linux
cargo ktstr coverage -- --workspace --lcov --output-path lcov.info

Requires cargo-llvm-cov and the llvm-tools-preview rustup component. Guest-side coverage is flushed over shared memory at VM exit and merged into the report automatically; multi-kernel runs merge every variant’s profraw into a single report.

Managed cargo ktstr coverage runs merge producer/build-script shards into one cached *.profdata; raw producer shards never enter the stable artifact tree or content cache. After the tests, ktstr merges that seed with the live runtime shards and removes the private raws only after cargo llvm-cov report succeeds. A failed merge/report or an interrupted report retains the available raws and merged profile below the ktstr cache’s coverage-profraw-recovery-v1/ directory and prints the exact recovery path. --no-report keeps cargo-llvm-cov’s ordinary user-owned artifact lifecycle. Host-side files from plain cargo ktstr test also remain next to the cargo-ktstr binary (an LLVM_PROFILE_FILE injection keeps default.profraw out of your kernel source tree; export LLVM_PROFILE_FILE yourself to opt out). To remove retained shards manually:

cargo ktstr llvm-cov clean --profraw-only          # only *.profraw under target/llvm-cov-target/
rm -f target/debug/llvm-cov-target/default-*.profraw
rm -f target/release/llvm-cov-target/default-*.profraw

Avoid bare cargo ktstr llvm-cov clean (wipes reports too) and --workspace (also runs cargo clean).

llvm-cov

Raw passthrough to cargo llvm-cov for subcommands that don’t fit the coverage flow (report, clean, show-env), with the same kernel-resolution plumbing:

cargo ktstr llvm-cov report --lcov --output-path lcov.info
cargo ktstr llvm-cov nextest --workspace

Always pass a subcommand: a bare cargo ktstr llvm-cov falls through to cargo test, which skips gauntlet variants and verifier cells entirely (they exist only under the nextest harness). The explicit nextest form gets targeted optional-ktstr feature inference; every other raw form (test, run, report, clean, show-env, or bare) preserves the supplied feature selection.

kernel

Manage cached kernel images: list, build, clean. The standalone ktstr kernel subcommands are identical.

How kernels are discovered

There are two flows, and they intentionally differ:

Explicit (--kernel ...) — the full pipeline. For a source-tree path: validate it is a kernel tree, look up the cache (clean git trees only — a cache hit short-circuits straight to tests), auto-configure (make defconfig if needed, append ktstr’s kconfig fragment, make olddefconfig), build with make -j$(nproc) KCFLAGS=-Wno-error, validate that the critical config options survived (CONFIG_SCHED_CLASS_EXT, CONFIG_DEBUG_INFO_BTF, CONFIG_BPF_SYSCALL, tracing options — the kernel build system silently drops options with unmet dependencies, and each failure gets a remediation hint), generate compile_commands.json, store the result in the cache, and run nextest with KTSTR_KERNEL (and KTSTR_KERNEL_LIST for multi-kernel) exported. Dirty or non-git trees build in place, get a _dirty kernel label, and never cache. Version, range, and git identifiers download/fetch + build + cache on miss.

Implicit (no --kernel) — discovery only, no builds. The test framework checks KTSTR_TEST_KERNEL, then KTSTR_KERNEL, then the newest valid cache entry, then local build trees (./linux, ../linux, /lib/modules/$(uname -r)/build), then host images (/boot/vmlinuz*). Whatever pre-built image it finds is used as-is — nothing is compiled and nothing new lands in the cache.

Tip

If you iterate on a local kernel tree, pass --kernel ../linux (or run cargo ktstr kernel build --kernel ../linux once) so the build is cached and reused. Implicit discovery will find the tree but never populate the cache for it.

kernel list

List cached kernels, newest first:

$ cargo ktstr kernel list
cache: ~/.cache/ktstr/kernels
  KEY                                              VERSION      SOURCE   ARCH    BUILT
  7.0.14-tarball-x86_64-kcabd40422                 7.0.14       tarball  x86_64  2026-07-04T23:06:12Z
  local-b4dc42d-x86_64-kcabd40422                  7.1.0        local    x86_64  2026-07-04T23:00:04Z
  local-5123e5a-x86_64-cfg1982ad42-kcabd40422      7.1.0        local    x86_64  2026-07-04T22:20:13Z
...
  local-c5d2724-x86_64-kcabd40422                  7.0.0        local    x86_64  2026-07-04T22:05:35Z
...
  7.0.13-tarball-x86_64-kc5f2631f0                 7.0.13       tarball  x86_64  2026-06-22T21:54:56Z (stale kconfig)
  7.1.1-tarball-x86_64-kc5f2631f0                  7.1.1        tarball  x86_64  2026-06-22T21:46:19Z (stale kconfig)
warning: entries marked (stale kconfig) were built against a different ktstr.kconfig. Rebuild with: kernel build --force --kernel <entry version> (add --extra-kconfig PATH if the entry also carries the (extra kconfig) tag).

(stale kconfig) marks entries built against a different ktstr kconfig fragment (they are rebuilt automatically when requested); (EOL) marks entries whose series has left kernel.org’s active releases list. --json emits the same data for scripting. With --kernel START..END, list switches to preview mode: it prints the versions the range expands to without downloading or building anything — the cheap answer to “what does 6.12..6.16 actually cover?”.

kernel build

Download (or use a source tree / git ref), build, and cache one kernel or a range:

cargo ktstr kernel build                        # latest stable from kernel.org
cargo ktstr kernel build --kernel 6.14.2        # specific version
cargo ktstr kernel build --kernel 6.12          # latest 6.12.x patch release
cargo ktstr kernel build --kernel 6.11..6.14    # every release in the range
cargo ktstr kernel build --kernel ../linux      # local source tree
cargo ktstr kernel build --kernel gke           # official GKE COS kernel + exact-source modules
cargo ktstr kernel build --kernel gke-129       # newest promoted revision in COS milestone 129
cargo ktstr kernel build --force --kernel 6.14.2

With no --kernel, it builds the latest stable series that has had at least 8 maintenance releases — keeping CI off brand-new majors whose early builds are more likely to break. Already-cached entries are skipped unless --force.

FlagDescription
--forceRebuild even if cached.
--cleanmake mrproper first (source-tree builds only).
--cpu-cap NReserve exactly N host CPUs for the build (build parallelism and a cgroup sandbox follow). See Resource Budget.
--extra-kconfig PATHExtra kconfig fragment merged over ktstr’s (user values win); lands in its own cache slot.
--skip-sha256Skip tarball checksum verification (emits a warning).
--include-eolWith a range, also build EOL series from the linux-stable mirror.

A bare relative name is read as a cache key — prefix a relative source directory with ./.

kernel clean

cargo ktstr kernel clean --keep 3                 # keep the 3 most recent valid entries
cargo ktstr kernel clean --corrupt-only --force   # drop only broken entries

Corrupt entries (missing metadata or image) never consume a --keep slot. --force skips the confirmation prompt (required non-interactively).

verifier

Sweep every declare_scheduler!-registered scheduler through the real kernel’s BPF verifier across topologies, checking verify + attach + dispatch per cell. Takes the same --kernel grammar as test — repeatable, ranges (6.12..6.14, with --include-eol for end-of-life series), paths, cache keys, git refs:

cargo ktstr verifier --kernel 7.0
cargo ktstr verifier --scheduler scx-ktstr     # one scheduler
cargo ktstr verifier --raw                     # no cycle collapse

Unlike ordinary test commands, a bare/unscoped verifier recursively widens its Cargo package selection to every compatible workspace package. Explicit package selectors remain scoped to the requested packages. The verifier reuses the targeted feature inference described under test, while excluding packages linked to an older ktstr that this verifier dispatcher cannot enumerate.

See BPF Verifier Sweep for the cell model, real output, and the kernels-filter contract.

shell

Boot any kernel ktstr can resolve and land in a busybox shell inside the VM — the fastest way to poke at a kernel you just built, check what a guest actually exposes, or reproduce a failing test’s environment by hand:

cargo ktstr shell --test my_failing_test     # boot the exact VM a test gets
cargo ktstr shell --kernel ../linux          # local tree
cargo ktstr shell --kernel ./arch/x86/boot/bzImage

--test NAME derives topology, memory, and include files from a registered #[ktstr_test] (mutually exclusive with --topology / --memory-mib; -i is additive) — so “why does this only fail on 9cpu-3llc-nosmt” becomes an interactive session in that exact machine.

For scripted checks, --exec runs a command and exits with its status:

$ cargo ktstr shell --kernel ../linux --no-perf-mode \
    --exec 'uname -r; cat /sys/kernel/sched_ext/state; nproc'
7.1.0-00001-g8cd2b47fb188
disabled
1

The standalone ktstr shell is the same boot flow minus the cargo integration (--test, raw image paths).

export

Export a registered test as a self-extracting .run file that reproduces the scenario on bare metal, no VM: the ktstr binary, the scheduler binary, and every declared include file, embedded in a bash preamble that validates root, sched_ext support, cgroup2, the no-other-scheduler-attached invariant, and topology compatibility before launching.

cargo ktstr export my_test -o /tmp/my_test.run
wrote /tmp/sched_basic_proportional.run (90074903 bytes archive, 0 include files)

The generated script opens with the frozen test specification and the preflight checks:

#!/bin/bash
# Generated by `cargo ktstr export`. Do not edit; regenerate to update.
set -euo pipefail

# --- frozen test specification ---
KTSTR_TEST_NAME=sched_basic_proportional
KTSTR_SCHED_NAME=ktstr_sched
KTSTR_GIT_HASH=73730e0
NEED_LLCS=1
NEED_CORES_PER_LLC=2
NEED_THREADS_PER_CORE=1
NEED_NUMA_NODES=1
TEST_DURATION_SECS=12
TEST_WATCHDOG_SECS=15

Scheduler choice, scheduler args, and topology are frozen at export time; --duration, --watchdog-timeout, and --quiet can be overridden at .run invocation. --package NAME scopes the workspace search (and disambiguates duplicate test names — otherwise the first matching binary in path order wins); --release builds the embedded binaries with the release profile to match how you will run them.

Not exportable (rejected with actionable errors): host_only tests, tests using bpf_map_write (they need the host-side probe surface), and KernelBuiltin schedulers.

completions

cargo ktstr completions bash >> ~/.local/share/bash-completion/completions/cargo
cargo ktstr completions zsh > ~/.zfunc/_cargo-ktstr
cargo ktstr completions fish > ~/.config/fish/completions/cargo-ktstr.fish

SHELL is one of bash, zsh, fish, elvish, powershell; --binary overrides the completion target (default cargo).

stats

Sidecar analysis for past runs: stats last-run (the gauntlet analysis — outlier scan, verifier stats, callback profile, KVM stats — of the newest run), stats list (run table), stats list-metrics (the regression metric registry), stats list-values (distinct filter values in the pool), stats show-host --run ID (archived host context), and stats explain-sidecar --run ID (why optional fields are absent). Bare cargo ktstr stats prints this subcommand list.

cargo ktstr stats last-run                 # newest run's analysis
cargo ktstr stats last-run --kernel 7.1.3  # newest run for that kernel
cargo ktstr stats last-run --dir PATH      # a specific run dir / archive

last-run resolves its source dir in precedence order: --dir, then KTSTR_SIDECAR_DIR, then --kernel (newest run dir whose leaf begins {LABEL}-), then the most-recently-modified run dir. cargo ktstr show-host prints the same host context live, and cargo ktstr show-thresholds TEST prints the merged assertion thresholds a test will run with. See Runs and Regression Gates.

perf-delta

Compare performance_mode test metrics between HEAD and a baseline commit, exiting non-zero when enough metrics regress to trip the gate:

cargo ktstr perf-delta --noise-adjust 5 --kernel 6.14

Baseline resolution, the --noise-adjust statistics, and the full flag set are documented in Runs and Regression Gates; the worked walkthrough is A/B Compare Branches.

affected

Emit the scheduler packages a base..HEAD diff affects, as a JSON array for a GitHub Actions dynamic matrix (["scx_lavd","scx_rusty"]). Attribution is fail-safe: any uncertainty widens to the full set, and only a strictly docs-only change emits []. --relevant is the same engine applied locally to narrow test / coverage / perf-delta runs. Both are documented with the complete CI workflow in CI.

locks

Enumerate every ktstr flock held on this host — read-only, never acquires anything. When a build or test is blocked behind a peer’s reservation, cargo ktstr locks names the peer without disturbing it:

$ ktstr locks
LLC locks:
 LLC  NODE  LOCKFILE               HOLDERS
 0    0     /tmp/ktstr-llc-0.lock  <none recorded>
 1    0     /tmp/ktstr-llc-1.lock  <none recorded>
...

Run-dir locks:
 RUN KEY               LOCKFILE                                       HOLDERS
 7.0.14-73730e0-dirty  target/ktstr/.locks/7.0.14-73730e0-dirty.lock  <none recorded>

An idle host shows <none recorded>; while a lock is held, the HOLDERS column names the holder’s PID and cmdline (cross-referenced against /proc/locks). Four lock-file roots are scanned:

  • {KTSTR_LOCK_DIR}/ktstr-llc-*.lock (default /tmp) — per-LLC reservations held by perf-mode test runs and --cpu-cap-bounded builds.
  • {KTSTR_LOCK_DIR}/ktstr-cpu-*.lock — per-CPU reservations from the same flow.
  • {cache_root}/.locks/*.lock — kernel-cache entry locks held during kernel build writes, plus per-source-tree locks held while building from a path.
  • {runs_root}/.locks/{kernel}-{project_commit}.lock — sidecar write locks serializing concurrent runs targeting the same run directory.
FlagDefaultDescription
--jsonoffJSON snapshot (pretty in one-shot mode; ndjson under --watch).
--watch DURATIONunsetRedraw at the interval until SIGINT (100ms, 1s, 5m).

Available identically as the standalone ktstr locks. The reservation model behind these locks is documented in Resource Budget.

The standalone ktstr binary

cargo install ktstr also installs a bare ktstr — the same engine without the cargo front-end. It is the guest-init binary (it runs as PID 1 inside every test VM) that doubles as a host CLI for the tools that don’t need a cargo project: ktstr topo, ktstr locks, ktstr ctprof, ktstr kernel, and ktstr shell. Each mirrors its cargo ktstr counterpart.

Reach for it when you are debugging outside a cargo project — poking at a kernel you just built, or inspecting a host with no ktstr workspace checked out. The common case is ktstr shell, an interactive KVM VM on any kernel ktstr can resolve:

ktstr shell --kernel ../linux
ktstr shell --kernel 6.14.2 --topology 1,2,4,1
ktstr shell -i ./my-binary -i strace --exec 'cat /proc/schedstat'

Files passed with -i land at /include-files/<name> inside the guest (directories walked recursively, dynamically-linked ELF binaries get their shared-library closure resolved). It takes the same --kernel grammar as cargo ktstr test, plus --topology N,L,C,T, --memory-mib, --disk, --dmesg, and --exec.

cargo ktstr is the primary interface: it resolves kernels from your workspace, accepts raw kernel-image paths, and adds --test NAME to reuse a registered test’s exact VM shape. Drop to ktstr only when you are not inside a project.