ktstr
Test Linux schedulers like code. Declare the topology, boot a real kernel in KVM, run real workloads, and let host-side checks explain what happened.
Scheduler bugs hide in topology: the fairness regression that only shows up on an odd LLC count, the no-progress case that needs SMT siblings, the crash that wants a NUMA crossing. ktstr turns those cases into ordinary Rust tests for sched_ext schedulers: the test declares the machine shape, the VM boots with that shape, and the host watches without instrumenting the guest.
Install the CLI, build a guest kernel, and run your first green test.
Register an scx_* binary and put it under test quickly.
Read the verdict, timeline, dumps, and auto-repro output in order.
Quick taste
use ktstr::prelude::*;
declare_scheduler!(MY_SCHED, {
name = "my_sched",
binary = "scx_mine",
});
#[ktstr_test(scheduler = MY_SCHED, llcs = 1, cores = 2, threads = 1)]
fn steady_under_my_sched(ctx: &Ctx) -> Result<AssertResult> {
scenarios::steady(ctx)
}
Run it against any target kernel. A path points at a local kernel tree;
a version such as 7.0 points at ktstr’s kernel cache:
cargo ktstr test --kernel ../linux --no-perf-mode
cargo ktstr: resolved kernel "path_linux_b5562c" cargo ktstr: BTF type anchor at target/ktstr_btf_anchor.h ... Nextest run ID d79da4e4-… with nextest profile: default Starting 1 test across 120 binaries (13597 tests skipped) PASS [ 8.601s] (1/1) ktstr::scx_cleanup_test ktstr/scx_empty_run_exits_under_watchdog ... Summary [ 8.642s] 1 test run: 1 passed, 13597 skipped
Without a scheduler attribute, tests run under the kernel’s default
scheduler (EEVDF) — useful for baselines and A/B comparisons.
llcs, cores, threads, NUMA, and gauntlet constraints become the VM's CPU topology.
Scenarios create cgroups, cpusets, workers, benchmarks, and mid-run operations inside the guest.
Opt-in assertions judge worker progress, spread, stuck gaps, throughput, temporal patterns, and regressions.
When it breaks, you see why
A crash log tells you where the scheduler died. ktstr also tells you what
the state looked like on the way there: on a crash it boots a second VM,
attaches BPF probes along the crash path, and reruns the scenario. Each
probed function prints decoded struct fields; → marks fields that
changed between entry and exit:
=== AUTO-PROBE: scx_exit fired === ktstr_select_cpu main.bpf.c:380 task_struct *p pid 40 cpus_ptr 0xf(0-3) dsq_id SCX_DSQ_INVALID → SCX_DSQ_LOCAL slice 19982063 → 20000000 weight 100 scx_flags RESET_RUNNABLE_AT|DEQD_FOR_SLEEP|ENABLED enqueue_task_scx kernel/sched/ext.c rq *rq cpu 0 task_struct *p pid 40 dsq_id SCX_DSQ_LOCAL scx_flags QUEUED|DEQD_FOR_SLEEP|ENABLED ... bpf_prog_9a11f2edaac0b52f_ktstr_dispatch+0x57/0x1db
Auto-repro is on by default and selects the kernel’s native scheduler-exit
hook (sched_ext_exit on the newest kernels, a raw scx_vexit entry/return
pair on the preceding generation, or scx_dump_state on global-era kernels) —
see Auto-Repro. For the
anatomy of ordinary failures (stats, timeline, monitor verdict), see
Reading Failure Output.
Real kernels under KVM
Each test gets a fresh micro-VM booting the exact kernel you target. Real cgroups, real BPF, no shared state.
How it works →Topology as code
NUMA nodes, LLCs, cores, SMT — declared on the test attribute, realized in the guest down to the ACPI tables.
Topology →Gauntlet
One test declaration fans out across a matrix of topology presets — odd LLC counts, SMT, NUMA crossings — with budget-aware selection for CI.
Gauntlet →Auto-repro
Crashes rerun themselves in a probe VM that captures function arguments and struct state along the crash path.
Auto-Repro →Design
Fidelity without overhead. Every test boots a real Linux kernel in a KVM VM with real cgroups and real BPF programs — no mocking, no containers, no state carried between tests. The VMM is purpose-built for this job; see VMM.
Direct access over tooling layers. The host-side monitor reads guest memory through BTF-resolved struct offsets — runqueues, DSQ depths, schedstat counters — loading nothing into the guest, so observation does not perturb the scheduler under test. See Monitor.
What it tests
- Fair scheduling — workers get CPU time without zero-work outliers or excessive scheduling gaps.
- Cpuset isolation — workers stay on assigned CPUs.
- Dynamic operations — cgroups created, destroyed, and resized mid-run.
- Affinity — the scheduler respects thread affinity constraints.
- Stress — many cgroups, many workers, rapid topology changes.
- Stuck-task detection — the scheduler doesn’t leave runnable tasks unrun.
Note
ktstr is pre-release. 0.x APIs change between releases, so pin the exact version — Getting Started shows how.
Next steps
- ktstr in Action — the full feature tour, with real output.
- Getting Started — install, build a kernel, first green test.
- Tutorial: Zero to ktstr — build up a real test suite step by step.
- Test a New Scheduler — already have a scheduler? Start here.