Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ktstr

Test Linux schedulers like code. Declare the topology, boot a real kernel in KVM, run real workloads, and let host-side checks explain what happened.

Real kernelsreleased, local, or git-built
Topology as codeNUMA, LLCs, cores, SMT
Actionable failuresstats, dumps, auto-repro

Scheduler bugs hide in topology: the fairness regression that only shows up on an odd LLC count, the no-progress case that needs SMT siblings, the crash that wants a NUMA crossing. ktstr turns those cases into ordinary Rust tests for sched_ext schedulers: the test declares the machine shape, the VM boots with that shape, and the host watches without instrumenting the guest.

New to ktstr?

Install the CLI, build a guest kernel, and run your first green test.

Already have a scheduler?

Register an scx_* binary and put it under test quickly.

Debugging a failure?

Read the verdict, timeline, dumps, and auto-repro output in order.

Quick taste

use ktstr::prelude::*;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_mine",
});

#[ktstr_test(scheduler = MY_SCHED, llcs = 1, cores = 2, threads = 1)]
fn steady_under_my_sched(ctx: &Ctx) -> Result<AssertResult> {
    scenarios::steady(ctx)
}

Run it against any target kernel. A path points at a local kernel tree; a version such as 7.0 points at ktstr’s kernel cache:

cargo ktstr test --kernel ../linux --no-perf-mode
cargo ktstr test --kernel ../linux --no-perf-mode
cargo ktstr: resolved kernel "path_linux_b5562c"
cargo ktstr: BTF type anchor at target/ktstr_btf_anchor.h
...
 Nextest run ID d79da4e4-… with nextest profile: default
    Starting 1 test across 120 binaries (13597 tests skipped)
        PASS [   8.601s] (1/1) ktstr::scx_cleanup_test ktstr/scx_empty_run_exits_under_watchdog
...
     Summary [   8.642s] 1 test run: 1 passed, 13597 skipped

Without a scheduler attribute, tests run under the kernel’s default scheduler (EEVDF) — useful for baselines and A/B comparisons.

Mental model
Describe the machine

llcs, cores, threads, NUMA, and gauntlet constraints become the VM's CPU topology.

Run real work

Scenarios create cgroups, cpusets, workers, benchmarks, and mid-run operations inside the guest.

Check behavior

Opt-in assertions judge worker progress, spread, stuck gaps, throughput, temporal patterns, and regressions.

When it breaks, you see why

A crash log tells you where the scheduler died. ktstr also tells you what the state looked like on the way there: on a crash it boots a second VM, attaches BPF probes along the crash path, and reruns the scenario. Each probed function prints decoded struct fields; → marks fields that changed between entry and exit:

cargo ktstr test — auto-repro output after a scheduler crash
=== AUTO-PROBE: scx_exit fired ===

  ktstr_select_cpu                                              main.bpf.c:380
    task_struct *p
      pid                 40
      cpus_ptr            0xf(0-3)
      dsq_id              SCX_DSQ_INVALID  →  SCX_DSQ_LOCAL
      slice               19982063         →  20000000
      weight              100
      scx_flags           RESET_RUNNABLE_AT|DEQD_FOR_SLEEP|ENABLED
  enqueue_task_scx                                              kernel/sched/ext.c
    rq *rq
      cpu                 0
    task_struct *p
      pid                 40
      dsq_id              SCX_DSQ_LOCAL
      scx_flags           QUEUED|DEQD_FOR_SLEEP|ENABLED
  ...
  bpf_prog_9a11f2edaac0b52f_ktstr_dispatch+0x57/0x1db

Auto-repro is on by default and selects the kernel’s native scheduler-exit hook (sched_ext_exit on the newest kernels, a raw scx_vexit entry/return pair on the preceding generation, or scx_dump_state on global-era kernels) — see Auto-Repro. For the anatomy of ordinary failures (stats, timeline, monitor verdict), see Reading Failure Output.

Real kernels under KVM

Each test gets a fresh micro-VM booting the exact kernel you target. Real cgroups, real BPF, no shared state.

How it works →

Topology as code

NUMA nodes, LLCs, cores, SMT — declared on the test attribute, realized in the guest down to the ACPI tables.

Topology →

Gauntlet

One test declaration fans out across a matrix of topology presets — odd LLC counts, SMT, NUMA crossings — with budget-aware selection for CI.

Gauntlet →

Auto-repro

Crashes rerun themselves in a probe VM that captures function arguments and struct state along the crash path.

Auto-Repro →

Design

Fidelity without overhead. Every test boots a real Linux kernel in a KVM VM with real cgroups and real BPF programs — no mocking, no containers, no state carried between tests. The VMM is purpose-built for this job; see VMM.

Direct access over tooling layers. The host-side monitor reads guest memory through BTF-resolved struct offsets — runqueues, DSQ depths, schedstat counters — loading nothing into the guest, so observation does not perturb the scheduler under test. See Monitor.

What it tests

  • Fair scheduling — workers get CPU time without zero-work outliers or excessive scheduling gaps.
  • Cpuset isolation — workers stay on assigned CPUs.
  • Dynamic operations — cgroups created, destroyed, and resized mid-run.
  • Affinity — the scheduler respects thread affinity constraints.
  • Stress — many cgroups, many workers, rapid topology changes.
  • Stuck-task detection — the scheduler doesn’t leave runnable tasks unrun.

Note

ktstr is pre-release. 0.x APIs change between releases, so pin the exact version — Getting Started shows how.

Next steps