Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Writing Tests

Tests are Rust functions annotated with #[ktstr_test]. Each test boots a KVM VM, runs the scenario inside it, and evaluates results on the host.

Attribute shape

Topology, timing, checks, snapshots, scheduler selection, and execution knobs.

Scheduler setup

Declare an scx_* binary once, then reuse it across tests and verifier sweeps.

Assertions

Checks are opt-in: choose worker progress, spread, stuck gaps, throughput, and temporal gates deliberately.

use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        ctx.cgroup_def("cg_0"),
        ctx.cgroup_def("cg_1"),
    ])
}

ctx.cgroup_def("name") is shorthand for CgroupDef::named("name").workers(ctx.workers_per_cgroup) — the common case. Use CgroupDef::named(...).workers(N).work_type(...) directly when the test needs to customize worker count or work type.

Run with cargo ktstr test --kernel ../linux or a released kernel version such as 7.0 (see Getting Started for setup). A passing run is one nextest line per test; the VM boot, scenario, and teardown all happen inside the reported duration — Getting Started shows a full transcript.

Every test gets the same machinery for free: a fresh VM per test (no state shared between tests), a failure dump with BTF-rendered scheduler BPF state if the scheduler crashes (see Reading Failure Output), and an automatic second-VM reproduction run with probes attached (Auto-Repro). Each test also expands into gauntlet variants across topology presets — see Gauntlet.

Warning

No worker checks run by default. The example above passes as long as nothing crashes — it does not assert worker progress, fairness, or stuck gaps. Opt in with not_stuck = true and the threshold attributes; see Checking for the model.

Scenarios

A scenario is the scheduling condition a test creates — which cgroups exist, which CPUs they may use, what their workers do, and what changes mid-run. The canned scenarios in scenarios::* exist so those conditions have names: scenarios::steady(ctx) produces the same reproducible condition against every scheduler you point it at, which is what makes results comparable across schedulers and commits.

Start simple

steady proves the scheduler survives ordinary balanced load.

Add movement

Cpuset, affinity, and cgroup-change scenarios exercise dynamic callbacks.

Mix signals

Specialized scenarios combine workload types when one primitive is too narrow.

Scenarios graduate in three stages; move down only when the stage above can’t express the condition:

  1. Canned — call a scenarios::* function. Zero setup, named, comparable.

    // Baseline: run the canned steady scenario under EEVDF.
    #[ktstr_test(llcs = 1, cores = 2, threads = 1)]
    fn baseline(ctx: &Ctx) -> Result<AssertResult> {
        scenarios::steady(ctx)
    }
    
    // Scheduler smoke test: crash/watchdog failures plus worker-progress checks.
    #[ktstr_test(scheduler = MY_SCHED, llcs = 2, cores = 2, threads = 1, not_stuck = true)]
    fn scheduler_smoke(ctx: &Ctx) -> Result<AssertResult> {
        scenarios::steady(ctx)
    }
  2. Your own cgroup layout — execute_defs(ctx, vec![...]) with CgroupDefs you declare: your worker counts, work types, cpusets, still one static phase.

    // Two cgroups pinned to different LLCs.
    #[ktstr_test(scheduler = MY_SCHED, llcs = 2, cores = 2, threads = 1)]
    fn split_llc(ctx: &Ctx) -> Result<AssertResult> {
        execute_defs(ctx, vec![
            CgroupDef::named("left").cpuset(CpusetSpec::Llc(0)),
            CgroupDef::named("right").cpuset(CpusetSpec::Llc(1)),
        ])
    }
  3. Steps — execute_steps / execute_scenario with Steps and Ops for anything that changes mid-run (cpuset swaps, scheduler replacement, snapshots, kernel-memory reads), plus a Backdrop for state that must outlive the steps.

Ops, Steps, and Backdrop documents stage 2 and 3 — the CgroupDef builder, every Op, and all the execute_* entry points. A custom scenario is just the #[ktstr_test] function body itself; Custom Scenarios covers writing bodies that go beyond Steps entirely. Start with the smallest shape that can fail for the behavior you care about — add topology, workers, snapshots, and temporal assertions only when they give you a new signal; otherwise they just make the run slower and the failure harder to read.

The canned catalog

FunctionCondition testedSetup
steadyBaseline fairness2 cgroups, no cpusets, equal CPU-spin load
steady_llcLLC-boundary scheduling2 cgroups on different LLCs (skips on 1-LLC topologies)
oversubscribedDispatch under oversubscription2 cgroups, 32 mixed workers each
cpuset_applyCpuset assignment on running tasksDisjoint cpusets applied mid-run
cpuset_clearCpuset removal on confined tasksCpusets cleared mid-run
cpuset_resizeCpuset resizing adaptationCpusets shrink then grow
cgroup_addScheduler reaction to a new cgroupCgroups created while others run
cgroup_removeScheduler reaction to cgroup removalCgroups torn down while others run
affinity_changeAffinity mask changesWorker affinities randomized mid-run
affinity_pinnedNarrow-affinity contentionWorkers pinned to a 2-CPU subset
host_contentionCgroup vs host-task fairnessRoot-cgroup workers beside managed cgroups
mixed_workloadsMixed workload fairnessHeavy + bursty + IO cgroups
nested_steadyNested cgroup hierarchyWorkers in nested sub-cgroups
nested_task_moveCross-level task migrationTasks moved between nested cgroups

More specialized custom_* functions live in the ktstr::scenario::{affinity, basic, cpuset, dynamic, interaction, nested, performance, stress} modules — see the API docs.

Against a new scheduler, run steady first — it is the smallest condition that can fail (two cgroups, spin load, nothing dynamic). Then steady_llc on a 2-LLC topology to see cache-boundary placement, then mixed_workloads and oversubscribed for load diversity. The dynamic scenarios (cpuset_*, cgroup_*, affinity_*) each isolate one reconfiguration path; reach for the one matching the code you changed.

How a scenario runs

A scenario body does not pick its own duration or topology — the #[ktstr_test] attribute does. The workload runs for the test’s duration_s (see the macro reference), on the topology the attribute declares, and a gauntlet run re-executes the same body across a whole topology matrix. Worker counts and cpusets come from the scenario’s own CgroupDefs.

Every scenario ends the same way: worker reports are collected and the opted-in checks run against them. A run’s stats roll-up looks like:

--- stats ---
2 workers, 4 cpus, 2 migrations, worst_spread=0.0%, worst_gap=21ms
  cg0: workers=1 cpus=2 spread=0.0% gap=10ms migrations=1 iter=209600
  cg1: workers=1 cpus=2 spread=0.0% gap=21ms migrations=1 iter=189252

Reading Failure Output walks the full anatomy.

Where to go next

  • The #[ktstr_test] Attribute — the full attribute reference: topology, timing, checking thresholds, execution knobs.
  • Scheduler Definitions — declare_scheduler!: how the scheduler under test is named, found, configured, and launched.
  • Payloads and Included Files — run benchmark binaries (schbench, fio, …) alongside workers and extract their metrics.
  • Custom Scenarios — scenario logic the ops system cannot express, written directly in the test body.
  • Snapshots and Live Capture — freeze and read scheduler BPF state on demand, when the kernel writes a chosen symbol, or on a cadence across the workload window.
  • Projections and Temporal Assertions — turn your scheduler’s own counters (scx_stats, BPF state, host timelines) into test signals, and assert on trajectories: counters that only advance, metrics that hold steady, systems that converge.