Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Checking

ktstr judges scheduler behavior through two channels: worker-side telemetry (every worker process reports what happened to it) and host-side monitoring (the monitor reads guest kernel state from outside). Both channels always measure; nothing asserts until the test opts in — a test with no checking attributes passes as long as the VM boots and the scenario completes.

Which API to reach for:

  • #[ktstr_test] attributes — cover most tests: not_stuck, max_gap_ms, max_spread_pct, min_iteration_rate, and every other threshold below has an attribute (see the macro reference).
  • Verdict + claim! — labeled assertions on values you compute inside a custom scenario body.
  • AbsoluteThresholds — a one-call multi-field bound check against collected reports, bypassing the config merge.
  • assert_scx_events_clean — bounds on SCX event counters (“no fallbacks fired”).

Worker checks

After each scenario, ktstr collects a WorkerReport from every worker and runs the opted-in checks against them:

  • Zero work units (not_stuck) — any worker with no measured work fails: tid N made no progress (0 work units).
  • Scheduling gaps (max_gap_ms) — the longest wall-clock gap observed at work-unit checkpoints. A violation renders as tid N stuck Xms on cpuY at +Zms (threshold Nms).
  • Fairness (max_spread_pct) — workers in one cgroup should get similar CPU time; the spread (max off-CPU% − min off-CPU%) must stay below the bound.
  • Cpuset isolation (isolation) — workers may only run on CPUs in their assigned cpuset; any excursion fails.
  • Throughput — max_throughput_cv bounds the coefficient of variation of per-worker work rate (some workers quietly slower); min_work_rate sets an absolute floor (all workers equally slow).
  • Benchmarking — max_p99_wake_latency_ns and max_wake_latency_cv bound wake-to-run latency for work types that block and measure it (see Work Types for which do); min_iteration_rate floors outer-loop iterations per delivered CPU-second per worker.

The loop, end to end

A test sets a threshold, the run violates it, the failure output names the check, the value, and the bound:

#[ktstr_test(
    scheduler = MY_SCHED,
    llcs = 1, cores = 2, threads = 1,
    min_iteration_rate = 50_000_000.0,  // deliberately unreachable floor
)]
fn throughput_gate(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("cg_a").workers(1).cpuset(CpusetSpec::disjoint(0, 2)),
        CgroupDef::named("cg_b").workers(1).cpuset(CpusetSpec::disjoint(1, 2)),
    ])
}
<!-- captured: cargo ktstr test --kernel 7.0 (throughput_gate demo test) | ktstr 0.23.0 | kernel 7.0.14 -->
cargo ktstr test --kernel 7.0
ktstr_test 'throughput_gate' [sched=scx-ktstr] [topo=1n1l2c1t] failed:
  worker 71 iteration rate 41903.3/s below floor 50000000.0/s
  worker 73 iteration rate 37834.5/s below floor 50000000.0/s

--- stats ---
2 workers, 4 cpus, 2 migrations, worst_spread=0.0%, worst_gap=21ms
  cg0: workers=1 cpus=2 spread=0.0% gap=10ms migrations=1 iter=209600
  cg1: workers=1 cpus=2 spread=0.0% gap=21ms migrations=1 iter=189252
...
--- monitor ---
samples=41 max_imbalance=2.00 max_dsq_depth=0 stuck=0
avg: imbalance=1.32 nr_running/cpu=1.2 dsq/cpu=0.0
events: fallback=0 (0.0/vcpu-s) keep_last=210 (52.5/vcpu-s) offline=0
...
verdict: monitor OK

Both channels report: the worker check that tripped, and the monitor verdict that did not. The full failure anatomy — timeline, scheduler log, dump sections — is in Reading Failure Output.

Monitor checks

The host-side monitor samples guest per-CPU runqueue state (via BTF offsets, no guest instrumentation) roughly every 100ms and evaluates:

  • Imbalance ratio — max(nr_running) / max(1, min(nr_running)) across CPUs.
  • Local DSQ depth — per-CPU dispatch queue depth.
  • Stuck-task detection — rq_clock not advancing on a CPU with runnable tasks; idle CPUs and preempted vCPUs are exempt.
  • Event rates — select_cpu_fallback and dispatch_keep_last counters per second.

Monitor violations always land in the failure report’s --- monitor --- section, but they flip the test result only when the test enforces them — set the corresponding attributes, call .with_monitor_defaults() on an Assert, or set enforce_monitor_thresholds. A monitor that produced no usable signal (empty samples, uninitialized guest memory) reports inconclusive, never a silent pass — a CI gate can always tell “verified OK” from “never measured”.

The defaults with_monitor_defaults() applies:

ThresholdDefaultRationale
max_imbalance_ratio4.0max(nr_running) / max(1, min(nr_running)) across CPUs (denominator clamped so an all-idle sample does not divide by zero). Lower values (2-3) false-positive during cpuset transitions.
max_local_dsq_depth50Per-CPU dispatch queue overflow. Sustained depth above this means the scheduler is not consuming dispatched tasks.
fail_on_rq_clock_stucktrueFail when rq_clock does not advance on a CPU with runnable tasks. Idle CPUs (NOHZ) and preempted vCPUs are exempt.
sustained_samples5At ~100ms sample interval, requires ~500ms of sustained violation. Filters transient spikes from cpuset reconfiguration.
max_fallback_rate200.0/vCPU-sselect_cpu_fallback events across all CPUs per delivered average-vCPU CPU-second. Sustained activity indicates systematic select_cpu failure.
max_keep_last_rate100.0/vCPU-sdispatch_keep_last events across all CPUs per delivered average-vCPU CPU-second. Sustained activity indicates the scheduler keeps reusing the previous dispatch target instead of making progress through the normal path.

Every monitor threshold uses the sustained_samples window — a violation must persist for N consecutive samples before it counts.

NUMA checks

For workers with a MemPolicy, three thresholds gate page placement:

  • min_page_locality — minimum fraction of pages on the expected NUMA nodes (the cgroup’s cpuset nodes, derived at evaluation time). Zero observed pages counts as zero locality, not a vacuous pass.
  • max_cross_node_migration_ratio — bound on migrated pages relative to allocated pages (from /proc/vmstat deltas).
  • max_slow_tier_ratio — bound on the fraction of pages landing on memory-only (CXL-tier) nodes.

Default thresholds

not_stuck = true also enables the built-in fairness and stuck-gap checks at these defaults:

CheckReleaseDebug
Scheduling gap2000 ms3000 ms
Fairness spread15%35%

Debug builds run with higher scheduling overhead, so thresholds are relaxed.

How configuration merges

Assert is the threshold-config struct; every field is an Option where None means “inherit”. Three layers merge, last-Some wins: the baseline (all None), then the scheduler’s assert, then the per-test attributes — so a scheduler-wide bound applies to every test and any single test can override or disable it. enforce_monitor_thresholds is the one sticky field: once any layer sets it, it stays set. Worked override recipes live in Customize Checking.

execute_steps_with(ctx, steps, Some(&assert)) bypasses the merged config with an explicit Assert for that scenario’s worker checks.

baseline every field None — inherit override scheduler's assert scheduler-wide bound override per-test #[ktstr_test] attributes max_gap_ms, max_spread_pct, … merged Assert → worker checks last Some wins execute_steps_with (ctx, steps, Some(&assert)) explicit Assert bypasses the merge enforce_monitor_thresholds is sticky: once any layer sets it, it stays set.

Verdicts and outcomes

Every assertion produces one of four outcomes, and a result’s terminal verdict is the fold over all of them, most severe first: Fail > Inconclusive > Pass > Skip.

OutcomeMeaning
Passthe assertion ran and the value satisfied the bound
Failthe assertion ran and the value violated the bound
Inconclusivethe assertion ran but had no signal to evaluate
Skipthe scenario couldn’t run (unmet precondition)
terminal verdict = fold over all outcomes (most severe wins) Fail > Inconclusive > Pass > Skip Fail ran · violated bound Inconclusive ran · no signal Pass ran · satisfied bound Skip couldn't run · precondition expect_err · clean pass → 1 allow_inconclusive → 0 exit 1 · Fail exit 2 · Inconclusive exit 0 · Pass / Skip --no-skip-mode: Skip → exit 1

Inconclusive exists for instrument-derived denominators — a ratio whose denominator (iterations, samples, wall-clock interval) legitimately reached zero because the workload produced no signal. Policy-derived denominators stay Fail on zero: under MemPolicy::Bind the policy says pages will exist, so their absence is a defect, not “couldn’t measure”.

CI gates read the verdict through four accessors:

if r.is_pass() { /* ship */ }
if r.is_fail() { /* block; surface r.failure_details() */ }
if r.is_skip() || r.is_inconclusive() { /* no verdict — triage */ }

is_pass() is deliberately strict: inconclusive and all-skip both read false.

Wall-latency ceilings under host contention

max_p99_wake_latency_ns measures a guest-wall interval — the time a request waited to run, as the guest saw it. There is no steal-adjusted guest clock that subtracts it out, so host-side scheduling delay during the measurement (a noisy neighbour starving the vCPU threads) inflates the reading directly. A plain measured <= T gate would blame the scheduler under test for the host’s noise.

So the wall-latency verdict is tri-state. Each run carries a contention witness over its Body (measurement) phase — host vCPU dilation D from lifecycle-boundary schedstat snapshots and a peak-window bound W(L) from ktstr’s host-cgroup CPU-pressure clock. CPU PSI some is the wall time during which at least one runnable task in that cgroup was stalled for CPU. It therefore sees a delayed vCPU even when the competitor is outside the cgroup, without charging arbitrary pressure elsewhere on a wide host. Other tasks sharing the runner cgroup can only bias the bound upward, toward indeterminate. ktstr removes that cross-cell noise by capping the localized PSI result with the target VM’s complete summed-vCPU run delay over the same conservative Body span. Both are upper bounds on one request’s host scheduling contamination, so taking their minimum is safe; if PSI is unavailable, the task-specific cap becomes a coarser whole-span fallback. The gate fires in the guest; the host re-judges it with the witness in hand:

VerdictConditionResult
Passmeasured <= Talways sound — contention only ever inflates a wall interval
Fail (confirmed)excess over T exceeds W(measured)no witnessed host contention explains it — a real failure at any load; the message gains a contention-checked note
Indeterminateexcess within W(measured)the failure might be host contention — non-blocking (indeterminate is a pass), surfaced as a contention-indeterminate note carrying measured / threshold / excess / W / D

An indeterminate demotion drops the failing outcome; if it was the only failure the whole result flips to Pass, and the annotation rides the notes channel so it stays visible in the passing run’s output and its sidecar. If the Body contention series saturated (it hit its interval cap and only a prefix survives), W is a lower bound, so the verdict never confirms on it — it is treated as indeterminate with a saturation note.

For an expect_err negative test, a run whose only expected failures became contention-indeterminate is a Skip, not a clean Pass: the environment prevented the harness from proving whether the expected failure remains. A confirmed failure still satisfies expect_err, an unrelated failure remains blocking, and --no-skip-mode promotes the environmental skip to a failure.

New witnesses sample the cumulative pressure clock at both lifecycle edges; even one coarse interval can therefore cover the whole Body. Each interval also carries its real wall width. The window calculation charges an interval’s entire pressure delta whenever the latency window could touch it, so a late monitor wake only makes W larger. ScenarioEnd carries the guest’s scenario duration: if host starvation queues the start and end frames and their accepted span is impossibly short, ktstr rejects the localized series and uses the complete whole-vCPU-thread lifetime as one conservative interval. If only pressure edge sampling fails, ktstr instead uses the tighter complete Body schedstat delta when available (or the lifetime when its opening snapshot is unavailable). If neither source can prove coverage, the failure is demoted rather than trusting an empty W = 0. Legacy sidecars without event-anchored widths retain the older ≥2-tick / ≥50%-span coverage gate.

Only ns-denominated latency exemplars take part. max_wake_latency_cv (a dimensionless ratio) and max_spread_pct (an off-CPU percentage) have no ns interval for W to bound, so they stay annotate-only: under measured dilation their failure carries a --- host dilation --- note (D and which ceilings include host preemption), but the verdict is unchanged.

Performance mode is different. A performance_mode cell pins each vCPU 1:1 to a host CPU, so it should read D ≈ 1. If its Body dilation exceeds the perf-mode isolation ceiling, the cell lost that isolation — every timing verdict this run produced is untrustworthy. That is a perf-mode isolation violated failure, recorded regardless of gate outcomes: it is an infra/isolation fault, not a scheduler-under-test fault. Default-mode cells get no such check — the tri-state already accounts for their host contention.

Beyond attributes

  • Verdict + claim! — the claim accumulator for custom scenario bodies. Labels come from the code itself (stringify!-derived), so they cannot drift from the value they describe:

    let mut v = Assert::default_checks().verdict();
    stats.claim_max_gap_ms(&mut v).at_most(100);
    claim!(v, iter_delta).at_least(1000);
    let result = v.into_result();
  • claim_better — the comparison primitive for “candidate beats baseline” gates (a scheduler vs EEVDF, this run vs a recorded number). It looks up the metric’s polarity in the registry, so you never write the wrong direction:

    // wakeup latency is lower-better; 60 vs 50 fails, correctly
    v.claim_better(BuiltinMetric::WakeupP99LatencyUs, cand).than(base);
    // require a 10% margin, not just any improvement
    v.claim_better(BuiltinMetric::TaobenchTotalOpsPerCpuSec, cand).than_by(base, 0.10);

    An unregistered metric yields Inconclusive, never a silent pass.

  • Claim variants and measurements — claim_present!(v, opt) fails loudly on None instead of vacuously passing; claim_set / claim_seq assert membership, length, and subset bounds over collections; note(msg) records free-text context and note_value(key, val) a typed measurement into AssertResult::measurements — triage payload attached to the verdict, not a metric (nothing recorded through claims or notes reaches the stats sidecar or perf-delta; see Assertable Metrics).

  • AbsoluteThresholds — flat per-run bounds (max_p99_wake_latency_ns, max_iteration_cost_p99_ns, max_migrations, min_work_units) checked in one call: assert_thresholds(&reports, &AbsoluteThresholds::strict()). Empty report slices return a skip rather than a vacuous pass.

  • assert_scx_events_clean(events, bound) — SCX event counters under a cap (None = exactly zero); negative counts always fail.

  • Composition — AssertResult::merge accumulates results in a loop; all_of / any_of fold sibling results as AND / OR.

Signatures, comparators, and construction details are in the ktstr::assert rustdoc. For phase-scoped checks over a stepped scenario, see Phases. To assert on your scheduler’s own counters — scx_stats fields, BPF globals and maps — see Projections and Temporal Assertions.