Checking
ktstr judges scheduler behavior through two channels: worker-side telemetry (every worker process reports what happened to it) and host-side monitoring (the monitor reads guest kernel state from outside). Both channels always measure; nothing asserts until the test opts in — a test with no checking attributes passes as long as the VM boots and the scenario completes.
Which API to reach for:
#[ktstr_test]attributes — cover most tests:not_stuck,max_gap_ms,max_spread_pct,min_iteration_rate, and every other threshold below has an attribute (see the macro reference).Verdict+claim!— labeled assertions on values you compute inside a custom scenario body.AbsoluteThresholds— a one-call multi-field bound check against collected reports, bypassing the config merge.assert_scx_events_clean— bounds on SCX event counters (“no fallbacks fired”).
Worker checks
After each scenario, ktstr collects a
WorkerReport from every worker and
runs the opted-in checks against them:
- Zero work units (
not_stuck) — any worker with no measured work fails:tid N made no progress (0 work units). - Scheduling gaps (
max_gap_ms) — the longest wall-clock gap observed at work-unit checkpoints. A violation renders astid N stuck Xms on cpuY at +Zms (threshold Nms). - Fairness (
max_spread_pct) — workers in one cgroup should get similar CPU time; the spread (max off-CPU% − min off-CPU%) must stay below the bound. - Cpuset isolation (
isolation) — workers may only run on CPUs in their assigned cpuset; any excursion fails. - Throughput —
max_throughput_cvbounds the coefficient of variation of per-worker work rate (some workers quietly slower);min_work_ratesets an absolute floor (all workers equally slow). - Benchmarking —
max_p99_wake_latency_nsandmax_wake_latency_cvbound wake-to-run latency for work types that block and measure it (see Work Types for which do);min_iteration_ratefloors outer-loop iterations per delivered CPU-second per worker.
The loop, end to end
A test sets a threshold, the run violates it, the failure output names the check, the value, and the bound:
#[ktstr_test(
scheduler = MY_SCHED,
llcs = 1, cores = 2, threads = 1,
min_iteration_rate = 50_000_000.0, // deliberately unreachable floor
)]
fn throughput_gate(ctx: &Ctx) -> Result<AssertResult> {
execute_defs(ctx, vec![
CgroupDef::named("cg_a").workers(1).cpuset(CpusetSpec::disjoint(0, 2)),
CgroupDef::named("cg_b").workers(1).cpuset(CpusetSpec::disjoint(1, 2)),
])
}
<!-- captured: cargo ktstr test --kernel 7.0 (throughput_gate demo test) | ktstr 0.23.0 | kernel 7.0.14 -->
ktstr_test 'throughput_gate' [sched=scx-ktstr] [topo=1n1l2c1t] failed: worker 71 iteration rate 41903.3/s below floor 50000000.0/s worker 73 iteration rate 37834.5/s below floor 50000000.0/s --- stats --- 2 workers, 4 cpus, 2 migrations, worst_spread=0.0%, worst_gap=21ms cg0: workers=1 cpus=2 spread=0.0% gap=10ms migrations=1 iter=209600 cg1: workers=1 cpus=2 spread=0.0% gap=21ms migrations=1 iter=189252 ... --- monitor --- samples=41 max_imbalance=2.00 max_dsq_depth=0 stuck=0 avg: imbalance=1.32 nr_running/cpu=1.2 dsq/cpu=0.0 events: fallback=0 (0.0/vcpu-s) keep_last=210 (52.5/vcpu-s) offline=0 ... verdict: monitor OK
Both channels report: the worker check that tripped, and the monitor verdict that did not. The full failure anatomy — timeline, scheduler log, dump sections — is in Reading Failure Output.
Monitor checks
The host-side monitor samples guest per-CPU runqueue state (via BTF offsets, no guest instrumentation) roughly every 100ms and evaluates:
- Imbalance ratio —
max(nr_running) / max(1, min(nr_running))across CPUs. - Local DSQ depth — per-CPU dispatch queue depth.
- Stuck-task detection —
rq_clocknot advancing on a CPU with runnable tasks; idle CPUs and preempted vCPUs are exempt. - Event rates —
select_cpu_fallbackanddispatch_keep_lastcounters per second.
Monitor violations always land in the failure report’s --- monitor --- section, but they flip the test result only when the test
enforces them — set the corresponding attributes, call
.with_monitor_defaults() on an Assert, or set
enforce_monitor_thresholds. A monitor that produced no usable
signal (empty samples, uninitialized guest memory) reports
inconclusive, never a silent pass — a CI gate can always tell
“verified OK” from “never measured”.
The defaults with_monitor_defaults() applies:
| Threshold | Default | Rationale |
|---|---|---|
max_imbalance_ratio | 4.0 | max(nr_running) / max(1, min(nr_running)) across CPUs (denominator clamped so an all-idle sample does not divide by zero). Lower values (2-3) false-positive during cpuset transitions. |
max_local_dsq_depth | 50 | Per-CPU dispatch queue overflow. Sustained depth above this means the scheduler is not consuming dispatched tasks. |
fail_on_rq_clock_stuck | true | Fail when rq_clock does not advance on a CPU with runnable tasks. Idle CPUs (NOHZ) and preempted vCPUs are exempt. |
sustained_samples | 5 | At ~100ms sample interval, requires ~500ms of sustained violation. Filters transient spikes from cpuset reconfiguration. |
max_fallback_rate | 200.0/vCPU-s | select_cpu_fallback events across all CPUs per delivered average-vCPU CPU-second. Sustained activity indicates systematic select_cpu failure. |
max_keep_last_rate | 100.0/vCPU-s | dispatch_keep_last events across all CPUs per delivered average-vCPU CPU-second. Sustained activity indicates the scheduler keeps reusing the previous dispatch target instead of making progress through the normal path. |
Every monitor threshold uses the sustained_samples window — a
violation must persist for N consecutive samples before it counts.
NUMA checks
For workers with a MemPolicy, three thresholds
gate page placement:
min_page_locality— minimum fraction of pages on the expected NUMA nodes (the cgroup’s cpuset nodes, derived at evaluation time). Zero observed pages counts as zero locality, not a vacuous pass.max_cross_node_migration_ratio— bound on migrated pages relative to allocated pages (from/proc/vmstatdeltas).max_slow_tier_ratio— bound on the fraction of pages landing on memory-only (CXL-tier) nodes.
Default thresholds
not_stuck = true also enables the built-in fairness and stuck-gap
checks at these defaults:
| Check | Release | Debug |
|---|---|---|
| Scheduling gap | 2000 ms | 3000 ms |
| Fairness spread | 15% | 35% |
Debug builds run with higher scheduling overhead, so thresholds are relaxed.
How configuration merges
Assert is the threshold-config struct; every field is an Option
where None means “inherit”. Three layers merge, last-Some wins:
the baseline (all None), then the scheduler’s assert, then the
per-test attributes — so a scheduler-wide bound applies to every
test and any single test can override or disable it.
enforce_monitor_thresholds is the one sticky field: once any layer
sets it, it stays set. Worked override recipes live in
Customize Checking.
execute_steps_with(ctx, steps, Some(&assert)) bypasses the merged
config with an explicit Assert for that scenario’s worker checks.
Verdicts and outcomes
Every assertion produces one of four outcomes, and a result’s
terminal verdict is the fold over all of them, most severe first:
Fail > Inconclusive > Pass > Skip.
| Outcome | Meaning |
|---|---|
Pass | the assertion ran and the value satisfied the bound |
Fail | the assertion ran and the value violated the bound |
Inconclusive | the assertion ran but had no signal to evaluate |
Skip | the scenario couldn’t run (unmet precondition) |
Inconclusive exists for instrument-derived denominators — a ratio
whose denominator (iterations, samples, wall-clock interval)
legitimately reached zero because the workload produced no signal.
Policy-derived denominators stay Fail on zero: under
MemPolicy::Bind the policy says pages will exist, so their absence
is a defect, not “couldn’t measure”.
CI gates read the verdict through four accessors:
if r.is_pass() { /* ship */ }
if r.is_fail() { /* block; surface r.failure_details() */ }
if r.is_skip() || r.is_inconclusive() { /* no verdict — triage */ }
is_pass() is deliberately strict: inconclusive and all-skip both
read false.
Wall-latency ceilings under host contention
max_p99_wake_latency_ns measures a guest-wall interval — the time
a request waited to run, as the guest saw it. There is no steal-adjusted
guest clock that subtracts it out, so host-side scheduling delay during
the measurement (a noisy neighbour starving the vCPU threads) inflates
the reading directly. A plain measured <= T gate would blame the
scheduler under test for the host’s noise.
So the wall-latency verdict is tri-state. Each run carries a
contention witness over its Body (measurement) phase — host vCPU
dilation D from lifecycle-boundary schedstat snapshots and a peak-window
bound W(L) from ktstr’s host-cgroup CPU-pressure clock. CPU PSI some is the
wall time during which at least one runnable task in that cgroup was stalled
for CPU. It therefore sees a delayed vCPU even when the competitor is outside
the cgroup, without charging arbitrary pressure elsewhere on a wide host.
Other tasks sharing the runner cgroup can only bias the bound upward, toward
indeterminate. ktstr removes that cross-cell noise by capping the localized
PSI result with the target VM’s complete summed-vCPU run delay over the same
conservative Body span. Both are upper bounds on one request’s host scheduling
contamination, so taking their minimum is safe; if PSI is unavailable, the
task-specific cap becomes a coarser whole-span fallback. The gate fires in the
guest; the host re-judges it with the witness in hand:
| Verdict | Condition | Result |
|---|---|---|
| Pass | measured <= T | always sound — contention only ever inflates a wall interval |
| Fail (confirmed) | excess over T exceeds W(measured) | no witnessed host contention explains it — a real failure at any load; the message gains a contention-checked note |
| Indeterminate | excess within W(measured) | the failure might be host contention — non-blocking (indeterminate is a pass), surfaced as a contention-indeterminate note carrying measured / threshold / excess / W / D |
An indeterminate demotion drops the failing outcome; if it was the only
failure the whole result flips to Pass, and the annotation rides the
notes channel so it stays visible in
the passing run’s output and its sidecar. If the Body contention series
saturated (it hit its interval cap and only a prefix survives), W is a
lower bound, so the verdict never confirms on it — it is treated as
indeterminate with a saturation note.
For an expect_err negative test, a run whose only expected failures became
contention-indeterminate is a Skip, not a clean Pass: the environment
prevented the harness from proving whether the expected failure remains. A
confirmed failure still satisfies expect_err, an unrelated failure remains
blocking, and --no-skip-mode promotes the environmental skip to a failure.
New witnesses sample the cumulative pressure clock at both lifecycle edges;
even one coarse interval can therefore cover the whole Body. Each interval
also carries its real wall width. The window calculation charges an interval’s
entire pressure delta whenever the latency window could touch it, so a late
monitor wake only makes W larger. ScenarioEnd carries the guest’s scenario
duration: if host starvation queues the start and end frames and their accepted
span is impossibly short, ktstr rejects the localized series and uses the
complete whole-vCPU-thread lifetime as one conservative interval. If only
pressure edge sampling fails, ktstr instead uses the tighter complete Body
schedstat delta when available (or the lifetime when its opening snapshot is
unavailable). If neither source can prove coverage, the failure is demoted
rather than trusting an empty W = 0. Legacy sidecars without event-anchored
widths retain the older ≥2-tick / ≥50%-span coverage gate.
Only ns-denominated latency exemplars take part. max_wake_latency_cv
(a dimensionless ratio) and max_spread_pct (an off-CPU percentage)
have no ns interval for W to bound, so they stay annotate-only:
under measured dilation their failure carries a --- host dilation ---
note (D and which ceilings include host preemption), but the verdict is
unchanged.
Performance mode is different. A performance_mode cell pins each
vCPU 1:1 to a host CPU, so it should read D ≈ 1. If its Body dilation
exceeds the perf-mode isolation ceiling, the cell lost that isolation —
every timing verdict this run produced is untrustworthy. That is a
perf-mode isolation violated failure, recorded regardless of gate
outcomes: it is an infra/isolation fault, not a scheduler-under-test
fault. Default-mode cells get no such check — the tri-state already
accounts for their host contention.
Beyond attributes
-
Verdict+claim!— the claim accumulator for custom scenario bodies. Labels come from the code itself (stringify!-derived), so they cannot drift from the value they describe:let mut v = Assert::default_checks().verdict(); stats.claim_max_gap_ms(&mut v).at_most(100); claim!(v, iter_delta).at_least(1000); let result = v.into_result(); -
claim_better— the comparison primitive for “candidate beats baseline” gates (a scheduler vs EEVDF, this run vs a recorded number). It looks up the metric’s polarity in the registry, so you never write the wrong direction:// wakeup latency is lower-better; 60 vs 50 fails, correctly v.claim_better(BuiltinMetric::WakeupP99LatencyUs, cand).than(base); // require a 10% margin, not just any improvement v.claim_better(BuiltinMetric::TaobenchTotalOpsPerCpuSec, cand).than_by(base, 0.10);An unregistered metric yields Inconclusive, never a silent pass.
-
Claim variants and measurements —
claim_present!(v, opt)fails loudly onNoneinstead of vacuously passing;claim_set/claim_seqassert membership, length, and subset bounds over collections;note(msg)records free-text context andnote_value(key, val)a typed measurement intoAssertResult::measurements— triage payload attached to the verdict, not a metric (nothing recorded through claims or notes reaches the stats sidecar orperf-delta; see Assertable Metrics). -
AbsoluteThresholds— flat per-run bounds (max_p99_wake_latency_ns,max_iteration_cost_p99_ns,max_migrations,min_work_units) checked in one call:assert_thresholds(&reports, &AbsoluteThresholds::strict()). Empty report slices return a skip rather than a vacuous pass. -
assert_scx_events_clean(events, bound)— SCX event counters under a cap (None= exactly zero); negative counts always fail. -
Composition —
AssertResult::mergeaccumulates results in a loop;all_of/any_offold sibling results as AND / OR.
Signatures, comparators, and construction details are in the
ktstr::assert rustdoc.
For phase-scoped checks over a stepped scenario, see
Phases. To assert on your scheduler’s own
counters — scx_stats fields, BPF globals and maps — see
Projections and Temporal
Assertions.