Module assert

Source

Expand description

Pass/fail evaluation of scenario results.

Key types:

AssertResult – pass/fail status with diagnostics and statistics
Assert – composable assertion config (worker + monitor checks)
ScenarioStats / CgroupStats – aggregated telemetry
NumaMapsEntry – parsed /proc/self/numa_maps VMA entry
Verdict – pointwise-claim accumulator (built via Assert::verdict / Verdict::new; comparators routed through ClaimBuilder / SetClaim / SeqClaim)

NUMA assertion functions:

parse_numa_maps – parse numa_maps content into per-VMA entries
page_locality – compute page locality fraction from entries
parse_vmstat_numa_pages_migrated – extract vmstat migration counter
assert_page_locality / assert_cross_node_migration – threshold checks

Assertion uses a three-layer merge: Assert::default_checks() -> Scheduler.assert -> per-test assert.

§Statistical conventions

Percentiles / medians: nearest-rank (see percentile), value at index ceil(n * p) - 1. Unlike interpolated percentiles, every reported p99 is an actual observed sample, not a synthetic midpoint. Consistent across every CgroupStats and ScenarioStats latency field.
CV (coefficient of variation) is stddev/mean computed over the pooled latency samples, not as a mean of per-worker CVs — see CgroupStats::wake_latency_cv for the masking caveat.

See the Checking chapter of the guide.

Re-exports§

pub use claim::ClaimBuilder;
pub use claim::SeqClaim;
pub use claim::SetClaim;
pub use claim::Verdict;
pub use temporal::EachClaim;
pub use temporal::FracPair;
pub use temporal::PhaseMapExt;
pub use temporal::SeriesField;

Modules§

claim: Pointwise-claim accumulator API: Verdict, ClaimBuilder, SetClaim, SeqClaim, and the claim! macro.
temporal: Temporal-assertion patterns over a periodic SampleSeries.

Structs§

AbsoluteThresholds: Threshold-preset bundle for assert_thresholds. Captures the guarantees a scheduler-under-test should meet on a healthy run: wake latency stays within bound, per-iteration compute cost stays within bound, CPU migrations stay within bound, and every worker makes some forward progress.
Assert: Unified assertion configuration. Carries both worker checks and monitor thresholds as a single composable type. Each Option field acts as an override — None means “inherit from parent layer”.
AssertDetail: A single diagnostic message from an assertion, paired with a structural DetailKind so filtering is robust to wording changes.
AssertDetailWithKind: Display adapter returned by AssertDetail::display_with_kind. Renders the detail as [<kind>] <message>. Held by reference so the helper allocates nothing on the formatting path; the lifetime is the borrow of the source AssertDetail.
AssertResult: Verdict for a single test scenario.
CgroupStats: Per-cgroup statistics from worker telemetry.
InfoNote: Informational annotation that does NOT contribute to the failure verdict — the structural counterpart to AssertDetail for “context surfaced alongside a result” emissions. Lives in its own type (not as a DetailKind variant of AssertDetail) so the “details = failures” mental model holds at the type level: AssertResult::details is the failure stream, AssertResult::info_notes is the context stream. Producers can no longer accidentally tag a note as a failure (the prior DetailKind::Note variant on AssertDetail made misclassification a one-character bug — every sidecar consumer that read details needed to remember to filter kind == Note to count real failures, and forgetting silently misreported failure counts).
NumaMapsEntry: Per-VMA entry parsed from /proc/self/numa_maps.
PassDetail: Structured record of a single passing claim — the positive counterpart to AssertDetail. Populated by Verdict’s record_pass_unary / record_pass_binary helpers at every comparator’s pass arm so the auto-repro renderer (and any other consumer that wants per-claim fidelity) can iterate passes alongside fails.
Phase: Identifier for a scenario phase. Newtype over u16 carrying the same 1-indexed encoding documented on every other phase-touching site: Phase::BASELINE is the pre-first-Step settle window (u16 0); Phase::step(k) is scenario Step k at 1-indexed u16 k + 1. The newtype catches the bug class where a raw u16 flows between sites that disagree about 0-indexed vs 1-indexed Step encoding, and gives operators readable construction at consumer sites (Phase::BASELINE / Phase::step(2) instead of magic 0u16 / 3u16).
PhaseBucket: Per-phase metric bucket — one entry per scenario phase in ScenarioStats::phases.
PhaseCgroupStats: Per-phase per-cgroup raw telemetry components — the per-phase analogue of CgroupStats. Holds RAW components (sample vectors + counters), NOT the reduced ratios/percentiles CgroupStats computes, so whole-run and cross-run aggregates RE-POOL from the components at every level (the per-phase telemetry thesis: an aggregate is recomputed over the pooled components, never averaged from ready-made per-phase reductions — a percentile or weighted ratio cannot be recovered from per-phase scalars). Covers every TYPED CgroupStats reduction: avg/min/max off-CPU% and spread from off_cpu_pcts; p99/median/CV wake latency from wake_latencies_ns; mean/worst run-delay from run_delays_ns; migration_ratio, iterations_per_cpu_sec, iterations_per_worker, page_locality, cross_node_migration_ratio from their counter components; the COUPLED worst gap (ms + the CPU that owned it) from max_gap_ms / max_gap_cpu; cpus_used / num_cpus from cpus_used. EXCLUDES CgroupStats::ext_metrics (the generic extensible map — a per-phase per-cgroup custom metric is a future extension, not part of the typed carrier). Lives in PhaseBucket::per_cgroup, keyed by cgroup name. The structural carrier is empty until a capture path populates it per phase.
PhaseGuard: RAII scope guard for the ACTIVE_PHASE thread-local. Install at scenario-driver run_step entry; the guard’s Drop restores the prior phase label, supporting cleanly-nested scenario dispatch (sub-scenarios layer over a parent’s phase context without leaking).
ScenarioStats: Aggregated statistics across all cgroups in a scenario.

Enums§

DetailKind: Category tag for an AssertDetail. Enables structural filtering (e.g. by AssertPlan) without matching on substrings of human-readable messages, which is fragile if wording changes.
NoteValue: Result of checking a scenario run.
Outcome: Terminal verdict for a single test scenario or merge fold — strict four-state enum that replaces the (passed, skipped) bool-pair encoding on AssertResult.
OutcomeRef: Borrowed view of an Outcome: same four discriminants but the Skip, Inconclusive, and Fail payloads borrow their AssertDetail in place. Returned by Outcome::as_ref and the zero-clone verdict-read fast path AssertResult::outcome_ref.

Constants§

COMPARATOR_VOCABULARY: Wire-canonical vocabulary of PassDetail.comparator tokens.
MAX_RECORDED_PASSES: Cap on AssertResult.passes (and the matching truncation sentinel) so a pathological test that fires millions of claims doesn’t balloon the wire-formatted result. Mirrors SnapshotBridge’s MAX_STORED_EVENTS truncation pattern: when the cap is hit, the cap-th record is replaced with a synthetic PassDetail { name: PASSES_TRUNCATION_SENTINEL_NAME, … } carrying the dropped-count, and further pushes are no-ops.
PASSES_TRUNCATION_SENTINEL_COMPARATOR: Comparator-string value used by the truncation sentinel record alone. Out-of-vocabulary by design — not in COMPARATOR_VOCABULARY — so the runtime debug_assert in record_pass_inner allows it explicitly without polluting the wire-canonical token set.
PASSES_TRUNCATION_SENTINEL_NAME: Sentinel PassDetail.name value used by the truncation record that replaces the MAX_RECORDED_PASSES-th slot when a test over-runs the cap. Consumers (the auto-repro renderer) match on this string to render [N passes truncated] instead of treating it as a real claim.

Traits§

CgroupStatsClaim: Pointwise-claim accessors generated by #[derive(Claim)] on CgroupStats. One claim_<field> method per public field, taking &mut Verdict as the accumulator; container fields (BTreeSet/Vec) route through SetClaim/SeqClaim. Method dispatch keys on the stats struct’s type, so identical field names across distinct stats structs do not collide. For prelude-exported stats types the trait is preluded, so use ktstr::prelude::* brings the accessors into scope; otherwise import the trait from the stats type’s module.
PhaseBucketClaim: Pointwise-claim accessors generated by #[derive(Claim)] on PhaseBucket. One claim_<field> method per public field, taking &mut Verdict as the accumulator; container fields (BTreeSet/Vec) route through SetClaim/SeqClaim. Method dispatch keys on the stats struct’s type, so identical field names across distinct stats structs do not collide. For prelude-exported stats types the trait is preluded, so use ktstr::prelude::* brings the accessors into scope; otherwise import the trait from the stats type’s module.
PhaseCgroupStatsClaim: Pointwise-claim accessors generated by #[derive(Claim)] on PhaseCgroupStats. One claim_<field> method per public field, taking &mut Verdict as the accumulator; container fields (BTreeSet/Vec) route through SetClaim/SeqClaim. Method dispatch keys on the stats struct’s type, so identical field names across distinct stats structs do not collide. For prelude-exported stats types the trait is preluded, so use ktstr::prelude::* brings the accessors into scope; otherwise import the trait from the stats type’s module.
ScenarioStatsClaim: Pointwise-claim accessors generated by #[derive(Claim)] on ScenarioStats. One claim_<field> method per public field, taking &mut Verdict as the accumulator; container fields (BTreeSet/Vec) route through SetClaim/SeqClaim. Method dispatch keys on the stats struct’s type, so identical field names across distinct stats structs do not collide. For prelude-exported stats types the trait is preluded, so use ktstr::prelude::* brings the accessors into scope; otherwise import the trait from the stats type’s module.

Functions§

assert_benchmarks: Check benchmarking metrics: p99 wake latency, wake latency CV, and minimum iteration rate.
assert_cross_node_migration: Check cross-node page migration ratio against threshold.
assert_isolation: Check that workers only ran on CPUs in expected.
assert_not_starved: Default fairness check for one cgroup’s worker reports: builds the per-cgroup telemetry (cgroup_stats) and records Starved / Unfair / Stuck against the framework default thresholds. Telemetry is ALWAYS populated — including a num_workers == 0 entry for empty reports — so r.stats.cgroups is never empty for a declared cgroup, independent of whether any fail outcome fired.
assert_page_locality: Check NUMA page locality against threshold.
assert_scx_events_clean: Assert that every SCX event counter in events is at or below max_count. events is a slice of (name, count) pairs sourced from the kernel’s per-task scx_event_stats (see kernel/sched/ext.c, SCX_EV_* macros) — typically aggregated and surfaced via monitor::ScxEventDeltas or sidecar GauntletRow.fallback_count / keep_last_count fields. Pass None for max_count to require zero (the strict default — error-class events should not fire under a healthy scheduler).
assert_thresholds: Run every check in thresholds against reports, merging results into a single AssertResult. A None field on the thresholds skips that check.
assert_throughput_parity: Check throughput parity across workers: coefficient of variation and minimum work rate.
build_phase_buckets: Build per-phase metric buckets from a sample series.
build_phase_buckets_with_stimulus: Phase buckets attributed against the guest stimulus timeline, then enriched with stimulus-event-derived per-phase iteration_rate.
cgroup_stats: Build per-cgroup telemetry (pure measurement, no assertions) from worker reports. This is the SINGLE telemetry builder on the assertion path: AssertPlan::assert_cgroup calls it unconditionally and assert_not_starved wraps it with the default fairness checks, so per-cgroup CgroupStats is never gated behind whether a worker-check assertion was configured. Empty reports yield a num_workers == 0 CgroupStats (the reduces below collapse to 0.0/0), so a declared cgroup that collected no reports surfaces as a zero-worker entry rather than silently vanishing from ScenarioStats::cgroups.
current_phase_label: Snapshot the active phase label installed by the most recent PhaseGuard::install on this thread. None outside any guarded scope. Construction sites for AssertDetail / PassDetail / InfoNote call this to auto-stamp the phase field; the test author can still override via the builder with_phase(...) chain when an explicit value is preferred.
page_locality: Compute page locality fraction from parsed numa_maps entries.
parse_numa_maps: Parse /proc/self/numa_maps content into per-VMA entries.
parse_vmstat_numa_pages_migrated: Extract numa_pages_migrated from /proc/vmstat content.
populate_run_distribution_metrics: Populate run-level DERIVED distributional metrics into stats.ext_metrics: every registered MetricKind::Distribution, MetricKind::WorstLowest, MetricKind::WakeLatencyTailRatio, and MetricKind::WorstCrossNodeRatio. This is the SOLE within-run producer of those metrics’ values — they carry no per-phase sample slice and no cross-cgroup merge fold, and their registry accessors are |_| None, so MetricDef::read reads the value written here from ext_metrics.
populate_run_ext_all: Run the FULL run-level ext_metrics population sequence into stats — the single source of truth shared by the eval layer (evaluate_vm_result) and crate::vmm::VmResult::run_metric, so the two produce byte-identical run-level ext maps for the same run. Reads samples + stats.phases + stats.cgroups, writes stats.ext_metrics, in the canonical order:
populate_run_ext_metrics: Populate cross-RUN aggregate entries for every registered crate::stats::MetricDef whose read_sample returns finite values across the entire sample series. Writes into target (typically ScenarioStats::ext_metrics) under the metric’s registry name — the same key the per-phase PhaseBucket::metrics uses, so cross-RUN and per-phase consumers reference the same name.
populate_run_ext_metrics_from_phases: Sibling of populate_run_ext_metrics that mines per-phase metrics back into the run-level ext_metrics map. Closes the gap for registered metrics whose values live in PhaseBucket.metrics but never reach ext_metrics via the SampleSeries path (their read_sample returns None): avg_imbalance_ratio (sourced from MonitorSample windowing inside build_phase_buckets), iteration_rate (sourced from stimulus event totals inside build_phase_buckets_with_stimulus), and system_time_ns / user_time_ns (per-thread-group CPU-time deltas injected by phase_group_cpu_delta inside buckets_from_grouped). The fold is generic over every key present on any phase, so it carries any such phase-only metric (the ext-metrics-only set whose read_sample returns None). Keys with a typed GauntletRow field (TYPED_FIELD_NAMES) are SKIPPED: their run-level value comes from the typed accessor (which wins on read), so re-injecting them here would double-source the run aggregate — the hazard the const’s doc describes. Their per-phase PhaseBucket value still feeds per-phase rendering.
populate_run_pooled_iterations_per_cpu_sec: Inject the run-level POOLED iterations_per_cpu_sec Rate’s two Counter components into stats.ext_metrics, summed across the cgroups that have measured on-CPU time — the cross-cgroup re-pool axis. Rather than routing the per-cgroup efficiency through AssertResult::merge’s worst-by-polarity ext_metrics fold (which picks the WORST cgroup’s value, not Σ, and has no derive post-pass), this reads the already-merged stats.cgroups vec directly: iterations_per_cpu_sec = Σtotal_iterations / Σ(total_cpu_time_ns/1e9) over cgroups with total_cpu_time_ns > 0 — the per-cgroup CgroupStats::iterations_per_cpu_sec re-pooled, NOT a mean of per-cgroup ratios, NOT the worst single cgroup.
populate_run_pooled_schbench: Inject the schbench whole-run Class-3 metrics — the loop Counter and the role-separate run-delay gate-Rate components — into stats.ext_metrics, summed across EVERY phase and EVERY cgroup from the per-phase SchbenchPhaseStats raw pairs (stats.phases[].per_cgroup[].schbench). The raw (run_delay_ns, pcount) pairs and loop_count are integer and associative, so summing across phases+cgroups gives the run-level totals; the two *_run_delay_ns_per_sched Rates then re-derive Σrun_delay/Σpcount (the sample-weighted per-schedule mean — NOT a mean of per-run means). The MESSAGE and WORKER thread roles pool SEPARATELY (different per-schedule wait populations — never cross-pool).
populate_run_pooled_schbench_distribution: Inject the schbench whole-run DISTRIBUTIONAL metrics (the wakeup / request latency percentiles + min/max and the achieved-rps percentiles) into stats.ext_metrics as the *_whole keys, re-pooled run-level by UNIONING the per-phase per-cgroup PlatStats histograms (stats.phases[].per_cgroup[].schbench.{wakeup,request,rps}) across EVERY phase and EVERY cgroup, then re-deriving each percentile / min / max over the merged histogram. PlatStats::combine is an associative bucket-count add, so the merged histogram is the FAITHFUL union and the re-derived percentile is the percentile OF the pooled sample set — NOT a mean of per-phase / per-cgroup percentiles (the percentile operator is non-linear). This is the schbench histogram analog of populate_run_distribution_metrics’s raw-sample union.
populate_run_pooled_taobench: Inject the whole-run taobench engine’s qps + hit Rate components into stats.ext_metrics, pooled across the run’s WorkType::Taobench cgroups. Each cgroup carries its workers’ merged whole-run aggregate (crate::assert::CgroupStats::taobench_whole); this folds those across cgroups (Σ ops, MAX wall window — the window is shared by the concurrent cohorts, per TaobenchStats::merge) and writes the six total_taobench_* Counter components (ops, fast_ops, slow_ops, wall_sec, plus the command-time get_cmds / get_hits), from which crate::stats::derive_rate_metrics derives taobench_total_ops_per_sec, taobench_fast_ops_per_sec, taobench_slow_ops_per_sec, the response-time taobench_hit_fraction (Σfast/Σcompleted), and the command-time taobench_command_hit_rate (Σhits/Σcmds). The whole-run Rate keys are registered METRICS, so — unlike the per-phase taobench_*_qps (MetricKind::PerPhase, invisible to the whole-run cross-run fold) — they reach the perf-delta --noise-adjust spread analysis. The open-loop serve-latency distribution is a SEPARATE pool (populate_run_pooled_taobench_distribution, the *_us_whole keys).
populate_run_pooled_taobench_distribution: Inject the taobench WHOLE-RUN open-loop serve-latency percentiles into stats.ext_metrics as the taobench_serve_*_us_whole keys, re-pooled run-level by UNIONING the per-phase per-cgroup serve PlatStats histograms (stats.phases[].per_cgroup[].taobench.serve_lat) across every step-attributed phase and every cgroup, then re-deriving each percentile / min / max over the merged histogram (PlatStats::combine is an associative bucket-count add, so the merged histogram is the faithful pooled sample set and the re-derived percentile is the percentile OF the union — NOT a mean of per-source percentiles). The taobench analog of populate_run_pooled_schbench_distribution, with the same source: the BASELINE (epoch 0) and inter-step-gap (u32::MAX) epochs are excluded (they are dropped from stats.phases by expand_backdrop_phase_buckets), so this is the steady-state serve distribution over the measured steps — a faithful PerRunDistribution, distinct from the standalone driver’s full-run histogram.

Module assert

Module assert Copy item path

§Statistical conventions

Re-exports§

Modules§

Structs§

Enums§

Constants§

Traits§

Functions§

Module assert