Benchmark Gates and Negative Tests

A performance gate you have never seen fail proves nothing. This recipe builds gates in pairs: a positive test that holds the scheduler to a floor, and a negative twin that degrades the scheduler on purpose and asserts the same gate trips. (Extracting metrics from benchmark payloads like schbench or fio is covered in Payloads and Included Files; cross-commit regression gates are A/B Compare Branches.)

Positive: gate a scenario

use ktstr::declare_scheduler;
use ktstr::prelude::*;

declare_scheduler!(MY_SCHED, {
    name = "my_sched",
    binary = "scx_my_sched",
    topology = (1, 1, 2, 1),
});

#[ktstr_test(
    scheduler = MY_SCHED,
    performance_mode = true,
    duration_s = 5,
    sustained_samples = 15,
)]
fn perf_positive(ctx: &Ctx) -> Result<AssertResult> {
    let checks = Assert::default_checks()
        .min_iteration_rate(5000.0)
        .max_gap_ms(500);
    let steps = vec![Step::with_defs(
        vec![CgroupDef::named("cg_0").workers(2)],
        HoldSpec::FULL,
    )];
    execute_steps_with(ctx, steps, Some(&checks))
}

Key points:

• performance_mode = true pins vCPUs to reserved host cores so the measurement isn’t host noise — see Performance Mode.
• Assert::default_checks() is all-None: every gate is opt-in, and nothing fails until you chain a setter. Threshold layering and merge order live in Customize Checking.
• execute_steps_with applies the Assert during worker checks.

A passing gated run looks like any passing run:

cargo ktstr: resolved kernel "7.0"
...
    Starting 1 test across 121 binaries (12531 tests skipped)
        PASS [  22.158s] (1/1) ktstr::ktstr_sched_tests ktstr/sched_basic_proportional
────────────
     Summary [  22.197s] 1 test run: 1 passed, 12531 skipped

Choosing the floor

Don’t guess thresholds. Run the scenario once with no gates, read the observed per-cgroup iteration counts from the --- stats --- block (or the run’s stats sidecar), and set the floor at a healthy margin below the observed rate — far enough down that run-to-run noise never trips it, close enough up that a real regression does. Tighten later once perf-delta --noise-adjust has shown you the actual run-to-run spread.

Negative: prove the gate fires

expect_err = true inverts the harness: the test must fail. An Ok return panics with expected test to fail but it passed, so a gate that silently stopped firing turns the negative test red. Skips are handled separately: a run that could not boot a kernel or lost the host-resources race emits a SKIP banner and does not count as the expected failure — an environment problem is never mistaken for proof that the gate fired (the skip-vs-fail taxonomy lives in Troubleshooting). Auto-repro is disabled automatically for expected-error tests.

#[ktstr_test(
    scheduler = MY_SCHED,
    performance_mode = true,
    duration_s = 5,
    extra_sched_args = ["--degrade"],
    expect_err = true,
)]
fn perf_negative(ctx: &Ctx) -> Result<AssertResult> {
    let checks = Assert::default_checks()
        .min_iteration_rate(5000.0)
        .max_gap_ms(500);
    let steps = vec![Step::with_defs(
        vec![CgroupDef::named("cg_0").workers(2)],
        HoldSpec::FULL,
    )];
    execute_steps_with(ctx, steps, Some(&checks))
}

extra_sched_args passes CLI args to the scheduler binary. --degrade is a real knob on ktstr’s fixture scheduler (scx-ktstr) that deliberately worsens its scheduling; the fixture also exposes --slow, --stall-after, and --fail-verify for other failure classes. Substitute your own scheduler’s equivalent — a degradation flag your scheduler ships for exactly this purpose is a feature, not a wart.

When the degraded run trips the gate, the failure the harness expects looks like the real thing, because it is:

    ktstr_test 'throughput_gate' [sched=scx-ktstr] [topo=1n1l2c1t] failed:
      worker 71 iteration rate 41903.3/s below floor 50000000.0/s
      worker 73 iteration rate 37834.5/s below floor 50000000.0/s

The in-repo pattern to copy is tests/assert_gate_matrix.rs: a macro stamps out a positive and a negative variant for each of ten worker gates (max_p99_wake_latency_ns, max_wake_latency_cv, min_iteration_rate, max_gap_ms, …), with the negative variant passing --degrade and setting expect_err. Each gate in the matrix is proven to fire by its own negative test — hold your gates to the same standard.

Related

• Payloads and Included Files — metric extraction from real benchmark binaries, declarative include files.
• A/B Compare Branches — the cross-commit complement: gates on the delta between two commits instead of an absolute floor.
• Customize Checking — threshold layers and merge order.