Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Investigate a Crash

When the scheduler under test dies mid-run, the test fails with a structured crash report: the scx exit reason, a kernel-side backtrace, per-CPU scheduler state, and — by default — the results of a second VM that ktstr boots automatically to replay the crash with probes attached. This recipe walks a real crash from report to regression test.

First step: rerun with full diagnostics

RUST_BACKTRACE=1 cargo ktstr test --kernel 7.0 -- -E 'test(my_test)'

RUST_BACKTRACE=1 (or full) does two things: it appends the --- diagnostics --- section (init stage, VM exit code, kernel console tail) to every failure — not only scheduler deaths — and it boots the guest with a verbose console (the same switch KTSTR_VERBOSE=1 flips).

The crash report

A failure prints as a header line plus sections, each present only when relevant: --- stats --- (per-cgroup worker results), --- diagnostics ---, --- timeline --- (phases with monitor samples), --- scheduler log --- (scheduler stdout+stderr, including the kernel’s DEBUG DUMP when the scheduler died), --- monitor --- (host-side observations and verdict), --- sched_ext dump --- (the same dump as traced by the guest kernel), and --- auto-repro ---. See Reading Failure Output for the full anatomy; this recipe focuses on the crash workflow.

A real crash, top to bottom

The crash below is real: ktstr’s fixture scheduler was told to call scx_bpf_error() via a host-written .bss trigger (the same mechanism step 7 of Test a New Scheduler uses). Trimmed to the load-bearing sections:

cargo ktstr test --kernel 7.0 -- -E 'test(=ktstr/bpf_crash_auto_repro_e2e)'
BUG SUMMARY: scx_bpf_error (src/bpf/main.bpf.c:424: ktstr: host-triggered crash)
ktstr_test 'bpf_crash_auto_repro_e2e' [sched=scx-ktstr] [topo=1n1l4c1t] failed:
  scheduler process died unexpectedly during workload (2.2s into test)
...
--- scheduler log ---
...
DEBUG DUMP
================================================================================

swapper/3[0] triggered exit kind 1025:
  scx_bpf_error (src/bpf/main.bpf.c:424: ktstr: host-triggered crash)

Backtrace:
  scx_exit+0x50/0x70
  scx_bpf_error_bstr+0x78/0x90
  bpf_prog_1fed99378f3a8055_ktstr_dispatch+0x4d/0x1cb
  bpf__sched_ext_ops_dispatch+0x4b/0xa7
  do_pick_task_scx+0x379/0x770
  __schedule+0x5ca/0xfc0
  schedule+0x44/0x1b0
  worker_thread+0xa2/0x2d0
  kthread+0xf3/0x130
  ret_from_fork+0x19b/0x260
  ret_from_fork_asm+0x1a/0x30

ktstr scheduler state:
  stall=0 crash=1 degrade_rt=0
  rodata: degrade=0 slow=0 scattershot=0 verify_loop=0 fail_verify=0
  ktstr_alloc_count=87 degrade_cnt=0 slow_cnt=0
...
Error: EXIT: scx_bpf_error (src/bpf/main.bpf.c:424: ktstr: host-triggered crash)

Three lines identify the cause:

  • The BUG SUMMARY / exit line carries the message and C source line your scheduler passed to scx_bpf_error().
  • The backtrace names the BPF program that raised it (…ktstr_dispatch) and shows it fired from the kernel’s pick path (do_pick_task_scx inside __schedule).
  • The ktstr scheduler state block is the scheduler’s own dump callback output — whatever your scheduler prints in its .dump() op appears here. In this case crash=1 confirms the host-written trigger was read.

Auto-repro

auto_repro defaults to true in #[ktstr_test]. When the scheduler crashes, ktstr automatically:

  1. Captures the crash stack trace from the scenario output.
  2. Boots a second VM with kprobes (kernel functions) and fentry probes (BPF callbacks) on each function in the crash chain, plus the kernel-selected scheduler-exit trigger (tp_btf/sched_ext_exit on the newest kernels, a raw scx_vexit entry/return pair on the preceding generation, or filtered fentry/scx_dump_state on global-era kernels).
  3. Reruns the scenario to capture function arguments at each crash point.

The --- auto-repro --- section starts with the probe pipeline’s own accounting, so you can always see how much of the chain attached. The same crash replayed under probes:

--- auto-repro --- (repro VM, probes on the crash chain)
--- probe pipeline ---
  extracted:   9 functions from crash backtrace
  traceable:   4 passed, 4 dropped: bpf_prog_9a11f2edaac0b52f_ktstr_dispatch, ...
  kprobes:     5 attached
  fentry:      11 attached
  trigger:     fired (fexit)
  events:      125 captured, 8 after stitch

=== AUTO-PROBE: scx_exit fired ===

  ktstr_select_cpu                                              main.bpf.c:380
    task_struct *p
      pid                 40
      dsq_id              SCX_DSQ_INVALID  →  SCX_DSQ_LOCAL
      slice               19982063         →  20000000
      scx_flags           RESET_RUNNABLE_AT|DEQD_FOR_SLEEP|ENABLED
  enqueue_task_scx                                              kernel/sched/ext.c
    rq *rq
      cpu                 0
    task_struct *p
      pid                 40
      dsq_id              SCX_DSQ_LOCAL
      scx_flags           QUEUED|DEQD_FOR_SLEEP|ENABLED

This is the chain to the error, not just the point of death: each probed function shows the task state it saw (fexit arrows mark what changed inside the call), so you can watch scheduling decisions progress right up to the frame where ktstr_dispatch raised scx_bpf_error(). Entry state that never got its exit pair (the crash interrupted it) still prints — the last rows are the scheduler’s final acts.

Without the tracepoint the trigger cannot attach and the section reports it plainly (trigger: attach failed ... No such process) while still appending the repro VM’s diagnostics — sched_ext dump, dmesg tail, and the failure dump when produced. Cost either way: one extra VM boot plus a scenario replay (~10-17 s here). Auto-repro is skipped when expect_err = true — an expected failure is not worth a repro VM — and can be turned off with auto_repro = false. See Auto-Repro for the two-VM cycle and kernel requirements.

Pin the bug as a regression test

Once the crash is understood, pin its signature so the same bug fails the next CI run instead of silently regressing. Two #[ktstr_test] attributes attach matchers to the captured scx_bpf_error text (the combined scheduler log and --- sched_ext dump --- corpus):

  • expect_scx_bpf_error_contains = "literal" — substring match. Use for the common case of pinning an exact error fragment without escaping regex metacharacters.
  • expect_scx_bpf_error_matches = "regex" — full regex match via the regex crate. Use for anchored patterns, character classes, and wildcards.

Both require expect_err = true, and both compose with AND semantics — set both to require both to match. The test body must actually set up the trigger; here the host-side BpfMapWrite from the crash above does it:

use ktstr::prelude::*;

// Host writes 1 into the scheduler's `crash` .bss global after
// load; the scheduler's dispatch path reads it and calls
// scx_bpf_error(...) — the crash this test pins.
static BPF_CRASH: BpfMapWrite = BpfMapWrite::new(".bss", "crash", 1);

#[ktstr_test(
    scheduler = MY_SCHED,
    bpf_map_write = BPF_CRASH,
    expect_err = true,
    expect_scx_bpf_error_contains = "host-triggered crash",
    expect_scx_bpf_error_matches = r"src/bpf/main\.bpf\.c:\d+",
)]
fn crash_regression(ctx: &Ctx) -> Result<AssertResult> {
    ktstr::scenario::basic::custom_crash_light(ctx)
}

The test fails if either matcher misses. A passing run means the scheduler still hits the pinned bug; a failure means the error text drifted (update the matcher) or the bug was fixed (delete the regression test).

Regex anchors use string-boundary semantics against the whole captured corpus: ^/$ match its start and end, and . does not cross \n. Opt in to line-level anchoring with (?m) and to dotall with (?s). Pattern-validity edge cases (empty patterns, trivially-matching patterns, builder vs attribute validation) are covered in the #[ktstr_test] reference.