Auto-Repro
The crash log tells you where the scheduler died; auto-repro tells
you what the state was on the way there. When a test fails because
the scheduler crashed or exited, ktstr boots a second VM, reruns the
scenario with BPF probes attached to the functions from the crash
backtrace, and prints each probed call with decoded arguments and
struct fields — entry and exit values side by side. The trail
appears in the --- auto-repro --- section of the failure output
(see Reading Failure Output); the end-to-end
debugging story is the
Investigate a Crash recipe.
Example output
The probe dump shows each function with decoded fields and source locations (DWARF for kernel functions, BPF line info for callbacks).
Where fexit captured post-mutation state, changed fields show an arrow between entry and exit values:
--- probe pipeline --- extracted: 9 functions from crash backtrace kprobes: 5 attached fentry: 11 attached trigger: fired (fexit) events: 125 captured, 8 after stitch === AUTO-PROBE: scx_exit fired === ktstr_select_cpu main.bpf.c:380 task_struct *p pid 40 cpus_ptr 0xf(0-3) dsq_id SCX_DSQ_INVALID → SCX_DSQ_LOCAL enq_flags NONE slice 19982063 → 20000000 vtime 0 weight 100 scx_flags RESET_RUNNABLE_AT|DEQD_FOR_SLEEP|ENABLED enqueue_task_scx kernel/sched/ext.c rq *rq cpu 0 task_struct *p pid 40 dsq_id SCX_DSQ_LOCAL slice 20000000 scx_flags QUEUED|DEQD_FOR_SLEEP|ENABLED ... scx_bpf_error_bstr+0x8a/0xd0 bpf_prog_9a11f2edaac0b52f_ktstr_dispatch+0x57/0x1db do_pick_task_scx+0x50e/0x940 __schedule+0x5a3/0x10c0
Reading it as a progression: pid 40 entered select_cpu off any
dispatch queue with a nearly expired slice; by exit it sat on the
local DSQ with a refilled 20 ms slice (SCX_DSQ_INVALID → SCX_DSQ_LOCAL, 19982063 → 20000000); the kernel’s
enqueue_task_scx then queued it. Healthy scheduling decisions,
captured right up to the frame where ktstr_dispatch called
scx_bpf_error() — the chain shows what the scheduler did with its
last tasks on the way into the error, not just where it died.
After the probe data, the section appends the repro VM’s wall time and, when non-empty, the last lines of its scheduler log, sched_ext dump, failure-dump JSON, and dmesg.
Enabling it — and what it costs
Auto-repro is on by default for every #[ktstr_test] with a
scheduler. Opt out per test:
#[ktstr_test(scheduler = MY_SCHED, auto_repro = false)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> { ... }
It fires only when the primary run fails, and it is disabled
automatically when expect_err = true (no point probing a
deliberately failing test). The cost is a second VM boot plus a full
scenario rerun — in the captured demo below, the repro VM added
about 17 seconds.
How it works
- Stack extraction — function names are parsed from the crash
trace in the scheduler log or kernel console. BPF program symbols
(
bpf_prog_*) are recognized and their short names extracted; generic frames (spinlocks, syscall entry, sched_ext exit machinery, trampolines) are filtered out. - BPF discovery — in the repro VM, loaded struct_ops programs
are discovered and added to the probe list along with their
kernel-side callers (e.g.
enqueue→enqueue_task_scx), so the pipeline still probes something when the crash produced no extractable stack. - BTF resolution — signatures come from vmlinux BTF and program
BTF; known structs (
task_struct,rq, dispatch queues) have curated fields resolved to offsets, and other struct pointers get scalar/enum/cpumask fields auto-discovered. - Probed rerun — the second VM reruns the scenario with kprobes
on kernel entry, fentry/fexit on BPF callbacks and kernel exits,
and the kernel-selected scheduler-exit trigger:
tp_btf/sched_ext_exiton the newest kernels, a rawscx_vexitentry/return pair on the preceding generation, or filteredfentry/scx_dump_stateon global-era kernels. - Stitching — events are filtered to the task that triggered the exit, sorted by timestamp, and rendered with decoded values.
If the primary VM failed before the scheduler ever attached and the
workload ever ran, the repro has nothing to reproduce — the framework
prepends a PRIMARY DID NOT REACH WORKLOAD label to the repro
verdict so you chase the primary’s startup failure (see its
--- diagnostics --- and --- timeline --- sections) instead of
reading the repro as evidence.
Kernel requirement
The probe trigger needs one compatible sched_ext target: the typed
sched_ext_exit tracepoint on the newest kernels, a raw entry/return pair
around scx_vexit on the preceding generation, or the two-argument
scx_dump_state function on global-era kernels. ktstr reads vmlinux BTF
before loading the probe object, picks the first compatible target in that
order, and disables every unused program before load. The tracepoint takes
precedence because it fires only after the exit claim succeeds and its
two-argument ABI does not inherit scx_vexit signature changes.
On a kernel without any compatible typed shape, the probed rerun fails
closed before scheduler launch and reports the concrete Phase-A attachment
error. The --- probe pipeline --- block records that failure; this is the
shape to recognize:
--- auto-repro ---
--- probe pipeline ---
extracted: 10 functions from crash backtrace
traceable: 7 passed, 3 dropped: bpf_prog_1fed99378f3a8055_ktstr_dispatch, bpf__sched_ext_ops_dispatch, ret_from_fork_asm
bpf_discover: 0 programs found
after_expand: 7 total probe targets
kprobes: 0 attached
trigger: attach failed (skeleton load (retry): No such process (os error 3); original error before retry: No such process (os error 3))
probe_data: 0 keys, 0 unmatched IPs
events: 0 captured, 0 after stitch
repro VM duration: 16.9s
Any diagnostic tail captured before the Phase-A failure is still appended, but the scheduler and workload are not started under a partially armed probe configuration.
Example test
bpf_crash_auto_repro_e2e in ktstr’s tests/scenario_coverage.rs
drives the path end to end: a host-side BPF map write sets the
fixture scheduler’s crash global, the scheduler calls
scx_bpf_error, and the auto-repro VM replays it.