Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Gauntlet

Some scheduler bugs only exist on topologies you don’t develop on: a per-LLC work-splitting heuristic that breaks on an odd LLC count, an idle-core picker that lands both SMT siblings, a migration policy that never crosses a NUMA boundary. The gauntlet expands every #[ktstr_test] into one variant per topology preset — up to 25 presets (14 on aarch64) — so those bugs surface as a named, re-runnable test case instead of a production report.

#[ktstr_test] one declaration one variant / preset 4cpu-1llc-nosmt 4cpu-2llc-nosmt 9cpu-3llc-nosmt 8cpu-2llc-smt 32cpu-4llc-smt 64cpu-8llc-smt 128cpu-4llc-smt 240cpu-15llc-smt 2numa-16cpu-4llc-nosmt filtered: constraints / host budget × each --kernel

Gauntlet variants are prefixed gauntlet/ and ignored by default:

# Run only base tests (default)
cargo ktstr test --kernel ../linux

# Run only gauntlet variants
cargo ktstr test --kernel ../linux -- --run-ignored ignored-only -E 'test(gauntlet/)'

# Run everything
cargo ktstr test --kernel ../linux -- --run-ignored all

# Run a single variant
cargo ktstr test --kernel ../linux -- --run-ignored ignored-only \
  -E 'test(=gauntlet/my_test/8cpu-2llc-smt)'

This is what the expansion looks like when nextest lists a test with min_llcs = 1 and default constraints on this host:

ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/12cpu-3llc-smt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/32cpu-4llc-nosmt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/32cpu-4llc-smt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/4cpu-1llc-nosmt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/4cpu-2llc-nosmt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/8cpu-2llc-smt
ktstr::worktype_coverage_fork_gauntlet_e2e gauntlet/worktype_fork_gauntlet_covers_all_arms/9cpu-3llc-nosmt

Under a multi-kernel run a {kernel_label} segment is appended — gauntlet/{name}/{preset}/{kernel}. See Test names and variants for the label format.

Topology presets

Note

Multi-NUMA and scale-boundary presets are opt-in. The default constraints (max_numa_nodes = 1, max_llcs = 12, max_cpus = 192) exclude the six *numa-* presets plus 240cpu-15llc-smt, 240cpu-15llc-nosmt, 252cpu-14llc-smt, and 252cpu-14llc-nosmt — 15 of the 25 gauntlet presets are active by default. Raise max_numa_nodes, max_llcs, or max_cpus on the test to opt in.

The catalog has one further preset, 192cpu-11llc-smt (26 in all), that is verifier-only: its non-uniform LLC sizing cannot be expressed through the gauntlet execution path, so it never fans out as a gauntlet/… variant — only the BPF Verifier Sweep boots it.

PresetTopologyCPUsLLCsNUMADescription
4cpu-1llc-nosmt1n1l4c1t411Single LLC
4cpu-2llc-nosmt1n2l2c1t421Minimal multi-LLC
9cpu-3llc-nosmt1n3l3c1t931Odd CPU count
15cpu-5llc-nosmt1n5l3c1t1551Prime LLC count
14cpu-7llc-nosmt1n7l2c1t1471Prime LLC count
8cpu-2llc-smt1n2l2c2t821SMT enabled
12cpu-3llc-smt1n3l2c2t1231SMT, 3 LLCs
32cpu-4llc-smt1n4l4c2t3241Medium topology
64cpu-8llc-smt1n8l4c2t6481Medium, many LLCs
128cpu-4llc-smt1n4l16c2t12841Large, few LLCs
128cpu-8llc-smt1n8l8c2t12881Large, many LLCs
240cpu-15llc-smt1n15l8c2t24015115-LLC SMT topology
252cpu-14llc-smt1n14l9c2t252141Near KVM vCPU limit
192cpu-11llc-smt1n11l*c2t192111Verifier-only. Non-uniform LLCs — ten of 18 CPUs + one of 12. Forces continuous overcommit (forced_cpu_budget = 96) so 192 vCPUs always time-slice. Targets schedulers assuming equal-sized caches.
32cpu-4llc-nosmt1n4l8c1t3241Medium, no SMT
64cpu-8llc-nosmt1n8l8c1t6481Medium, many LLCs, no SMT
128cpu-4llc-nosmt1n4l32c1t12841Large, no SMT
128cpu-8llc-nosmt1n8l16c1t12881Large, many LLCs, no SMT
240cpu-15llc-nosmt1n15l16c1t24015115-LLC topology, no SMT
252cpu-14llc-nosmt1n14l18c1t252141Near KVM vCPU limit, no SMT
2numa-32cpu-2llc-smt2n2l8c2t3222Multi-NUMA, 2 nodes, one LLC per node, SMT
2numa-16cpu-4llc-nosmt2n4l4c1t1642Multi-NUMA, 2 nodes
2numa-128cpu-8llc-smt2n8l8c2t12882Multi-NUMA, 2 nodes, SMT
2numa-128cpu-8llc-nosmt2n8l16c1t12882Multi-NUMA, 2 nodes, no SMT
4numa-32cpu-8llc-nosmt4n8l4c1t3284Multi-NUMA, 4 nodes
4numa-192cpu-12llc-smt4n12l8c2t192124Multi-NUMA, 4 nodes, SMT

Topology format: {numa_nodes}n{llcs}l{cores_per_llc}c{threads_per_core}t — 1n2l4c2t is 1 NUMA node, 2 LLCs, 4 cores per LLC, 2 threads per core = 16 CPUs. Note that llcs is the total across the machine, not per node. A * in the cores field (1n11l*c2t) marks a non-uniform machine whose LLCs are not all the same size — 192cpu-11llc-smt is ten LLCs of 9 cores plus one of 6 (packing width 9); the guest observes the uneven layout via fixed-width, partially-populated APIC-ID blocks.

aarch64: ARM64 CPUs do not have SMT. Presets with threads_per_core > 1 are excluded on aarch64, leaving 14 presets (the 5 small presets, 6 -nosmt variants, and 3 non-SMT NUMA presets).

Constraint filtering

#[ktstr_test] topology constraints filter which presets a test runs on. A preset is skipped when any constraint is not met:

  • num_numa_nodes() < min_numa_nodes
  • max_numa_nodes is set and num_numa_nodes() > max_numa_nodes
  • num_llcs() < min_llcs
  • max_llcs is set and num_llcs() > max_llcs
  • requires_smt and threads_per_core < 2
  • total_cpus() < min_cpus
  • max_cpus is set and total_cpus() > max_cpus

See The #[ktstr_test] Attribute for the attribute table.

Authoring gauntlet-ready tests

Worked example

A test with min_llcs = 2, requires_smt = true, and default max_numa_nodes = 1 against the preset table above:

  • 4cpu-1llc-nosmt (1 LLC): excluded — below min_llcs
  • All non-SMT presets (4cpu-2llc-nosmt, 9cpu-3llc-nosmt, *-nosmt): excluded — requires_smt
  • 240cpu-15llc-smt (15 LLCs): excluded — above default max_llcs = 12
  • 252cpu-14llc-smt (252 CPUs, 14 LLCs): excluded — above default max_cpus = 192 (also above default max_llcs = 12)
  • All *numa-* presets: excluded — above default max_numa_nodes = 1

Result: 6 of 25 gauntlet presets survive (8cpu-2llc-smt, 12cpu-3llc-smt, 32cpu-4llc-smt, 64cpu-8llc-smt, 128cpu-4llc-smt, 128cpu-8llc-smt). (192cpu-11llc-smt is verifier-only and never in the gauntlet count.) On aarch64, none survive — all aarch64 presets lack SMT.

Variant count

The total number of gauntlet variants for a test is valid_presets × resolved_kernels: the 6 surviving presets above produce 6 variants under a single kernel and 12 under --kernel A --kernel B.

Tests that skip gauntlet

Entries with host_only = true never produce gauntlet variants — they run on the host without booting a VM, so topology variation carries no signal. Tests whose names start with demo_ are ignored by default, gauntlet variants included.

Operator notes

  • Wall time. Each variant boots its own VM and runs the full scenario, so a sweep costs roughly (surviving presets × the per-run wall time you observe for the base test). nextest runs variants in parallel within your host’s budget. For a coverage-per-second subset under a deadline, use budget-based selection.
  • Memory. Each gauntlet VM gets max(cpus × 64 MiB, 256 MiB, entry.memory_mib) of guest RAM (plus an initramfs-derived floor). For the 252cpu-14llc-* presets that is at least 16128 MiB — the host needs that much free memory to run the variant.