Run Modes
Every ktstr VM boots in one of three run modes, and the mode decides how much of the host the run reserves, how its vCPU threads are placed, and therefore how much a timing measurement can be trusted. The mode is not a tuning knob you reach for mid-test — it is the contract the run makes with the host and with every concurrent ktstr process sharing it.
Probes a 1:1 shared-LLC/exclusive-CPU placement once, converts a success to lifetime CPU-SH, then immediately uses a shared vcpus + 1 pool if the probe fails. Default peers may overlap; 1:1 is an optimization, not a promise.
Whole LLCs and their CPUs held exclusively, vCPUs pinned 1:1 under SCHED_FIFO with a reserved service CPU, NUMA-bound hugetlb memory. Measurement fidelity at the cost of host sharing.
A shared subset of LLCs and exact shared CPU footprint sized to a budget, with vCPUs masked rather than pinned. Coordination without the full isolation contract.
Two of these modes have dedicated deep-dive pages — Performance Mode for the isolation contract and Resource Budget for the coordination mechanics. This page is the concept that unifies all three: what each promises, how the shared reliability machinery behaves under each, and how to choose.
The three modes
| Default | Performance | No-perf | |
|---|---|---|---|
| Selected by | neither switch set | performance_mode = true | --no-perf-mode / KTSTR_NO_PERF_MODE, or a cpu_budget |
| LLC lock | shared (LOCK_SH), exact candidate or fallback pool | exclusive (LOCK_EX), one per virtual LLC | shared (LOCK_SH), on a budgeted LLC subset |
| Per-CPU lock | exact admission probes with LOCK_EX, then retains LOCK_SH; fallback retains LOCK_SH on the admitted pool | exclusive (LOCK_EX) on every CPU covered by a whole-LLC claim; exact pinned/service CPUs at per-CPU grain | shared (LOCK_SH) on the exact budgeted pool; the cgroup cpuset enforces placement |
| vCPU placement | best-effort 1:1 pin; otherwise masked to a shared vcpus + 1 pool | pinned 1:1 to reserved cores | masked to the reserved CPU pool, not pinned |
| vCPU scheduling | SCHED_OTHER | SCHED_FIFO priority 1 | SCHED_OTHER |
| Memory | anonymous, THP via MADV_HUGEPAGE | NUMA-bound 2 MB hugetlb (MAP_HUGETLB, strict MPOL_BIND) | anonymous, THP via MADV_HUGEPAGE |
| What it promises | safe coexistence with performance isolation; default peers may time-slice | hard non-interference — a gap in the guest is the scheduler’s | bounded host footprint, not isolation |
The same contrast as a host sketch — how each mode’s vCPU threads land on host CPUs:
Default
The default path builds a bounded, deterministic set of complete 1:1
placements from the host’s actual CPU/LLC/NUMA shape and gives every
candidate one nonblocking probe. An immediately free candidate proves
LOCK_SH on its LLCs plus LOCK_EX on assigned host CPUs, converts the
CPU locks to LOCK_SH under the registry transition fence, and pins 1:1
(acquire_default_run_locks).
That exact result is deliberately opportunistic: its lifetime claim is
shared, so later default, no-perf, and build peers may overlap it. If a
compatible shared peer makes the instantaneous CPU-EX probe
unavailable—or if no exact topology maps—the run
immediately enters the same LLC-SH/CPU-SH admission machinery as no-perf,
requesting a Spread pool of vcpus + 1 CPUs clamped to its allowed set.
The returned pool is the mask used by every vCPU and worker. Exact-pinned
defaults, default fallbacks, and no-perf/build reservations can overlap;
default does not wait merely to recover exclusive CPUs.
Only hard performance exclusion changes that rule. A performance LOCK_EX
holder, or an earlier incompatible EX claim, cannot be crossed. The normal
test path may wait for enough shared capacity to become legal; interactive
and no-wait paths surface contention rather than booting without a
reservation. If the clamped pool is narrower than the guest vCPU count,
the sidecar records the actual pool and ktstr reports overcommit.
Performance
performance_mode = true takes a whole host LLC per virtual LLC under
LOCK_EX, publishes matching exclusive CPU claims for every CPU in those
domains, pins each vCPU thread 1:1 to a core within them, and reserves one
extra host CPU that no vCPU shares for the monitor and watchdog. The CPU
claims are deliberately redundant with whole-LLC ownership: they make the
cross-mode exclusion explicit to every default/no-perf shared claim
participants. On top of the pinning it runs vCPUs under SCHED_FIFO,
binds guest memory to the
pinned vCPUs’ NUMA nodes (strict MPOL_BIND, no silent remote fallback),
backs it with 2 MB hugetlb pages, and — on x86_64 — suppresses PAUSE/HLT
exits and hints KVM_HINTS_REALTIME so the guest drives its own
haltpoll instead of paying wakeup latency
(validate_performance_mode,
vcpu.rs,
numa_mem.rs).
It never runs unisolated: if the host cannot honor the guarantee the build
fails before boot and the test skips visibly. The full contract, failure
modes, and host-sizing are in Performance Mode.
No-perf
--no-perf-mode (or a cpu_budget) forces performance_mode = false and
routes the run through the budgeted path: the planner reserves a subset of
host LLCs sized to a CPU budget, takes LOCK_SH on them and on exactly the
budgeted CPU pool, and masks every vCPU thread onto that pool via a cgroup
v2 cpuset — no pinning, no RT scheduling, no hugetlb. Shared CPU claims
let compatible no-perf and default peers overlap deliberately
while still respecting exclusive performance owners. Placement is Spread: concurrent
no-perf VMs fan out across available LLCs rather than stacking onto the
same low-LLC prefix. Build-time planning is shape-only and owns no resource
fds, so immutable image preparation and other cold setup cannot sequester
host capacity. After image preparation, run() re-plans against live
holder counts and acquires the fresh plan’s exact LLC and CPU set; that
same set drives every vCPU and worker affinity mask
(acquire_run_locks,
no-perf arm in
builder.rs).
The CPU-budget resolution and the cpuset sandbox live in
Resource Budget.
Who selects what
Selection is by two switches, resolved at build time: performance_mode
on the test attribute or builder, and --no-perf-mode / KTSTR_NO_PERF_MODE
(any non-empty value). A per-test cpu_budget attribute or an operator
--cpu-cap implies the budgeted no-perf path. --no-perf-mode wins — it
forces performance_mode = false — so an operator can always strip
isolation off a test that asked for it.
The verifier sweep hardcodes no_perf_mode(true), and the reason is
instructive
(collect_verifier_output):
it only loads the scheduler’s BPF and reads the kernel verifier’s
load-time verified_insns counts, a value fixed at BPF load and wholly
independent of every perf-mode tuning (pinning, RT priority, hugepages,
NUMA mbind, exit suppression). It needs none of that. Disabling perf mode
selects a bounded no-perf placement whose LLC and CPU claims are both
shared. Compatible verifier cells can therefore enter together—even when
their budgeted masks overlap—while the admission registry still routes
them around disjoint capacity and fences an exclusive performance
owner. A performance_mode peer holding LOCK_EX can defer an
overlapping verifier cell, which is correct: the verifier must not perturb
an isolated peer’s pinned CPUs.
Shared reliability machinery
The safety and observability machinery runs in every mode; some of it reads the mode to decide how hard to press.
The progress watchdog is mode-independent
The three-tier progress watchdog
(watchdog_step.rs)
makes the same decisions regardless of mode — it reasons about a phase’s
CPU burn and runnable demand, not about pinning:
- Tier-1 (spinning wedge) — the busiest single vCPU’s in-phase CPU burn crossed the phase’s budget without reaching a milestone. Boot reuses the width-scaled boot-headroom allowance; attach, dispatch, and teardown keep flat, width-independent budgets, while Body has Tier-1 disabled. The evidence is the max per-vCPU burn, so a wide idle guest’s diffuse background CPU does not trip it; a lone spinner does. The budget is widened 3/2 under the pthread CPU currency to absorb VM-exit overhead.
- Tier-2 (silent idle wedge) — an INFRA phase sat past its wall backstop with live evidence channels and no runnable demand. The runnable conjunct is the load-bearing idea: a starved-but-alive cell with work always shows queued-or-running tasks in its own runqueue memory (readable regardless of host scheduling), so it is exempt at any dilation, while a cell with nothing runnable is not starved of anything — idle in an infrastructure phase past the backstop is the wedge. Deliberately no CPU term: a wide idle guest’s housekeeping-CPU burn scales with vCPU count, so no width-stable CPU floor exists.
- Tier-3 (deadman deferral) — the guest-derived hard deadline fires at the wall if the monitor is dead or the cell is inert (its busiest single vCPU’s CPU trickle stalled below a currency-dependent floor for two consecutive 10 s windows, and no milestone within a 60 s grace). An active cell may outlive that wall deadline under contention, but not forever: deferral also ends once the current phase has consumed more busiest-vCPU CPU than the VM’s entire effective deadline budget. That backstop is width-independent and dilation-immune — starvation stretches the wall needed to spend it instead of causing a false timeout.
Only the CPU currency differs by host, not by mode: the PMU task-clock (guest-only time) uses the tight floors, and the pthread fallback (which charges VM-exit overhead to an idle guest) uses widened floors so an idle wedge stays killable without a starved cell being misjudged.
FIFO-2 sensing threads
The watchdog and hang detector run at SCHED_FIFO priority 2
unconditionally, in every mode
(freeze_coord/mod.rs).
The detector’s own sensing must not dilate with the load it measures:
under extreme host dilation a SCHED_OTHER watchdog can miss its own
100 ms tick, latch monitor_live = false, and fire the deadman against a
cell that was actively progressing. FIFO-2 grants scheduling immunity at
~microseconds per tick. In perf mode the vCPUs sit at FIFO-1, below it; in
default and no-perf mode it outranks the SCHED_OTHER vCPUs — intended,
sensing must win. The service-CPU pin on those threads is the only
mode-gated part: it applies only in perf mode, where a service CPU was
reserved for them to pin to.
Halt-poll policy is mode-aware
KVM_CAP_HALT_POLL is set per-VM from the resolved mode plus the run-time
overcommit outcome
(halt_poll_policy):
| Mode / outcome | Halt-poll | Why |
|---|---|---|
| No-perf | 0 | the guest deliberately shares host CPUs; polling burns CPU that belongs to peers |
| Performance | module default | perf mode disables HLT exits and drives the guest’s own haltpoll — host polling is redundant |
| Default, exact pin or shared fallback | 0 | both retain CPU-SH, so a compatible peer may overlap later and polling would burn its time |
Hugepages
Perf mode reserves explicit 2 MB hugetlb pages (MAP_HUGETLB) and
NUMA-binds them; MADV_HUGEPAGE is not applied there (and is rejected on
hugetlb mappings anyway). Default and no-perf mode use anonymous memory
with MADV_HUGEPAGE, so THP applies subject to the host’s THP=madvise
policy — a hint, not a reservation
(numa_mem.rs).
Dilation reporting
Every run samples host-side vCPU scheduling dilation
D = 1 + Σrun_delay / Σon_cpu over its vCPU host threads
(HostVcpuSchedstat::dilation),
measured host-side deliberately so it captures the CPU the vCPU thread was
starved of by the host, orthogonal to the guest scheduler under test.
Its meaning shifts by mode: in perf mode, where vCPUs are pinned 1:1 to
reserved cores, D should read ~1.0, so a departure is an
isolation-violation indicator — something got onto those cores. In
default and no-perf mode, where vCPUs share host CPUs, D quantifies
the overcommit the mode accepted.
The contention witness and the wall-latency verdict
The Body (measurement) phase’s dilation D — plus a variable-width series of
runner-cgroup CPU-pressure deltas during that phase — form the contention
witness the wall-latency ceilings consume. Full vCPU schedstat sweeps happen
only at lifecycle transitions; the Body hot path reads one PSI counter on the
monitor’s already-existing wake, independent of vCPU count. A complete
task-specific schedstat delta over the widened Body span caps the PSI result,
so another concurrently running cell in the same runner cgroup cannot inflate
W beyond this VM’s own accumulated delay. If the opening snapshot races vCPU
startup, the complete whole-vCPU-thread lifetime total provides a looser but
sound enclosing cap. Each AP records that cumulative total once immediately
before its host thread exits, avoiding a teardown race with disappearing proc
entries without adding any periodic work. ScenarioEnd’s guest duration also
detects start/end lifecycle frames delivered together after host starvation;
that case discards the falsely tiny localized span and uses the same complete
lifetime as a conservative interval. From the series ktstr
derives W(L), the worst host delay any interval of length L could have
absorbed; pairing that with a gate’s measured latency turns
max_p99_wake_latency_ns from a plain threshold into the contention-aware
tri-state (pass always sound / fail only when refutation-proof /
indeterminate = a non-blocking annotated pass) described under
Wall-latency ceilings under host contention.
The same page covers the perf-mode counterpart to the isolation-violation
indicator above: when a performance_mode cell’s Body D exceeds the
perf-isolation ceiling, that is not a scheduler failure but an infra
fault, and the run’s timing verdicts are marked untrustworthy regardless
of any gate outcome.
Lock interaction across concurrent modes
The flock semantics compose so the modes can share a host safely. Shared holders coexist; an exclusive holder blocks every shared acquirer and vice versa. The practical matrix, per resource:
| Concurrent runs | LLC lock | Per-CPU lock | Result |
|---|---|---|---|
| Default + Default | both LOCK_SH — coexist | retained claims are LOCK_SH; exact admission probes use momentary LOCK_EX | an initially free run may pin; later shared peers may overlap it |
| Default + No-perf | both LOCK_SH — coexist | exact attempt bounces on no-perf LOCK_SH, then default uses LOCK_SH | shared pools may overlap deliberately |
| No-perf + No-perf | both LOCK_SH — coexist | both LOCK_SH — coexist | deliberate shared CPU pool |
| Any shared holder + Performance | LOCK_SH vs LOCK_EX — block | shared/exclusive CPU claims also block | perf waits for every overlapping shared holder |
| Performance + Performance | LOCK_EX each — mutually exclusive | LOCK_EX bridge mirrors the LLC ownership | serialized per LLC |
So exact-pinned defaults, default fallbacks, no-perf runs, and kernel builds may intentionally share the same CPUs. A perf-mode run waits for every overlapping holder, and while it holds its LLCs nobody else touches those CPUs. The per-resource lock table and the planning phases are in Resource Budget.
Tradeoffs
| Default | Performance | No-perf | |
|---|---|---|---|
| Measurement fidelity | reservation only — host noise remains | highest — host variance removed | lowest — vCPUs float on shared CPUs |
| Host sharing | shares LLCs and CPUs; may pin 1:1 when initially unshared | owns whole LLCs exclusively | shares a budgeted LLC subset |
| Contention behaviour | exact probe never waits; shared admission waits only for overlapping performance EX | wait + retryable fail (ResourceContention) | shared pool — compatible peers coexist; wait only for LOCK_EX |
| Cost | none beyond the reservation | needs (llcs·cores·threads)+1 CPUs, free LLCs, hugepages, CAP_SYS_NICE | one CPU budget, no privileges |
| Use when | correctness tests where pass/fail is binary | timing thresholds — gaps, spreads, wake-latency, A/B against the same host | multi-tenant CI, kernel builds beside perf runs, deliberate oversubscription |
The tradeoff space at a glance — each mode trades measurement fidelity against how densely the host can be shared, and each fails contention differently. Positions are ordered by the documented mechanisms; the dilation figures are the measured reference points:
Correctness tests (cpuset isolation, zero-work detection) do not need
perf mode — their verdict is binary and unaffected by host noise. Timing
tests (max_gap_ms, max_spread_pct, max_p99_wake_latency_ns) do, and
their results are comparable only against runs on the same host, since
guest-side jitter from shared caches and memory bandwidth remains even
under full isolation. See Checking for which checks are
timing-sensitive.
Validation evidence
The mode machinery is measured, not assumed. The committed evidence is
dilation_validation.md;
the raw numbers live there, summarized here.
The reliability rework does not shift the measurements it annotates. The full matrix was re-run against the pre-branch baseline at the branch’s final code state (release builds, N=6 per side, one run at a time on a load-gated host): across the perf-mode steady and across-detach cells, the default-mode pipe, split, and iteration-rate cells, and the narrow wakeup/request percentiles, every metric lands inside the baseline’s run-to-run envelope or within integer-µs bucket noise, and negative-detection is 12/12 on both sides. The systematic departures are improvements: the default-mode split request tail and the wide request median are reproducibly faster — flagged for re-baselining, not blunted thresholds.
The dilation samples read as expected, and are the reference points for what each mode’s placement produces on a quiet host:
- ~1.0x — a perf-mode 1:1-pinned cell (near-ideal, the isolation contract holding).
- ~1.08x — the steady perf schbench cell; ~1.13x a near-1:1 wide no-perf cell; both undiluted.
- ~1.2x — the per-phase-across-detach cell, where a mid-run scheduler swap widens the tail.
The wide-topology finding, re-measured on the final code. An interim capture on wide (56 vCPU) near-1:1 no-perf cells showed the unconditional FIFO-2 sensing threads pushing wakeup-p99 up ~40%. Re-measured at the branch’s final code state against the same baseline (release builds, N=6 per side), that tail is gone: wide wakeup-p99 lands at 0.983x the baseline with a far tighter spread. What remains is a small, constant cost in its place — wide wakeup-p50 shifts from ~6.5 µs to ~15.8 µs (+9.3 µs, spreads disjoint), the per-wakeup preemption charge of the RT sensing threads spread uniformly instead of erratically. It surfaces only at width (narrow wakeups are identical to the µs on both sides), only on the wakeup median, and only in no-perf mode, whose contract masks rather than pins and does not promise timing isolation. Throughput is unaffected (rps +0.9%, loop count +0.5%, in-envelope) and the wide request median is reproducibly ~8% faster. A cross-commit absolute baseline on wide-cell wakeup-p50 should be re-taken; a threshold test, which gates on metric presence, is unaffected.
The contention witness no longer scales its hot-path file reads with VM width. Earlier validation laddered the original per-tick O(vCPU) schedstat sweep and found no measurable steady-schbench shift, but the shape was still an avoidable observer cost on wide cells. The current path takes those sweeps only at lifecycle transitions and uses one cumulative runner-cgroup CPU-pressure read per Body monitor tick. That scope retains delayed-vCPU coverage without charging pressure from unrelated host cgroups, and a task-specific lifecycle schedstat cap removes noise from other cells sharing the runner cgroup. If scoped PSI or the opening lifecycle snapshot is unavailable, a complete whole-vCPU-thread lifetime total provides a coarser enclosing fallback. The guest-reported scenario duration selects that fallback too when queued lifecycle frames under-cover the real workload. The worker’s per-checkpoint CPU-clock read remains; its historical worst-case SpinWait cost and the original ladder are retained in the validation record.
The watchdog catches real wedges fast and leaves healthy cells alive.
The injected-wedge fixtures
(progress_watchdog_e2e.rs)
kill a spinning Teardown wedge on Tier-1 once its lone hot vCPU burns the
flat CPU budget (measured 16–35 s total, run-to-run wedge-start variance)
and a silent idle wedge on Tier-2 at the 15 s phase wall backstop
(measured ~19 s total) — both far below the ~150 s Tier-3 deadman, and at
1 vCPU and 64 vCPUs alike: the wide idle wedge, formerly bounded only by
the harness terminate-after because the old summed-CPU-trickle conjunct
stayed above any fixed floor at width, dies on the same backstop as the
narrow one (measured 19.2 s) now that Tier-2 carries no CPU term. No
watchdog tier fired on any of the healthy wide runs, and the
previously-false-killed 256-vCPU CI shapes survive their long idle
phases.
Related
- Performance Mode — the full isolation contract, prerequisites, and failure modes.
- Resource Budget — the CPU budget, the cpuset
sandbox, lock planning, and
ktstr locks. - Checking — which checks are timing-sensitive and so want perf mode.
- Monitor — the host-side observer the watchdog and dilation sampling build on.