Benchmarks & gates
Numbers with provenance.
Marketing benchmarks are usually unfalsifiable. These aren't: every figure below is produced by code in the repository, most of them gate CI, and each one links to exactly where it comes from. Host-timing numbers are labeled indicative — certified timing awaits the QNX target.
Scenario KPI gate · WS-3.1
Safety KPIs that block merges
A deterministic corpus — 248 doer scenarios and 71 perception frames — runs on every PR with hard floors, and a 40,000-sample Monte-Carlo campaign runs nightly with 95% Wilson confidence bounds. Negative controls must breach, proving the gate can see the faults it claims to catch.
Admissibility & safety floors — per-PR gate
Fraction of proposals admitted without MRC fallback; observed vs gated floor.
ci/scenario_kpi_thresholds.jsoncrates/kirra-kpi-gate/src/lib.rs
The hard zeroes
Some floors aren't percentages — they're absolutes. Per-PR: unsafe_miss_rate ≤ 0.0,
hazard_recall ≥ 1.0, differential MissedTighten = 0 and
PhantomTighten = 0. Fusion loosening the corridor is a hard zero, always.
Nightly Monte-Carlo (seed 424242424242, 40k doer + 20k perception samples) gates the 95% Wilson bounds: geometric admissibility lower bound observed at 0.9636 against a 0.93 floor; unsafe-miss upper bound 0.0126 against a 0.03 ceiling; hazard-recall lower bound 0.9852 against 0.95; negative-control breach lower bound above 0.90.
ci/scenario_kpi_montecarlo.json.github/workflows/kpi-campaign-nightly.yml
Test inventory
Where 3,305 tests live
Test functions by tree
#[test] + #[tokio::test] counts, whole repository.
Beyond example-based testing
32 property-based test suites (proptest) with committed regression seeds · 12 Kani proof harnesses · 4 Loom concurrency models · 4 fuzz targets, deep-fuzzed 90 minutes each weekly with persisted corpora · mutation testing against checker diffs · the same storage conformance suites run against three backends including live Postgres 16.
proptest-regressions/verification/kani/crates/kirra-loom-models/src/lib.rsfuzz/fuzz_targets/.github/workflows/fuzz-deep-weekly.yml
Decision coverage — gated floor vs measured
Branch/decision coverage on the checker crates (true MC/DC is toolchain-blocked upstream, tracked as #65).
Transport read latency — p50, host-indicative
From the repository's own benches; not certified WCET.
Doer evaluation
Measuring the untrusted half
The doer-eval harness scores planners on two axes the safety case cares about: admissibility (how often the checker admits their plans without MRC fallback) and plan quality against a reference. The committed scorecard shows a v2 model trading strict-acceptance for progress — visible, quantified, and safe to ship because the checker holds either way.
crates/kirra-doer-eval/src/lib.rsartifacts/doer-eval/scorecard.json
READ THE SOURCE
Reproduce any number on this page.
Each chart names the file that produces it. Clone, run, compare — that's the point.