Diagnostics & observability
You can't govern
what you can't see.
A safety governor that goes dark under failure is worse than none — operators must be able to tell “locked out” from “down” at 2 AM. Kirra's observability surface is engineered for exactly that moment: reads survive lockout, denials explain themselves, and the audit trail proves its own integrity.
The operator surface
Observability that survives the bad day
Observability GETs are deliberately posture-exempt: a read cannot actuate, and blocking it would remove fleet visibility exactly when an operator needs it most. Writes on the same prefixes stay fully gated.
Fleet posture, live
Point-in-time posture per fleet and per node, time-series history, flap detection, and a server-sent-events stream of every posture transition — each event carrying its generation number, so consumers can detect reordering and staleness.
Denials that explain themselves
Every denied actuator command mints a verdict id and becomes a signed, human-readable artifact:
the machine deny-code, an operator-language explanation, the exact recorded inputs, and the audit-chain
fields — served to the auditor role at GET /verdicts/{id}.
An audit trail that proves itself
Chain verification and export endpoints let an operator — or a regulator — confirm ledger integrity on demand, and the shipped off-box stream re-verifies independently with no access to the source database.
Metrics
A scrape that survives lockout
The Prometheus endpoint is posture-exempt so monitoring keeps working while the fleet is locked out — the moment dashboards matter most. Fleet-safety series sit alongside rollout and credential-lifecycle series.
| Series | What it tells an operator |
|---|---|
| kirra_ota_campaigns_total{state} | Rollout campaigns by lifecycle state — a halted campaign is visible fleet-wide |
| kirra_ota_campaign_rollout_percent | Staged rollout progress per campaign |
| kirra_ota_campaign_applied_nodes | Adoption numerator — which nodes actually run the artifact |
| kirra_cert_principals{state} | mTLS credential census: active, revoked, expired, expiring soon |
Operator tooling
Three consoles, one honesty rule
Operator UIs never overstate what happened. The air-gapped console renders a clearance grant as “GRANT RECORDED — DELIVERY PENDING” rather than implying a vehicle was released — the fail-closed philosophy applied to pixels.
Runs entirely in your browser on a simulated fleet — clearly labeled demo data, no backend, nothing to install. The same Next.js console operators run against a live verifier.
Fleet console
The full operator application: posture trends, envelope plots, fleet topology and position maps, version-adoption views, incident replay maps, and audit evidence browsing.
Air-gapped console
A single self-contained HTML file — inline styles and scripts, no CDN, no build — for secured environments where the recovery surface must work with nothing but a browser.
Ops dashboard
A lightweight dashboard shipped alongside the verifier in Docker Compose, with its own isolated request pool so an API flood can't starve the operator's view — and vice versa.
Health & drift
The quiet failures, made loud
Config drift detection
Every boot commits a SHA-256 digest of the effective configuration to the audit chain, and unknown
KIRRA_* variables warn at startup — a typo'd setting that silently isn't taking effect
becomes visible instead of latent.
Credential lifecycle
An hourly census warns on expiring and lapsed mTLS principals with hash-chained audit entries — pure observability, because the auth path already fail-closes expiry on its own.
Liveness watchdogs
Telemetry silence, stale posture caches, and silent redundancy channels all surface as posture changes within their sweep budgets — absence of data is itself a monitored signal.
READ THE SOURCE
See the whole system's state in one place.
Bring up the verifier and dashboard with Docker Compose, watch the posture stream, and pull a verdict explanation yourself.