Fleet Dashboard Plan
⚠️ SUPERSEDED 2026-10-20 — consolidated into
[fleet-dashboard-design-of-record-2026-10-20](fleet-dashboard-design-of-record-2026-10-20)(Nova). Kept for the record.Three of this plan's premises are now wrong (measured 2026-10-20, not inferred):
- The board source is retired. §"Verified data sources" and §5.2 C1 name
Pulse/Kanban/PULSE-BOARD.kanban+Docs/Kanban/SOLOS-V3.mdas the goals/tasks input.SOLOS-V3.mdwas deleted by Young (5b48edc), its generator cron is paused, and its script now fails closed. Task truth ishermes kanban. See05-operations/kanban-projection-decision-2026-10-20.md.- The GO window source is NOT blocked. §6.1 records a 401 and concludes "a collector script cannot read the GO window percentage today". The credential is a readable
.envon Pulse, not the OS keyring.GET /zen/go/v1/usagereturns live tiers — measured 2026-10-20: rolling 40% / weekly 38% / monthly 48%, all[OK]. §6.1's "renders unavailable" fallback is no longer needed.- §5.3's colour rules are in force but the "stuck" threshold is not 300 s. Hermes ships
DEFAULT_CLAIM_TTL_SECONDS= 15 min andstranded_threshold_seconds= 30 min; a 300 s local rule fires on healthy workers. Use the native thresholds.The three design questions (§Phasing, §Lanes, §Open questions) and the collector spec §5.2 are reconciled, not discarded, in the design of record. The cost panel §6.2 (real
opencode stats --cost --jsonnumbers) and the defects §6.4 D1/D2 are carried forward verbatim — they are still open.
Fleet Dashboard — plan for team review (2026-10-16)
Young: "build our dashboard — display goal and tasks, agents health, system health. Before we build it, plan it out with the Team." This note is that plan. No build until Sol confirms the Stage 8 data contract and lanes ack.
Grounding — this is Stage 8's surface, not a new invention
Sol's stage-8-observability.md already owns "how we see a running V3": control-in-force readout, audit trail, model-window readout, task-age readout, emptiness-is-loud. The dashboard is the human face of Stage 8 — the plan below maps Young's three panels onto Sol's five steps, so we build one data layer that serves both. PLAN.md rule holds: a phase is done when its proof runs — each collector below ships with a probe.
Verified data sources (probed today — real surfaces, no invention)
| Panel | Data | Source (verified live) | Notes |
|---|---|---|---|
| Goals & Tasks | Kanban board + task states | vault/Pulse/Kanban/PULSE-BOARD.kanban, Docs/Kanban/SOLOS-V3.md (git-synced, already mirrored to docs-site) | board = the one task store (per kanban-as-the-one-board decision); A2A in-flight tasks via tasks/get for the live view |
| Agents health | endpoint up/down, last exchange, skills non-empty | A2A ports per agent (fleet-ssh-tunnels.md registry: laptop 18301-18313, LAB→laptop 18441-18443), gateway listeners (today's live check: 8311-8313 UP, 8301/8305 DOWN — that IS the health signal), ~/.hermes/a2a_audit.jsonl for last-exchange ages | emptiness-is-loud: agent serving skills: [] or endpoint down shows RED, never silently absent |
| System health | node RAM/disk/services/tunnels/crons/vault-parity | prod: free/df/systemctl (23Gi/180G free, solos-docs+pulse-site active), docker 29.7.2; tunnel forward probes (the one-liners in fleet-ssh-tunnels.md); vault git parity LAB↔origin↔prod (sha check like my sync chain); cron last-run markers (/tmp/vault-pull.log, delivery=local quiet-failure rule) | GO/ZEN model windows: GET opencode.ai/zen/go/v1/usage (provider-authoritative, Stage 8.3) — Mira's cost panel rides here |
Backend contract = Stage 8's observe.py (exists, proof hasn't run). The dashboard consumes it; we do NOT build a second collector stack.
Architecture (build-on-top, consistent with the Docker design)
collectors (Stage 8 observe.py + thin shims) read-only
→ one JSON status surface: /api/status.json loopback-only
→ nginx (auth_basic) serves a dashboard page dash section or subdomain (Young's call)
Frontend: thin page (same Nuxt/Docus stack as docs-site, or a tiny static page
with fetch+render) — Nova's lane once the docs nav lands.
- Read-only. The dashboard reports; it does not control. No restart buttons in v1.
- Loopback + nginx gate only (same pattern as docs/pulse sites; per-tenant rules don't change).
- Every panel declares its source + staleness. A panel that can't reach its source shows "STALE since
" in red — absence announces itself (Stage 8's whole premise). - Refresh: collectors run on cron (2–5 min), dashboard renders the last snapshot — no heavy live polling from the browser.
The three panels (spec)
- Goals & Tasks — Kanban board projection (columns/cards/owners from the .kanban file, auto-synced via git — the board Young ruled "the one board"); in-flight A2A tasks with age vs the 300 s rule (Stage 8.4) shown separately as "Live tasks".
- Agents Health — one row per agent (11): endpoint status (probed), last successful exchange (from a2a_audit.jsonl), skills count (empty = loud), lane, model, GO window % (from usage endpoint, Stage 8.3). Red/green/amber rules defined per column.
- System Health — hosts (laptop, LAB, prod): RAM/disk/services/tunnels-up/crons-last-run/vault-parity; plus backup freshness (gdrive archive latest date — ties to Stage 9/Mira).
Lanes
| Lane | Who | Scope |
|---|---|---|
| Data contract + observe.py proof | Sol (Stage 8 owner) | confirm the status.json schema covers 8.1–8.5; run the first proof |
| Collectors + wiring + ops (hosts, tunnels, cron, vault-parity probes) | Tam | I already probe all of these manually today — script them into collectors |
| Cost/window panels | Mira | GO/ZEN window display rules, cost framing |
| Frontend | Nova | after docs-nav fix (her current lane); thin UI over status.json |
| Build support | Ace | schema + probes implementation with me |
| Coordination | Min | lane sequencing |
Phasing (each with its proof)
- P0 — data contract (Sol + me):
status.jsonschema v0 (agents, tasks, systems, windows), collectors for what exists TODAY (ports, kanban, disk/ram, audit ages, vault parity). Proof:status.jsonserved on loopback with real values, STALE markers work. - P1 — first dashboard page: static render of status.json behind nginx auth. Proof: Young opens it and sees all three panels with live data.
- P2 — refresh + alerts: cron cadence, GO-window warnings (Mira's thresholds), stuck-task alarm at 300 s (Stage 8.4). Proof: kill a tunnel → panel goes red within one cycle without human probing.
Open questions for Young
- Placement: a section inside docs.youngle.au, or its own host (dash.youngle.au later with tenants)? Default: docs-site section first (zero new infra).
- Auth: same auth_basic gate? Default: yes.
- Anything that must NOT be on the dashboard (secrets surface discipline — tokens/keys never rendered).
Register
- This note = the plan of record for the dashboard until Young rules.
- Next action: Sol confirms/annotates the Stage 8 contract (or delegates), lanes ack, then P0 starts with my collectors.
§5 Design — collector spec (Tam lane, appended 2026-10-16 evening)
Concrete design for the data layer: every probe, field, cadence and colour rule. Sol: this is what status.json v0 carries — amend or rule, then P0 is mechanical.
5.1 status.json v0 schema (draft)
{
"generated_at": "<iso8601>",
"stale": false,
"tasks": { "board": { "columns": {...}, "cards": [...] },
"live": [ {"id","agent","age_s","stuck":bool,"state"} ] },
"agents": [ {"name","endpoint","up":bool,"last_exchange_s","skills_count",
"lane","model","window":{"model","percent","status","resets_at"} } ],
"systems": [ {"host","ram_used_gb","ram_total_gb","disk_free_gb",
"services":[{"name","active"}],"tunnels":[{"name","up"}],
"crons":[{"name","last_ok_s"}],"vault_parity":bool} ],
"backups": { "gdrive_latest": "<date>", "age_h": N }
}
Rules: every array element carries its own checked_at; a failed probe fills the field with "error: <reason>" and sets that element's stale=true — the panel renders red, never omits.
5.2 Collectors (what each probes, verified sources only)
C1 — board (tasks):
- Source:
Pulse/Kanban/PULSE-BOARD.kanban+Docs/Kanban/SOLOS-V3.mdvia the prod clone (already git-synced). - Parse: cards per column, owner, due-ish dates. Cadence: every 5 min (cheap: one file read).
- Live A2A tasks: poll each agent's endpoint with
tasks/getfor recent ids froma2a_audit.jsonl; age vs 300 s →stuck. Cadence: 2 min.
C2 — agents health:
- Endpoint probe: HTTP GET per A2A port from
fleet-ssh-tunnels.mdregistry (loopback forwards). Expect any HTTP response = up (200/405/JSON-RPC error all prove the server listens); connection refused/timeout = down. Cadence: 2 min. last_exchange_s: tail~/.hermes/a2a_audit.jsonlper agent (LAB + laptop copies). 0-30 min green, 30 min–6 h amber, >6 h red (tunable by Min).skills_count: from the agent card (/.well-knownunavailable → fromtasks/getagent metadata or the skills registry projection once Stage 1 wires it) — emptiness is LOUD (Stage 8.5).window:GET /zen/go/v1/usage(provider-authoritative; Stage 8.3) — needs one credential from Mira's pool config; percent ≥80 amber, ≥95 red (Mira tunes).
C3 — systems health:
- prod (via server key):
free -h,df -h /,systemctl is-activefor solos-docs + pulse-site + vault-sync timer, docker ps count. Cadence: 5 min. - Tunnels: the one-liners in
fleet-ssh-tunnels.md§5 (mesh 18311/18312/18313 + reverse 18443; tam-tunnel 8433) — run from the laptop collector. Cadence: 2 min. - Crons: last-run markers — vault-sync log mtime, delivery=local cron outputs (
state.dbexecution query or the cron registry API). Silent-failure rule: a cron that hasn't stamped in >2× its period = red. - Vault parity:
git rev-parse HEADon LAB vs origin vs prod clone — mismatch = amber; push/pull broken >1 h = red.
C4 — backups (Mira/Stage 9 tie-in):
rclone lsjson gdrive:vault-archivelatest date + on-prod/home/ubuntu/archivepresence. Age >24 h = amber, >48 h = red.
5.3 Colour rules (v0 defaults — Min/Mira tune)
| Signal | Green | Amber | Red |
|---|---|---|---|
| Agent endpoint | responds | — | refused/timeout |
| Last exchange | <30 min | <6 h | >6 h |
| In-flight task | <300 s | — | ≥300 s (stuck) |
| RAM (prod) | <70% | <85% | ≥85% |
| Disk free | >40G | >20G | <20G |
| Tunnel | up | — | down |
| Cron freshness | ≤2× period | ≤4× | >4× or silent fail |
| Vault parity | all 3 equal | drift <1 h | drift >1 h |
| GO window | <80% | <95% | ≥95% |
| Backup age | <24 h | <48 h | ≥48 h |
5.4 Ops shape
- Collector = one script per host section (bash+python, no new framework), writes
status.jsonatomically (tmp+mv), runs from cron on the laptop (it can reach LAB via mesh and prod via tunnels — the hub of the hub-and-spoke). - Serving: prod nginx serves the JSON + dashboard page behind the existing auth gate; the JSON lives at
dash.youngle.au/status.jsonpath or docs-site/dashboard/section (Young's pending call). - Security: collectors read-only; no tokens in status.json (GO window shows percent only, never the key); nginx gate mandatory.
- Failure rule: collector cron is
deliver=local, quiet-failure(Young's standing rule); the dashboard's own staleness banner is the alert.
5.5 P0 proof (ready to run on Sol's ack)
status.json generated with real values for: 8 agents probed (including nova/ace RED — today's live outage as the demo), board snapshot, prod RAM/disk/services, 3 tunnel probes, vault parity. Served loopback + behind nginx. Young opens it and sees the fleet as it is this minute — that's the P0 exit proof.
§6 Cost & window panels — Mira's lane (appended 2026-10-16, probed live)
Tam's §5.2 C2 says the window column comes from GET /zen/go/v1/usage and "needs one
credential from Mira's pool config". I probed it: that source does not work as
specified, and the credential is not in a pool config. Everything below is measured
tonight, not assumed.
6.1 The window source — measured, and blocked
| Probe | Result |
|---|---|
GET opencode.ai/zen/go/v1/usage | 401 AuthError — path exists, wants auth |
GET opencode.ai/zen/v1/usage | 404 — does not exist on ZEN |
GET opencode.ai/zen/go/v1/models | 200 (catalogue only — no usage) |
opencode auth list | 2 workspaces stored: SolOS, Pulse |
| Credential location | OS keyring (~/.local/share/keyrings/), not config, not env |
Consequence: a collector script cannot read the GO window percentage today. The
credential lives in the OS keyring (the E5 property — auth is user-level, found by
uid-fallback), and there is no python3 -m keyring on this host and no env var. So
§5.2 C2's window.percent field cannot be filled with provider truth, and
§5.3's <80 green / ≥95 red row has nothing to colour.
I will not put a fabricated percentage on the dashboard. Per Stage 8's own premise — absence announces itself — the window panel ships v1 as:
window: { model: "go-allowance", percent: null,
status: "unavailable", reason: "no script-readable credential path (401 on /zen/go/v1/usage; key in OS keyring)" }
…rendered grey with the reason shown, never green-by-default. A panel that says "I cannot see this" is honest; one that shows 0% is the V2 defect wearing a UI.
Path to real numbers (P1, needs a decision): one of —
(a) a opencode CLI subcommand that prints usage (none exists today — stats is the
only read), (b) a small helper that pulls the key from the keyring under the right uid,
or (c) accept derived signals only (below) and drop the provider-truth claim.
6.2 What DOES work today — the derived cost panel
opencode stats --cost --json is real, script-readable, needs no credential, and
carries everything a cost panel needs. Probed tonight:
| Signal | Measured now | Source |
|---|---|---|
| Lifetime spend | $22.38 | stats --cost --json |
| Per-model cost | qwen3.8-flash $11.37, deepseek-v4-flash $7.57, gpt-6-luna $0.74, kimi-k3 $0.72 | models[] (24 paid models in 30d) |
| Cache read ratio | 91.8% (6.32b cached / 561m input) | tokens.cache.read |
| Free vs paid | free models carry the bulk of tokens at $0.00 | models[].cost == 0 |
v1 cost panel = these four rows. No credential, no new endpoint, real numbers.
6.3 Colour rules (my tuning of §5.3)
Replaces the single GO-window row with four implementable ones. Thresholds from the ledger's own alert rule and tonight's measurements:
| Signal | Green | Amber | Red |
|---|---|---|---|
| Daily paid spend | < $1.00 | $1.00–5.00 | > $5.00 (= ledger THRESHOLD) |
| Cache ratio | ≥ 80% | 50–80% | < 50% (91.8% today — a drop is the early warning) |
| Free-share of tokens | ≥ 95% | 80–95% | < 80% (paid crept in) |
| GO window % | — | — | renders "unavailable" until §6.1 is solved |
| Backup age | < 24 h | < 48 h | ≥ 48 h (Tam's, unchanged) |
6.4 Two defects found in my own lane while probing this
D1 — the spend ledger can report $0.00 while the fleet is working. spend-ledger.py
queries opencode stats --days 1, which counts opencode CLI sessions only. The
Brain agents run on Hermes, so their model traffic is not in opencode's local DB.
Measured tonight: ledger rows for 10-13 → 10-15 all read $0.0000, while lifetime
spend is $22.38 and 24 paid models appear in a 30-day window. This is the same
defect class as V2's "$0.00000 across 30,721 attempts" — a cost record that is
structurally blind to where the money goes.
Test to close it: compare stats --cost deltas against a known Hermes-routed paid
call. Until then the ledger's zero is "not measured", not "nothing spent".
D2 — the ledger's date column is UTC while its cron runs in AEDT. The 09:00 AEDT
run is 22:00 UTC the previous day, so rows are labelled one day earlier than the run
that wrote them (today's 09:00 run wrote a 2026-10-15 row). Harmless for trends,
misleading for "what did we spend today". One-line fix in the script.
6.5 What I am NOT doing
- Not editing §5.1–§5.5 (Tam's spec) — this section annotates, it does not overwrite.
- Not touching the collector scripts until Sol confirms the §5.1 schema — lane rule.
- Not fixing D1/D2 in
spend-ledger.pyuncommitted alongside Tam's in-flight work; both are registered here for her or Min to sequence.

