Fleet Dashboard — consolidated design of record + implementation plan (2026-10-20)

This is the design of record for the fleet dashboard. It reconciles three documents into one and re-verifies every load-bearing claim against the live system.

Lane note: sources 1–3 are Tam's and Sol's authority documents. This file does not overwrite them silently — it states what merged, what changed, and what is now false. Min owns the register; Young owns the ruling.

Status of this document: design-for-review. Section 11 lists what Young must rule on before Phase 1 starts.

0. What changed since the three source docs (measured, 2026-10-20)

Re-verifying the plan's premises against the live system moved three things. Two of them change the implementation plan; one closes an open item outright.

C2 correction, stated plainly: Mira's §6.1 "blocked" finding was correct on the evidence she had (a 401 with no script-readable path) but its conclusion — "a collector script cannot read the GO window percentage today" — is now false, and the reason it was false is that the credential was on a host she was not standing on. Per design-record-authoring: re-measure a register item's premise before reporting it as still pending. It is superseded, not open.

1. Overview & Goals

Young: "build our dashboard — display goal and tasks, agents health, system health."

The dashboard is the human face of Stage 8 (observability) — a read-only web surface projecting the fleet's live state.

Core principles (unchanged across all sources):

• Read-only. The dashboard reports; it does not control. No restart buttons in v1.
• No false green. Missing/stale data shows unknown (grey) or STALE (red) — never a default-green that reads as healthy.
• Absence announces itself. Every panel declares its source and its staleness. A failed probe fills the field with "error: <reason>" and sets stale=true. The panel renders red; it never omits.
• One data layer. All panels consume one surface. No per-panel collectors.
• Zero new infra. Same Docus/Nuxt stack as docs.youngle.au, same auth gate.

2. Architecture

Stage 8 observe.py + thin shims + native Hermes kanban API    read-only
  → one JSON surface: /api/status.json (projection)            loopback-only
      → nginx (auth_basic) → /dashboard on docs.youngle.au
Frontend: thin Docus/Nuxt page consuming status.json           Nova's lane

Placement: /dashboard on docs.youngle.au (Young's ruling, 2026-10-16). Full-width layout, excluded from the docs/pulse aside.

Auth: auth_basic against /etc/nginx/.htpasswd, single user young. If the vault grows secrets or tenant-specific docs, split the viewer from the dashboard before tenant #1 — a hard boundary, not a style choice.

Refresh: Collectors run on cron (2–5 min). The page renders the last snapshot and refreshes client-side every 2 min (fetch + swap, no reload). No heavy live polling from the browser.

Security constraints (Sol's annex §2.3 — retained verbatim in force):

• Never render: tokens/keys, per-tenant .env content, Telegram tx hashes, provider keys, JWTs, raw a2a_audit.jsonl events.
• Only slices cross the gate: generated_at, agent endpoint status, model window %, task counts by state, vault parity sha, host ram/disk.
• GO window shows percent only, never the key.

3. The data layer — reconciled

The three sources proposed two different collection strategies. Both are recorded here with their measured state.

3.1 Goals & Tasks — source changed (C1)

Old: parse PULSE-BOARD.kanban + SOLOS-V3.md (git-synced markdown). New: read the kanban kernel directly.

• Task truth = hermes kanban (per Young, 2026-10-20). The vault holds no board artifact that claims to be generated.
• Boards are now split per workstream: LAB docker-vhost / research / daily-operation / solos-v3; Pulse daily-report-brief / research / podcast.
• The board is single-host by design (Hermes docs), with a documented cross-home sharing mechanism (shared kanban.db + unique profile names + kanban.dispatch_profiles per home as a fail-closed claim allowlist).

Live today (measured): hermes kanban boards list → default (done=1, ready=1, running=3), daily-operation (current), docker-vhost, research, solos-v3.

Consequence for the design: the Goals & Tasks panel must be board-aware — it needs a board selector, because a fleet view spans five LAB boards and three Pulse boards. Hermes ships no cross-board aggregate view (Echo measured this: the dispatcher sweeps all boards per tick, but that is dispatch, not display; all dashboard API endpoints are board-scoped via ?board=<slug>). "Fleet board as the view" is a build, not a configuration.

3.2 Agents Health — source changed, and is now the biggest gap

Old (Tam's §5.2 C2): HTTP GET probe per A2A port; last_exchange_s from ~/.hermes/a2a_audit.jsonl; skills_count from the agent card.

Measured reality:

Measured /metrics payload (8311, live):

{"uptime_seconds": 211448.2, "inbound_total": 48, "outbound_total": 35,
 "streams_started": 0, "push_sent": 0, "push_failed": 0,
 "tasks_completed": 35, "tasks_failed": 13, "anti_loop_triggers": 0,
 "rate_limit_triggers": 0, "avg_latency_ms": 44190.5}

This is a design problem, not a build problem, and it must be ruled on (Young/Min). The Heart plane — three of the eleven agents — has no health surface a collector can read. The V2 audit (source 5) already named the shape of this trap: "Can answer" ≠ "is running", and a /health that probes the process rather than the wire is exactly the artifact the design forbids. Tam's plan assumed any HTTP response proves the server listens; on the Heart plane, the correct liveness probe is a JSON-RPC 2.0 SendMessage with protocolVersion: "1.0" — a real wire exchange, not a socket check. Recommendation: add a read-only GET /health to the V3 A2A server (operation/runtime/a2a_server.py) that reports wire-observed state, mirroring the Hermes adapter's /metrics. Owner: Sol (V3 engineering lane). Until that lands, the Heart rows on the dashboard render unknown, never green.

Also measured: heartbeats.json does not exist anywhere on LAB. Sol's annex assumed a heartbeat file. It does not exist. Per design-record-authoring §3, this is recorded as deliberately absent — a later session finding it missing and building it re-introduces a husk.

3.3 System Health — source unchanged, still unbuilt

Per Tam's §5.2 C3: prod free -h / df -h / / systemctl is-active / docker count; tunnel probes from fleet-ssh-tunnels.md; cron last-run markers; vault parity (git rev-parse HEAD on LAB vs origin vs prod).

Measured today: solos-docs.service and pulse-site.service are running on prod (161.33.71.67) under nginx; solos-docs is a Docus/Nuxt Nitro server on 127.0.0.1:8091 with EnvironmentFile=/etc/solos-docs.env; the docs app source lives at /home/ubuntu/docs-site with app/pages/index.vue and app/pages/[lang](lang)/[...slug].vue — i.e. a /dashboard page does not exist yet and must be authored.

Crons: on LAB, kanban-obsidian-sync is paused (retired, per C1) and solos-v3-board-projection runs every 5 min on profile min — the latter is the only board→V3-store dispatcher, and it is a third dispatch path that does not read kanban.dispatch_profiles (Echo's finding, Sol's addendum 2 §3). Any collector that "reads cron last-run" must know which cron it is reading and whether that cron is the one that matters.

3.4 Backups + Cost

• Backups (C4): rclone lsjson gdrive:vault-archive latest date + on-prod /home/ubuntu/archive presence. Age >24 h = amber, >48 h = red.
• Cost (Mira's §6.2): opencode stats --cost --json — lifetime spend, per-model cost, cache read ratio, free-vs-paid share. Real, script-readable, no credential needed.
• Window (now live): GET /zen/go/v1/usage → rolling/weekly/monthly percent. Measured just now: 40% / 38% / 48%, all [OK].

4. Panel specifications

4.1 Fleet status strip (top — the 5-second glance)

Host dots · tunnel count · "oldest stale source". If generated_at > 15 min old the strip turns amber with "COLLECTOR STALE" (the dashboard never lies).

┌────────────────────────────────────────────────────────────┐
│ FLEET STATUS STRIP  —  generated 2 min ago · all sources   │
│ [🧠 pulse-laptop OK] [🖥 LAB OK] [🌐 prod OK] [🔒 tunnels   │
│  4/4] [⏱ oldest stale source: 3m]                          │
└────────────────────────────────────────────────────────────┘

4.2 Goals & Tasks (left card)

• Board selector (new — see §3.1): the panel is scoped to one board, defaulting to the current one. A "show all boards" aggregate is a build (Hermes has no cross-board view) and is out of v1.
• Cards as compact rows: title, owner chip, status. Owner chips use team colours (Brain violet, Heart green, Pulse blue).
• Click a card → link through to the vault file on the site (the card's own vault path — it is already mirrored).
• Live tasks sub-block: in-flight A2A tasks with age; anything ≥ the stuck threshold shows a red STUCK 6m12s badge.

4.3 Agents Health (right card — the dense one)

• One row per agent (11, from agents.yaml — the registry, not a hand list).
• Sort: red first, then amber, then green (fleet triage at a glance).
• Columns: name · endpoint dot (green=responds, red=refused, grey=unknown — not probeable) · last exchange (from a2a_audit.jsonl) · skills count (0 = red "EMPTY") · model · GO window bar with % (amber ≥80, red ≥95).
• Row click → expands: lane, A2A port, instrument, checked_at.
• Heart-plane rows render unknown until the V3 /health surface lands (§3.2).

4.4 System Health (3 host cards)

• Per host: RAM bar (amber ≥70%, red ≥85%), disk bar (free-based: amber <20G, red <10G), services as dots (click = name + state), tunnels as dots with names, crons "last OK: 12m ago", vault parity ✓ (or "LAB a1b2c3 / prod 9f8e7d — DRIFT").
• prod card also shows solos-docs + pulse-site + docker container count.

4.5 Backups + Cost strip

• gdrive archive latest date ("Sep-03 backup: 2d ago" style), age-coloured.
• GO/ZEN window bars per model (percent only — never keys), Mira's thresholds.
• Cost rows: lifetime spend · per-model cost · cache read ratio · free-share of tokens.

5. Colour rules (merged from Tam's §5.3 + Mira's §6.3)

No false green is the rule that decides every grey cell above. A value we cannot measure is grey, never green.

6. Visual language

• Matches the site: Nuxt UI tokens, dark-mode aware (the site is dark by default).
• Green brain accent for OK (#22c55e — the brand), amber #f59e0b, red #ef4444, muted zinc text.
• Panels are cards (rounded-xl border bg-elevated), consistent with the homepage cards already shipped.
• Bars: single-height (6px) rounded, fill = value, colour = the rule in §5.
• Typography: numbers in mono (tabular), labels in the site's sans.
• Empty/absent states: never blank — "— (source error: …)".
• Page title: Young's Mind — Fleet Dashboard.
• Footer: "read-only view — data refreshes every 2–5 min".

7. Collector specification (Tam's §5.2, re-verified)

C1 — Board (tasks) — source rewritten, see §3.1

• Source: the kanban kernel (hermes kanban, board-scoped).
• Cadence: 5 min (one cheap read). Live A2A tasks: poll per-agent GetTask; age vs the stuck threshold → stuck. Cadence: 2 min.

C2 — Agents health — source partly rewritten, see §3.2

• Endpoint probe: Brain plane → GET /metrics on 8311–8315 (verified live). Heart plane → no readable surface exists; needs the V3 /health shim, or a JSON-RPC 1.0 SendMessage liveness exchange as the interim probe.
• last_exchange_s: tail ~/.hermes/a2a_audit.jsonl (219 KB, live-appended). 0–30 min green, 30 min–6 h amber, >6 h red.
• skills_count: from the Hermes profile roster — the native /profiles endpoint returns skill_count per profile (measured present in the plugin API). Emptiness is LOUD.
• window: GET /zen/go/v1/usage — LIVE (§3, C2).

C3 — Systems health

• prod (via server key): free -h, df -h /, systemctl is-active, docker ps count. Cadence: 5 min.
• Tunnels: one-liners in fleet-ssh-tunnels.md §5. Cadence: 2 min.
• Crons: last-run markers. Silent-failure rule: >2× period = red. Read the cron registry, not a log file — and know which cron is which (see §3.3).
• Vault parity: git rev-parse HEAD on LAB vs origin vs prod clone.

C4 — Backups + Cost + Window

• Backups: rclone lsjson gdrive:vault-archive + on-prod /home/ubuntu/archive.
• Cost: opencode stats --cost --json.
• Window: GET /zen/go/v1/usage → percent only.

Ops shape

• One script per host section (bash+python, no new framework), writes status.json atomically (tmp+mv), runs from cron on the laptop (it can reach LAB via mesh and prod via tunnels — the hub of the hub-and-spoke).
• Serving: prod nginx serves the JSON + the page behind the existing auth gate.
• Collectors are read-only; no tokens in status.json; nginx gate mandatory.
• Collector cron is deliver=local, quiet-failure (Young's standing rule); the dashboard's own staleness banner is the alert.

8. Known defects (carried forward, still open)

D1 — the spend ledger can report $0.00 while the fleet is working.spend-ledger.py queries opencode stats --days 1, which counts opencode CLI sessions only. The Brain agents run on Hermes, so their model traffic is not in opencode's local DB. Same defect class as V2's "$0.00000 across 30,721 attempts". Test to close it: compare stats --cost deltas against a known Hermes-routed paid call. Until then the ledger's zero is "not measured", not "nothing spent". Owner: Mira/Min to sequence.

D2 — the ledger's date column is UTC while its cron runs in AEDT. The 09:00 AEDT run is 22:00 UTC the previous day, so rows are labelled one day earlier than the run that wrote them. One-line fix in the script. Owner: Mira/Min.

D3 (new) — the Heart plane has no health surface. Three of eleven agents cannot be probed by a collector without a live message exchange. See §3.2. Owner: Sol.

D4 (new) — there is no cross-board fleet view. "Fleet board as the view" is a build. See §3.1. Owner: Min (if wanted).

9. What is deliberately absent

• heartbeats.json — does not exist on LAB. Not built. The dashboard reads liveness from the audit log and endpoint probes instead.
• A vault board projection — retired by decision (C1). The dashboard reads the kernel.
• A second collector stack — the plan explicitly forbids it; the dashboard consumes Stage 8's observe.py plus thin shims.
• Cross-board aggregate view — a build, not v1 (D4).
• Write actions — out of scope for v1, permanently. No restart buttons, no task mutation, no board writes from the page.

10. Implementation plan

Phase 0 — Resolve the build-vs-extend decision (Young + Min) — NEW, and first

Hermes ships a native kanban dashboard plugin: a 1847-line FastAPI backend (plugins/kanban/dashboard/plugin_api.py) and a 210 KB built frontend (dist/index.js + dist/style.css), with these read endpoints:

Two options, and the choice belongs to Young/Min — not to Nova:

My recommendation, for the record: Option B for the public page, Option A's endpoints reused as the data source. Reasoning: the native plugin solves task state perfectly but is a LAB-local web app on a different auth model, while Young's ruling is a section of docs.youngle.au behind the existing gate. A Docus page that reads the same kanban_db module the plugin reads gets the kernel without forking the UI — and keeps the placement Young already ruled on. But this is a placement-and-auth ruling, and it is Young's to make; §11 lists it as R1.

Phase 1 — Data contract (Sol + Tam)

• Sol confirms/annotates the merged §3 schema
• Add a read-only GET /health to the V3 A2A server (D3) — wire-observed, never process-liveness
• Tam scripts C1 (kernel board read) → real board snapshot
• Tam scripts C2 (agents) → Brain /metrics probes + audit ages + profile skills
• Tam scripts C3 (systems) → prod RAM/disk/services + tunnel probes + vault parity
• Tam scripts C4 (backups + cost + window) → gdrive + opencode stats + usage tiers
• Emit the snapshot to /api/status.json on loopback

Proof: status.json served on loopback with real values; STALE markers work; a Heart-plane row renders unknown rather than green.

Phase 2 — The page (Nova)

• One Vue/Docus page component + a tiny composable consuming status.json
• Board selector (§4.2) — a real build, not a config flip
• Fleet status strip (§4.1)
• Goals & Tasks panel (§4.2)
• Agents Health panel (§4.3) — with the grey unknown path
• System Health panel (§4.4)
• Backups + Cost strip (§4.5) — window now with real numbers
• Client refresh every 2 min (fetch + swap, no reload)
• Empty/absent states per the staleness doctrine

Proof: Young opens /dashboard and sees every panel with real values, including a grey unknown where a value genuinely cannot be measured.

Phase 3 — Refresh + alerts (Tam + Mira)

• Collector cron wired (2–5 min cadence)
• Stuck-task alarm at the native threshold
• GO-window warnings (Mira's thresholds) — now possible
• Staleness banner ("COLLECTOR STALE") when generated_at > 15 min

Proof: kill a tunnel → the panel goes red within one cycle, without human probing.

Phase 4 — Hardening

• Security review: no secrets in status.json; the auth gate is enforced
• Viewer/dashboard split decision before tenant #1
• D1/D2 fixes in spend-ledger.py
• Decide the Heart-plane health surface's permanence (D3)

11. Open decisions — Young's ruling needed

For Tam/Sol/Mira (no ruling needed — these are their lanes, recorded here so the plan is actionable):

• Tam: C1–C4 collector scripts + the cron cadence.
• Sol: the V3 /health shim (R3) and the §3 schema confirmation.
• Mira: cost/window panel rules; D1/D2 sequencing.

12. Lanes (preserved from the source plan, unchanged)

Lane note: Nova (this document's author) designs and specifies; she does not implement. The build belongs to Tam, Sol, Ace and Min.

13. Historical context — NOT part of this design

Sources 4 and 5 describe a different dashboard: the SolOS V2 heart-plane ops console (atlas, ace, nova, echo, alpha) with failover, steward, rate limiting and enforcement audit. It was built (ops/dashboard.py, 907 lines) and audited by Echo on 2026-10-07 with 18 findings, including a critical self-DoS (a 1.18 GB log read per request, 2.49 GB peak RSS) and several false-clean panels. It is not the fleet dashboard, it was never merged into this plan, and nothing in §1–§11 depends on it.

Two lessons from it are carried forward, because they are design constraints, not project history:

1. A panel that reports "no data" is not the same as a panel that reports "no problems." (Echo F2/F3 — except Exception swallowed MemoryError and rendered "All systems nominal".) This is why §5 has a grey unknown and why the stale flag is per-element.
2. The data layer must be bounded. The V2 build read a 1.18 GB file whole on every request. Any collector here that touches a growing log must seek from EOF or use an index — a2a_audit.jsonl is already 219 KB and live-appended.

Nova — Graphics & UX Designer, brain plane. Read-only on code. This document is the consolidated design of record + implementation plan for the fleet dashboard. The build belongs to Tam, Sol, Ace and Min per §12.