Architecture evidence pack — 2026-09-28
Architecture evidence pack — 2026-09-28
Purpose: the evidence reference for Sol's ARCHITECTURE.md v3 rewrite. Every row was probed live by Tam (laptop + LAB on-host) and, where noted, cross-checked by Sol on-host. Three of my own claims are recorded here as OVERTURNED — a note that hides its own errors is worse than no note.
Subject doc: /home/lab/SolOS/docs/vault/solos/architecture/ARCHITECTURE.md — 14,123 bytes, mtime 2026-09-26 10:32, status string claims "verified 2026-09-22" (stale by 6 days).
Render pipeline is NOT on LAB (:8091 = 000 from tailnet and LAN) — the vault is the source of truth; prod renders.
0. Probe-path trap (record first — it caused two of my three errors)
~/.solos/core is a real directory, and there is a second live-looking copy at SolOS/runtime/home/.solos/core. Both contain agent_app.py. Grepping cold lands on the wrong one and produces a wrong answer.
/home/lab/.solos/core/agent_app.py— exists/home/lab/SolOS/runtime/home/.solos/core/agent_app.py— exists/home/lab/SolOS/core/— empty/absent (this is the path I wrongly reported from)/home/lab/SolOS/.displaced-20260926-122825/core/agent_app.py— displaced archive, not live
Rule for the v3 doc and for any future audit: resolve the running entrypoint first (systemctl --user show -p ExecStart), then follow imports from there. Do not grep a path you found by find.
1. Host model
| Host | Identity | Reachability | Notes |
|---|---|---|---|
| LAB | Ultra 5 / 32 GB / 512 GB, Ubuntu 24.04, user lab (no sudo) | LAN 192.168.4.60 · tailnet 100.94.188.127 | Fleet home. Tailnet works without a tailscale0 interface — tailscaled runs rootless in userspace mode (--tun=userspace-networking), so ip -4 addr shows only 127.0.0.1 + 192.168.4.60/22 while 100.x traffic succeeds. Do not conclude the tailnet is dead from ip addr. |
| PROD | 161.33.71.67 / tailnet 100.124.146.127 | ssh (tam_server_key, read-only) | Being redesigned as production. Still serves: prod OpenCode web :4090 + API :49374 (Alpha's only surface), nginx domains, the old laptop-tunnel board :8432 (loopback). |
2. Verified live state (2026-09-28 ~22:45–22:50)
A2A mesh — 7/7 green + alpha recovered
agent-card 200 on all eight ports; message/send real completions 2–4 s on 8300/8301/8302/8303/8304/8306/8307 (model space-bunny-free).
| Port | Agent | Unit | State |
|---|---|---|---|
| 8300 | Sol | lab-a2a-sol | active |
| 8301 | Ace | lab-a2a-ace | active |
| 8302 | Mira | lab-a2a-mira | active (self-healed — was alive-not-serving twice today; a /health 200 hides that class) |
| 8303 | Nova | lab-a2a-nova | active |
| 8304 | Echo | lab-a2a-echo | active |
| 8305 | Alpha | lab-a2a-alpha | active but disabled — will not return on reboot |
| 8306 | Atlas | lab-a2a-atlas | active (one transient starvation window observed, self-drained) |
| 8307 | Min | lab-a2a-min | active |
Live entrypoint: lab-a2a-* units run runtime/venv/bin/python runtime/agents-runner.py; that file does from core.agent_app import build_app (line 11) and calls build_app(...) (line 16).
Core infra
| Service | Port | Verified | Class |
|---|---|---|---|
| Central Registry | 8432 | 200 (tailnet + LAN) | live, authoritative board |
| CostGov | 8430 | 200 on-host; /v1/check/ and /v1/route/model answer for all 8 | live, loopback only (tailnet 200 / LAN 000) |
| A2A Discovery | 8433 | 000 — dead on LAB, no unit | GAP (Min's C7) |
| Conductor | 8320 | 000 — dead, no unit | GAP — doc §6 lists it healthy |
| solos-memory | 8437 | not yet listening (in build, Ace) | planned |
| Hermes gateway | — | hermes-gateway.service active (user scope); :9900–:9904 all 000 | port layer gone, process alive |
CostGov enforcement — the real finding
- Routing IS enforced in the A2A lane. Live
core/agent_app.py:308→GET :8430/v1/route/model?agent=,timeout=4, before static config;tier=="go"→opencode-go/. Confirmed by both Tam and Sol at the same line. - The 50%-law cap is enforced NOWHERE.
switch_required(from/v1/check) has zero code consumers — grep across the live tree returns onlysolos.dband backups, no.pyreader. (Tan's earlier claim thatbudget-check.py/llm_core.pyconsume it was OVERTURNED by Sol's grep; the signal is written and never read.) - All eight agents report
pct: 0.0, reason"free model — no spend cap, never breaches"— so the cap has never had the opportunity to fire. Two independent reasons it is untested: no reader, and nothing to cap. - The real exposure: the CostGov-unreachable path sets a fallback reason but emits no log and no metric — a CostGov outage degrades to static config silently. This is the invariant worth writing, because it stays true after the cap is fixed.
/v1/tiersand/v1/pools→ 404 (not implemented).
Surfaces (my claim was OVERTURNED here)
| Surface | Port | LAB | PROD |
|---|---|---|---|
| Flight Desk | 4100 | 200 on-host AND 200 from laptop over tailnet — dual-resident | yes |
| Web chat | 3000 | 000 | yes |
| Docs site | 8091 | 000 (render pipeline not migrated) | yes |
| Prod OpenCode | 4090 / 49374 | n/a | loopback + nginx vhost |
| Eve / Eva / Tam doors | 8442 / 8443 / 8447 | 200 (via lab-a2a-eva-relay → prod → laptop) | relay hop |
Eva direct :8441 | 8441 | 000 — no unit; the relay is the live path | — |
Personas
Source of truth ~/.solos/personas/*.md (8 agents) + shared/ (base, team-manual, voice-anchors) + fleet/, synced every 30 min by solos-persona-sync.timer. ADK2 fully retired 2026-09-28 (42 files archived, runnable unit references rewritten, no runtime/process/unit/persona/registry residue).
Hermes on LAB
Profiles: exactly sol, min, mira + the default root (~/.hermes/, which is the running gateway's HERMES_HOME and multiplexes the three). Default root identity = Sol. memory_durable in solos.db: 30 rows (8 persona/ + hermes-migration/* + default_model).
3. Memory plane (for §7.5 of v3)
- Existing:
solos.db—memory_durable30 ·memory_longterm321 (0 embeddings) ·memory_episodic1536 · legacymemories230. - Fork hazard, verified: two divergent DB files exist.
SolOS/runtime/home/.solos/data/solos.db(422 MB, longterm 321 / episodic 1536) is the one the units use (HOME=/home/lab/SolOS/runtime/homeinlab-systemd.env);/home/lab/.solos/data/solos.db(428 MB, longterm 309 / episodic 1129) is a stale copy that is the easier path to write to by accident. The service must pin the systemd HOME path and openmode=rwso a wrong path fails loudly. - Building:
solos-memoryon127.0.0.1:8437(MCP + REST). v1 writesmemory_shortonly —memory_longterm/memory_episodicare read-only, enforced by aWRITE_TABLESallowlist. Cross-agent recall returns provenance(agent, key, created_at)withforeign: true. A2A hook timeout 300 ms. Embedder deferred to v1.1 (Sol reversed the "no embedder" call: silent multi-word LIKE failure is indistinguishable from amnesia). - Isolated venv
runtime/venv-memoryexists with realmcp 2.2.0(FastMCP→MCPServer;httpx→httpx2). Sharedruntime/venvuntouched.
4. Claims in v2 that are STALE (rewrite targets)
| § | v2 says | Truth (2026-09-28) |
|---|---|---|
| §1.2, §6 | :8433 discovery owns agent cards | dead on LAB |
| §1.3, §6 | CostGov routes every call | routing yes; caps unread; silent fallback |
| §2 diagram | :9900–:9904 Hermes layer in core infra | contradicts §5 "decommissioned"; process alive, ports gone |
| §2/§3 | Alpha is 8th native, card-served, :8305 | card-served but unit disabled — reboot-survivability defect |
| §4 | Eva :8441 | dead; live path is the relay :8443/:8447/:8442 |
| §6 | Conductor :8320 healthy | dead, no unit |
| §5 | Hermes gateway DECOMMISSIONED | blanket claim false in one direction (service active) |
| §2/§9 | Web chat/Flight Desk/Docs as one host's surfaces | Flight Desk dual-resident; chat+docs prod-only |
| whole doc | single-host model | LAB vs PROD split (§2 of v3) |
| whole doc | no memory plane | add §7.5 |
| §2/§3 | port map with :8305 as native | Alpha retired from Hermes; prod-OpenCode-only + A2A :8305 card |
5. My overturned claims (recorded deliberately)
- "Alpha
:8305FAILED / 000 everywhere." True at ~22:05, stale by 22:28 — the unit recovered on its own and I did not re-probe before reporting. Real defect isdisabled, not down. - "No human surfaces on LAB." I probed
:3000and:8091, saw 000, and generalised without probing:4100. Flight Desk is 200 on LAB. - "CostGov routing is not consulted by the A2A lane." I found
agent_app.pyin the displaced archive and reported its line as live-but-then-doubted it. Routing is enforced atcore/agent_app.py:308. (Sol's counter-claim that it was not consulted was also wrong — corrected by both, confirmed by the symlink/directory resolution in §0.)
6. Open items for the v3 doc to name explicitly
- Enforcement gap: CostGov fallback is silent (no log/metric) — a real exposure that survives the cap fix.
- Cap gap:
switch_requiredhas no reader; v1 of the memory spec cannot be declared "gated" until that exists. - Alpha
:8305disabled — reboot survivability. :8433dead (Min C7, Atlas, after C10).:8320Conductor dead — no owner assigned yet.- Skills-tree drift (Min C9) advertises ADK2 + a retired port map; the doc and skills must converge or v3 rots the same way.
- Two
core/trees + a displaced 484 MB archive — the audit trap in §0. - Two divergent
solos.dbfiles — the fork hazard in §3.
LAB A2A Self-Heal Watchdog — spec (Young directive: "crons for self-healing, get Sol and Ace on it")
SPEC — Cross-Agent Memory Bus v0.1 (Tam, research/spec lane)
For: Atlas + Ace (solos-memory engineering). Review: Young. Coordination: Min. Status: spec only. No code, no registry schema changes proposed here.

