SolOS V3 — Production Documentation

SolOSV3 is the A2A-native agent fleet. One registry, everything derived from it, cost-first model routing, and every control falsifiable by design. V2 (/home/lab/solos2) is reference only.

Suite: 457 passed · 2 skipped · 2 xfailed at 751286d — the state this document describes.

1. What it is, in one paragraph

Eleven agents across three planes, all declared in one YAML file. The Heart (5 agents) runs tasks; the Brain (3) orchestrates and reports; Pulse (3, another host) is the human-facing plane. A task arrives → the router picks the cheapest capable model (free first, paid last) → the executor runs it through the opencode CLI → every attempt is priced into a cost ledger → a stuck task escalates without a human. Nothing restated: the registry is the one source and every projection derives from it. Each module names the V2 defect it exists to close.

2. Where everything lives

3. The agents — from the registry, at 751286d

The rule: a task is classified first (cortex/classification.py). A confidential task routes only to a confirmed zero-retention model — with none confirmed, it is refused outright rather than assumed safe. RETENTION_BLOCKLIST additionally refuses muse-spark (trains on request data) even if someone confirms it.

4. How a task flows

submit --> TaskStore
             |
     classify (cortex/classification.py)
             |  confidential? -> confirmed zero-retention only, else REFUSED
             v
     route (core/config/router.py)
             |  ladder: GO free -> ZEN free -> GO paid -> ZEN paid
             |  per (endpoint, model_id); cost rung beats instrument rung
             v
     invoke (operation/runtime/invoke.py, OpencodeInvoker)
             |  opencode CLI; output-classified (rc==0 is NOT success)
             |  E5 boundary: no provider key reaches the child process
             v
     record (core/config/cost_ledger.py)
             |  every attempt priced; unknown is never summed as zero
             v
     stuck rule (cortex/tasks/stuck.py)
             |  300s without progress -> escalate, no human decides
             v
     observe (operation/runtime/observe.py)
                state counts · per-model ledger · cost · heartbeat

5. Operating commands

Environment (every command):

SOLOS_V3_HOME=/home/lab/.solos

Run the whole test suite (the health check):

cd /home/lab/.solos && SOLOS_V3_HOME=$PWD python3 -m pytest tests/ -q

Run one Heart worker (what the systemd unit does):

python3 operation/runtime/worker.py ace --poll-interval 5 --sweep-interval 30

Flags: --max-parallel N · --once (one cycle, for smoke tests) · --db PATH.

Serve one agent's A2A endpoint:

python3 operation/runtime/serve.py --agent ace          # port from the registry
python3 operation/runtime/serve.py --agent ace --port 8320 --log-level DEBUG

Install the systemd unit:

mkdir -p ~/.config/systemd/user && cp etc/systemd/solos-v3-worker.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now solos-v3-worker
journalctl --user -u solos-v3-worker -f

Submit a task (via A2A, the supported path):

json   {"jsonrpc":"2.0","id":1,"method":"SendMessage",
        "params":{"message":{"role":"user","parts":[{"text":"<goal>"}]}}}

Observe a running fleet:

python3 operation/runtime/observe.py            # state counts, ledger, heartbeats

Probe the model catalogue live:

SOLOS_V3_HOME=$PWD python3 -c "from core.config import models; [print(p.declaration.address, p.reachable) for p in models.probe()]"

Backup and restore rehearsal (--no-agent, zero tokens):

hermes cron list                                  # solos-backup-github 03:00 daily
bash ~/.hermes/profiles/mira/scripts/restore-proof.sh

6. Design rules, and the defect each one closes

7. Modules — what each one is for

8. Cost model — the numbers an operator needs

A free model on ZEN still costs $0.00; a paid GO model still costs money — which is why ZEN free outranks GO paid.

Measured posture (7 days to 2026-10-11): spend $11.60, down from $19.49; free models did 4.8b tokens at $0.00; one paid model carried everything. Live figures at any time: opencode stats --days 7 --cost --models.

9. Production status — honest, and dated

So: every capability V3 defines is built and tested. What production adds is the operator action in §5 — installing the unit and binding the ports. That is a morning's work, not a build phase, and it is the one line in this document that ages first.

10. Known constraints

• operation/state/ is disposable by design — task .db files do not survive to the backup. The cost ledger gets a second copy on each backup run.
• Pulse planes 3 agents on 192.168.4.34 — a different host; not reachable from LAB's loopback.
• provider_routing/data_collection: deny solves retention only via OpenRouter, and no model is on OpenRouter today. The blocklist is the control until then.
• operation/state/ is disposable by design — task .db files are not backed up; the cost ledger is copied out on each backup run.
• The model sheet (05-operations/) is a 2026-10-11 snapshot — prices are quoted, not probed. core/config/models.py::probe() answers it live.
• Batch processing (batch_runner.py) is out of scope — training-data generation, not fleet operation.