Fleet probe — 2026-10-21

Owner: Sol (engineering). Lane: operation/runtime/. Probed on LAB.

A direct probe request over A2A. Method: live probes only — every claim below is backed by a command in this repo's shell history or by a file on disk.

1. Summary

Fleet verdict: 7 of 8 LAB agents healthy. Atlas (8303) is degraded, not dead.

2. A — Heart listener starves its own event loop (HIGH, OPEN)

Symptom

atlas on 8303 answers TCP (the port is bound, connect() succeeds) but never returns a response. The agent-card route times out even though the card route does no model work at all.

$ ss -ltn | grep 8303
LISTEN 21  2048  127.0.0.1:8303  0.0.0.0:*      # Send-Q 21 and climbing

$ curl -m 10 -o /dev/null -w '%{http_code} t=%{time_total}s' \
    http://127.0.0.1:8303/.well-known/agent-card.json
000 t=10.002369s                                  # timeout, every time

Recv-Q on a LISTEN socket is the accept backlog. A rising Send-Q with a bound port and successful connect() means: the kernel accepts the TCP connection, and the application never runs the handler.

connect() succeeding is therefore not evidence of a working listener. It is evidence only that a socket is bound.

Root cause

operation/runtime/a2a_server.py:209, inside async def on_message_send:

outcome = self.executor.run(task_id)      # <-- synchronous, blocking

Executor.run reaches operation/runtime/invoke.py:150:

proc = subprocess.run(argv, timeout=self.timeout_s, ...)   # timeout_s = 300

subprocess.run is blocking. It is called directly from an async def with no await asyncio.to_thread(...) / run_in_executor wrapper. The call therefore blocks uvicorn's single event-loop thread for up to 300 seconds, and during that window every route on that listener is unresponsive — including the card route and /healthz, which do no model work.

This is the classic "one slow request takes the whole server down" defect, and it is invisible to every probe that only checks the port is bound.

Proof by mechanism

The blocking was observed directly — the atlas listener had a live child process:

$ ps --ppid 2822158 -o pid,stat,etime,cmd
431354 Sl 01:35 /home/lab/.opencode/bin/opencode run --model opencode-go/longcat-2.5-preview-free ...

opencode run was hung for 1m35s → 4m21s while the listener's event loop was blocked waiting on it. The listener's only Python thread was in poll_schedule_timeout — i.e. parked inside the blocking call, not serving.

Proof by recovery

When the 300 s subprocess.run timeout finally fired, the child exited and the listener recovered — then immediately re-blocked because another request arrived:

CHILD EXITED after wait ~20s
  -> 000 t=12.002353s     # still blocked (next request in flight)
  -> 200 t=9.818329s      # served, backlog draining
  -> 000 t=12.002394s

Backlog drained 21 → 1 → 8 with a new opencode child, and 0/3 card probes succeeded in the recovery window (a second run measured 0/5). Self-healing is not a fix; the listener is permanently degraded under model traffic.

The recovery window is narrow enough that one lucky probe reads as "working" — a single window showed one 200 between two 000s. Judge by the backlog trend and by repeated probes, never by one success.

Blast radius

Every agent served by operation.runtime.serve — ace (8301), alpha (8302), atlas (8303). ace and alpha are currently healthy only because no slow model call happens to be in flight when probed. The defect is latent in all three.

ace is not exempt — measured round-trip, 8.0 s:

$ curl -m 200 ... 8301 ... "Reply with exactly: PONG-ace-8301"
HTTP 200 time 8.032391s
"artifacts":[{"parts":[{"text":"PONG-ace-8301"}]}]

A free-tier model call taking 8 s blocks that listener's event loop for 8 s.

Not a regression

git show HEAD:operation/runtime/a2a_server.py has the identical line 209 — the defect is committed at HEAD, not introduced by the uncommitted working tree. git diff operation/runtime/a2a_server.py is empty.

Fix (not applied — outside the scope of a probe request)

Two independent changes, both in Sol's lane:

1. Do not block the event loop. In on_message_send, either
   • outcome = await asyncio.to_thread(self.executor.run, task_id), or
   • honour params.configuration.return_immediately (already implemented at line 205!) and let the polling path drive execution.
   
   Note line 205 already supports non-blocking submission via return_immediately; the default path is the one that blocks.
2. Bound the subprocess timeout. timeout_s = 300 is too long to hold an event loop hostage. The worker service escalates on a 300 s no-reply (solos-v3-worker.service: "no reply for 423s (>= 300s)"), so 300 s is the house timeout — but it must be enforced off the event loop.

3. B — a2a-shim points at a dead address (MEDIUM, OPEN)

operation/a2a_shim.py:31 hardcodes:

TAM = "http://192.168.4.34:8443/"

That address refuses connections. The live route to Tam is the SSH tunnel on 18443, which works:

direct 8443  -> 000  (curl exit 7, connection refused)
tunnel 18443 -> 200  (card: Tam | Research & Reports)

The shim has no tunnel reference (grep '1844' a2a_shim.py → nothing), so every request it forwards dies:

journalctl --user -u a2a-shim.service
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

A 200 from the shim would still not prove success — the shim catches HTTPError but not URLError, so a connection failure raises out of forward() and the request dies before a response is written. Callers that only check the HTTP status would see a dropped connection, not a clean error.

Fix: point TAM at http://127.0.0.1:18443/ (the proven tunnel), and resolve it from the registry's pulse endpoint rather than a literal.

4. C — Pulse is tunnel-only (MEDIUM, environmental)

The host pings (3.0 ms) and the tunnel ports are bound, so the link is up; only the direct A2A ports are not exposed on the LAN side. Card names confirm identity through the tunnel: Eva / Eve / Tam, matching the registry.

Pulse is not in the "fleet down" bucket — it is reachable, through the tunnel.

5. D — Cortex is a library, not a service (LOW, doc drift)

cortex/ has no HTTP server — no uvicorn, no FastAPI, no http.server anywhere under /home/lab/.solos/cortex/. It is imported as a library (from cortex.registry import agent as registry_agent, operation/runtime/serve.py:54).

:8320 is not listening on LAB. The registry (cortex/registry/agents.yaml) is the live source of truth for identity and is read directly from disk.

Note: older vault docs list a "Conductor :8320" governance service — that belongs to the retired V2 topology and describes old-box loopback survivors, not LAB. cortex/README.md is explicit that assembly and memory are "Not started".

6. E — Provider transiently unavailable (LOW, self-recovered)

During the probe the upstream provider failed:

openai.APIConnectionError: Connection error.
⚠️ Model fallback: longcat-2.5-preview-free via opencode-go unavailable (request timeout); using cat-jam-free via opencode-go.
⚠️ Model fallback: cat-jam-free via opencode-go unavailable (rate limit); using minimax-m3.1-flash-free via opencode-go. Primary retry eligible in ~61402 s

https://opencode.ai/zen/go/v1/models returned HTTP 200 in 0.62 s when probed directly — the outage was transient, not a config fault. It did inflate A2A round-trip latency (observed 5.7 s, 14.1 s, 35.1 s, 91.7 s, 186.2 s, 380.2 s in the gateway log).

7. What actually works (verified round-trips)

A sentinel was sent to each listener and asserted verbatim — a plausible sentence about the sentinel would not count.

Brain plane — sol (8313), hermes gateway:

SendMessage "Reply with exactly: PONG-sol-8313"
-> TASK_STATE_COMPLETED
-> status.message.parts[0].text = "PONG-sol-8313"
-> artifacts[0].parts[0].text   = "PONG-sol-8313"     # 14.1 s

Heart plane — ace (8301), V3 serve.py:

SendMessage "Reply with exactly: PONG-ace-8301"
-> TASK_STATE_COMPLETED
-> artifacts[0].parts[0].text = "PONG-ace-8301"      # 8.0 s

The reply lands in both status.message.parts and artifacts[].parts — read both, as the listener skill says.

Protocol facts (V3 speaks A2A v1.0 only)

Registry (cortex/registry/agents.yaml) is the authority and matches the live fleet:

• Heart: ace 8301, alpha 8302, atlas 8303
• Brain: min 8311, mira 8312, sol 8313, nova 8314, echo 8315
• Pulse: eva 8441, eve 8442, tam 8443 (tunnel 18441-18443)

Brain ports are served by the hermes gateway process; heart ports by operation.runtime.serve via systemd units. ss -ltnp shows the owning process — the unit list alone reads short.

8. Not done / still blocked

• Defect A is diagnosed, not fixed. A probe request does not authorise a production-affecting change to the serving path. The fix is written above and belongs to Sol's lane (operation/runtime/).
• Defect B is diagnosed, not fixed. Same reason; same lane.
• Did not restart or modify any systemd unit.
• Did not touch PROD (161.33.71.67).
• Did not query the Pulse agents for task state beyond a card read.

9. Second probe pass — 2026-10-21 17:20–18:15 AEDT (Sol)

Same probe ID (SOL-PROBE-20261021), a second pass with a corrected instrument. This section corrects two claims in §7 above and adds one finding of my own.

9.1 The instrument had five defects of its own — all mine, all fixed

Each early run produced a confident wrong answer, and each was a probe bug, not a fleet bug. Recording them because the same five will bite the next session:

A probe that cannot fail correctly is worse than no probe: four of these five reported a defect in the fleet that did not exist.

Defect 6 is the dangerous direction. Defects 1–5 made the fleet look broken; defect 6 made a working tier look broken — it printed v1.0 'SendMessage' accepted on 0/3 for Heart while the row directly above showed ACCEPTED TASK_STATE_COMPLETED. Heart's c[2] cell is REFUSED -32009 because the header is required, so the summary counted a header-dependent refusal as a method failure. A headline that contradicts its own table is a probe bug until proven otherwise — re-derive from the raw rows, never from the summary line. The finding in §9.2 was written from the raw rows and is unaffected.

9.2 CORRECTION to §7 — the Brain tier does NOT "accept" v0.3

§7 records v0.3 content string → TASK_STATE_REJECTED "Empty task", which is right, but the framing "Brain accepts v0.3" is wrong. A 2×2 probe on :8311/:8312/:8313, fresh contextId per arm so the anti-loop guard could not be what was measured:

The discriminator is the PAYLOAD SHAPE, not the method name. The Brain tier dispatches the v0.3 method but cannot read a v0.3 payload, so a v0.3 caller gets HTTP 200 + a task id + a task that never ran. That is a hollow success — the same shape as Pulse's "200 that did not run" in a2a-capability-ping-t_echo_cap_A-2026-10-20.md. A caller that checks only result.taskId records a success.

This also corrects Tam's claim in Pulse/research/tam/tech/2026-10-21-a2a-roster-reconciliation.md §2: "A lowercase message/send now fails everywhere, on every tier." Measured (n=2 per arm, body parsed):

Heart refuses the v0.3 name with a guard that names the right call. Brain accepts the v0.3 name and then fails the v0.3 body — a softer failure that the "fails everywhere" phrasing hides. The practical rule is unchanged and is what the fleet should follow: always SendMessage + A2A-Version: 1.0 + a v1.0 parts[] payload.

9.3 CORRECTION to §7 — the Brain card-name defect has a code-free fix

§7 records "Card name is hermes-LAB for every agent" as a fact. It is correct, and the cause is the one Tam named (a2a-roster-reconciliation §2b): adapter.py:275 sets self.agent_name = _default_agent_name(), which reads only env A2A_AGENT_NAME. The per-profile key platforms.a2a.extra.agent_name is never read by this build.

But the fix does not need a code change to the vendored repo. The adapter already reads A2A_AGENT_NAME from the profile's own scoped .env. Proven in isolation (sandbox copies of each profile home; the live gateway was not touched, restarted, or reconfigured):

• Control (MUST FAIL): the defect reproduced on 5/5 live profiles before any change. A fix that cannot show its own failure proves nothing.
• MUST PASS: 5/5 with the key set in the profile's own .env.
• Isolation: sol's scope reads sol, min's reads min — no leak.
• Auth is unaffected: security.py has 0 references to agent_name; peer identity comes from credentials (A2A_PEER_TOKENS / bearer / loopback IP). The advertised name is display-only, so it cannot weaken auth.
• Cost: one env key per profile, then a gateway restart — which interrupts all five Brain agents, because one process (pid 3733570) serves all five ports. That is why it is scheduled, not hot-patched.

Applied 2026-10-21 (card t_b3a9bc3c). The keys are written to all five Brain profile .env files and the change is live and verified — see §9.3a for the live state and the correct runnable form.

# per profile, append to {profile}/.env  (A2A_AGENT_NAME={name})
# then:  systemctl --user reload hermes-gateway   # SIGUSR1 -> drain + relaunch
# verify: curl -s http://127.0.0.1:8313/.well-known/agent-card.json | jq -r .name

Use reload, not restart. systemctl --user restart (and hermes gateway restart) is a hard kill: it drops in-flight turns with no drain. reload maps to ExecReload=/bin/kill -USR1 $MAINPID, and the gateway's own SIGUSR1 handler drains active turns, marks them resume_pending, and exits for the supervisor to relaunch. It is also the only route that is not policy-blocked in a single-query run.

9.3a The fix is APPLIED and VERIFIED LIVE (2026-10-21)

The restart landed at 18:55:44 (new pid 2224) and the five Brain cards now serve their own names. Verified by live probe, not by reading a config:

$ for p in 8311 8312 8313 8314 8315; do
    curl -s http://127.0.0.1:$p/.well-known/agent-card.json \
      | python3 -c 'import sys,json;print(json.load(sys.stdin)["name"])'
  done
min
mira
sol
nova
echo

Each profile's own gateway.log records the before → after in one place — the same adapter, the same port, the same process tree:

min   2026-10-20 23:04:17  A2A: serving Agent Card ... on http://127.0.0.1:8311 ... as 'hermes-LAB'
min   2026-10-21 18:56:05  A2A: serving Agent Card ... on http://127.0.0.1:8311 ... as 'min'

5/5 distinct card names, and the card now agrees with /health and with cortex/registry/agents.yaml. A card probe can finally identify which Brain agent it reached.

Control — the Heart plane is unchanged. ace Alpha Atlas on 8301-8303 still carry their own names, so the change is Brain-scoped.

No regression in the A2A client path. A live SendMessage to nova through the a2a toolset after the restart returned A2A-POSTCHECK-OK; a2a_list still resolves all 12 peers by name; a2a_discover on http://127.0.0.1:8313/ now reports Agent: sol.

No test regression. tests/test_peer_dialects.py — 17/17 pass, both before and after the restart. It reads Brain identity from /healthserved_agents[].profile (not the card), so it never depended on the defect. The comment that said "every Brain card carries the same name: hermes-LAB" was stale after this change and has been reworded to say the card is fixed while /health stays the identity channel.

Wider suite: 11 pre-existing failures, none from this change. The suite must run on /usr/bin/python3 (the Hermes toolchain interpreter has no PyYAML and no a2a SDK). On that interpreter: 535 passed, 11 failed, 3 skipped, 2 xfailed. None of the 11 reference the agent card or A2A_AGENT_NAME, and none of them are in a file this change touched. The three distinct causes:

Nothing in the runtime routes on the Brain card name. a2a_orchestrate matches peers by configured capability, and a2a_call addresses peers by configured name — neither reads the peer's advertised card name. The advertised name is display-only, so nothing downstream silently changed meaning. security.py still has 0 references to agent_name: auth is untouched.

How the change actually reached the live fleet — it was a HOST REBOOT, not the reload. The card asked for a gateway restart; what happened at 18:55 was a reboot of the whole machine, which restarted the gateway as a side effect. The journal proves it, and the difference matters:

Proof it was a reboot, not a restart: the boot id changed (journalctl --list-boots shows 61900713… ending 18:55:14 and ff2159a2… starting 18:55:40), the GNOME session and the desktop were torn down and re-login'd, apt-daily-upgrade had just installed packages, and systemd-reboot.service ran. A systemctl --user restart cannot do any of that. The reboot's own trigger is not attributable from the journal — no systemctl reboot call, no sudo reboot at that time, no reboot-request entry, and Automatic-Reboot is commented out in /etc/apt/apt.conf.d/50unattended-upgrades. It follows a manual apt upgrade + reboot by a human at the console; the sudo reboot in ~/.bash_history is from 2026-10-11, not today.

What that means for the drain. The SIGUSR1 handler had deferred the restart on 3 active A2A DM turns with a 1800 s cap, and the reboot pre-empted that drain entirely — the machine went down, so the in-flight turns were dropped rather than resumed. The lifecycle ledger's "previous life exited UNCLEANLY" is the corroboration. The fix landed, but it landed by accident of a reboot, not by the planned drain. If a future card needs this restart again, the drain path still has to be made to work, because a reboot is not a substitute for it: it interrupts the whole host, not just the gateway.

Nothing is left armed. hermes-gateway shows NRestarts=0 since the boot, restart_loop.json holds a single boot entry (the breaker needs 3 chained entries), and no profile log has a pending "Restart deferred".

Scope note — the default profile is not in this card. The default profile still serves hermes-LAB on :9900; it is not one of the five Brain agents in the registry (min/mira/sol/nova/echo), and the card did not cover it. Left as-is deliberately; a separate change would be needed to rename it, and nothing routes on it either. (:9900 has also been serving with bearer auth since 2026-10-20 23:03 — before this change — so its card is not the only thing that differs from the Brain plane.)

Proven in-process with the real adapter rather than a sandbox copy, because the mechanism that matters is the scope: gateway/run_adapters.py:1300 builds each profile's adapter inside _profile_runtime_scope(profile_home), so adapter.py:275's _default_agent_name() reads that profile's own .env. Control used the pre-edit .env backups and reproduced the defect on 5/5.

9.4 Routing: no rotation, all 8 LAB agents answer as themselves

SendMessage "reply with your agent name", sent to each port, the answering agent read from the reply — not the card, not the port map:

8/8 route to their own identity. No rotation. A rotation is the failure that returns a successful response from the wrong agent, so this table is read from the reply text, not from the card.

The Brain card names in this table are the pre-fix values. They were captured before §9.3a's restart; the card now names each agent (§9.3a), and the card and the reply agree. The routing verdict is unchanged either way — that was the point of reading the reply.

Registry ↔ live: cortex/registry/agents.yaml lists 8301/8302/8303 and 8311–8315; all 8 are listening; 0 registry ports not listening, 0 listening ports not in the registry.

9.5 Two latencies that a short timeout would call "down"

• :8303 atlas — 90 s on the first attempt (my probe's own ceiling fired, and I recorded it as a timeout before re-probing with a 600 s ceiling: instant). This is Defect A in this same document — the blocking subprocess.run holding the event loop — showing up as latency, and it is the same listener §2 describes. My first reading ("atlas timed out") was the probe's ceiling, not the fleet.
• :8314 nova — 187.6 s for a one-line reply. Under the old 60 s a2a_agents timeout in the peer configs, that call is a timeout, not a slow success. Same class as Echo's 150 s in Tam's note.

So a "peer is down" report is more likely a timeout ceiling than a dead port. The a2a_agents peer timeout of 60 s in every profile's config is shorter than a real Brain turn. Worth widening to 300 s (the house timeout already used by solos-v3-worker).

9.6 Not done

• DID apply the A2A_AGENT_NAME change to all five live Brain profiles and DID restart the gateway — card t_b3a9bc3c, verified live in §9.3a. (This line previously read "did not"; the state changed.)
• Did not rename the default profile's card (:9900 still says hermes-LAB). It is not one of the five Brain agents and was outside the card's scope.
• Did not fix Defect A (blocking subprocess.run on the Heart event loop) or Defect B (the shim's dead Pulse address) — both diagnosed in §2/§3.
• Did not correct Tam's note at its source. §9.2 records the measured disagreement; Tam owns his note and the correction belongs to him, so this is filed as a finding, not an edit to his file.
• Did not create a second probe document — this section is inside the existing fleet-probe-2026-10-21.md (same topic, one home per fact).