Handoff — lab-a2a-selfheal build lane (Tam → Ace, verbatim spec + measured facts)

Ace has no vault mount, so the spec is pasted verbatim below. Everything under "ADDENDUM (Tam, measured 2026-09-27)" is not in the spec — it is live-measured LAB context Ace needs to avoid collisions and false positives.

PART 1 — VERBATIM SPEC (unchanged, sha256 2445c203…50da)

The following block is the file solos/research/tam/tech/2026-09-27-lab-self-heal-watchdog-spec.md byte-for-byte.

---
date: 2026-09-27
tags: [lab, watchdog, self-healing, a2a, reliability, fleet, spec]
status: spec-ready — awaiting Sol/Ace build sign-off
source links:
  - reproduced live on LAB 192.168.4.60 across 4 sweeps on 2026-09-27
  - board: task-a86d1f6f (prod→LAB mirror), Sol commit 5325818 (cron design)
related: "[2026-09-27-lab-connectivity-audit](2026-09-27-lab-connectivity-audit)"
---

# LAB A2A Self-Heal Watchdog — spec (Young directive: "crons for self-healing, get Sol and Ace on it")

## The problem, measured

Agents wedge on a **hung provider call with no task-level timeout**. The signature is
consistent and reproducible across every sweep today:

- `systemctl --user is-active lab-a2a-` → `active`
- `GET /health` → `{"status":"running"}`
- registry `/agents` → `online`, `fresh: true`
- `ps` → **0.2–0.3 % CPU** (idle, blocked on I/O — not busy-looping)
- `POST message/send` → **times out** (never returns)

A `systemctl --user restart ` clears it instantly, every time. No work is lost
(0.2 % CPU = nothing in flight).

**Sweep history today (real dispatch, not /health):**

| Sweep | Result | Wedged |
|---|---|---|
| 1 | 8/8 | min (14.3 s, near-limit) |
| 2 | 7/8 | min (20 s timeout) |
| 3 | 7/8 | sol (25 s timeout) |
| 4 | 8/8 | — (all <6 s) |
| 5 | 5/8 | nova, echo, alpha (all 20 s+) |

Wedged agents differ every sweep — it is per-turn I/O, not a fixed bad agent.

## Why the existing heartbeat does not cover this

`lab-eva-heartbeat.timer` / `lab-eve-heartbeat.timer` (5-min interval) cover **only
Eve and Eva**, and per Sol's earlier finding they *write* `status='online'` straight
into SQLite rather than probing the door — so they **cannot report a fault by
construction**. There is no heartbeat for the 8 A2A agents at all, and nothing that
probes them by real dispatch.

## The spec — `lab-a2a-selfheal.timer` (Sol/Ace build)

**Unit:** `lab-a2a-selfheal.timer` → `lab-a2a-selfheal.service`, user-level
(same layer as the `lab-a2a-*` units it repairs), `OnBootSec=3min`,
`OnUnitActiveSec=2min`, `Persistent=false`.

**Script:** `/home/lab/.solos/scripts/a2a_selfheal.sh` — bash, no new deps.

Algorithm per cycle:

1. For each `lab-a2a-{sol,ace,mira,nova,echo,alpha,atlas,min}`:
   - `curl -fsS -m 4 http://127.0.0.1:/health` (liveness only, cheap)
   - **real dispatch probe:** `POST /` with a trivial `message/send`, `--max-time 12`
   - if health OK **and** dispatch completes → `continue` (no action)
   - if health OK **but** dispatch times out → **wedged** → `systemctl --user restart `
   - if health fails → **down** → `systemctl --user restart `
2. Concurrency guard: `flock -n /run/lock/lab-a2a-selfheal.lock` — never overlap runs.
3. Restart budget: **max 1 restart per unit per 10 min** (a unit that keeps dying is a
   different problem; log it, don't thrash).
4. Log one line per action to `~/.solos/logs/a2a-selfheal.log` with a rolling tail.
5. **Alert only on restart** (Young's standing rule: no alert fatigue) — deliver to
   the board/A2A as one line: `a2a-selfheal: restarted  ()`.

**Exit 0 always** unless the script itself fails (so a wedged agent never marks the
timer failed).

**Why 2 min / 12 s:** the fastest observed human-visible wedge recovery was an
immediate manual restart; 12 s is above the slowest healthy lane (Min 14.3 s observed
once, typical 3–5 s) — so use **15 s** probe timeout to avoid false positives, and
2 min cycle so worst-case deafness is ~2 min instead of "until Young notices".

**Safety:** restart is the same reversible action used manually today; no data loss
observed because wedged units are idle. Rollback = `systemctl --user disable --now
lab-a2a-selfheal.timer`.

## Acceptance (how we know it works)

1. Deliberately wedge a lane (SIGSTOP the runner) → within one cycle the timer
   restarts it and dispatch returns to <6 s. **Sweep returns 8/8.**
2. Two consecutive sweeps 5 min apart both 8/8 **without manual intervention**.
3. `grep restart ~/.solos/logs/a2a-selfheal.log` shows the action, and no more than
   1 restart per unit per 10 min.
4. Zero alert noise: no dispatch for units that were healthy.

## The real fix (not the watchdog)

The watchdog is symptom control. The cure is **task-level provider timeouts** in
`SolOS/runtime/agents-runner.py` so a hung call aborts instead of blocking the door
forever — that is Sol's lane and belongs in the v2 scaffold
(`agent_loop` timeouts, already tracked in the rebuild blueprint acceptance set).

🎯 **Tam's Read:** the wedge is a per-turn I/O hang with no timeout; a 2-minute
dispatch-probing watchdog converts today's manual restarts into automatic ones and caps
deafness at ~2 minutes, while the timeout fix in the runner removes the cause.

PART 2 — ADDENDUM (Tam, measured on LAB 2026-09-27 ~22:15 local, NOT in the spec)

✅ Greenfield — zero collision on the name

systemctl --user list-unit-files | grep -i selfheal   ->  (empty)
ls -la /home/lab/.solos/scripts/a2a_selfheal.sh      ->  No such file or directory

Nothing named selfheal exists as unit, script, or service. lab-a2a-selfheal.timer is free.

🔴 BLOCKER for the spec's "real fix" line — the timeout is NOT in agents-runner.py

The spec says the cure is task-level timeouts in SolOS/runtime/agents-runner.py. Measured, that file is a 657-byte uvicorn launcher with no agent loop at all:

# /home/lab/SolOS/runtime/agents-runner.py  (657 bytes, 0700)
app = build_app(tools=list(tools), agent_name=name, agent_port=port)
uvicorn.run(app, host="127.0.0.1", port=port, log_level="info", timeout_graceful_shutdown=30)

The real call path — and where the timeouts already live:

Consequence for Ace's build: a provider call can legitimately run up to 180 s, and the door's wait_for cap is 300 s. The watchdog's proposed 15 s probe timeout is therefore 12×–20× shorter than a legitimate in-flight call. A busy-but-healthy agent mid-LLM-call will look "wedged" to a 15 s probe.

→ Build implication: the probe must distinguish busy from wedged before restarting. /health already returns the fields you need:

{"agent":"sol","status":"running","circuit_open":false,"circuit_failures":0,
 "sessions":2,"current_task":"[Young] Self-he..."}

Recommended gate (needs Young sign-off, do not ship unilaterally): only restart when /health returns 200 and current_task is empty/unchanged across two consecutive cycles and the dispatch probe timed out. Otherwise you will SIGTERM real work every 2 minutes and the 0.2 % CPU "nothing in flight" assumption breaks.

This also directly contradicts board task-ff3cec20 (fleet jam: DISPATCH_STUCK backlog 800–968 min) — those agents were busy with stuck work, not idle.

⛔ Sol's scope — do not collide

Sol's lane is the cause fix, not the watchdog:

• Board task-0f67574 (in_progress) — "WATCHDOG / AUTO-RESTART — root fix for recurring agent hangs… run_*.py loops wedge on poll_schedule_timeout (Sol :8300 + Alpha :8305 hung 2026-09-22)… Add supervision: systemd WatchdogSec + sd_notify, or a health-check poller that auto-restarts any agent with no agent-card/health response for N seconds. Owner: Sol (arch) + Ace (eng)."
• Board task-fe702d26 (completed) — V2 Phase 1 scaffold (Sol). There is no /home/lab/SolOS/v2 directory on the box — the scaffold is not on disk in the location the spec implies.
• Sol's own file: sd_notify / WatchdogSec supervision inside the agent units (the Type=notify route). grep sd_notify /home/lab/.solos/core/agent_app.py → no match. If Sol lands Type=notify + WatchdogSec, the poller half of task-0f67574 is the same work as this spec — you two are converging on one task.

Safe split, proposed, needs Young/Min to arbitrate:

Ace: do not edit agent_app.py, agent_loop.py, agents-runner.py, or any lab-a2a-*.service unit. That is Sol's file. If you change the agent units you break Sol's Type=notify work and your own restart path at the same time.

📌 Port map — measured from the live unit files, do not guess

lab-a2a-sol.service    AGENT_NAME=sol    AGENT_PORT=8300
lab-a2a-ace.service    AGENT_NAME=ace    AGENT_PORT=8301
lab-a2a-mira.service   AGENT_NAME=mira   AGENT_PORT=8302
lab-a2a-nova.service   AGENT_NAME=nova   AGENT_PORT=8303
lab-a2a-echo.service   AGENT_NAME=echo   AGENT_PORT=8304
lab-a2a-alpha.service  AGENT_NAME=alpha  AGENT_PORT=8305
lab-a2a-atlas.service  AGENT_NAME=atlas  AGENT_PORT=8306
lab-a2a-min.service    AGENT_NAME=min    AGENT_PORT=8307

All 8: Restart=on-failure, RestartSec=5, TimeoutStopSec=20, NRestarts=0, ActiveState=active.

/home/lab/.solos/scripts/a2a_send.py has a stale port map — it lists atlas:8306 but omits alpha/8305 entirely, and has no eva. Do not reuse it as the probe client without fixing; its own timeout=300 is also wrong for a watchdog probe.

📐 Facts you need that the spec leaves implicit

• Unit file location: /home/lab/.config/systemd/user/lab-a2a-selfheal.{timer,service} (confirmed — lab-eva-heartbeat.timer lives there).
• Unit template to copy (lab-a2a-echo.service, the shape to mirror): After=network.target lab-central-registry.service / Wants=network.target, Type=simple, EnvironmentFile=/home/lab/SolOS/runtime/lab-systemd.env, NoNewPrivileges=true, PrivateTmp=true, journal output.
• Restart cost: TimeoutStopSec=20 + RestartSec=5 ⇒ worst case 25 s to return to serving, plus uvicorn timeout_graceful_shutdown=30 and _GRACEFUL_DRAIN_SECONDS=15. A 2-minute cycle must not restart two units simultaneously, or agents will cross-talk during drain.
• Lock file path is valid: /run/lock is writable by uid 1001 (lab) — the spec's flock -n /run/lock/lab-a2a-selfheal.lock works as written. (Alternative with the same guarantee: $XDG_RUNTIME_DIR/lab-a2a-selfheal.lock = /run/user/1001.)
• State file — spec has none, budget needs one. Spec rule 3 (max 1 restart per unit per 10 min) is unimplementable without persistent state. Use /home/lab/.solos/state/a2a-selfheal-restarts (dir exists, lab:lab 0755, precedent: .solos/state/tunnel_watchdog_state). One line per unit: unit epoch.
• Log path valid: /home/lab/.solos/logs/ exists and is writable; currently holds auto-pilot.log (1.1 MB) — your log needs its own tail cap.
• Alert channel — concrete, verified. Board is 127.0.0.1:8432, live (/health → {"status":"ok","service":"central-registry"}). POST /reports schema (read from its own openapi.json): {"agent": str, "kind": str = "report", "content": str, "project": str = ""} — agent and content required. That is the most reliable "board/A2A" target. There is no POST /board write route; /board is GET-only.
• Registry writes status='online' are useless as a fault signal (already in the spec) — confirmed, /agents freshness is not derived from a real probe.

⚠️ Two spec bugs to fix before you build

1. Probe timeout is self-contradictory. Algorithm step 1 says --max-time 12; the "Why 2 min / 12 s" paragraph then says use 15 s (because Min was observed at 14.3 s — 15 s is already only 0.7 s of headroom over one observation). Neither value is above the real 180 s provider cap. Resolve explicitly and write the number in one place. Recommend 15 s for detection + the busy-gate from the blocker above before any restart.
2. TimeoutStopSec mismatch in the code comments. core/agent_app.py:23 and :462 reason about TimeoutStopSec=130; every live lab-a2a-*.service says TimeoutStopSec=20. The drain logic was tuned for a 130 s window and is running with 20 s. Flag to Sol — it is a real shutdown-path bug, not a watchdog issue, but it can make a restart look like a SIGKILL.

🎯 Acceptance deltas for the spec's test 1

Spec acceptance 1 says "SIGSTOP the runner". With Restart=on-failure + 25 s stop window, SIGSTOP → systemctl --user restart = SIGTERM to a stopped process, which cannot be handled ⇒ the cycle burns the full 25 s. Also SIGSTOP is the only clean way to reproduce the real wedge (real wedges are I/O blocks, not stops), so keep it — just budget the expectation, and add: a unit that is SIGSTOPped at the moment the watchdog restarts it must not be double-restarted inside the 10-min budget.

Verbatim paste ends here.

Sources: live LAB systemctl --user cat/show/list-unit-files, curl against :8300/:8304/:8306 /health, board :8432 /health, /openapi.json, /tasks, /tasks/; file reads of /home/lab/SolOS/runtime/agents-runner.py, /home/lab/.solos/core/agent_app.py, /home/lab/.solos/core/agent_loop.py, /home/lab/.solos/scripts/a2a_send.py; sha256 pull-back verification of the spec file (local laptop copy == LAB canonical copy, 2445c203…50da).

🎯 Tam's Read: the name is free and the port map is pinned, but the spec's "real fix" points at the wrong file (agents-runner.py has no agent loop) and its 15 s probe would restart agents that are legitimately mid-180 s LLM call — Ace should build the timer with a busy-gate and leave agent_app.py/agent_loop.py to Sol, since task-0f67574 has both their names on it.