LAB A2A Self-Heal Watchdog — spec (Young directive: "crons for self-healing, get Sol and Ace on it")

The problem, measured

Agents wedge on a hung provider call with no task-level timeout. The signature is consistent and reproducible across every sweep today:

• systemctl --user is-active lab-a2a-<x> → active
• GET /health → {"status":"running"}
• registry /agents → online, fresh: true
• ps → 0.2–0.3 % CPU (idle, blocked on I/O — not busy-looping)
• POST message/send → times out (never returns)

A systemctl --user restart <unit> clears it instantly, every time. No work is lost (0.2 % CPU = nothing in flight).

Sweep history today (real dispatch, not /health):

Wedged agents differ every sweep — it is per-turn I/O, not a fixed bad agent.

Why the existing heartbeat does not cover this

lab-eva-heartbeat.timer / lab-eve-heartbeat.timer (5-min interval) cover only Eve and Eva, and per Sol's earlier finding they write status='online' straight into SQLite rather than probing the door — so they cannot report a fault by construction. There is no heartbeat for the 8 A2A agents at all, and nothing that probes them by real dispatch.

The spec — lab-a2a-selfheal.timer (Sol/Ace build)

Unit: lab-a2a-selfheal.timer → lab-a2a-selfheal.service, user-level (same layer as the lab-a2a-* units it repairs), OnBootSec=3min, OnUnitActiveSec=2min, Persistent=false.

Script: /home/lab/.solos/scripts/a2a_selfheal.sh — bash, no new deps.

Algorithm per cycle:

1. For each lab-a2a-{sol,ace,mira,nova,echo,alpha,atlas,min}:
   • curl -fsS -m 4 http://127.0.0.1:<port>/health (liveness only, cheap)
   • real dispatch probe: POST / with a trivial message/send, --max-time 12
   • if health OK and dispatch completes → continue (no action)
   • if health OK but dispatch times out → wedged → systemctl --user restart <unit>
   • if health fails → down → systemctl --user restart <unit>
2. Concurrency guard: flock -n /run/lock/lab-a2a-selfheal.lock — never overlap runs.
3. Restart budget: max 1 restart per unit per 10 min (a unit that keeps dying is a different problem; log it, don't thrash).
4. Log one line per action to ~/.solos/logs/a2a-selfheal.log with a rolling tail.
5. Alert only on restart (Young's standing rule: no alert fatigue) — deliver to the board/A2A as one line: a2a-selfheal: restarted <unit> (<reason>).

Exit 0 always unless the script itself fails (so a wedged agent never marks the timer failed).

Why 2 min / 12 s: the fastest observed human-visible wedge recovery was an immediate manual restart; 12 s is above the slowest healthy lane (Min 14.3 s observed once, typical 3–5 s) — so use 15 s probe timeout to avoid false positives, and 2 min cycle so worst-case deafness is ~2 min instead of "until Young notices".

Safety: restart is the same reversible action used manually today; no data loss observed because wedged units are idle. Rollback = systemctl --user disable --now lab-a2a-selfheal.timer.

Acceptance (how we know it works)

1. Deliberately wedge a lane (SIGSTOP the runner) → within one cycle the timer restarts it and dispatch returns to <6 s. Sweep returns 8/8.
2. Two consecutive sweeps 5 min apart both 8/8 without manual intervention.
3. grep restart ~/.solos/logs/a2a-selfheal.log shows the action, and no more than 1 restart per unit per 10 min.
4. Zero alert noise: no dispatch for units that were healthy.

The real fix (not the watchdog)

The watchdog is symptom control. The cure is task-level provider timeouts in SolOS/runtime/agents-runner.py so a hung call aborts instead of blocking the door forever — that is Sol's lane and belongs in the v2 scaffold (agent_loop timeouts, already tracked in the rebuild blueprint acceptance set).

🎯 Tam's Read: the wedge is a per-turn I/O hang with no timeout; a 2-minute dispatch-probing watchdog converts today's manual restarts into automatic ones and caps deafness at ~2 minutes, while the timeout fix in the runner removes the cause.