A2A replies are DROPPED, not lost — the 300 s reply timeout discards a finished answer

Owner: Echo (research & security lane). Measured on LAB, 2026-10-21 ~17:42–18:10 AEDT. Lane: plugins/platforms/a2a/ (vendored Hermes A2A plugin) + the Brain fleet's a2a_agents peer timeouts.

Trigger: a peer sent five probe messages to the five Brain ports in eight seconds. Two of five were reported as failures. Both agents had answered in full, and both answers were destroyed. This note records the mechanism, the measurement, and the two levers.

1. Summary

Fleet verdict: the Brain tier is healthy and answers correctly. The A2A reply path is where the loss happens.

2. A — The reply timeout destroys a finished answer (HIGH, OPEN)

The measurement

Five probe sends, one per Brain port, 8 s apart (17:42:01 → 17:42:33 AEDT). Two succeeded, three returned [agent did not reply in time]:

The gap is exactly the timeout, every time. From ~/.hermes/a2a_audit.jsonl:

task-599a8734391a4386  -> [agent did not reply in time]  after 300.0 s
task-12a2e298acf34561  -> [agent did not reply in time]  after 300.0 s
task-bcfc0aae091145fe  -> [agent did not reply in time]  after 300.0 s

300.0 s to the decimal on all three. The agents were not slow to start; they were slow to finish, and the door had already stopped listening.

Proof the answer existed and was thrown away

Mira's own session store, read directly (profiles/mira/state.db, session 20261021_174209_30e3a164):

17:42:09 user      [A2A inbound …]
17:49:26 assistant Probe complete. Findings, with evidence: **A2A reachable — I'm alive.** …

Mira produced a complete, substantive answer at 17:49:26. The door wrote [agent did not reply in time] at 17:47:09 — 137 s earlier. The answer existed; it was thrown away.

The persisted conversation record shows the same thing, and shows what the caller actually received:

$ cat ~/.hermes/a2a_conversations/ctx-a23c8ea0302f432f.jsonl
{"ts": 1792564929.61, "role": "user",  "text": "probe",                        "task_id": "task-599a8734391a4386"}
{"ts": 1792565229.61, "role": "agent", "text": "[agent did not reply in time]", "task_id": "task-599a8734391a4386"}

The agent row contains the timeout string, not Mira's answer. The caller received a lie of omission: the agent is told the peer never replied.

The mechanism, in code

plugins/platforms/a2a/adapter.py:

# line 66-71 — the window
def _reply_timeout() -> float:
    return max(1.0, float(os.getenv("A2A_REPLY_TIMEOUT", "300")))

# line 663-665 — the wait
def _await_reply(self, pending, keepalive=None):
    return self._await_future(pending["future"],
                              pending["started"] + _reply_timeout(), keepalive,
                              (protocol.STATE_FAILED, "[agent did not reply in time]"))

# line 834-838 — where the answer goes
async def send(self, chat_id, content, reply_to=None, metadata=None):
    if not (metadata or {}).get("notify"):
        logger.debug("A2A: ignoring non-final send for context %s", chat_id)
    elif not self._resolve_oldest_for_context(chat_id, protocol.STATE_COMPLETED, content or ""):
        logger.debug("A2A: send() for context %s had no pending waiter", chat_id)  # late chunk / out-of-band

The task's Future is resolved by the gateway's final reply send. When the 300 s window expires, _await_reply returns the failure tuple and _finalize_task pops the pending entry (adapter.py:643). When the agent's real answer finally arrives, send() looks for a pending waiter, finds none, and the reply is dropped — at logger.debug, so it is invisible at the default log level.

The audit row is written from the same discarded value, so the record says the agent never replied. Both the caller and the audit log are wrong, and they are wrong in the same way, which is why this reads as a peer fault rather than a transport fault.

Why the 300 s window is not generous — measured across the fleet

All Brain profiles' gateway logs, response ready … time=N s:

total response-ready events: 1727   max 17963.3 s
  <30s   477
  30-60  242
  60-120 304
  120-300 354
  300-600 176      <-- these arrive AFTER the reply window
  >600   174      <-- and these
over 300s: 350 (20%)
over 600s: 174

One in five real turns finishes outside the window, and 174 turns took more than ten minutes. The window is not a tail-case allowance; it is inside the normal distribution.

3. B — The peer-side timeout is shorter than the door's own window (HIGH, OPEN)

Every profile's a2a_agents block sets timeout: 60 for every peer (profiles/echo/config.yaml and the same in min/mira/sol/nova). The one exception is tam-lab at 120.

The door waits 300 s; the caller's socket gives up at 60 s. So even when the agent answers inside the door's window, the caller has already disconnected. Two independent timers, and the outer one is five times shorter.

Measured consequence — a peer that took 187.6 s is a timeout, not a slow success, for every caller in the fleet. Sol's §9.5 in fleet-probe-2026-10-21.md reached the same conclusion from the other direction.

⇒ The two timers must be ordered, and the outer one must be the longer. Caller timeout > door reply window, or the caller must poll GetTask.

4. C — The drop leaves no trace (MEDIUM, OPEN)

The only signal that an answer was discarded is logger.debug("A2A: send() for context %s had no pending waiter").

• It is DEBUG. The gateway runs at INFO (grep setLevel gateway/run.py), so the line never appears.
• grep -rn "no pending waiter" ~/.hermes/profiles/*/logs/*.log → 0 hits, across a window in which at least three answers were dropped.

Nothing in the audit log distinguishes "the agent never replied" from "the agent replied too late". The caller cannot tell, the operator cannot tell, and the record is indistinguishable from a dead peer.

5. D — A timed-out task disappears from GetTask (LOW, by design)

TaskStore is in-memory and capped:

# protocol.py:292-296
class TaskStore:
    _MAX_TERMINAL = 500
# protocol.py:418-421 — _trim_locked evicts the oldest terminal records

A task that has timed out is terminal, so it is eligible for eviction immediately. Measured:

17:50:24  GetTask task-599a8734391a4386 -> task not found   (port 8315 — wrong port, see below)
18:01:13  GetTask task-599a8734391a4386 -> task not found
18:01:13  GetTask task-bcfc0aae091145fe -> TASK_STATE_FAILED  "[agent did not reply in time]"

The task store is per-adapter (per port), not global — adapter.py:289 constructs protocol.TaskStore() per adapter instance. So a task is only retrievable from the port that created it. My first poll used :8315 for a task created on :8312 and got task not found — my probe error, not a defect. The corrected poll, against the right port, returns the failure state and the timeout string. Recording it because "the task vanished" is the natural reading and it is wrong.

6. What is NOT the cause

• Not truncation. No payload was cut. The bodies were 5 bytes (probe).
• Not a misroute. Each probe landed in the correct profile's session — verified by reading the sessions table of min/mira/sol/nova/echo directly: each probe has its own ctx-* and its own session id, and the 17:42:01/09/17/25/33 sends map 1:1 onto min/mira/sol/nova/echo.
• Not a protocol/dialect error. v1.0 SendMessage + parts[{kind:text}] works: a control send to :8315 returned TASK_STATE_COMPLETED with the sentinel OK-A in both status.message.parts[0].text and artifacts[0].parts[0].text. The v0.3 content shape still returns TASK_STATE_REJECTED "Empty task" — matches fleet-probe-2026-10-21.md §9.2.
• Not the provider. opencode-go did fail once during the window (APIConnectionError, 17:46:14, retried and recovered). That is real but incidental — Mira answered 17 s after that retry.

7. The two levers (NOT applied — a probe request does not authorise a

production-affecting change to a shared serving path)

Both require a gateway restart (pid 3733570 serves all five Brain ports), which interrupts every Brain agent. That is Young's decision, not a probe's.

Lever 1 — widen the door's reply window

# system-level env for the gateway unit, or the profile .env
A2A_REPLY_TIMEOUT=900

adapter.py:69 reads it; there is no other knob. 900 s covers the 300–600 s band and most of the >600 s band. Caveat: the orphan sweep (adapter.py:369, self.tasks.fail_orphans(timeout, …)) uses the same window, so widening it also widens the orphan grace — which is the correct coupling, and is what the vendored README describes (floor 300 s, never fails a task while a request is still waiting).

Lever 2 — make the caller wait longer than the door

In every profile's config.yaml, a2a_agents.<peer>.timeout:

timeout: 60    ->    timeout: 900

Rule to encode: caller timeout > door reply window. If the door is 900 s, the caller needs ≥ 900 s; if the door stays 300 s, the caller still needs > 300 s, and 600 s is the safe minimum.

Lever 3 (not a timeout) — make the drop visible

Promote the drop site to a warning with the discarded text, so a late answer is distinguishable from a silent one:

# adapter.py:838
logger.warning("A2A: late reply for context %s had no pending waiter — DISCARDED: %r",
               chat_id, (content or "")[:200])

Lever 4 (design) — a terminal task should still carry the late answer

TaskStore.complete() is idempotent and refuses to overwrite a terminal record (protocol.py:376-378). So a late answer cannot repair the task even if it were captured. The durable fix is to let a late send() write into the terminal record (as result_output / a late_reply field) so a GetTask after the fact shows the truth, and to have the audit row record late_reply rather than the timeout string.

8. What this means for a caller

1. A TASK_STATE_FAILED with [agent did not reply in time] is not proof the peer is dead, slow, or unresponsive. It is proof the peer did not answer within the door's window. On this fleet that is 20 % of turns.
2. Before resending, read the peer's own store. Mira answered. If you resend, you duplicate real work and you may cause a second 300 s wait.
3. Poll GetTask on the port that created the task — the store is per-adapter.
4. Do not trust the audit summary to tell you what the peer received.

9. Not done

• Did not change A2A_REPLY_TIMEOUT, any a2a_agents timeout, or any code.
• Did not restart hermes-gateway.
• Did not touch PROD (161.33.71.67).
• Did not modify fleet-probe-2026-10-21.md — Sol owns that file; this finding is filed separately rather than edited into it.