Agent memory provider — hindsight

Card t_af604d1b asked for a ruling that nobody re-litigates: which memory provider does the fleet run, why the other one is rejected, what makes the choice exclusive, and how it gets installed.

The decision: LAB runs hindsight, served bare-metal on 192.168.4.60:8888, reached by agents in local_external mode. mem0 is rejected.

Every claim below was re-measured on LAB on 2026-10-21, not carried forward from the study. Where I found the study's framing to be imprecise, I say so and give the corrected mechanism — that correction is the most useful thing in this note.

1. The decision, in one block

2. Why hindsight, and why mem0 is rejected

The licence test — both pass, and that is not the deciding test

Neither stack contains AGPL, SSPL or BUSL. Both are licence-clean, so the licence is not what separates them. Echo's study said the same, and it is worth restating because it is the part people remember wrongly: the licence was never the discriminator.

The capability gate on mem0 (verified in source, not from a README badge)

mem0's own plugin, integrations/hermes-plugin-mem0/_backend.py:94:

# rerank is platform-only; the self-hosted /search ignores it.

and mem0/configs/prompts.py:474:

"You extract from BOTH user and assistant messages."

with prompts.py:886 carrying a worked example titled "Example 12: Multi-Speaker Conversation — Extract From ALL Speakers."

So self-hosting mem0 means running the tier the vendor sells as the reduced one, and its extraction prompt is built to ingest assistant-authored content as first-class memory. In a single-user product that is a design choice. In our topology — eleven agents, one shared store, where a peer agent's message arrives in the assistant role — it makes another agent's assertion first-class extractable memory. That is precisely the false-memory mechanism our own 2026-10-11 memory-architecture study identified, and the one speaker-attested grounding (SAG) fixes.

mem0's custom_instructions is not a substitute. It is a real field (mem0/configs/base.py:54) that threads into the extraction prompt — but it sits in the same prompt as "extract from BOTH roles" and "extract from ALL speakers". It is a conflicting instruction, not a gate: the LLM decides. Our own study concluded write-path validation must be structural. A competing sentence in a system prompt is not structural.

What mem0 does have: per-write agent_id and metadata attribution (_backend.py:17 — _add_kwargs(user_id, agent_id, infer, metadata)), which lets a reader filter or downgrade agent-authored memories at query time. That is a real structural mitigation, and it is why the read path matters more than the provider.

The symmetric check on hindsight — and it comes out better

hindsight's Hermes plugin keeps the speaker as a first-class field, not as prose (hindsight-integrations/hermes/__init__.py:1684):

def _build_turn_messages(self, user_content, assistant_content):
    return [
        {"role": role, "content": f"{prefix}: {content}", "timestamp": now}
        for role, prefix, content in (
            ("user",      self._retain_user_prefix,      user_content),
            ("assistant", self._retain_assistant_prefix, assistant_content),
        )
    ]

plus per-write metadata: retained_at (UTC audit time), message_count, turn_index, source.

The precise claim, so it is not overstated: hindsight preserves the speaker label. I found no evidence it filters or down-weights by role on the retain path. So hindsight is not SAG either. The difference is that hindsight hands the read path the field it needs, where mem0's extraction prompt has already dissolved the distinction into prose before anything is stored. Neither provider implements speaker-attested grounding; only one of them leaves the evidence intact for us to implement.

That is the decision I would make on, and it is a shape difference, not a benchmark.

The things hindsight also wins on

• MIT covers the whole tree; no "platform-only" markers found anywhere in the repo.
• A consolidation operation ("reflect", knowledge graph) — the layer mem0's ADD-only store structurally lacks. mem0's memory path is add-only for extraction: grep -c 'event.*UPDATE\|event.*DELETE' over mem0/memory/main.py returns matches, but all six are capture_event(...) telemetry calls, not store mutations. The store is still ADD-only, and its dedup is delegated to the LLM via a 20-item "recently extracted" window.
• No staleness metadata: grep for written_at, last_confirmed_at, validity_basis across mem0/ returns nothing.
• trusted_project_dirs — a trusted-source check on which directory's config may set the memory bank. A class of control mem0's plugin does not have.

One thing mem0 wins on, honestly

user_id sharing across gateways is a documented one-value config — the Pulse↔LAB requirement with the least ceremony. hindsight needs the server plus local_external wiring. That is a real cost, and it is a real cost of the topology, not of the software. We are paying it anyway because the server is running either way (§4).

Where mem0 was ruled out on top of all this

Anonymous telemetry is ON by default. mem0/memory/telemetry.py:14:

MEM0_TELEMETRY = os.environ.get("MEM0_TELEMETRY", "True")

The catalog description flags it too. A memory server holds everything every agent has said; it is the highest-value target in the fleet. Shipping it with a vendor phone-home on by default, into a product whose own self-host tier is the reduced one, is the wrong default to accept.

3. The exclusivity constraint — and the mechanism is not the protobuf pin

This is the correction that matters most, and it is the one a future session must not get backwards.

The card was migrated from fleet-board.md carrying: "they cannot coexist — protobuf conflict." The protobuf conflict is real, and it is not the reason they cannot coexist.

The real, enforced reason: Hermes' own exclusive-provider rule

agent/memory_manager.py:434:

def add_provider(self, provider: MemoryProvider) -> None:
    """Register a provider; builtin always accepted, only ONE external allowed."""
    if provider.name != "builtin" and self._has_external:
        ...
        logger.warning(
            "Rejected memory provider '%s' — external provider '%s' is "
            "already registered. Only one external memory provider is "
            "allowed at a time. Configure which one via memory.provider "
            "in config.yaml.", provider.name, existing,
        )
        return

Hermes' CLI says it in one line (hermes_cli/subcommands/memory.py:18):

"Only one external provider can be active at a time."

and the plugin loader enforces it structurally: memory providers are kind: exclusive (hermes_cli/plugins_manifest.py:270 — memory-provider markers imply exclusive), and the general discovery path skips exclusive plugins (hermes_cli/plugins_discovery.py:279 — "exclusive plugin — activate via <category>.provider config").

So the two providers are exclusive by Hermes' architecture, at the config level, before any dependency is resolved. One memory.provider: value. One active external provider. This is not a quirk of these two packages; it is how Hermes models memory.

The protobuf conflict is real, but it binds the server layer, not the plugin layer

The pins, verified against PyPI metadata today:

protobuf<7.0.0 and protobuf>=7.35.1 cannot both be satisfied. The constraint is genuinely unsatisfiable.

But — and this is the part the study's framing gets wrong — that conflict does not bind the thin Hermes plugins. The plugins are thin:

hindsight plugin 1.2.3  →  hindsight-client>=0.10.1,<1  +  hindsight-embed>=0.10.1,<1
mem0 plugin      1.3.0  →  mem0ai>=2.0.10,<3             +  httpx>=0.27,<1

Neither pulls protobuf directly. I resolved and installed both thin plugin dependency sets into one venv on LAB, and it succeeded:

protobuf 6.33.6 · mem0ai 2.2.1 · hindsight-client 0.10.3 · hindsight-embed 0.10.3

The protobuf conflict bites at the server layer — hindsight-api-slim (the full server tree) cannot resolve into a venv that already carries mem0ai. That is exactly what hindsight's own plugin README says it is avoiding: the server is deliberately not a Hermes venv dependency; local_embedded runs it as a separate process, and local_external points at a server we run ourselves.

So the accurate statement of exclusivity is two-layered, and both layers point the same way:

1. Hermes' exclusive-provider rule — binding, enforced, and the reason we pick one. One memory.provider: value; one active external provider.
2. The server-layer dependency conflict — real, and it means the two providers cannot be run as two servers against one store on one host without a dependency split.

Neither layer is "the protobuf pin alone." A future session that reads only the pin and concludes "hindsight and mem0 could coexist as plugins" would be right about the plugin and wrong about the product — they still cannot both be the active provider, and they still cannot both serve the same fleet from one host. The decision is unaffected. The stated mechanism was.

A correction to the catalog pin, found while checking

The live catalog (https://hermes-agent.nousresearch.com/docs/api/plugin-catalog.json, 754 entries) and the on-disk plugin-catalog/hindsight.yaml disagree:

hermes plugins search hindsight shows the live value (1.2.3 @ d952b6e6). The on-disk yaml is a stale local copy. The pins in the study (d56c4acd for hindsight, 43849c6e for mem0) are the old ones. Echo's study recorded hindsight as 1.2.3 — right on version, but on the old sha. The mem0 pin 43849c6e still matches the live catalog, and that commit no longer resolves on GitHub (No commit found for SHA: 43849c6e…) — the mem0 repo has moved it.

Consequence for this decision: none. The mem0 plugin's dependency set is unchanged at main (mem0ai>=2.0.10,<3 — I re-read it live). But the record must not carry a dead SHA forward.

4. The install path — and it decides whether Docker is needed on LAB at all

The install path is the decision's second load-bearing consequence, and it is now settled by measurement, not by argument.

The answer: no Docker on LAB, and no root either

hindsight-api-slim installs and runs on LAB as the unprivileged lab user, with an embedded PostgreSQL it provisions itself. I re-proved this end-to-end today, from a clean venv — the sibling card t_ac60925f proved it on 2026-10-21 and I did not take that on trust.

Install (no root, no container runtime):

uv venv --python 3.14 .venv
uv pip install --python .venv/bin/python \
    "hindsight-api-slim[embedded-db]" "hindsight-api-slim[local-onnx]"
# → hindsight-api-slim 1.0.0 · pg0-embedded 0.15.2 · onnxruntime 1.31.0 · protobuf 7.36.2

The three install traps reproduce exactly, and each is a silent startup failure:

Run (embedded Postgres, LAN-bound, no daemon to install):

export HINDSIGHT_API_HOST=192.168.4.60 HINDSIGHT_API_PORT=8888
export HINDSIGHT_API_EMBEDDINGS_PROVIDER=onnx HINDSIGHT_API_RERANKER_PROVIDER=rrf
export HINDSIGHT_API_LLM_PROVIDER=opencode-go \
       HINDSIGHT_API_LLM_MODEL=deepseek-v4-flash \
       HINDSIGHT_API_LLM_API_KEY="$OPENCODE_GO_API_KEY"
exec .venv/bin/hindsight-api --host 192.168.4.60 --port 8888

A finding beyond the sibling card: opencode-go is a first-class provider with our gateway baked in. hindsight_api/config.py:1114 lists it in PROVIDER_DEFAULT_MODELS, and engine/providers/openai_compatible_llm.py:899 sets its default base URL to https://opencode.ai/zen/go/v1 — our exact endpoint, with no custom base_url and no LLM_BASE_URL env var. I verified this path end-to-end, and it is simpler and less fragile than the generic openai + LLM_BASE_URL recipe the sibling card recorded.

Proof it is really running (live probes, today)

$ curl -s http://192.168.4.60:8888/health
{"status":"healthy","database":"connected","db_acquire_ms":0.7,
 "db_pool_waiting":0,"db_pool_in_use":0,"db_pool_max":100,"db_pool_idle":5}

$ ss -ltn | grep 8888
LISTEN 0  2048  192.168.4.60:8888  0.0.0.0:*

$ curl -s http://127.0.0.1:8888/health      # loopback correctly refuses — the bind is explicit
rc=7

A real write, a real LLM extraction, a real embedding, a real recall — over the LAN address:

$ curl -s -X POST http://192.168.4.60:8888/v1/default/banks/probe2/memories \
    -H 'Content-Type: application/json' \
    -d '{"items":[{"content":"Memory provider decision 2026-10-21: LAB runs hindsight
         bare-metal; mem0 is rejected because its protobuf<7.0.0 pin conflicts with
         hindsight-api-slim protobuf>=7.35.1, making the two providers mutually exclusive."}]}'
{"success":true,"bank_id":"probe2","items_count":1,"async":false,
 "usage":{"input_tokens":3018,"output_tokens":204,"total_tokens":3222}}

$ curl -s -X POST http://192.168.4.60:8888/v1/default/banks/probe2/memories/recall \
    -H 'Content-Type: application/json' \
    -d '{"query":"which memory provider was chosen for LAB and why is mem0 excluded?","budget":"low"}'
{"results":[
  {"text":"LAB runs hindsight bare-metal; mem0 is rejected due to protobuf version conflict
   | When: 2026-10-21 | mem0's protobuf<7.0.0 pin conflicts with hindsight-api-slim's
   protobuf>=7.35.1 requirement, making the two providers mutually exclusive",
   "scores":{"final":0.852,"semantic":0.883,"keyword":0.600}},
  ...]}

The recalled text is the fact we stored, with entity extraction (protobuf, mem0, hindsight, hindsight-api-slim all resolved as entities) — not an echo of the query.

The measured cost

The local_external wiring — the Pulse↔LAB shape

hindsight's plugin has three modes: cloud, local_embedded, local_external (hindsight-integrations/hermes/README.md:155,189). local_external is "point the plugin at an existing Hindsight instance you're already running" — a URL and an optional API key, no daemon management. That is exactly our shape: a server on LAB bound to the LAN, with Pulse (192.168.4.34, same /24) and the LAB agents pointed at it.

local_embedded would start the daemon at http://localhost:8888 — loopback, unreachable from Pulse. So local_external is required for the topology, not merely preferred.

The consequence for Docker, stated plainly

Choosing hindsight is what removes Docker from LAB's critical path for memory.[lab-container-runtime-bare-metal-2026-10-21](lab-container-runtime-bare-metal-2026-10-21) recorded that Docker is not installable on LAB as this host is configured — no root, and AppArmor denies the user namespace that rootless containers require. It is one command Young runs, or one SUDO_PASSWORD, and it remains available. But the memory stack does not need it, and hindsight is the only candidate with a Docker-free path: mem0's shared-store path wants Postgres, and its self-host path is a Docker-first compose file.

This is the answer to the card's second question: the provider's install path decides whether Docker is needed on LAB at all, and on hindsight the answer is no.

5. The two constraints that shape the deployment

Embeddings must be local (ONNX) — our gateway has no embedding model

https://opencode.ai/zen/go/v1/models returns 38 chat models and zero embedding models, and our key is rejected by api.openai.com. So a remote embedding endpoint is not available to us at all. ONNX is the only working embeddings path, and it needs no key. That is the second install trap above, and it is why embeddings_provider=onnx is not a preference.

The LAN bind is explicit, and that is a security decision

HINDSIGHT_API_HOST=192.168.4.60 → LISTEN 192.168.4.60:8888, and loopback correctly refuses (rc=7). The bind is explicit, so the server is not on every interface. That is right for a service holding every agent's memory, and it must stay that way: do not bind 0.0.0.0, and do not set AUTH_DISABLED on a LAN-reachable instance.

6. What this decision does NOT settle

This card decides the provider. It does not enable anything, and it must not be read as permission to.

• The read path is the gate — not the provider. Min's note (t_3841e128, [memory-read-path-2026-10-21](memory-read-path-2026-10-21)) landed the same day and is the harder constraint: today memory is provably written and unprovably read. The memory tool is write-only by construction (add/replace/remove — no query), the one conditional read is the provider prefetch path, and it is switched off on every profile. The metric is recalls_with_results — conditional reads returning ≥1 result, per agent, 7-day trailing — and it is unmeasurable today, because describe_recall() emits a UI status line, not a log line. Min's recommendation: do not enable a provider until that counter exists (t_c5c2964b, Sol, running).
• The September prior art still stands. Our cross-agent memory bus accumulated 1,548 rows with zero live readers and died quietly. Adopting a provider does not fix that. The read path is the product. A provider without a named reader repeats that failure with better tooling.
• Nothing is installed as a service. The stack I proved is a foreground process. No systemd unit, no pinned dependency file, no backup of ~/.pg0, no restart-on-boot. Making it a service is the next card, and it should be someone else's.
• The Hermes plugin side is untested. hermes plugins install hindsight + hermes memory setup → memory.provider: hindsight → local_external → api_url: http://192.168.4.60:8888 — the server works; the agent-side wiring has not been exercised. Note the plugin's recall_types now defaults to observation only, and auto_recall is on by default, so the read path would be live the moment the provider is enabled — which is exactly why the counter comes first.
• The trust rule is not yet set. Before the first write in production, the read path needs the rule that treats user-attested facts as evidence and agent-authored content as lower-trust. hindsight preserves the role field that makes this implementable (§2); mem0 does not.

7. Not verified (read as limits, not findings)

• I did not install the Hermes hindsight plugin, and I did not run hermes memory setup. The server is proven; the agent-side wiring is not.
• The retain/recall proof used one bank and a handful of facts. It proves the stack runs and round-trips, and that recall returns the stored fact with entity extraction. It does not prove recall quality, multi-agent attribution, or the speaker-attested grounding that §2 turns on.
• I did not measure cost. Every write calls the opencode-go gateway. Observed: ~3,018 in / ~204 out tokens for a one-fact retain. I did not price a real workload.
• Pulse→LAB was not probed from Pulse's side. The LAN bind is measured from LAB; the far side is Tam's already-closed step 1. The 8889/8888 instances I ran were scratch probes, not services.
• I did not test mem0's plugin on this host. Its rejection rests on source read at main and on PyPI metadata, not on an install.
• I did not re-read hindsight's server-side extraction/consolidation prompts. I read the plugin's retain path. The asymmetry Echo flagged is therefore narrowed but not gone — and it still points the same way.
• Vendor benchmarks are their claims. mem0's 92.5/94.4 and hindsight's LongMemEval numbers are not reproduced here and were not a deciding input. Echo's step 6 — spike real sessions and measure whether recall changed an answer — remains the only honest test.

8. The one-paragraph version

LAB runs hindsight, served bare-metal on 192.168.4.60:8888, and agents reach it in local_external mode; mem0 is rejected. The rejection rests on three measured grounds: mem0's self-host tier is explicitly the reduced one (its own plugin says "rerank is platform-only"), its extraction prompt is built to ingest assistant-authored content as first-class memory ("extract from BOTH roles", "Extract From ALL Speakers") — the exact false-memory mechanism our own study identified — and its anonymous telemetry is on by default. hindsight keeps the speaker as a first-class role field, so the read path can implement speaker-attested grounding itself; mem0's extraction prompt has already dissolved that distinction into prose before anything is stored. The two are mutually exclusive because Hermes allows one external memory provider at a time (kind: exclusive, enforced in agent/memory_manager.py:434) — and separately because mem0ai pins protobuf<7.0.0 while hindsight-api-slim needs protobuf>=7.35.1, which means they cannot both serve one host. The install path decides the Docker question: pip install hindsight-api-slim[embedded-db, local-onnx] with embedded PostgreSQL and ONNX embeddings — no Docker, no root — and it is proven running on LAB today, health, retain and recall, over the LAN address. Nothing is enabled by this note: the read path is the gate, recalls_with_results is the metric, and the counter must exist before any provider is switched on.

🎯 Sol's Read: hindsight, decisively, on shape rather than benchmarks — it preserves the speaker label, so our read path can implement the trust rule ourselves, and mem0's prompt dissolves it before storage. mem0 is rejected on its reduced self-host tier, its both-roles-extraction prompt, and default-on telemetry. The exclusivity claim in the card was mechanically wrong and I corrected it in the record: Hermes' one-external-provider rule is what makes them mutually exclusive; the protobuf conflict is real but binds the server layer — I installed both thin plugin dependency sets into one venv successfully. The install path settles Docker: bare-metal, no root, no container runtime, proven live on 192.168.4.60:8888 today — and opencode-go is a built-in provider with our gateway as its default base URL, which is simpler than the recipe the sibling card recorded. Nothing is enabled. The read path is the gate, not the provider.