The ledger was blind, and it was writing into a dead tree
The ledger was blind, and it was writing into a dead tree
Young's question, 2026-10-21: "can't be right everything is zero?" — about the
daily spend-ledger-daily cron reporting total=$0.0000 paid=$0.0000 for both
LAB and prod, every day, with alert=OK.
He was right, and the zero was not a fact about spending. It was a fact about which client the query could see.
1. What was actually wrong
Three separate defects, found by measuring rather than reading.
D1 — the ledger could not see the fleet (the big one)
spend-ledger.py ran opencode stats --cost --json --days 1. That reads the
opencode CLI's session store (~/.local/share/opencode/opencode.db). The
brain fleet runs on Hermes, whose usage lives in
~/.hermes/profiles/<name>/state.db — a different database, a different
harness.
Measured on LAB, last 24h: 9,031 Hermes API calls across five profiles (mira/min/sol/echo/nova), every one of them invisible to the query.
This is the same defect class as V2's "$0.00000 across 30,721 attempts" — a cost record structurally blind to where the money goes.
D2 — the live cron scripts wrote to a deleted directory
Docs/SolOSV3/ was renamed to Docs/Fleet/ on 2026-10-18. The residue purge
fixed the vault copies of my scripts, but the live copies the cron
actually executes still pointed at Docs/SolOSV3/...:
spend-ledger.py:23 LEDGER_DIR = ".../Docs/SolOSV3/05-operations"
solos-backup.sh:41 KEEP_LOG=".../Docs/SolOSV3/05-operations/backup-history.log"
solos-backup.sh:101 LEDGER_DST=".../Docs/SolOSV3/05-operations/ledger-backup"
restore-proof.sh:42 KEEP_LOG=".../Docs/SolOSV3/05-operations/restore-proof-history.log"
Every run recreated the dead tree and wrote there. Proof: the ledger's
backup-history.log and restore-proof-history.log had been dark since
2026-10-18, while Docs/SolOSV3/ reappeared on disk with a fresh
spend-ledger.tsv in it. The ledger was writing to a file nobody reads, and
the canonical TSV in Docs/Fleet/ was frozen at 10-17.
D3 — the date column was UTC while the cron ran in AEDT
The 09:00 AEDT run is 22:00 UTC the previous day, so every row was labelled a day early. Harmless for a trend, misleading for "what did we spend today".
2. What the fleet actually spent (measured, not asserted)
Recomputed independently from the stores, then matched by the script to the cent:
| Local day (AEDT) | Hermes | opencode CLI | v1 recorded |
|---|---|---|---|
| 2026-10-12 | $7.1829 (opencode-go/glm-5.3-flash) | $0.0022 | $0.0000 |
| 2026-10-13 | $0.0374 (muse-spark-1.3-contributor) | $0.0000 | $0.0000 |
| 2026-10-14 | $0.0000 | $0.0000 | $0.0000 |
| 2026-10-15 | $0.9418 (muse-spark-1.3-contributor) | $0.0003 | $0.0000 |
| 2026-10-16 | $0.0000 | $0.0000 | $0.0000 |
| 2026-10-17 | $0.0000 | $0.0000 | $0.0000 |
| 2026-10-18 | $0.0000 | $0.0000 | — |
| 2026-10-19 | $0.0408 (muse-spark-1.3-contributor) | $0.0009 | — |
| 2026-10-20 | $0.0000 (8,492 calls, 95.9% cached) | $0.0017 | $0.0000 |
$8.20 of real paid spend was invisible, and one day ($7.18) would have tripped the $5 alert.
The zeros that remain are real: 10-14, 10-16, 10-17, 10-18 ran on free models (11M–55M tokens/day at 95–98% cache reads). That is the free-first ladder working. The days that read $0.00 were free; the days that read $0.00 while money moved were not.
3. The fix — spend-ledger.py v2
The design rule is one sentence: price from the source that can account for it, and never let a gap read as a zero.
| Rule | Why | Evidence |
|---|---|---|
| Read both harnesses | The fleet IS Hermes; v1 saw one store | 9,031 calls/day invisible |
| The harness's own dollar figure wins | The CLI records what it was charged — a measurement | $0.0144 charged for opencode-go/space-bunny-free across 597 sessions, with no catalogue row |
| The catalogue fills what Hermes cannot measure | Hermes records cost_status='unknown' and $0.00 on every row | 111 rows in mira's store, every one $0.00 |
Exact (provider, model) pair only | A cross-provider fallback read 33% under the truth | GO's retired deepseek-v4-flash priced at another vendor's rate |
No rate AND no measurement = unknown, never $0.00 | The V2 defect in a new costume | cost_state column + call count |
A -free name is not proof of a free tier | Reading a name as a price is how a ledger lies | opencode-go/space-bunny-free: no catalogue row, real charges |
| Non-overlapping local days, AEDT | v1's rows were labelled a day early | v1's UTC column vs the 09:00 AEDT cron |
cache_ratio = cr/(in+cr) | v1 divided by input alone — no ceiling, read as nonsense | 2.75 ratios in v1's own history |
| LAB and prod never summed | Standing rule | separate rows, no combined total |
GO allowance windows from the provider's own /usage | The binding constraint is the window, not the bill | rolling=0% weekly=0% monthly=17% |
Validation. The pricing engine was checked against the CLI's own accounting on a 30-day window: all 16 catalogue-priced pairs agree within 2%, and every day above was re-derived independently from the stores and matches to the cent.
13 regression tests pin the rules (scripts/test-spend-ledger.py, all
pass) — one per defect above, so the ledger cannot quietly go back to lying.
4. The other thing this exposed
opencode-go/space-bunny-free — a model whose name says free — has no
catalogue row, and the CLI charged for it 597 times. Had v2 treated the
name as proof of $0, it would have repeated the original defect exactly.
So v2 prices the pair, not the name. Where the harness recorded a charge,
the money is in paid_cost and the missing rate is flagged as a rate-sheet
gap, not a measured spend. Where nothing measured it, it is unknown with
its call count, and it never enters a total.
The GO window today: rolling 0%, weekly 0%, monthly 17% — the $10 allowance is barely touched, because the ladder is free-first and the free tier is working.
5. The durable bits
- The nightly backup now commits the spend ledger and fails loud if the ledger has not been written for 48h. Verified by simulation: a stale ledger is indistinguishable from a fleet that stopped spending, which is the same blindness that hid this for a week.
- The live cron scripts were synced from the vault copies — zero
SolOSV3references remain in any of my scripts. The residue purge had fixed the vault, not the live copies; that is the half that was still broken. - v1's 11-column TSV was rotated aside as
spend-ledger-v1-legacy.tsvrather than being appended to with v2 columns.
6. What I did not do
- Prod is still partly unmeasured. Prod has no Hermes store and no GO
key, so its window state is
unavailableand its Hermes column readsNOT MEASURED. Prod's opencode store IS read, and its last activity was 2026-10-15. Getting prod onto a real measurement path needs a decision about whether prod should run agents at all — a call for Young/Min, not for me to assume. - I did not change the free-first ladder or the $5 threshold. Both are sound; the threshold now actually works because the numbers are real.
- I did not touch the V3
model-cost.jsonlledger (it has never held a row). That is Sol's lane and a separate build. - I did not register this on the board — Min registers, and there is a card to open (see below).
Evidence: commit fea6d16 (vault) + dba4dd3 (ledger TSV + script).
Ledger: Docs/Fleet/05-operations/spend-ledger.tsv (28 rows, 10-12→10-20).
7. Ruling brief — live verification 2026-10-22 (Mira, for Young/Min)
Card t_6cd15ebd asks whether prod is SUPPOSED to spend. Verified by live
probe, not by reading config:
- Prod has no Hermes at all. No
~/.hermes, nostate.dbanywhere on the host. The Hermes column is structurally unmeasurable — confirmed, not assumed. - Prod's opencode store IS measured and IS honest. 51 sessions,
lifetime $3.7399; last model activity 2026-10-15. Nothing has spent
on prod since the 10-18 decommission. Its
$0.0000rows are a true idle, not a gap. - LAB and prod share ONE GO account. Key fingerprints identical on
both hosts (SHA-256:12
d7fcd6e1bf77), and live/usageprobes from both hosts return the IDENTICAL window (rolling 1%, weekly 0%, monthly 17%, sameresetsAt). "No GO key on prod" is a host-placement gap, not a credential gap — the ledger can report the shared window for prod using the LAB key today, no new secret needed. - Prod today is a serve host that happens to host an idle worker:
docs.youngle.au + pulse-site.service live,
v3-heart-prodworker alive since 10-12 but its last task transitions were 10-12. Consistent with both the topology doc (prod = serve + Heart) and the census (opencode fully decommissioned 10-18). - Stale ALERT flags on 10-12→10-18 rows are history, not signal. A
--dry-runre-run of those dates under current code writes clean rows (10-13: zero alerts; 10-12: only the genuine $7.18 LAB alert). The TSV flags predate the current alert semantics. One-command fix if Young wants history cleaned:for d in 2026-10-12 ... 2026-10-18; do spend-ledger.py --date $d; done(upsert by date/host/source, commit4a644bb— cannot double-count).
Recommendation (unchanged): (a)-with-facts. Keep prod's opencode row
(it exists and is measured), keep Hermes as NOT MEASURED with the
structural reason, and report prod's window from the SHARED account
(LAB key) instead of unavailable — or drop the prod column per (c) if
prod is docs-only by policy. Either way the remaining call is prod's
ROLE, which is Young/Min's, not Mira's. No code changed for this brief.

