Young's Mind
05 Operations

The ledger was blind, and it was writing into a dead tree

Young's question, 2026-10-21: "can't be right everything is zero?" — about the daily spend-ledger-daily cron reporting total=$0.0000 paid=$0.0000 for both LAB and prod, every day, with alert=OK.

The ledger was blind, and it was writing into a dead tree

Young's question, 2026-10-21: "can't be right everything is zero?" — about the daily spend-ledger-daily cron reporting total=$0.0000 paid=$0.0000 for both LAB and prod, every day, with alert=OK.

He was right, and the zero was not a fact about spending. It was a fact about which client the query could see.

1. What was actually wrong

Three separate defects, found by measuring rather than reading.

D1 — the ledger could not see the fleet (the big one)

spend-ledger.py ran opencode stats --cost --json --days 1. That reads the opencode CLI's session store (~/.local/share/opencode/opencode.db). The brain fleet runs on Hermes, whose usage lives in ~/.hermes/profiles/<name>/state.db — a different database, a different harness.

Measured on LAB, last 24h: 9,031 Hermes API calls across five profiles (mira/min/sol/echo/nova), every one of them invisible to the query.

This is the same defect class as V2's "$0.00000 across 30,721 attempts" — a cost record structurally blind to where the money goes.

D2 — the live cron scripts wrote to a deleted directory

Docs/SolOSV3/ was renamed to Docs/Fleet/ on 2026-10-18. The residue purge fixed the vault copies of my scripts, but the live copies the cron actually executes still pointed at Docs/SolOSV3/...:

spend-ledger.py:23   LEDGER_DIR = ".../Docs/SolOSV3/05-operations"
solos-backup.sh:41   KEEP_LOG=".../Docs/SolOSV3/05-operations/backup-history.log"
solos-backup.sh:101  LEDGER_DST=".../Docs/SolOSV3/05-operations/ledger-backup"
restore-proof.sh:42  KEEP_LOG=".../Docs/SolOSV3/05-operations/restore-proof-history.log"

Every run recreated the dead tree and wrote there. Proof: the ledger's backup-history.log and restore-proof-history.log had been dark since 2026-10-18, while Docs/SolOSV3/ reappeared on disk with a fresh spend-ledger.tsv in it. The ledger was writing to a file nobody reads, and the canonical TSV in Docs/Fleet/ was frozen at 10-17.

D3 — the date column was UTC while the cron ran in AEDT

The 09:00 AEDT run is 22:00 UTC the previous day, so every row was labelled a day early. Harmless for a trend, misleading for "what did we spend today".

2. What the fleet actually spent (measured, not asserted)

Recomputed independently from the stores, then matched by the script to the cent:

Local day (AEDT)Hermesopencode CLIv1 recorded
2026-10-12$7.1829 (opencode-go/glm-5.3-flash)$0.0022$0.0000
2026-10-13$0.0374 (muse-spark-1.3-contributor)$0.0000$0.0000
2026-10-14$0.0000$0.0000$0.0000
2026-10-15$0.9418 (muse-spark-1.3-contributor)$0.0003$0.0000
2026-10-16$0.0000$0.0000$0.0000
2026-10-17$0.0000$0.0000$0.0000
2026-10-18$0.0000$0.0000—
2026-10-19$0.0408 (muse-spark-1.3-contributor)$0.0009—
2026-10-20$0.0000 (8,492 calls, 95.9% cached)$0.0017$0.0000

$8.20 of real paid spend was invisible, and one day ($7.18) would have tripped the $5 alert.

The zeros that remain are real: 10-14, 10-16, 10-17, 10-18 ran on free models (11M–55M tokens/day at 95–98% cache reads). That is the free-first ladder working. The days that read $0.00 were free; the days that read $0.00 while money moved were not.

3. The fix — spend-ledger.py v2

The design rule is one sentence: price from the source that can account for it, and never let a gap read as a zero.

RuleWhyEvidence
Read both harnessesThe fleet IS Hermes; v1 saw one store9,031 calls/day invisible
The harness's own dollar figure winsThe CLI records what it was charged — a measurement$0.0144 charged for opencode-go/space-bunny-free across 597 sessions, with no catalogue row
The catalogue fills what Hermes cannot measureHermes records cost_status='unknown' and $0.00 on every row111 rows in mira's store, every one $0.00
Exact (provider, model) pair onlyA cross-provider fallback read 33% under the truthGO's retired deepseek-v4-flash priced at another vendor's rate
No rate AND no measurement = unknown, never $0.00The V2 defect in a new costumecost_state column + call count
A -free name is not proof of a free tierReading a name as a price is how a ledger liesopencode-go/space-bunny-free: no catalogue row, real charges
Non-overlapping local days, AEDTv1's rows were labelled a day earlyv1's UTC column vs the 09:00 AEDT cron
cache_ratio = cr/(in+cr)v1 divided by input alone — no ceiling, read as nonsense2.75 ratios in v1's own history
LAB and prod never summedStanding ruleseparate rows, no combined total
GO allowance windows from the provider's own /usageThe binding constraint is the window, not the billrolling=0% weekly=0% monthly=17%

Validation. The pricing engine was checked against the CLI's own accounting on a 30-day window: all 16 catalogue-priced pairs agree within 2%, and every day above was re-derived independently from the stores and matches to the cent.

13 regression tests pin the rules (scripts/test-spend-ledger.py, all pass) — one per defect above, so the ledger cannot quietly go back to lying.

4. The other thing this exposed

opencode-go/space-bunny-free — a model whose name says free — has no catalogue row, and the CLI charged for it 597 times. Had v2 treated the name as proof of $0, it would have repeated the original defect exactly.

So v2 prices the pair, not the name. Where the harness recorded a charge, the money is in paid_cost and the missing rate is flagged as a rate-sheet gap, not a measured spend. Where nothing measured it, it is unknown with its call count, and it never enters a total.

The GO window today: rolling 0%, weekly 0%, monthly 17% — the $10 allowance is barely touched, because the ladder is free-first and the free tier is working.

5. The durable bits

  • The nightly backup now commits the spend ledger and fails loud if the ledger has not been written for 48h. Verified by simulation: a stale ledger is indistinguishable from a fleet that stopped spending, which is the same blindness that hid this for a week.
  • The live cron scripts were synced from the vault copies — zero SolOSV3 references remain in any of my scripts. The residue purge had fixed the vault, not the live copies; that is the half that was still broken.
  • v1's 11-column TSV was rotated aside as spend-ledger-v1-legacy.tsv rather than being appended to with v2 columns.

6. What I did not do

  • Prod is still partly unmeasured. Prod has no Hermes store and no GO key, so its window state is unavailable and its Hermes column reads NOT MEASURED. Prod's opencode store IS read, and its last activity was 2026-10-15. Getting prod onto a real measurement path needs a decision about whether prod should run agents at all — a call for Young/Min, not for me to assume.
  • I did not change the free-first ladder or the $5 threshold. Both are sound; the threshold now actually works because the numbers are real.
  • I did not touch the V3 model-cost.jsonl ledger (it has never held a row). That is Sol's lane and a separate build.
  • I did not register this on the board — Min registers, and there is a card to open (see below).

Evidence: commit fea6d16 (vault) + dba4dd3 (ledger TSV + script). Ledger: Docs/Fleet/05-operations/spend-ledger.tsv (28 rows, 10-12→10-20).

7. Ruling brief — live verification 2026-10-22 (Mira, for Young/Min)

Card t_6cd15ebd asks whether prod is SUPPOSED to spend. Verified by live probe, not by reading config:

  1. Prod has no Hermes at all. No ~/.hermes, no state.db anywhere on the host. The Hermes column is structurally unmeasurable — confirmed, not assumed.
  2. Prod's opencode store IS measured and IS honest. 51 sessions, lifetime $3.7399; last model activity 2026-10-15. Nothing has spent on prod since the 10-18 decommission. Its $0.0000 rows are a true idle, not a gap.
  3. LAB and prod share ONE GO account. Key fingerprints identical on both hosts (SHA-256:12 d7fcd6e1bf77), and live /usage probes from both hosts return the IDENTICAL window (rolling 1%, weekly 0%, monthly 17%, same resetsAt). "No GO key on prod" is a host-placement gap, not a credential gap — the ledger can report the shared window for prod using the LAB key today, no new secret needed.
  4. Prod today is a serve host that happens to host an idle worker: docs.youngle.au + pulse-site.service live, v3-heart-prod worker alive since 10-12 but its last task transitions were 10-12. Consistent with both the topology doc (prod = serve + Heart) and the census (opencode fully decommissioned 10-18).
  5. Stale ALERT flags on 10-12→10-18 rows are history, not signal. A --dry-run re-run of those dates under current code writes clean rows (10-13: zero alerts; 10-12: only the genuine $7.18 LAB alert). The TSV flags predate the current alert semantics. One-command fix if Young wants history cleaned: for d in 2026-10-12 ... 2026-10-18; do spend-ledger.py --date $d; done (upsert by date/host/source, commit 4a644bb — cannot double-count).

Recommendation (unchanged): (a)-with-facts. Keep prod's opencode row (it exists and is measured), keep Hermes as NOT MEASURED with the structural reason, and report prod's window from the SHARED account (LAB key) instead of unavailable — or drop the prod column per (c) if prod is docs-only by policy. Either way the remaining call is prod's ROLE, which is Young/Min's, not Mira's. No code changed for this brief.

Copyright © 2026