Cron & Vault-Sync Inventory Audit — LAB vs Laptop (2026-09-27)

Joint review with Eva for Young. Every claim below is verified by direct command output on the named host, not inferred. Three claims in the incoming inventory are wrong and one live silent failure is running right now.

🔴 Headline: the 30m vault sync is green and pointed at the wrong box

/home/eva/tam_vault_mirror/sync_vault.sh line 9:

RHOST="ubuntu@100.124.146.127"          # OLD prod box
RBASE="$RHOST:/home/ubuntu/.solos/docs/vault/solos/research/tam"

Canonical LAB is lab@192.168.4.60:/home/lab/SolOS/docs/vault/solos/research/tam.

The job reports success every 30 minutes while writing to a legacy host:

2026-09-27 13:38:04 AEST OK pushed=1

Consequence — the canonical vault is behind. Full 30-file sha pull-back, local vs LAB:

Missing from canonical LAB, present on old prod:

• tech/2026-09-27-lab-connectivity-audit.md (local sha 3a3ec36269a6…, mtime 12:34)
• tech/2026-09-27-lab-cutover-phase2-review.md (local sha 73b3a433c9f0…, mtime 10:39)

So the newest two notes — the very ones that document the cutover — are absent from the source of truth. This is the exact failure mode Young called unacceptable: green status, no delivery. It has been running since the 09-27 cutover.

⚠️ Correction 1 — LAB's 4 timers are USER-level, not system

systemctl list-timers (system scope) shows 20 unit files, all stock Ubuntu (sysstat, apt-daily, fstrim, logrotate, fwupd, e2scrub, man-db, motd-news…). Zero SolOS timers in system scope.

All SolOS automation is in systemctl --user list-timers, enabled via loginctl Linger=yes for lab (so it survives logout/reboot). Checking system scope alone reports "nothing here" — which is likely how a wrong picture formed.

lab has no crontab; /etc/cron.d holds only distro files (anacron, e2scrub_all, sysstat, .placeholder). sudo -n is not available to this key, so root/other-user crontabs were not enumerated — residual unknown, low risk.

⚠️ Correction 2 — there is no "snap cache clean" timer

The 4th user timer is launchpadlib-cache-clean — it prunes the Launchpadlib API cache (find /home/lab/.launchpadlib/api.launchpad.net/cache -mtime +30 -delete), 1/day. Unrelated to snap.

A stock snap.firmware-updater.firmware-notifier.timer (hourly) does exist and is the likely source of the mix-up. It is Snap Inc. stock, not fleet automation.

Net effect: LAB genuinely has 3 SolOS timers, not 4. Eva's substantive claim ("no content/publish crons") is ✅ confirmed — all three timers are heartbeat/keypool only.

⚠️ Correction 3 — the central cron registry does not exist

Fleet rule says scheduled jobs live on the central registry (:8436) and never as silent local-only schedules. There is no registry to live on:

• http://192.168.4.60:8436/ → HTTP 000, no listener
• http://192.168.4.60:8430/ → HTTP 000, no listener
• LAB ss -ltn fleet ports: 8300-8307, 8432, 8442, 8443, 8447. No 8436, no 8430.
• Laptop holds 127.0.0.1:8430 only — a loopback listener, no 8436.

So 0 of 16 jobs are centrally registered; every schedule is a local profile job. Confirms the standing note that governance ports are ABSENT on LAB. Whatever gets built needs the registry stood up first, or it will reproduce the same blind spot on LAB.

📋 Correction 4 — laptop cron counts (Eva's list undercounted)

Authoritative source: each profile's cron/jobs.json.

Eva — 14 defined, 11 enabled, 3 disabled (Eva reported 9 active):

Two things Eva's list missed: the Economic Watch weekly job, and the whole 4-job paper/alert cluster (S32, SEMI exit, MINI entry, MINI status). All four of those deliver to origin — in this CLI session that means no message is ever delivered; output is written to cron/output/ and read by nobody. Trading-alert jobs that alert nowhere.

Eve — 1 job: eve-daily-principle, 0 7 * * *, deliver=telegram. Tam — 1 job: 305a8f2d68a3 tam-vault-sync, every 30m, deliver=local.

🔴 Second silent failure: eve-daily-principle has not run in 4 days

Its own incident record, still acked_at=None:

2026-09-27T07:00:57  RuntimeError: [drift_skip:silent] Skipped to prevent
unintended spend: global inference config drifted since this job was created
(provider 'openrouter' -> 'opencode-go'; model
'nvidia/nemotron-3.5-lightning:free' -> 'glm-5.3-flash'), and this job is
unpinned. No inference call was made.

Still skipping today. First seen 2026-09-23 → four missed daily runs. Fix is one command: hermes cron edit 73fd3fff38a6 --provider opencode-go --model glm-5.3-flash. The alert fired once and then nothing — classic silent stop.

🧯 Unclosed incident debt

5 incidents across profiles, all acked_at=None, all unclosed:

• eva bad0e4099b06 + 77f6d5273b4a — alerted, HTTP 400 provider "name" is not supported by this endpoint, Sep 17-19, still firing
• tam 305a8f2d68a3 ×3 — detected/alerted, script exit 1, all Sep 21 (the 30m sync failure burst: 140 FAIL lines that day)
• eve 73fd3fff38a6 ×2 — detected/alerted, drift_skip, Sep 23-27

Sync FAIL density by day, from vault_sync.log: 18 (09-10), 9 (09-15), 22 (09-16), 20 (09-20), 140 (09-21), 25 (09-22), then clean. Stable ~46-52 OK runs/day since 09-23 (≈every 30 min, as expected).

🗺️ LAB vault state (canonical, for whoever builds)

• Path: /home/lab/SolOS/docs/vault/solos/research/tam/
• Git repo, HEAD 267a6f1 "docs: Pulse-laptop LAB transport map (Sol 2026-09-27)"
• Remotes: origin = github.com/youngframeworks/solos-vault.git; prod = ubuntu@161.33.71.67:/home/ubuntu/.solos/docs/vault — 161.33.71.67 is not the host the sync script targets (100.124.146.127). Three hostnames for one fleet across one config. Worth pinning while the builder is in there.
• Untracked in git, needs a commit: solos/research/tam/2026-09-27-lab-access-lane-verification.md, solos/research/tam/tech/2026-09-27-lab-route-confirm.md, plus 3 .obsidian/ plugin dirs that should be gitignored.
• Obsidian Local REST API (27124) is loopback-only on LAB → agent access needs ssh -N -L 27924:127.0.0.1:27124 + HTTPS. Not exercised this pass.

✅ What the builder must fix (ordered)

1. Re-point sync_vault.sh RBASE to lab@192.168.4.60 + canonical path; backfill the 2 missing notes; re-verify all 30 by sha pull-back. Do this first — it is a live gap in the source of truth.
2. Pin eve's 73fd3fff38a6 provider/model to stop the drift-skip.
3. Re-deliver the 4 Eva jobs using deliver=origin onto a gateway-connected platform, or accept they are write-to-nowhere.
4. Stand up the :8436 cron registry (or declare the laptop profile the registry) and migrate all 16 jobs onto it with delivery target + failure signal + read-back, per the cron-discipline rule.
5. Ack/close the 5 stale incidents so the queue means something.
6. Add a 2-minute staleness watchdog on the canonical vault: compare newest mtime in solos/research/tam/ against local SRC and alert on drift. This audit found the drift by hand; nothing automated would have.

🎯 Tam's Read

The sync cron is the urgent item — it has been reporting OK pushed=1 every 30 minutes while writing to a dead prod host, leaving the two newest notes missing from the canonical LAB vault, and LAB's registry port :8436 that all 16 jobs are supposed to live on does not exist, so every schedule on this fleet is currently ungoverned and local-only.

2026-10-21 — retirement verified for 305a8f2d68a3, and the successor's real defect

Follow-up section (same topic, same home). Card t_7114d166 — "Cron — Tam Vault Sync (paused)". Every claim below is from a live probe on this host, 2026-10-21.

✅ 1. The retired job is correctly retired — no action needed

No run has occurred in 16 days. The last_status: error on the card is the pre-pause failure from 2026-10-05, not a current fault — a paused job keeps its last status forever, which is why a paused card keeps reading as "error".

It is also genuinely fail-closed. Executed the profile script directly (~/.hermes/profiles/tam/scripts/sync_vault.sh, exit 78):

BLOCKED: LAB root moved to /home/lab/SolOS/vault; local-to-LAB mapping is not reconciled, no files written.

Exit 78 with zero writes — vault_sync.log and last_commit.txt mtimes unchanged (both 2026-10-20 22:59). So even an accidental un-pause cannot write to the vault until someone sets LAYOUT_MAP_CONFIRMED=1.

✅ 2. No zombie runner — the 10-20 log write was a different script

_logs/vault_sync.log was last written 2026-10-20 22:56, five days after the pause. Not a resumed cron. Two scripts share that log path and are not interchangeable:

The 22:56 lines are pulse/…-prefixed → they came from the prod-leg script run by hand, not from the paused cron. No scheduler anywhere references either script (crontab -l empty, no systemd unit, only ROLLBACK_vault_sync.sh mentions the name). Two scripts sharing one log file is a diagnosis trap worth naming: it makes a manual run look like a resurrected cron.

⚠️ 3. The supersession claim is only half true

The card says "superseded by pulse-lab-sync-30m (4780fda2d4ee)". That job is enabled and scheduled — but it was failing every 30 minutes for 33 consecutive runs (2026-10-20 22:14 → 2026-10-21 13:48, FAIL push(new) on one file). A paused job superseded by a silently-broken successor means the lane was dark twice over.

Root cause, proven — not the file it names. The mkdir phase's ssh consumes the while read loop's stdin, so the loop dies almost immediately and never creates most remote directories:

• loop as written (no -n): 2 iterations of 98 directories
• same loop with ssh -n: 98 of 98
• A/B against a throwaway LAB root: current logic created 2 dirs, ssh -n created 82 (the full allowlisted set in_allow passes)

Sorted, agents/eve/briefs sits at position 12 of 98, so it is never created — first-ever push into any new directory fails permanently. The named file was innocent: direct scp -O returns rc=0 and the pull-back sha256 matches (f270c230d33a…). Fix: ssh -n in the mkdir phase. The file loop needs no change — scp does not consume stdin (verified 460/461 iterations).

Handed to the owning worker on t_1b4ab60b as a comment; I did not edit their script mid-patch. Also flagged there at the time: the manifest had dropped to 13 entries where pre-patch runs logged skipped=407. It has since recovered to 408 (16:46 run) — so that symptom was transient, cause not established, and needs no action. Recorded rather than asserted.

⚠️ The lane went green at 16:46 — that is a FALSE GREEN

By 16:46 4780fda2d4ee reported last_status: ok (pushed=2 skipped=406) and the manifest recovered to 408 entries. This is not evidence of a fix. Verified at 17:0x: the mkdir line is still ssh $SSH_OPTS … with grep -c "ssh -n" = 0.

The ok status is partly an artifact of this audit: while reproducing the bug I ran ssh … "mkdir -pv '$RDIR/agents/eve/briefs'" by hand at 14:55, manually creating the directory the loop could never create. The subsequent pushes succeeded into a directory that would not exist without that test. Recording it here because a green tick bought with a manual fix is the exact "green status, no delivery" failure this note opened with.

Three probe files also remain in the canonical vault at lab:/home/lab/SolOS/vault/Pulse/agents/eve/briefs/ — ZZ_probe_hidden.md, ZZ_hold.md, ZZ_hold2.md (2726 bytes each, 15:26–16:38). None exist locally, and the sync is one-directional so it will never remove them. Not deleted: they look like the owning worker's live probes.

Still required: ssh -n in the mkdir phase, remove the ZZ_* probes, then re-run and confirm a genuinely new directory is created. The current green proves nothing.

Guarded by card t_db52b256 (child of t_1b4ab60b, promotes only when that lane closes), so if the owning worker closes on the false green, the fix is still tracked and cannot be lost.

⚠️ 4. Documentation defects — both fixed in this pass

1. ✅ FIXED 2026-10-21. sync_pulse_lab.sh's header points at sync_vault.sh.RETIRED, and that file did not exist. Created it at ~/.hermes/profiles/tam/scripts/sync_vault.sh.RETIRED (2207 bytes): a documentation marker recording the retirement, the exit-78 fail-closed proof, the successor job, and the two-same-name warning. It restores and replaces no script.
2. ✅ FIXED 2026-10-21. The crons registry note (agents/tam/research-mirror/automation/cron-registry.md) read tam (pulse): (none registered yet) while Tam's profile runs 8 jobs — stale since 2026-09-07. Its tam (pulse) section now carries all 8 jobs, generated from a live cronjob list. SHA-verified on LAB: bce8a31a7932….

🎯 Tam's Read (2026-10-21)

305a8f2d68a3 is honestly retired — paused 16 days, fail-closed at exit 78, no zombie runner — but "paused, superseded" was masking a live outage: its successor had been red every 30 minutes for 33 runs on a one-character ssh -n defect, so the vault lane was dark while both cards read as administratively tidy. The lane's 16:46 green tick changed none of that — the defect is still in the script, and part of that green was bought by a manual mkdir during this audit.

Sources: live probe 2026-10-21 — cronjob list, ~/.hermes/profiles/tam/cron/{jobs.json,executions.db,output/}, direct execution of the retired script, lab@192.168.4.60 (8 ssh/scp probes), loop A/B against /home/lab/_synctest_Pulse (removed after test).