Young's Mind
Tech

Cron & Vault-Sync Inventory Audit — LAB vs Laptop (2026-09-27)

Joint review with Eva for Young. Every claim below is verified by direct command output on the named host, not inferred. Three claims in the incoming inventory are wrong and one live silent failure is running right now.

Cron & Vault-Sync Inventory Audit — LAB vs Laptop (2026-09-27)

Joint review with Eva for Young. Every claim below is verified by direct command output on the named host, not inferred. Three claims in the incoming inventory are wrong and one live silent failure is running right now.

🔴 Headline: the 30m vault sync is green and pointed at the wrong box

/home/eva/tam_vault_mirror/sync_vault.sh line 9:

RHOST="ubuntu@100.124.146.127"          # OLD prod box
RBASE="$RHOST:/home/ubuntu/.solos/docs/vault/solos/research/tam"

Canonical LAB is lab@192.168.4.60:/home/lab/SolOS/docs/vault/solos/research/tam.

The job reports success every 30 minutes while writing to a legacy host:

2026-09-27 13:38:04 AEST OK pushed=1

Consequence — the canonical vault is behind. Full 30-file sha pull-back, local vs LAB:

ResultCount
Match on LAB28
Missing on LAB2

Missing from canonical LAB, present on old prod:

  • tech/2026-09-27-lab-connectivity-audit.md (local sha 3a3ec36269a6…, mtime 12:34)
  • tech/2026-09-27-lab-cutover-phase2-review.md (local sha 73b3a433c9f0…, mtime 10:39)

So the newest two notes — the very ones that document the cutover — are absent from the source of truth. This is the exact failure mode Young called unacceptable: green status, no delivery. It has been running since the 09-27 cutover.

⚠️ Correction 1 — LAB's 4 timers are USER-level, not system

systemctl list-timers (system scope) shows 20 unit files, all stock Ubuntu (sysstat, apt-daily, fstrim, logrotate, fwupd, e2scrub, man-db, motd-news…). Zero SolOS timers in system scope.

All SolOS automation is in systemctl --user list-timers, enabled via loginctl Linger=yes for lab (so it survives logout/reboot). Checking system scope alone reports "nothing here" — which is likely how a wrong picture formed.

User timerIntervalScriptResult
lab-eva-heartbeat.timer5 min (OnBootSec=2min)runtime/bin/eva-heartbeat.shsuccess, NRestarts=0
lab-eve-heartbeat.timer5 min (Persistent)runtime/bin/eve-heartbeat.shsuccess, NRestarts=0
lab-opencode-keypool.timer6 h (Persistent)runtime/bin/opencode-keypool-refresh.shsuccess, never re-run since boot
launchpadlib-cache-clean.timer1 dinline find ~/.launchpadlib/… -mtime +30 -deletesuccess

lab has no crontab; /etc/cron.d holds only distro files (anacron, e2scrub_all, sysstat, .placeholder). sudo -n is not available to this key, so root/other-user crontabs were not enumerated — residual unknown, low risk.

⚠️ Correction 2 — there is no "snap cache clean" timer

The 4th user timer is launchpadlib-cache-clean — it prunes the Launchpadlib API cache (find /home/lab/.launchpadlib/api.launchpad.net/cache -mtime +30 -delete), 1/day. Unrelated to snap.

A stock snap.firmware-updater.firmware-notifier.timer (hourly) does exist and is the likely source of the mix-up. It is Snap Inc. stock, not fleet automation.

Net effect: LAB genuinely has 3 SolOS timers, not 4. Eva's substantive claim ("no content/publish crons") is ✅ confirmed — all three timers are heartbeat/keypool only.

⚠️ Correction 3 — the central cron registry does not exist

Fleet rule says scheduled jobs live on the central registry (:8436) and never as silent local-only schedules. There is no registry to live on:

  • http://192.168.4.60:8436/ → HTTP 000, no listener
  • http://192.168.4.60:8430/ → HTTP 000, no listener
  • LAB ss -ltn fleet ports: 8300-8307, 8432, 8442, 8443, 8447. No 8436, no 8430.
  • Laptop holds 127.0.0.1:8430 only — a loopback listener, no 8436.

So 0 of 16 jobs are centrally registered; every schedule is a local profile job. Confirms the standing note that governance ports are ABSENT on LAB. Whatever gets built needs the registry stood up first, or it will reproduce the same blind spot on LAB.

📋 Correction 4 — laptop cron counts (Eva's list undercounted)

Authoritative source: each profile's cron/jobs.json.

Eva — 14 defined, 11 enabled, 3 disabled (Eva reported 9 active):

JobScheduleEnabledDeliver
Market Watch (AU→HK→US)0 8,18 * * *✅telegram:8302448891
Daily Market Briefing0 8 * * *✅telegram
Economic Watch — Rates & Housing0 10 * * 1✅telegram
Evening Report — Scan + MA5030 18 * * 1-5✅telegram
ASX200 Scan30 17 * * 1-5✅telegram
Content Watchdog (Pulse)0 10,16 * * 1-5✅telegram
Portfolio Tracker DM0 8 * * 1-5✅telegram
S32 Grid Bot (PAPER)35 17 * * 1-5✅origin
SEMI Exit Alert ($40+)every 15m✅origin
MINI Entry Alerts (Paper)every 15m✅origin
MINI Desk Daily Status15 8 * * 1-5✅origin
Watchlist — SEMI0 9 * * 1-5⛔ disabledtelegram
MA50 Alerts (ASX)30 18 * * 1-5⛔ disabledtelegram
BUB Price Alert ($0.105)every 15m⛔ disabledorigin

Two things Eva's list missed: the Economic Watch weekly job, and the whole 4-job paper/alert cluster (S32, SEMI exit, MINI entry, MINI status). All four of those deliver to origin — in this CLI session that means no message is ever delivered; output is written to cron/output/ and read by nobody. Trading-alert jobs that alert nowhere.

Eve — 1 job: eve-daily-principle, 0 7 * * *, deliver=telegram. Tam — 1 job: 305a8f2d68a3 tam-vault-sync, every 30m, deliver=local.

🔴 Second silent failure: eve-daily-principle has not run in 4 days

Its own incident record, still acked_at=None:

2026-09-27T07:00:57  RuntimeError: [drift_skip:silent] Skipped to prevent
unintended spend: global inference config drifted since this job was created
(provider 'openrouter' -> 'opencode-go'; model
'nvidia/nemotron-3.5-lightning:free' -> 'glm-5.3-flash'), and this job is
unpinned. No inference call was made.

Still skipping today. First seen 2026-09-23 → four missed daily runs. Fix is one command: hermes cron edit 73fd3fff38a6 --provider opencode-go --model glm-5.3-flash. The alert fired once and then nothing — classic silent stop.

🧯 Unclosed incident debt

5 incidents across profiles, all acked_at=None, all unclosed:

  • eva bad0e4099b06 + 77f6d5273b4a — alerted, HTTP 400 provider "name" is not supported by this endpoint, Sep 17-19, still firing
  • tam 305a8f2d68a3 ×3 — detected/alerted, script exit 1, all Sep 21 (the 30m sync failure burst: 140 FAIL lines that day)
  • eve 73fd3fff38a6 ×2 — detected/alerted, drift_skip, Sep 23-27

Sync FAIL density by day, from vault_sync.log: 18 (09-10), 9 (09-15), 22 (09-16), 20 (09-20), 140 (09-21), 25 (09-22), then clean. Stable ~46-52 OK runs/day since 09-23 (≈every 30 min, as expected).

🗺️ LAB vault state (canonical, for whoever builds)

  • Path: /home/lab/SolOS/docs/vault/solos/research/tam/
  • Git repo, HEAD 267a6f1 "docs: Pulse-laptop <-> LAB transport map (Sol 2026-09-27)"
  • Remotes: origin = github.com/youngframeworks/solos-vault.git; prod = ubuntu@161.33.71.67:/home/ubuntu/.solos/docs/vault — 161.33.71.67 is not the host the sync script targets (100.124.146.127). Three hostnames for one fleet across one config. Worth pinning while the builder is in there.
  • Untracked in git, needs a commit: solos/research/tam/2026-09-27-lab-access-lane-verification.md, solos/research/tam/tech/2026-09-27-lab-route-confirm.md, plus 3 .obsidian/ plugin dirs that should be gitignored.
  • Obsidian Local REST API (27124) is loopback-only on LAB → agent access needs ssh -N -L 27924:127.0.0.1:27124 + HTTPS. Not exercised this pass.

✅ What the builder must fix (ordered)

  1. Re-point sync_vault.sh RBASE to lab@192.168.4.60 + canonical path; backfill the 2 missing notes; re-verify all 30 by sha pull-back. Do this first — it is a live gap in the source of truth.
  2. Pin eve's 73fd3fff38a6 provider/model to stop the drift-skip.
  3. Re-deliver the 4 Eva jobs using deliver=origin onto a gateway-connected platform, or accept they are write-to-nowhere.
  4. Stand up the :8436 cron registry (or declare the laptop profile the registry) and migrate all 16 jobs onto it with delivery target + failure signal + read-back, per the cron-discipline rule.
  5. Ack/close the 5 stale incidents so the queue means something.
  6. Add a 2-minute staleness watchdog on the canonical vault: compare newest mtime in solos/research/tam/ against local SRC and alert on drift. This audit found the drift by hand; nothing automated would have.

🎯 Tam's Read

The sync cron is the urgent item — it has been reporting OK pushed=1 every 30 minutes while writing to a dead prod host, leaving the two newest notes missing from the canonical LAB vault, and LAB's registry port :8436 that all 16 jobs are supposed to live on does not exist, so every schedule on this fleet is currently ungoverned and local-only.

Copyright © 2026