Stage 2 completion — health gate, LIVE marker, soak on a real tenant (2026-10-21, Sol, t_c1cb1012)

Card: t_c1cb1012 — Stage 2 completion: prove the health gate + LIVE marker + soak on test-tenant Split from: t_3f8c9735 (criteria 1, 2, 4, 5 already proven there) Host: prod 161.33.71.67 (aarch64, Docker 29.7.2, Compose 5.4.0) Script: /home/ubuntu/.solos-infra/onboard_tenant.sh — final sha256 b7af1f2d283848f4addb8d2127cdddd337522f67e5498bb11d6afeb9b9e56d18 (36,974 B) Status: ✅ all five acceptance criteria proven by live probe. One real defect was found and fixed in the process.

1. Headline

The gate now passes on a real tenant and fails on a broken one — both proven live, both with the shipped script. Proving the FAIL path is what found the defect: the gate did not report a verdict at all when a service was dead. It aborted silently. Details in §4.

2. The successful run (AC1, AC2)

Tenant finalproof (finalproof.invalid), onboarded with the final script, SOAK_SECONDS=600. Log: /tmp/finalproof-090732.log on prod.

  predeploy verified: medusa-migrate exited 0, 148 tables in medusa-finalproof
  admin user created and verified in the tenant DB (admin@finalproof.invalid)
  nginx vhost written → /etc/nginx/sites-available/tenant-finalproof.conf
  running health gate...
  /health (medusa-server): 200 ✓
  /eve/v1/health (eve-agent): 200 ✓
  /app (medusa admin): 200 ✓
  medusa-server: healthy ✓ (restarts=0)
  medusa-worker: healthy ✓ (restarts=0)
  storefront: healthy ✓ (restarts=0)
  eve-agent: healthy ✓ (restarts=0)
  postgres: healthy ✓ (restarts=0)
  redis: healthy ✓ (restarts=0)
  orphan containers: 0 ✓
  health gate PASSED
  soaking for 600s (restart count must stay 0)...
  soak start restarts: medusa-server=0 medusa-worker=0 storefront=0 eve-agent=0 postgres=0 redis=0
  soak end restarts:   medusa-server=0 medusa-worker=0 storefront=0 eve-agent=0 postgres=0 redis=0
  soak PASSED: 600s elapsed, 0 restarts across 6 services ✓
  ✅ Tenant finalproof onboarded successfully and marked LIVE

The soak was observed, not assumed

Start and end counts are both recorded by the script, and both are all-zero across all six services. Independently confirmed after the run with a separate docker inspect (not the script's own read):

cib-finalproof-eve-agent-1     restarts=0 health=healthy status=running
cib-finalproof-medusa-migrate-1 restarts=0 health=n/a     status=exited   (one-shot, by design)
cib-finalproof-medusa-server-1 restarts=0 health=healthy status=running
cib-finalproof-medusa-worker-1 restarts=0 health=healthy status=running
cib-finalproof-postgres-1      restarts=0 health=healthy status=running
cib-finalproof-redis-1         restarts=0 health=healthy status=running
cib-finalproof-storefront-1    restarts=0 health=healthy status=running

The LIVE marker (AC1)

/home/ubuntu/.solos-infra/tenants/finalproof/LIVE, mode 600:

slug=finalproof
domain=finalproof.invalid
project=cib-finalproof
network=tenant-finalproof
compose=/home/ubuntu/.solos-infra/docker-compose.finalproof.yml
secrets=/home/ubuntu/.solos-infra/tenants/finalproof/secrets.env
admin_email=admin@finalproof.invalid
cert=false
onboarded_at=2026-10-21T09:19:41Z

Every field the card names is present and matches the tenant. cert=false is correct, not a defect: finalproof.invalid does not resolve, so no certificate can be issued (§5).

The card's named target, test-tenant

tenants/test-tenant/LIVE exists and was re-proved end-to-end with the final script through the documented ONBOARD_FORCE=1 override (log: /tmp/force-test-tenant-092245.log on prod):

  health gate PASSED
  soaking for 600s (restart count must stay 0)...
  soak start restarts: medusa-server=0 medusa-worker=0 storefront=0 eve-agent=0 postgres=0 redis=0
  soak end restarts:   medusa-server=0 medusa-worker=0 storefront=0 eve-agent=0 postgres=0 redis=0
  soak PASSED: 600s elapsed, 0 restarts across 6 services ✓
  ✅ Tenant test-tenant onboarded successfully and marked LIVE

Marker rewritten at 2026-10-21T09:33:34Z, contents correct, all six services healthy with restarts=0. So test-tenant satisfies AC1 and AC2 with the shipped script, and the refusal in §3 applies to it.

ONBOARD_FORCE=1 is the documented escape hatch for updating a LIVE tenant in place. It is explicit, and it is not the default — the default is refusal.

3. The refusal (AC3)

Step 1 refuses to re-onboard a tenant that is marked LIVE. Proven on both LIVE tenants, with the final script:

=== Onboarding tenant: finalproof (finalproof.invalid) ===
ERROR: tenant finalproof is already marked LIVE (.../tenants/finalproof/LIVE).
       Refusing to re-onboard a LIVE tenant — the LIVE marker is the
       record that this tenant passed the health gate, and re-onboarding
       it in place would rewrite its compose/vhost/upstreams silently.
       To update a live tenant deliberately, re-run with ONBOARD_FORCE=1
       ...
EXIT=1

The marker was not touched by the refused run (mtime unchanged), and the tenant's containers were not restarted. The override is explicit and deliberate (ONBOARD_FORCE=1), not a default.

4. The regression (AC4) — and the defect it found

The defect: the gate aborted instead of reporting a verdict

The first fault-injection run (ONBOARD_FAULT_PRE=medusa-server:kill) died with no output at all after pre-gate: injecting fault → medusa-server (kill). No FAIL message. No verdict. The script simply stopped.

Root cause — set -euo pipefail meeting a pipe:

probe_http() {
    docker exec "$1" node -e "..." 2>/dev/null | tail -1
}

When the container is dead, docker exec exits 1. Under pipefail the pipeline inherits that non-zero status, and under set -e the whole script aborts — silently, before any verdict is printed. Reproduced in isolation on prod:

old -> exit=1 out=[|]          # aborted, no output, no verdict
new -> exit=0 out=[REACHED_NEXT code=[000]]   # returns 000, script continues

The LIVE marker was still not written — but only by accident, because the script died before step 10. A gate that aborts instead of reporting FAIL is not a control: it cannot distinguish "the tenant is broken" from "my probe broke", and the operator gets a run that stops for no stated reason. The card's own words apply: a control that cannot fail is not a control — and neither is one that cannot report.

The fix

probe_http now runs in a subshell with set +e, always exits 0, and returns a status code (000 when there is no response). The caller compares the code and decides PASS/FAIL. A probe must never be able to kill the script that reports its verdict.

The same silent-abort class was fixed in two more places (TABLES and ADMIN_COUNT, where a dead Postgres would have skipped the very diagnostic written to explain the failure).

Proof the gate now fails correctly

Variant A — fault before the gate (ONBOARD_FAULT_PRE=medusa-server:kill, tenant failproof2, log /tmp/failproof2-085448.log):

  running health gate...
  pre-gate: injecting fault → medusa-server (kill)
  /health (medusa-server): 000 ✗
  /eve/v1/health (eve-agent): 200 ✓
  /app (medusa admin): 000 ✗
  medusa-server: NOT STARTED ✗
  medusa-worker: healthy ✓ (restarts=0)
  storefront: healthy ✓ (restarts=0)
  eve-agent: healthy ✓ (restarts=0)
  postgres: healthy ✓ (restarts=0)
  redis: healthy ✓ (restarts=0)
  orphan containers: 0 ✓
  ❌ Health gate failed — tenant NOT marked live

Variant B — fault during the soak (ONBOARD_FAULT=medusa-worker:kill at t+32 s, tenant failproof3, log /tmp/failproof3-090007.log):

  health gate PASSED
  soaking for 120s (restart count must stay 0)...
  soak start restarts: medusa-server=0 medusa-worker=0 storefront=0 eve-agent=0 postgres=0 redis=0
  soak(t+32s): injecting fault → medusa-worker (kill)
  soak: medusa-worker is NOT RUNNING after 32s ✗
  soak end restarts:   medusa-server=0 medusa-worker=gone storefront=0 eve-agent=0 postgres=0 redis=0
  soak FAILED after 120s window — tenant NOT marked live
  ❌ Health gate did NOT pass — tenant NOT marked live
     (health gate + soak both must pass; the LIVE marker is only written after both)

No LIVE marker was written by any failing run:

failproof:  NO-LIVE
failproof2: NO-LIVE
failproof3: NO-LIVE
finalproof: /home/ubuntu/.solos-infra/tenants/finalproof/LIVE

A second, smaller fix found by the same runs

docker inspect "" (empty container id — which is what a dead service resolves to) prints a multi-line error to stdout and exits 0. That error text was being spliced into the soak report line, which printed medusa-worker= + newline + ? instead of medusa-worker=?. Fixed with a restart_count helper that never inspects an empty id. The soak lines in §2 are now clean single lines.

A false claim in the script's own comments, corrected

The fault-injection header claimed docker kill would be restarted by the daemon (restart: unless-stopped) and that RestartCount would become 1. Measured live: it does not. A manual docker kill counts as a manual stop, so the daemon leaves it EXITED and RestartCount stays 0. The gate catches it as not running / unhealthy, not as a restart. The comment was wrong because it had never been measured — it now states the measured behaviour.

5. What was SKIPPED, and why

• certbot / TLS — SKIPPED on every tenant. test.youngle.au, finalproof.invalid and failproof*.invalid do not resolve to 161.33.71.67, so no certificate can be issued. The script detects this, writes the vhost, does not enable it, and says so. cert=false in the LIVE marker is the honest record of that. Adding the DNS A record is a change to a domain Young/Young Le controls — not mine to make.
• nginx vhost enabled — NOT enabled (same reason). Verified: no tenant vhost is in sites-enabled, and nginx -t passes.
• public hostname health check — NOT performed. It needs the DNS record that does not exist.
• admin login through the browser — NOT performed. The admin user is asserted to exist in the tenant DB and /app is asserted to return 200. A real browser login needs a reachable admin hostname; the Medusa admin sits behind Authelia forward-auth in the vhost, and there is no enabled vhost.
• backup/rollback (stage 2.3 / 2.4) — NOT part of this card.

6. Files changed on prod

The Medusa image was not rebuilt; no defect was in the image.

7. Left undone / for someone else

• Throwaway proof tenants are still up on prod — cib-failproof2-*, cib-failproof3-* and cib-finalproof-* (plus their networks tenant-failproof2/3, tenant-finalproof and the sites-available/tenant-failproof*.conf vhost files). docker compose down was blocked by this session's approval gate (container-lifecycle commands need approvals.single_query_mode: approve, or an interactive session). They are harmless — isolated bridge networks, no host ports, no vhost enabled — but they should be removed. Command: docker compose -f docker-compose.finalproof.yml -p cib-finalproof down --remove-orphans (same for failproof2, failproof3).
• migproof (the earlier throwaway proof tenant, migproof.invalid, LIVE) is also still running and is likewise a candidate for removal.
• test-tenant is LIVE and serving — that is the intended end state, not debris.

8. The generalisable lesson

A control has to be able to fail, and it has to be able to SAY that it failed.

The first attempt at proving the FAIL path looked like it worked — no LIVE marker was written. But the script had not refused; it had crashed. The marker was absent for the wrong reason. The same instinct that produced "a control that cannot fail is not a control" produces the sharper version: a control that cannot report its verdict is not a control either. The tell was a run that stopped with no stated reason — the machine equivalent of an operator shrugging.

The fix was one line of reasoning applied consistently: the probe reports, the caller decides. Every assertion in the script now has to be reachable in order to explain its own failure, and the thing that observes can never abort the thing that is observed.