Stage 2 proof — onboard_tenant.sh end-to-end (2026-10-21, Sol, t_3f8c9735)

Card: t_3f8c9735 — Stage 2 proof: run onboard_tenant.sh end-to-end on a test tenant Host: prod 161.33.71.67 (aarch64, Docker 29.7.2, Compose 5.4.0) Image: solos-medusa:latest — Medusa 2.21.2, Node 24.22.0 Status: ✅ the script now runs end-to-end to a healthy, LIVE tenant.

1. The headline: three real defects, each found by live probe, each fixed at the cause

The script had never been run end-to-end. It did not reach "healthy" — and the reasons were not the ones the card assumed. The card's measured state (medusa-server crash-looping, 719 restarts; tenant DB with 1 table) was correct but incomplete: the missing migration was real, but it was not the first failure. Behind it sat two more, each of which alone would have kept the tenant down. All three are now fixed and each fix is proven by a probe.

Defect A — the Medusa server could not find the admin build (boot failure)

Symptom. With a migrated DB, the server still exited 1:

Error starting server: Could not find index.html in the admin build directory.
Make sure you run 'medusa build' before starting the server.

Root cause. @medusajs/medusa resolves the admin build as

// @medusajs/medusa/dist/loaders/admin.js:31
outDir: path.join(rootDirectory, utils_1.ADMIN_RELATIVE_OUTPUT_DIR)
// ADMIN_RELATIVE_OUTPUT_DIR = "./public/admin"

rootDirectory is the process CWD. The image's CMD was cd apps/backend && npx medusa start, so it looked for /app/apps/backend/public/admin/index.html — but medusa build writes the admin to /app/apps/backend/.medusa/server/public/admin/index.html. The build was in the image all along; the server was looking in the wrong place.

DISABLE_MEDUSA_ADMIN=true and ADMIN_DISABLED=true do not help: neither is read by the loader (the loader's only disable knob is configModule.admin.disable, and the image's config does not set it). Both were tested and both still died with the same error.

Fix. Run the server with CWD=/app/apps/backend/.medusa/server — the directory the build actually populated. Applied to medusa-server and medusa-worker in the template.

Proof (live). Same image, same DB, same env — only CWD differs:

Defect B — the migrator demanded SSL from a Postgres that has SSL off

Symptom. npx medusa db:migrate failed after ~10 s with a message that points at the wrong thing:

Could not connect to the database while running migrations. The connection
timed out after 10 seconds, which usually indicates an incorrect database URL
or an SSL configuration issue.

Root cause. The tenant's Postgres is stock postgres:15-alpine with ssl = off (PROVEN live: show ssl → off; no snakeoil cert in the image). But Medusa's module-level database config — the one the migrator uses — defaults to SSL for any URL that is not localhost:

// @medusajs/utils/dist/modules-sdk/load-module-database-config.js
getDefaultDriverOptions(clientUrl):
  clientUrl.match(/localhost|127\.0\.0\.1|ssl_mode=(disable|false)|sslmode=(disable)/i)
    ? { connection: { ssl: false } }                                   // local
    : { connection: { ssl: { rejectUnauthorized: false } } }          // REMOTE

So the migrator opens an SSL connection, Postgres answers "The server does not support SSL connections", and the knex pool can never create a connection. knex.raw("SELECT 1") then hangs until Medusa's hardcoded 10 s race in verifyMigrationConnection fires and misreports the SSL error as a timeout. The real error is never surfaced.

This is why the failure looked like a network/DB problem and was not: the runtime connection uses ssl: false (via pg-connection-loader.js) and talks to the same DB fine. The bug was invisible until the DB was empty and the migrator had to run.

Proof. The error was captured by instrumenting the tarn pool:

[POOL createFail] 7  The server does not support SSL connections
[POOL createFail] 8  The server does not support SSL connections
... (51 consecutive createFail events) ...

Fix. ?sslmode=disable on DATABASE_URL. That string matches the sslmode=(disable) branch of the matcher above, so the migrator uses ssl: false and connects. It is a URL fix, not a code change.

Proof (live, same image, fresh DB each time).

Note the masking: with MEDUSA_DB_MIGRATION_CONNECTION_TIMEOUT=120000 the error changed to "Knex: Timeout acquiring a connection. The pool is probably full." — still not the SSL error. The SSL error only surfaced by instrumenting the pool directly. Anyone tuning the timeout would have chased the wrong thing.

Defect C — no predeploy step at all, and the migrate ran in the wrong place

Symptom. The server crash-looped with relation "currency" does not exist (719 restarts on medusa-server, 3852 on medusa-worker). The template had no migration step, and the script had none either.

Fix. The template now carries a medusa-migrate one-shot compose service, and the server depends on it:

medusa-migrate:
  restart: "no"
  command: ["sh", "-c", "cd /app/apps/backend/.medusa/server && npx medusa db:migrate"]
  deploy: { resources: { limits: { cpus: "1.0", memory: 2G } } }

medusa-server:
  depends_on:
    medusa-migrate: { condition: service_completed_successfully }

Why a compose service and not a hand-run container:

• compose owns it. The 56 orphan cib-test-tenant-*-run-* containers on prod were hand-run docker run leftovers that compose never cleaned up. A compose service is labelled, tracked and removed by compose down.
• restart: "no" makes it one-shot — it runs, migrates, exits 0, and stays exited. It cannot restart-loop.
• service_completed_successfully means the server cannot start until the migration succeeded.
• 2G memory. The 1G limit OOM-killed the migrator (exit 137) mid-migration on prod. 2G is the measured floor that completes.

2. The three script bugs found by the first end-to-end run

The first run of the fixed script reached "all 7 services Healthy" and then failed its own assertion. Both were real bugs in the script, found only by running it:

1. compose ps -q does not list exited containers. The migrate one-shot has exited by the time the script checks it, so ps -q medusa-migrate returned empty and the script reported "the template lost its predeploy" while the migration had in fact succeeded with 148 tables. Fixed to ps -a -q.
2. The user table is a Postgres reserved word. The admin-user assertion used from user, which failed with relation "user" does not exist — the table exists, it just must be double-quoted. Fixed to from "user".
3. The admin user must be created with CWD = .medusa/server — the same root cause as Defect A. npx medusa user run from /app/apps/backend did not create the row; run from .medusa/server it logged User created successfully. and the row is verifiable in the tenant DB.

3. What the script does now — the full ten steps

docker-systems-design.md §4 specifies ten steps. All ten are now in onboard_tenant.sh, and every gate is a real probe:

Two deliberate design points:

• The health gate probes inside the containers (docker exec … wget), so it asserts the service's own /health, not a proxy's opinion of it.
• The vhost points at the containers' bridge IPs, not the network aliases. Docker's embedded DNS (127.0.0.11) is container-only — the host cannot resolve medusa-<slug>. The host can route to the bridge subnet: PROVEN live, curl http://172.19.0.4:9000/health from the host → 200.

4. Live probe results (prod 161.33.71.67)

4.1 The end-to-end run

=== Onboarding tenant: test-tenant (test.youngle.au) ===
  secrets generated → /home/ubuntu/.solos-infra/tenants/test-tenant/secrets.env
  compose rendered → /home/ubuntu/.solos-infra/docker-compose.test-tenant.yml
  network: compose will create and own tenant-test-tenant
 Container cib-test-tenant-medusa-migrate-1 Exited
 Container cib-test-tenant-medusa-worker-1 Healthy
 Container cib-test-tenant-medusa-server-1 Healthy
 Container cib-test-tenant-redis-1 Healthy
 Container cib-test-tenant-postgres-1 Healthy
 Container cib-test-tenant-storefront-1 Healthy
 Container cib-test-tenant-eve-agent-1 Healthy
  services started

All 7 services healthy on the first run, with the migrate one-shot exited 0 and the server gated behind it.

4.2 Predeploy

migrate: exited exit=0 oom=false
tables:  148

4.3 Health endpoints — probed from inside the containers

medusa-server/health     -> HTTP/1.1 200 OK
medusa-server/app        -> HTTP/1.1 200 OK        (admin HTML served)
eve-agent/eve/v1/health  -> HTTP/1.1 200 OK
storefront/              -> <!DOCTYPE html>&lt;html&gt;&lt;head&gt;&lt;meta charset="UTF-8"&gt;

4.4 Health endpoints — probed from the HOST through the bridge IPs

host->storefront http=200
host->medusa/health http=200
host->eve/health http=200

This is the path the nginx vhost uses, so the vhost's upstreams are proven reachable before the vhost is enabled.

4.5 Restart counts and health

medusa-server: restarts=0 health=healthy
medusa-worker: restarts=0 health=healthy
storefront:    restarts=0 health=healthy
eve-agent:     restarts=0 health=healthy
postgres:      restarts=0 health=healthy
redis:         restarts=0 health=healthy

Before: medusa-server 1141 restarts, medusa-worker 6261 restarts. After: 0 and 0.

4.6 Orphans

orphan count before: 56
orphan count after:  0

The 56 cib-test-tenant-medusa-server-run-* containers were removed. They were com.docker.compose.oneoff=True leftovers from hand-run migrate containers. The new medusa-migrate is a compose service, so compose down owns it — the orphan class is closed at the source, not just cleaned up. A live assertion in the health gate (orphan containers: 0 ✓) now catches a regression.

5. What was SKIPPED, and why

Stated plainly, because the card requires it:

• certbot / TLS — SKIPPED. test.youngle.au does not resolve (dig +short test.youngle.au A → NXDOMAIN; only docs, pulse, eve, sol, web, yukonit and www exist, and www points elsewhere). A certificate cannot be issued for a domain that does not resolve. The script detects this, writes the vhost, does not enable it, and says so. Add the DNS A record → 161.33.71.67 and re-run the script to finish step 8. This is a DNS action on a domain Young/Young Le controls — not mine to make.
• nginx vhost enabled — NOT enabled (same reason: no certificate, and an SSL vhost without a cert fails nginx -t).
• public hostname health check — NOT performed. The gate probes the services directly (in-container and via the host→bridge path). A check through a public hostname needs the DNS record that does not exist yet.
• admin login through the browser — NOT performed. The admin user is asserted to exist in the tenant DB and /app is asserted to return 200. A real browser login needs a reachable admin hostname; the Medusa admin is behind Authelia forward-auth in the vhost, and there is no vhost yet.
• backup/rollback (stage 2.3 / 2.4) — NOT part of this card. Untouched.
• the db:migrate predeploy's own card — this card proves the whole path end-to-end, as its body specifies.

6. Files changed on prod

The Medusa image was not rebuilt: all three defects are fixed in the compose template and the script, none of them in the image.

7. The generalisable lesson

The 10 s timeout was a mask, not a cause. Medusa's verifyMigrationConnection races knex.raw("SELECT 1") against a 10 s timer and reports the loser as "the connection timed out … incorrect database URL or an SSL configuration issue". The URL was correct, the DB was reachable (a direct pg connect returned in 38 ms), and the real fault — an SSL handshake against a ssl=off server — was invisible until the pool was instrumented.

The general rule this earns us: when a tool reports a timeout, find the underlying error before touching the timeout. Raising MEDUSA_DB_MIGRATION_CONNECTION_TIMEOUT from 10 s to 120 s did not fix anything; it only revealed a different misleading message ("The pool is probably full") and burned a cycle. The bug was one layer below the one the error pointed at.

The second half of the lesson: the first end-to-end run found two more bugs in the script itself (ps -q vs ps -a -q; the reserved word user). Neither was visible from reading the code. A script that has never run is not a script; it is a hypothesis. The only way to close that gap is to run it and read the output, which is what this card did.