Provider-complete image promoted to solos-medusa:latest

Card: t_177d36cb (migrated from fleet-board.md, Min, 2026-10-21) Owner: sol (engineering) · Date: 2026-10-21 Prod: 161.33.71.67, aarch64, Docker 29.7.2, Compose 5.4.0

1. The gap (reproduced live, not read from a report)

The tenant compose files run image: solos-medusa:latest. That image predated all of the notification work.

No image matched the source. The config that declares local→feed and the per-tenant email provider was not in the image any tenant ran.

2. Two independent root causes

2.1 The build swallowed its own failure

The shipped Dockerfile ended its build step with:

RUN npm run build 2>&1 | tail -5 || echo "build attempted"

A failed medusa build became a successful docker build and a shipped image. "Build succeeded" was never evidence of anything.

2.2 No .dockerignore — the host's node_modules became the image's

The build context included the host's node_modules (879 MB) and .medusa, and COPY . ./ ran after the install step, so it overwrote the builder's lockfile-driven install with the host's state.

That is why each image froze a different dependency set:

Each was a snapshot of whatever the host happened to hold at that minute.

3. The fix

/home/ubuntu/builds/medusa-backend/Dockerfile (rewritten) + a new .dockerignore:

• npm ci --workspaces --include-workspace-root — deterministic, from the lockfile. The npm ci … || npm install fallback is gone; a failed install is now a failed build.
• npx turbo build --force with no || echo.
• A build-stage gate that asserts the ARTIFACTS, not the exit code:
  • the compiled medusa-config.js exists and contains notification-local, src/modules/resend, src/modules/o365
  • resend/service.js, resend/index.js, o365/service.js, o365/index.js all exist
  • resend, @azure/identity, @microsoft/microsoft-graph-client all require.resolve inside the image
  
  Any one of these failing fails the docker build.

4. The proven image

solos-medusa:latest = sha256:42a3fd44d23c472e4d8860b5ef53e40f3e11ed4f60a4e38b25c51108cd1e448c
                      created 2026-10-21T18:08:48+11:00   arm64/linux
solos-medusa:rollback-20261021-2101
                    = sha256:c64518326eea83a3a709f1d95ee0b0c4815cd0b7987437545001b630c26bb550
                      (the previous latest, created 2026-10-20T21:01:54+11:00)

Image content is byte-identical to the source tree (md5):

5. Boot proofs (the IMAGE, not the source tree)

Isolated DB t177_probe_2026 in a card-scoped postgres (t177-pg). The tenant DB medusa-test-tenant was not touched. Evidence read from the notification / notification_provider tables, not from log lines.

TENANT_EMAIL_PROVIDER=resend

[notification] email provider: resend (channel=email)
provider rows : local | enabled=true | channels={feed}
                resend| enabled=true | channels={email}
step success  -> status=success provider=resend external_id=stub-t177-msg-000001
step rejected -> status=failure provider=resend external_id=null
feed          -> status=success provider=local
sms           -> "Could not find a notification provider for channel: sms"

The stub logged a real POST /emails with Authorization: Bearer … and a 9950-byte rendered React Email body. A rejected send is recorded as failure — not a false success.

TENANT_EMAIL_PROVIDER=none (the fail-closed default)

[notification] TENANT_EMAIL_PROVIDER=none — the `email` channel has NO provider. …
[notification] email provider: none (channel=email)
step success  -> status=failure provider=null   ("Could not find a notification provider for channel: email")
step rejected -> status=failure provider=null
feed          -> status=success provider=local     ← feed stays ALIVE

Boots cleanly. Email is dead by design and loudly; feed is alive.

TENANT_EMAIL_PROVIDER=o365

Boots; registers o365 | enabled=true | channels={email}.

Send path proven against the local fake Entra/Graph harness from card t_154c2421 (no Entra tenant exists on our side). The only difference from the shipped config was two test-only options added to the config inside the container's writable layer — the image on disk is untouched:

+ graph_base_url: process.env.O365_GRAPH_BASE_URL
+ authority_host: process.env.O365_AUTHORITY_HOST

What crossed the wire:

GET  /probe-tenant/v2.0/.well-known/openid-configuration
POST /probe-tenant/oauth2/v2.0/token?client-request-id=…      (417 bytes)
POST /v1.0/users/orders@probe.invalid/sendMail   auth=present  (3355 bytes)
     subject=Order confirmation  from=orders@probe.invalid
     to=probeT177-success@example.invalid  body=2939 bytes

What the DB recorded:

step success -> status=success provider=o365 external_id=fake-request-id-accepted
feed         -> status=success provider=local
sms          -> "Could not find a notification provider for channel: sms"

6. Rollback (one line)

docker tag solos-medusa:rollback-20261021-2101 solos-medusa:latest
docker compose -f /home/ubuntu/.solos-infra/docker-compose.test-tenant.yml -p cib-test-tenant \
  up -d --force-recreate medusa-server medusa-worker

7. A finding worth keeping: medusa exec loads the SOURCE config

Measured, not assumed. A load marker in apps/backend/.medusa/server/medusa-config.js never printed; the same marker in apps/backend/medusa-config.ts printed immediately.

medusa exec loads apps/backend/medusa-config.ts. medusa start loads the compiled .medusa/server/medusa-config.js. A probe that patches the compiled config to change provider options will silently patch the wrong file. The proof harness now patches both and asserts the anchor exists in each.

This independently confirms §4.2 of tenant-notification-providers-2026-10-20, from a different angle: that section established it for the image; this card establishes it for the medusa exec probe path, where it cost a real debugging cycle.

It is a chokepoint fact (guard rule): the control belongs where the config is actually loaded, and the two load paths differ.

8. What I did NOT do

• The test tenant's own database is still unmigrated (1 table). Its medusa-server crash-loops on relation "tax_provider" does not exist. That predates this card; it is a tenant-migration task, not an image task. Promoting the image does not fix it and does not worsen it.
• No real Resend key and no real Entra tenant exist on our side. The sends were proven against local stubs over real HTTP/TLS, with the real provider code. This proves the image's wiring, the boot registration and the recording of send outcomes. It does not prove deliverability to a real inbox.
• latest was not pushed to a registry. The tenant compose files reference a local tag on this host, so a local retag is the whole promotion. There is no registry to push to (the registry is the source tree — see stage1-finish-verification-2026-10-20).
• No tenant was restarted onto the new image. That is an operator action, and the tenant DB needs migrating first or it will still crash-loop.
• The two old probe images were left in place — harmless, and the historical record of the half-populated states.
• The latest tag was never pushed or pulled by a registry; docker tag is the whole promotion path here.

9. Where the artefacts are

On prod, under /home/ubuntu/t177d36cb/:

Files changed on prod:

10. The lesson

"Build succeeded" is not a result. The earlier build had a || echo "build attempted" in it, and that single clause shipped an image that silently lacked the entire feature while reporting success.

A build gate must assert the artifacts, not the exit code. And a build context must not carry the host's node_modules in — otherwise the image freezes a snapshot of the host instead of the lockfile.