Docker systems design — prod node (2026-10-13, with Min)

Young: "Min study with Tam for Docker systems design, how to apply to our current production server." This is the measured design for the actual prod host (not a generic guide). All numbers probed live today.

Measured prod state (ground truth for the design)

The design: one node, N compose projects, nginx include-dir routing

Rule 1 — publish to loopback only. Every tenant port publishes 127.0.0.1:9x0x→container and nginx is the only public face (matches the current pattern: 8091/8092 are already 127.0.0.1-bound). Medusa/PG/Redis none ever exposed beyond the compose network; not even to localhost except the medusa server proxy port. Postgres and Redis are NOT published at all.

Traffic: client → nginx :443 (tls) → 127.0.0.1:9201 → tenant's medusa server / storefront
                                  → 127.0.0.1:9203 → tenant's EVE agent (/eve/ + /.well-known/workflow/ forwarded RAW)

• Solos-docs/nginx pattern extends trivially: /etc/nginx/conf.d/tenants/<tenant>.conf generated by the onboarding script; nginx -t && systemctl reload nginx (reload = no connection loss for docs/pulse sites).
• TLS/DNS: one subdomain per tenant (<shop>.youngle.au or client's domain) → certbot per tenant OR wildcard cert (DNS choice — flag to Sol/Ace; wildcard is less reload churn).
• EVE nginx special case (from eve docs): proxy must forward /eve/ AND /.well-known/workflow/ unrewritten or workflow callbacks stall — put both paths in the tenant block; eviction = - no path rewrite anywhere.

Rule 2 — names/volumes/ports are per-project namespaced. Compose -p cib-<tenant> enforces network+volume prefix; the template drops fixed container_name: (the official docs' container_name: medusa_backend guidance is a dev artifact; per-tenant stacks need the drop to avoid name collisions exactly as the " recommendations" section states).

• Ports per tenant (design): medusa → 9200+3k per index? NO — simply allocate next doc-block: port table is a single JSON register managed by the onboarding script (.infra/ports.json consumed by both compose env and nginx template). Never hardcode.
• Volumes: cib-<tenant>_pg_data, cib-<tenant>_redis, etc. (compose project prefix does this).

Rule 3 — one shared image, per-tenant env. Build images once per template release (Dev/Ops pipeline via GitHub Actions on an x86+arm64 build host or docker buildx on this node — buildx exists in Docker 29 ✅) → push to the fleet registry (GHCR, private) → tenant compose references tag. Medusa per-tenant builds NEVER run on prod (that's the "builds saturate 4 vCPU" defect from the sizing study).

Rule 4 — env-file = tenant secrets register. infra/tenants/<tenant>/{.env,ports.json} 0600 root; script generates PG/Redis passwords + JWT/COOKIE secrets (openssl rand -hex 32); never in source. Staging copy = separate project cib-<tenant>-stg with its own env + ports.

Rule 5 — the update/rollback loop is script-gated.

deploy: pull tag → compose pull → predeploy (migrate) → compose up -d
gate:  :9000/health=OK AND /eve/v1/health=OK else previous-tag rollback

Cron: nightly pg_dump per tenant → rclone gdrive: dated dir (this box is the GDrive lane already); retention mirror = vault-archive semantics.

Capacity (final, this node)

20G usable after measured baseline (see co-host study): ~5 Starter (3G) / 3–4 Pro (4.8G incl. agent) / 2 Market; 7.1G anomaly recovery adds another ~2 tenants if cleaned up. Keep 15% headroom at rule-of-thumb; recheck after anomaly cleanup.

Ops / pre-tenant checklist (before the stack lands)

1. Investigate the 7.1G user.slice anomaly (40 cgroups; likely stale sessions/relays) — reclaim if stale (Min/Ace).
2. fail2ban burns 687M RSS — acceptable; watch.
3. Reverse-proxy wildcard TLS decision (wildcard vs per-domain certbot) — Sol's spec.
4. Port range convention (9200+) — reserve in .infra/ports.json, keep clear of the 8442–8448 A2A band and 52243/49374 ephemerals.

• Onboarding script does NOT run docker builds; tenants only PULL images from the registry — image builds happen once per template release on the build host, never on prod.

Open items

• EVE/ Nitro route double-prefix forwarding test (verify /eve/v1/health through staged nginx block before next tenant).
• ARM64 + x86 shared images: buildx multiarch or per-arch tags (GHCR private, ~/.ssh token-free pulls on prod? gitops credentials canonical list in Sol unit's lane).
• Oracle free-tier confirmation: Sol verifies via the cloud console whether our 4 vCPU / 24 GB / 200 GB shape is within the Always Free inclusions (Mira's economics model both cases).

Young's node-architecture Q&A (2026-10-13, later)

Q1 — "Each node the same customer workspace? Do we need to install each repo per node?" No per-node repo installs. One template repo (Medusa dtc-starter fork + EVE agent template + Agilo POS client + compose/prod files) pinned by git tag; images built once per release, pulled from private GHCR by any node (ARM64 multi-arch). A node carries only: the template checkout at the pinned tag (small, per-node identical), per-tenant env/secrets, and tenant volumes/data. Every tenant stack on every node is the same pinned release — drift impossible by construction (node = git checkout <tag> + docker compose pull, never local builds). "Same customer workspace" per node = yes in content (identical template release), per-tenant in instance (each tenant = own compose project + own data + own keys).

Q2 — "Public face only nginx, everything else local?" Confirmed exactly: nginx :80/:443 is the sole public listener; every tenant container publishes to 127.0.0.1 only (medusa/storefront/EVE), Postgres and Redis publish nothing — they exist only on the compose project's internal network. Same pattern the two live sites already use (8091/8092 are 127.0.0.1-bound). Only SSH (22) + nginx (80/443) stay public on the node.

Q3 — "Backup the node — how?" Two layers, matching what this node already runs:

1. Per-tenant data (the non-reconstructable layer): nightly pg_dump per tenant cron → dated dir via the proven rclone gdrive:vault-archive/<date> lane (already running on this node for the vault; extend the same script pattern to tenants) + Redis SAVE snapshot weekly (reconstructable from Postgres anyway — snapshots are a speed-up, not the safety net).
2. The reconstructable layer (no backup needed, by design): tenant containers/config/secrets are re-creatable from template tag + env file — but the env/secrets files themselves are precious: weekly tar of infra/tenants/ (0600) → encrypted (age or gpg) → same offsite lane. Plus node config snapshot: /etc/nginx/conf.d/tenants/ + crontab → same weekly bundle.
3. Restore drill = the gate: a backup is only real after a restore test — spin a scratch compose project from tag + restored dump, health-check, destroy. Quarterly cadence, tracked like our vault pull-back sha-verify discipline. RPO ≤ 24h (nightly), RTO ≈ 1h (pull image + restore dump + nginx block). Backup growth is the disk ceiling to watch (per sizing study).

Sources

• Live probe 2026-10-13 (prod): docker/compose versions, nginx -T server blocks + proxy_pass, ss -tlnp port map, free -h, systemd-cgtop, iptables DOCKER chain count, ufw empty status.
• docs.medusajs.com docker/deployment guides (fetched 2026-10-13, .md endpoints) — compose topology, worker mode, env matrix, predeploy, /health.
• unified-package-deepdive tiers; hermes-eve-cohosting-study co-host budget; medusa-hosting-docker-study image discipline.