O365 / Microsoft Graph notification provider — the per-tenant opt-in email channel
O365 / Microsoft Graph notification provider — the per-tenant opt-in email channel
Card: t_154c2421 (build task 4 of 8 from t_c9b50255).
Method: live probes on prod (161.33.71.67, aarch64, Node v24.19.0) against
the pinned @medusajs/medusa 2.21.2, with the real Microsoft client
libraries (@azure/identity 4.14.0, @microsoft/microsoft-graph-client 3.0.7).
Isolated scratch DB (o365_e2e_2026 on the pg-temp container).
No tenant DB was written to.
🔴 Read this first: no real Entra tenant was used
No email was delivered, and none could be. Microsoft Graph and Microsoft Entra ID were local stand-ins (a fake AAD + fake Graph over real TLS/HTTP). The provider code, the Microsoft client libraries, the token flow, the HTTP and the TLS are all real; the far end is not Microsoft.
What is therefore proven: the provider's boot registration, its request shape, its 202 handling, its 429/403 handling, and that the Notification Module records the truth in the DB.
What is NOT proven: that mail reaches a real inbox, that a real tenant's
Mail.Sendconsent works, or that DKIM/DMARC align for a real domain. A live send against a real Entra tenant is still owed. §8 says exactly what it would take.
1. What landed
| File (on prod) | Change |
|---|---|
builds/medusa-backend/apps/backend/src/modules/o365/service.ts | The provider: AbstractNotificationProviderService, identifier = "notification-o365", one send(). |
builds/medusa-backend/apps/backend/src/modules/o365/index.ts | ModuleProvider(Modules.NOTIFICATION, { services: [...] }). |
builds/medusa-backend/apps/backend/src/modules/o365/emails/order-placed.ts | Order-confirmation template (plain HTML, no renderer dependency — §4). |
builds/medusa-backend/apps/backend/src/modules/o365/emails/order-shipped.ts | Shipping-confirmation template. |
builds/medusa-backend/apps/backend/package.json | @azure/identity ^4.14.0, @microsoft/microsoft-graph-client ^3.0.7 in dependencies. |
builds/medusa-backend/package-lock.json | Resolved: identity 4.14.0, graph 3.0.7, @typespec/ts-http-runtime 0.3.9, http-proxy-agent 7.0.2 — all [prod], not dev. |
The medusa-config.ts provider block and the compose env contract were
already landed by card t_7b0702c4 and are unchanged by this card:
TENANT_EMAIL_PROVIDER=o365 selects this provider; the env names are
O365_TENANT_ID, O365_CLIENT_ID, O365_CLIENT_SECRET, O365_FROM,
O365_MAILBOX.
2. The provider's contract (the parts that are not obvious)
2.1 send() MUST throw on failure — a swallowed error is a false success
The Notification Module records status=failure on a notification row only
when the provider's send() rejects. It never inspects the return value
(@medusajs/notification/dist/services/notification-module-service.js, v2.21.2,
lines 94–101):
const res = await this.notificationProviderService_.send(provider, entry.data)
.catch((e) => { entry.data.status = FAILURE; ...; throw ... })
entry.data.external_id = res.id
entry.data.status = SUCCESS
A provider that logs the error and returns {} produces a row with
status=success and external_id=NULL — a false success, which the
per-channel health gate (card t_7e2d75cb, which reads the DB) would read as
a healthy email channel. This provider throws on every failure, including
a throttle.
A throw does not crash the worker: the Notification Module converts it to a
MedusaError and the shipped subscriber catches that and logs it. The row in
the notification table is the evidence an operator can see. Proven in §3.
2.2 202 Accepted is NOT delivery
Graph returns 202 Accepted for sendMail, meaning "accepted for
processing", not "delivered" — Microsoft's own wording: "a 202 Accepted response code indicates that the request has been accepted; however,
it doesn't indicate that the request processing has completed. Delivery of the
message is subject to Exchange Online limitations and throttling."
So the provider logs it as an acceptance and never as a delivery. The actual line emitted on the success path:
O365 Graph ACCEPTED (202, queued for delivery — NOT delivered) email to <to> from <from> (template "<t>")
The delivery truth is the notification row status plus the tenant's own bounce
handling — the same "approved must not read as delivered" discipline as the EVE
queue. Graph returns no message id for sendMail; the correlation id that
is available is the request-id response header, so that is what the
provider stores in external_id.
2.3 The send path is a USER mailbox, never /me
POST /users/{mailbox}/sendMail with from set explicitly. There is no
signed-in user in a daemon: the app token acts as the application.
/me/sendMail would fail for an application permission. The from is always
the tenant's own mailbox, so receipts carry the tenant's branding and
DKIM/DMARC alignment.
2.4 saveToSentItems — decided and documented
Default: true (Graph's own default). Rationale: a tenant's staff sending
transactional mail from a shared mailbox generally want the copy — it is the
audit trail the tenant expects, and it costs nothing at the Graph API. A tenant
that wants a clean mailbox sets save_to_sent_items: false.
Note the Graph contract: saveToSentItems is a top-level sibling of
message, not a member of it (§4.1 — this was a real bug, caught live).
2.5 The retry story — decided and documented
A throttle is surfaced as a visible failure the operator sees. It is not retried in-process, and it is not silently dropped.
The reason is measured, not stylistic: the Graph client's RetryHandlerhonours the Retry-After header on a 429 and SLEEPS for that long
in-process (RetryHandler.ts — RETRY_STATUS_CODES = [429, 503, 504],
getDelay() reads Retry-After, capped at 180s per attempt, 3 retries by
default). A Medusa subscriber that blocks for up to 3 × 180s is a stalled
worker, and the Notification Module has no retry queue, so a "successful"
retry after a stall is not guaranteed either.
So max_retries defaults to 0 — fail fast, record the failure, let the
operator (and the health gate) see it. A tenant that prefers the SDK's waiting
behaviour can raise max_retries (capped at 10) and retry_delay_seconds
(capped at 180).
This is a deliberate decision, and it is a weaker story than a queue+backoff
would be. The honest framing: a throttled send is visible and recorded,
but it is not automatically retried — re-sending is an operator action
until a queue lands. The per-channel health gate (card t_7e2d75cb) is the
detector.
2.6 Scoping to one mailbox
Graph's Mail.Send application permission means "send mail as any user in
the organization" and requires tenant admin consent — so it cannot be
pre-provisioned by us, and it is the main operational reason O365 is opt-in
rather than the default. The tenant's admin registers the app, grants consent,
and narrows the blast radius to one service mailbox with RBAC for
Applications in Exchange Online (Microsoft's own guidance). The provider
takes the mailbox as a required option (O365_MAILBOX) and always sends from
it.
The provider does not attempt to enforce the RBAC scope — only Entra can. What it does is refuse to start without a mailbox, so a tenant cannot accidentally send as "any user".
3. The proof (all on the pinned runtime, DB-read)
3.1 End-to-end (medusa exec on the isolated scratch DB)
| Case | createNotifications threw? | notification row | verdict |
|---|---|---|---|
| accepted (202) | no | status=success, provider_id=o365, external_id=fake-request-id-accepted | ✅ |
| throttled (429) | yes | status=failure, provider_id=o365, external_id=NULL | ✅ |
| rejected (403) | yes | status=failure, provider_id=o365, external_id=NULL | ✅ |
| shipped template | no | status=success, external_id set | ✅ |
| unknown template | yes | status=failure | ✅ |
feed | no | status=success, provider_id=local | ✅ coexists |
sms (no provider) | yes | status=failure | ✅ fails loudly |
The failure rows are the point. The throttle produced:
O365 Graph failed to send email to probe-throttled@example.invalid:
HTTP 429 | code=TooManyRequests | Your tenant has exceeded its sending limit.
Retry after 1 second. | request-id=fake-request-id-throttled
— the HTTP status, the Graph error code, the operator-facing message and the request id, all in the recorded failure.
The worker did not crash: 0 unhandled rejections / uncaught exceptions in the probe log, and the steps after the throttle still succeeded. The throttle is handled, recorded, and visible.
3.2 Boot registration (the DB is the evidence)
id | handle | is_enabled | channels
-------+--------+------------+----------
local | local | t | {feed}
o365 | o365 | t | {email}
o365 registers with channels={email} and coexists with local→["feed"].
One provider per channel is enforced by the framework at boot — a duplicate is
a boot failure, and per the card that is correct and was not
"fixed".
3.3 What actually went over the wire (the fake Graph's request log)
GET /probe-tenant/v2.0/.well-known/openid-configuration
POST /probe-tenant/oauth2/v2.0/token
client_id=probe-client-id&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default
&grant_type=client_credentials&client_secret=...
POST /v1.0/users/orders@probe.invalid/sendMail auth: Bearer <token>
top-level keys: message,saveToSentItems
message.subject: Order confirmation
message.toRecipients[0]: probe-accepted@example.invalid
message.from: orders@probe.invalid
saveToSentItems: true
The bearer token is attached, the scope is https://graph.microsoft.com/.default,
the grant is client_credentials, and the path is the user mailbox.
3.4 Boot probes — 12/12 pass
Missing tenant_id / client_id / client_secret / from / mailbox each
fail at boot (the module-provider loader calls validateOptions before
registering). A malformed mailbox address ("not-an-email", "orders @probe.invalid",
"") also fails at boot. A complete, valid config passes.
4. Two bugs the live proof caught (neither would have been caught by a "mock" that always returns 202)
4.1 The request body shape was WRONG
The first end-to-end run reported status=success for the throttled and
rejected cases too — because the fake Graph was answering 202 to everything.
Making the stand-in behave like real Graph (validating the body, rejecting a
malformed one with 400) exposed the real bug:
Graph's sendMail body is
{ "message": { "subject": …, "body": {…}, "toRecipients": […], "from": {…} },
"saveToSentItems": true }
message is a wrapper, and saveToSentItems is a top-level sibling of
it. The provider was posting the message fields at the top level, with
saveToSentItems inside what should have been the wrapper. Real Graph
rejects that with 400. Fixed and re-proven.
The lesson is the same one the EVE queue taught: a control that cannot report failure is not a control. A stand-in that returns 202 to everything would have shipped a provider that Graph rejects on every single send.
The DB keeps the evidence of both runs — the first run's throttled and
rejected rows say success; the second run's say failure.
4.2 customHosts needs the BARE hostname, not host:port
The Graph client only attaches the Authorization header to a host it
recognises (isValidEndpoint in GraphRequestUtil.ts strips the port
before comparing against the allowed set). Passing localhost:18445 in
customHosts therefore meant no bearer token on the wire — a silent
auth-failure trap for anyone pointing the provider at a non-Microsoft base
URL. The provider now registers both the bare hostname and the
host:port form.
5. Controls (8/8 pass) — the stand-in can actually report failure
| # | Control | Result |
|---|---|---|
| 1 | the pre-fix body shape (no message wrapper) is REJECTED | 400 ✅ |
| 2 | a message with no from is REJECTED | 400 ✅ |
| 3 | a non-JSON body is REJECTED | 400 ✅ |
| 4 | the fixed body shape is ACCEPTED | 202 ✅ |
| 5 | the throttle branch fires | 429 ✅ |
| 6 | the rejection branch fires | 403 ✅ |
| 7 | the OIDC discovery carries all 4 fields MSAL requires | 200 ✅ |
| 8 | the token endpoint issues a bearer token | 200 ✅ |
Control 1 is the important one: it proves the stand-in would have caught the §4.1 bug, so the pass in §3.1 means something.
6. Two Microsoft-library facts that cost real time (measured)
6.1 MSAL's OIDC discovery must carry FOUR fields
isOpenIdConfigResponse() in @azure/msal-common requires
authorization_endpoint, token_endpoint, issuer and jwks_uri. A
discovery document missing any one of them fails with
endpoints_resolution_error. A real Entra tenant serves all four; the local
stand-in had to as well.
6.2 The client-credentials flow against a non-Microsoft authority
ClientSecretCredential accepts authorityHost + disableInstanceDiscovery,
and then resolves the authority as
{authorityHost}/{tenantId}/v2.0/.well-known/openid-configuration. The token
request that follows is the ordinary
grant_type=client_credentials&scope=…/.default form. This is why the
provider can be pointed at a local stand-in at all — and it is also the
hook a sovereign-cloud or private-cloud deployment would use.
7. Licence and supply chain
@microsoft/microsoft-graph-client and @azure/identity are MIT — no AGPL
exposure, as the card recorded. Both are in dependencies (not devDeps), and
the resolved transitive closure is [prod] in the lockfile.
One supply-chain note that is NOT a licence issue but is a real one:@azure/identity pulls in @typespec/ts-http-runtime and
http-proxy-agent/https-proxy-agent transitively. The probe app initially
failed to boot with Cannot find module '@typespec/ts-http-runtime/internal/logger'
because those packages were not installed — the real build tree has them
(verified in the lockfile), but any tree that copies the node_modules by
hand must copy the whole closure, not just @azure and @microsoft.
8. What I did NOT do — and what is still owed
- I did not perform a real Graph send. No Entra tenant exists on our side.
A real send needs: an Entra tenant, an app registration with the
Mail.Sendapplication permission, admin consent, RBAC for Applications scoping the app to one mailbox, and a verified sending domain. The provider code is proven; the tenant onboarding path is documented, not exercised. - I did not prove deliverability. A 202 is an acceptance, not a delivery (§2.2). Bounce handling is the tenant's own Exchange policy, not ours.
- I did not build a retry queue. The throttle story is "visible failure", not "automatic retry" (§2.5). A queue + backoff is a follow-up.
- I did not change the default provider selection.
TENANT_EMAIL_PROVIDERstill defaults tonone; the fleet default staysresend(cardt_1419d598). This card does not move that. - I did not build the customer-facing in-app thread (card
t_b7f5cd57). - I did not add a duplicate-channel guard — the framework already fails closed at boot, and the card said not to write one.
- I did not prove a production-mode boot. The probes ran
medusa execunderNODE_ENV=development(noREDIS_URL→ fake redis). The Notification Module does not use redis, so routing is unaffected — but this is not amedusa startof a rebuilt image.
9. Where the artefacts are
| Artefact | Where (prod 161.33.71.67) |
|---|---|
| Landed provider | /home/ubuntu/builds/medusa-backend/apps/backend/src/modules/o365/ |
| Probe app | /home/ubuntu/o365-e2e/app/ (its own node_modules, own scratch DB) |
| Fake Entra + fake Graph | /home/ubuntu/o365-e2e/fake-graph.cjs |
| End-to-end probe | /home/ubuntu/o365-e2e/app/o365-e2e-probe.ts |
| Controls | /home/ubuntu/o365-e2e/controls.cjs |
| Boot probes | /home/ubuntu/o365-e2e/boot-probes.cjs |
| Runner | /home/ubuntu/o365-e2e/run-e2e.sh |
| Result | /home/ubuntu/o365-e2e/result.json |
| Wire log | /home/ubuntu/o365-e2e/fake-graph-state.json |
| Scratch DB | o365_e2e_2026 on the pg-temp container (droppable) |
10. Sources
1 @medusajs/notification/dist/services/notification-module-service.js (v2.21.2) — the send() catch that records status=failure2 @medusajs/modules-sdk/dist/loaders/module-provider-loader.js — validateOptions is called at boot, before registration
3 @medusajs/notification/dist/loaders/providers.js — one provider per channel, enforced at boot
4 @microsoft/microsoft-graph-client/src/middleware/RetryHandler.ts — 429/503/504 retried, Retry-After honoured, 180s cap
5 @microsoft/microsoft-graph-client/src/GraphRequestUtil.ts — isValidEndpoint strips the port before the host comparison
6 @azure/msal-common/dist/authority/OpenIdConfigResponse.mjs — the four required discovery fields
7 @azure/identity/dist/commonjs/msal/utils.js — getKnownAuthorities / disableInstanceDiscovery8 https://learn.microsoft.com/en-us/graph/api/user-sendmail — the message wrapper, saveToSentItems placement, and "202 Accepted … doesn't indicate that the request processing has completed"
9 https://learn.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits — 10,000/day, 30 msg/min, 500 recipients/message, TERRL
10 https://learn.microsoft.com/en-us/graph/outlook-send-mail-from-other-user — app tokens send as any user
Provider-complete image promoted to solos-medusa:latest
Card: t_177d36cb (migrated from fleet-board.md, Min, 2026-10-21) Owner: sol (engineering) · Date: 2026-10-21 Prod: 161.33.71.67, aarch64, Docker 29.7.2, Compose 5.4.0
Prod decommission — opencode-web.service + tailscaled
Young, 2026-10-18: decommission opencode-web.service (sol.youngle.au, :4090) and tailscaled.

