Mathematical Introduction to Deep Learning — Jentzen, Kuckuck, von Wurstemberger

arXiv 2310.20360v3 (2025) — 737 pp, 6 Parts / 18 chapters, deterministic↔stochastic mirrored (every Ch.6 GD method has a Ch.7 SGD twin). 45 runnable Python listings (pp. 681–682), repo github.com/introdeeplearning/book. 177 numbered definitions, 87 exercises.

Part I — ANNs (Ch. 1–2)

• 16 activation functions rigorously defined (ReLU, GELU, Swish/SiLU, ELU, softplus, sine …) (pp. 28–58).
• Def 1.3.1/1.3.4: structured FC-ANN Φ = ((W₁,B₁),…,(W_L,B_L)); realization as nested affine + activation (pp. 56–58).
• FC-ResNets (Def 1.5.1, p. 72); vectorized RNN/LSTM (pp. 75–79); brief coverage of autoencoders, transformers/attention, GNNs, neural operators (§1.7, pp. 79–82).
• Ch. 2 glue lemmas for compositions/parallelizations of ANNs (pp. 83–105) — foundation for all constructive proofs downstream.

Part II — Approximation theory (Ch. 3–4, pp. 113–180)

• Cor 3.3.5 (p. 130): 1-D, f Lipschitz ⇒ depth-1 ReLU net, D₁ ≤ L(b−a)/ε + 2, P(F) ≤ 3L(b−a)/ε + 7 — fully constructive, backprop-friendly.
• Cor 4.3.14 (p. 164): d-dim Lipschitz ⇒ P(Φ) ≤ C·ε^(−2d) (curse of dimensionality).
• Prop 4.4.12 (pp. 176–177): refined rectified-clipped error ≤ 3dL(b−a)A^(−1/d).
• Remark 4.4.14 (p. 180) honesty flag: the explicit constant cannot be tracked in d ⇒ bound practically void for d ≥ 5; state-of-the-art high-dim results live in the PDE literature (refs 14, 15, 28, 73, 127, 167).

Part III — Optimization (Ch. 5–11, pp. 181–558)

Ch. 5 Gradient flow (ODE) (pp. 183–240)

• GF ODE Θ_t = ξ − ∫₀ᵗ∇L(Θ_s)ds; loss zoo: MSE, Huber, cross-entropy, KL (§5.4).
• Coercivity ⟨θ−ϑ,∇L(θ)⟩ ≥ c‖θ−ϑ‖² ⇒ Thm 5.9.4 (p. 239): ‖Θ_t−ϑ‖ ≤ e^(−ct)‖ξ−ϑ‖.
• Cor 5.9.5 (p. 240): 0 ≤ (c/2)‖Θ_t−ϑ‖² ≤ L(Θ_t)−L(ϑ) ≤ (L/2)e^(−2ct)‖ξ−ϑ‖².

Ch. 6 Deterministic GD — the optimizer encyclopedia (pp. 241–360)

Verbatim captured update rules:

• GD: Cor 6.1.11 (p. 251) L(Θₙ)−L(ϑ) ≤ (L/2)(1−2cγ+γ²L²)ⁿ‖ξ−ϑ‖² for γ ∈ (0, 2c/L²).
• Momentum (Def 6.3.x, pp. 274–276): mₙ = αₙmₙ₋₁ + (1−αₙ)∇L; Θₙ = Θₙ₋₁ − γₙmₙ. Two conventions distinguished (§6.3.1).
• Nesterov (pp. 312ff); Adagrad (p. 334).
• RMSprop (Def 6.6.5, pp. 338–339): M⁽ⁱ⁾ₙ = βM⁽ⁱ⁾ₙ₋₁ + (1−β)|G⁽ⁱ⁾|²; Θₙ₋₁ − γₙε + (Mₙ/(1−∏β))^½⁻¹Gₙ₋₁.
• Adam (Def 6.8.1, pp. 342–343): dual decay streams with both bias corrections (1−∏α), (1−∏β) — update rule (6.584–6.585) captured exactly.
• AdamW (§6.10, p. 349): decoupled weight decay λΘₙ₋₁ applied OUTSIDE the √M̂ scaling — the precise Adam-vs-AdamW distinction.
• Shampoo (§6.11, p. 352): preconditioned DJ = Lₙ^(−1/4)∇L Rₙ^(−1/4) (exact Jacobi updates Jₙ = (1−λ)Lₙ + ∇L Dᵀ, analogous right side).
• Muon (§6.12, pp. 353–354): momentum + Newton–Schulz orthogonalization NS_{a,b,c,ε}(mₙ, K) with both idealized (exact Π_K) and polynomial versions — algorithm boxes given. Cited to Jordan et al. 245. → Practitioner flag: first time Muon is formalized in a textbook; solos Shampoo in the same chapter.
• Ch. 7: SGD twins of every Ch. 6 method (expected losses, martingale arguments), incl. AMSGrad (§7.14, p. 449), summary table §7.15 (p. 450).

Ch. 8 Backpropagation (pp. 457–468)

Two-layer structure: backprop for general parametric functions (adjoint recursion G = (D_y F)ᵀ Λ), then specialized to ANNs. The adjoint-structure derivation is the reusable core.

Ch. 9 Kurdyka–Łojasiewicz (KL) inequalities (pp. 469–526)

• Standard KL functions, regular regime, tame regularity, nonsmooth KL, gradient-convergence vs value-convergence, Fréchet/limiting subgradients, generalized KL.
• §9.17 (p. 525): non-convergence counterexample for SGD — classical convex analysis fails for ANN training; the KL tame-optimization framework is what makes convergence tractable.

Ch. 10 Batch norm (pp. 527–534) — training/inference formulations + connection.

Ch. 11 Random initializations (pp. 535–574) — optimization error via complementary distribution functions; gamma/beta function machinery; strong convergence rates.

Part IV — Generalization (Ch. 12–13, pp. 575–612)

• Metric entropy: packing/covering inequalities (pp. 575–586), finite-dim covering entropy bounds (p. 580).
• ERM: concentration for random fields (p. 587), uniform statistical-learning bounds (p. 592).
• Ch. 13: Monte Carlo estimates, uniform strong error estimates, strong generalization rates (p. 605). Fewer explicit statements published here than in Parts II/III.

Part V — Composed error analysis (Ch. 14–15, pp. 613–634)

• Ch. 14: bias-variance decomposition; overall error = approximation + estimation/optimization + generalization.
• Ch. 15 (p. 621): full strong error analysis of ANN training; §15.2 adds SGD-with-random-initialization composition. THE bridge from theory to end-to-end reality; strongest statements of the book.

Part VI — Deep learning for PDEs (Ch. 16–18, pp. 635–676)

• PINNs (Ch. 16): PDE → stochastic optimization reformulation; derivation of PINN/DGM (pp. 637–640); implementation listings.
• Deep BSDE methods (Ch. 17). Ch. 18: strong/weak formulations + stochastic representations; error-analysis taxonomy for deep-PDE (§18.4, p. 675).

Practitioner flags

1. Muon + Shampoo now have textbook-rigorous treatment (first) — relevant if/when SolOS training or fine-tuning moves past off-the-shelf AdamW.
2. The approximation caveats (p. 180) are an honest map of theory-vs-practice: high-dim in the wild relies on PDE-adjacent special structure, not generic ANNs.
3. The deterministic (Ch. 6) ↔ stochastic (Ch. 7) mirroring is a clean mental model: every optimizer has two mathematically distinct worlds.
4. All 45 code listings are runnable against the published repo — usable as a course for teaching, not just a reference.

Extraction provenance

~55 representative pages read via pypdf incl. all TOC/list/introduction pages; every sampled page cleaned. Minor OCR-honesty notes: Ch. 10 batch-norm proofs and Ch. 17 FE proofs skimmed; several auxiliary pages flagged OCR-required earlier and skipped. All page cites from the v3 (2025-07-15) PDF.