AI/ML Engineering Playbook — SolOS v2 Rebuild

Distilled for Sol (engineering lead) + Ace (developer) from two sources:

• Designing Machine Learning Systems (Chip Huyen, O'Reilly) — production MLOps.
• Artificial Intelligence: Foundations of Computational Agents — Python edition (Poole & Mackworth, v0.9.18) — AIPython reference implementation.

Focus: practical, buildable, testable, observable engineering — not theory.

How the two books complement each other: Huyen supplies the production lens (monitoring, data pipelines, deployment, retraining loops); Poole & Mackworth supply the agent/reasoning lens (agent architecture, search, RL, uncertainty, multi-agent). Read Book 1 for "how do I keep a running system from silently degrading," Book 2 for "how do I structure agents, state, and learning so the system is testable." The Top-10 below fuses both into concrete SolOS v2 build tasks.

How to use: Skim the two book sections (a/b/c each), then drive the sprint off the Top-10. Each action item names its source chapter so you can drill into the original text when a decision needs the full reasoning.

Book 1 — Designing Machine Learning Systems (Chip Huyen)

1a. Core engineering principles

1. Design the whole system, not the model. Changing data, features, or metrics affects every other component. Optimize end-to-end behavior, not model accuracy in isolation.
2. ML models fail silently. Code compiles, loss decreases, predictions are wrong — and nobody notices. Assume every deployed model is failing until your monitoring proves otherwise.
3. Data is the system. "It depends" on the data: two teams with the same problem build different systems. Invest in data quality before model sophistication.
4. Track everything you can about experiments. Experiment tracking gives observability into model state; without it you cannot reproduce a good result or debug a bad one.
5. Observability is a prerequisite for monitoring. Instrument first (timers, NaN counters, input/output logging); only then can you alert on what goes wrong.
6. Separate operational metrics from ML-specific metrics. Uptime/latency (system health) and accuracy/features/predictions/inputs (model health) both matter; a fast wrong answer is still a failure.
7. Use feedback as natural labels. Clicks, ratings, edits are free labels — log them, use them to detect degradation and to re-label for retraining.
8. Minimize training-serving skew. The features at inference must be computed by the same pipeline/code as at training, or your eval is meaningless.
9. Alert only on what warrants action; avoid alert fatigue. Too many false-positive alerts train teams to ignore all alerts. Tune thresholds against real signal.
10. Degradation is inevitable; plan for it. Data drifts, environments change. Build the assumption of continual retraining/monitoring into the design, not bolted on later.

1b. Directly-applicable patterns/techniques (multi-agent relevance)

• Instrument the mesh, not just the model. Add timers to every agent/tool call, count NaNs/empty outputs, log unusual inputs (very long prompts, malformed JSON). Observability = the memory/state you need to debug.
• Log the four artifacts per pipeline stage: raw inputs → features → predictions → accuracy/feedback. Deeper artifacts are more structured and easier to alert on; shallow ones catch upstream errors.
• Track user feedback as labels for the agent mesh. Upvote/downvote/correct on agent outputs; aggregate to detect a drifting agent and to seed re-labeling/re-prompting.
• Adopt the training/eval/production split discipline. Static test sets for offline eval (golden evals), plus shadow/online eval for deployed behavior — same input, compare outputs.
• Monitoring = metrics pipeline with SLO-style thresholds. Define "up" explicitly (e.g. p50 latency < X, p99 < Y, 2xx rate); alert when breached. This maps directly to CostGov + mesh health.
• Feature-store-style shared state for the mesh. Centralize computed context (embeddings, tool results, memory keys) so every agent reads from one source of truth — avoids per-agent recompute and skew.
• Continual learning loop for prompts/agents. Treat prompt/agent iteration as retraining: keep a queue of production failures, batch-relabel, re-eval offline before promotion.
• Schema-validate inputs and outputs at the boundary. Mismatched schema is a leading false-alert source; validate at ingestion and at each agent handoff.

1c. Warnings / anti-patterns

• Building two pipelines (train vs serve). The classic source of skew; if training and inference features are computed by different code, eval results don't transfer to production.
• Monitoring every feature for drift. Most drift is benign; monitoring all features floods you with false positives and triggers alert fatigue.
• Optimizing model metrics over business outcomes. A better offline score that doesn't move user behavior (completion, retention, correctness) is wasted effort.
• Deploying with no feedback channel. If you can't observe how predictions are used or whether they were right, you're flying blind and can't detect silent failure.
• Treating MLOps as a one-time setup. Without a retrain/redeploy loop and ownership, the system decays as soon as data shifts.

Book 2 — Artificial Intelligence: Foundations of Computational Agents (Poole & Mackworth, Python ed.)

2a. Core engineering principles

1. Readability over efficiency. "Think of it like a model of an engine made of glass." Optimize for correctness and clarity first; asymptotic complexity second; micro-optimization never.
2. Represent agents and environments as separate, swappable objects. The agent/environment boundary is the fundamental abstraction — you can test an agent against many environments and vice versa.
3. Hierarchical control beats monolithic reasoning. Split the agent into layers (body/middle/top) with distinct time scales and abstraction levels; each layer has a narrower, more testable job.
4. Generic interfaces with pluggable strategies. One Searcher, one Learner, one Agent interface; variants (A*, naive Bayes, Q-learning) plug in. This is the template for a plugin/MCP layer.
5. Explicit representation beats implicit magic. Represent states, actions, and models explicitly as data; avoid hidden control flow the reader can't see.
6. Test at every layer, not just end-to-end. The book ships unit tests for search, CSP, RL, and games; each component is verified in isolation before composition.
7. Simulate before you deploy. Run agents against a simulated environment to generate data, validate behavior, and debug — cheap, repeatable, deterministic when seeded.
8. Explicit uncertainty is a feature. Represent belief as probability distributions (belief networks, HMM/DBN, particle filters) rather than a single point estimate; keep the uncertainty visible.
9. Exploration is part of the algorithm. RL/learning agents must balance exploit vs explore; a pure greedy agent converges to mediocrity. Bake exploration in, don't assume it.

2b. Directly-applicable patterns/techniques (multi-agent relevance)

• Agent/Environment interface for the mesh. Model every SolOS agent as (percept -> action) and the mesh/world as the environment. Enables deterministic simulation, replay, and unit testing of individual agents.
• Hierarchical controller = orchestration layer. Top layer plans, middle decomposes, body executes (tool calls). Mirrors SolOS's mesh + OpenCode runtime; give each layer its own state and failure handling.
• A generic "Learner" interface. One interface for naive Bayes, decision trees, boosting, NN. For SolOS: a generic "Skill/Agent" interface so prompts, tools, and models are swappable implementations.
• Evaluation discipline from ch. 7: hold-out test sets, k-fold cross-validation, parameter tuning on a validation set, never on test. Apply directly to prompt/agent eval (golden sets, no leakage).
• Experience replay (RL ch. 13). Store (state, action, reward) transitions in a buffer, sample for training. For SolOS: log agent trajectories to memory plane, replay to improve prompts/policies.
• Model-based learning. Learn an explicit world model, then plan in it, then act — separates "learning" from "deciding." For SolOS: keep an explicit world/task model (vault/docs) that agents reason over rather than re-discovering.
• Minimax/α-β for adversarial or multi-agent negotiation. Prune dominated branches; for a multi-agent mesh, explicitly model what other agents will do instead of treating them as a black box.
• Stochastic simulation + sampling for uncertainty. When exact inference is too costly, sample (particle filtering, Gibbs). For CostGov: estimate cost/risk distributions by sampling rather than exact accounting.

2c. Warnings / anti-patterns

• Conflating training and test data. Any information from the test set that leaks into training/feature/tuning choices invalidates your evaluation (aipython keeps test/train split explicit — ch. 7).
• Greedy-only agents without exploration. Pure exploit agents get stuck in local optima; you must encode exploration strategy explicitly (RL ch. 13).
• Monolithic agents that do everything. The book's refactor from single agent to hierarchical controller exists precisely because one big decision loop is brittle and untestable.
• Implicit/hidden state. When state lives in scattered globals or mutated objects, agents become non-reproducible and impossible to test. Keep state explicit and observable.

Top-10 Action Items — SolOS v2 (Python, OpenCode runtime, MCP plugin layer, multi-agent mesh, CostGov, memory plane, vault/docs)

1. Define a single Agent/Environment interface before building. Every SolOS agent is (percept -> action); the mesh is the environment. Gives you deterministic simulation, replay, and per-agent unit tests (AIPython ch. 2/13 + DMLS "system as a whole"). The MCP plugin layer becomes just implementations of one interface.
2. Instrument the mesh from day one. Timers on every tool/agent call, NaN/empty-output counters, malformed-JSON logging, and per-stage artifact logging (input → feature → output → feedback) (DMLS ch. 8). Observability is the memory you need to debug silent failures.
3. Log agent trajectories to the memory plane with feedback labels. Store (state, action, reward/outcome) transitions for experience replay (AIPython RL ch. 13) and user thumbs/corrections as natural labels (DMLS). This feeds both CostGov accounting and prompt/agent retraining.
4. Build a golden-eval harness with strict train/test separation. Static golden sets for offline eval, k-fold/validation for tuning, never tune on test (AIPython ch. 7). Add shadow/online eval so a new agent or prompt is compared against the incumbent on live inputs before promotion.
5. Make CostGov a sampling-based estimator, not exact accounting. Model cost/risk as a distribution, sample to estimate (particle filtering / Gibbs analog, AIPython ch. 9). Alert on thresholds with SLO-style definitions of "within budget," and tune thresholds to avoid alert fatigue (DMLS ch. 8).
6. One feature/context pipeline, shared by train and serve. Centralize computed context (embeddings, tool results, memory keys) so training/eval and production read from the same code path — eliminates training-serving skew (DMLS). Schema-validate at the MCP boundary and at every agent handoff.
7. Design the mesh as a hierarchical controller. Top layer plans, middle decomposes, body executes tool calls, each with its own state and failure handling (AIPython ch. 2). Narrow, single-responsibility agents are testable; monoliths are not.
8. Write explicit, swappable, readable code over clever code. Mirror AIPython's "engine made of glass": explicit state, generic interfaces, minimal magic. The MCP plugin layer should be a set of pluggable strategies behind one interface, not ad-hoc branching.
9. Ship a retrain/redeploy (prompt/skill iteration) loop, not a one-time build. Keep a queue of production failures, batch-relabel, re-eval offline, promote on measured improvement (DMLS continual-learning loop). Assign ownership for monitoring + iteration.
10. Treat the vault/docs as an explicit world model. Let agents reason over a persistent, versioned world/task model instead of re-discovering context (AIPython model-based learning + knowledge graphs ch. 16). Ground every agent's context in one source of truth to reduce drift and hallucination.

Cross-cutting checklist (one-liners)

• Every deployed agent has: a feedback signal, an SLO threshold, a golden eval, and an owner.
• No test-set leakage; no train/serve skew; no silent failures without a monitor.
• State is explicit, interfaces are generic, uncertainty is visible, exploration is baked in.
• When in doubt: simulate it, replay it, and measure it before shipping it.