Test-Time Compute Scaling

What

Spend more compute at inference time instead of training: sample many answers (repeated sampling), search over reasoning paths, or auto-select inference techniques.

Papers

• Large Language Monkeys (Brown et al. 2024) — repeated sampling at scale: sampling N completions and picking the best gives big gains on hard tasks; error rate falls roughly with a power law in samples.
• Scaling LLM Test-Time Compute Optimally (Snell et al. 2024) — the compute-optimal way to spend test-time budget: a learned verifier + search beats naive majority voting; optimal policy is "solve-verify" style search (sequential revisions vs parallel sampling, chosen per difficulty).
• Archon (Saad-Falcon et al. 2024) — automatic architecture search over inference-time techniques (which prompt strategy + verifier + sampling combo), then composes the winning modules.

Key ideas

• Verifier quality dominates: with a good verifier, more samples → monotonically better.
• "Compute-optimal" scaling curves: test-time compute can substitute for some model size.
• Architecture search automates the "which technique stack" question.

SolOS mapping

• Sample-2-verify-ship for high-stakes briefs: generate 2 candidate briefs, run a cheap verifier (per note 02), ship the best. ~2x brief cost, big quality gain.
• Our MA50 alert engine is a deterministic verifier already — extend the pattern to prose outputs.
• Archon-style thinking applies to our prompt stacks: we version skills, not prompt combinations — treat a skill + model choice as the "module".