Train-Time Scaling & RL

What

Scale the training side of reasoning: bootstrap models on their own good reasoning (STaR), reward correct-but-verbose reasoning at scale (DeepSeekMath), and do RL at open-source scale (DAPO).

Papers

• STaR — Bootstrapping Reasoning With Reasoning (Zelikman et al. 2022) — generate rationales, keep the ones that produce correct answers, fine-tune on them, repeat. Self-improvement via filtering + retraining.
• DeepSeekMath — group relative policy optimization (GRPO) with a verifier (math answers) scales reasoning RL without a critic model — the recipe behind DeepSeek-R1.
• DAPO (2025) — open-source large-scale RL system for LLMs: stable, reproducible scaling of RL beyond math into agentic behavior.

Key ideas

• Self-bootstrapping: the model's own good outputs become its training data (STaR loop).
• RL with outcome verification (can check the answer) is enough — no reward model needed.
• This is the "train-time" counterpart to test-time compute: both are levers.

SolOS mapping

• We can't train models — but the spirit lands in our ops: harvest successful session outputs into skills (the curator/auto-skill-extraction direction) = STaR at the skill level.
• Outcome-verifiable tasks first: alerts, brief numbers, API responses are verifiable — perfect targets for future "RL-style" auto-improvement (e.g., the MA50 engine tuning its touch threshold from realized outcomes).
• DAPO lesson: stable loops need clear verifiers — our acceptance-criteria verifier pass (note 02) is the prerequisite.