Research Index — Self-Improving AI Agents

Studied from Stanford CS329A (Autumn 2025) — https://cs329a.stanford.edu/ Compiled by Min for the SolOS fleet, 2026-08-07. Each note: what → why → key idea → results → SolOS mapping.

Notes

1. Test-time compute scaling — repeated sampling, optimal compute, inference-time architecture search → 01-test-time-compute
2. Robust verification (PRMs) — verifier models, step-by-step reward → 02-verifiers
3. Reasoning + acting + feedback — ReAct, execution-feedback RL, Constitutional AI → 03-reasoning-acting-feedback
4. Planning & search agents — LATS, SPRINT, ADaPT, Search-o1, AlphaCode → 04-planning-search
5. Train-time scaling / RL — STaR, DeepSeekMath, DAPO → 05-train-time-rl
6. Self-improving agents — The AI Scientist, AlphaEvolve, CodeMonkeys → 06-self-improving-agents
7. Memory architectures — MemGPT, Cartridges, CacheBlend → 07-memory
8. The Reflection Loop — Reflexion, Quiet-STaR, Self-Rewarding Models → 08-reflexion
9. 2026 update — what changed since the study — scaling plateau + verification gap, generative PRMs, RSI demonstrated, reflection caveat confirmed → 09-2026-update

Fleet review (Echo + Sol, 2026-08-07)

Both agents reviewed the notes. Consensus and final adoption priority:

1. Verifier pass (Sol: highest ROI + prerequisite for everything; Echo: builds trust) — build FIRST
2. Execution-feedback → skill loop (Echo: biggest visible quality signal) — needs one process rule
3. Memory tiering rules (Sol: lowest effort — hierarchy exists, add promotion rules)
4. Search-enhanced Ask AI (both: accept; Sol: web results need their own verifier gate)
5. Test-time compute (Sol: scope to critical-path/client-facing only — cost)
6. Auto-skill extraction (both: long play — needs #1 + eval suite first)

Echo's additions: Reflexion, CoT (2201.11903), ToT (2305.10601), Quiet-STaR, SWE-bench, GAIA, Voyager, Self-Rewarding Models. Constitutional AI caveat applied (note 03, 08). Sol's risk flags: skill bloat (add dedup/consolidation), weak-verifier tuning (precision-first), cost creep (gate on task confidence).

Fleet adoption shortlist (from the study)

• Verifier pass (note 2) — cheap-model check on every deliverable before sign-off
• Execution-feedback loop (note 3) — failures → skill patches same-day
• Search-enhanced Ask AI (note 4) — tool-use + retrieval before answering
• Memory tiering (note 7) — STM→LTM promotion rules
• Test-time compute (note 1) — sample-2-verify-ship for high-stakes briefs
• Eval suite (all) — weekly regression for fleet outputs

Full reading list (27 papers, abstracts captured)

• 2110.14168 — 2110.14168 Training Verifiers to Solve Math Word Problems
• 2210.03629 — 2210.03629 ReAct: Synergizing Reasoning and Acting in Language Models
• 2212.08073 — 2212.08073 Constitutional AI: Harmlessness from AI Feedback
• 2305.20050 — 2305.20050 Let's Verify Step by Step
• 2310.04406 — 2310.04406 Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
• 2310.08560 — 2310.08560 MemGPT: Towards LLMs as Operating Systems
• 2311.05772 — 2311.05772 ADaPT: As-Needed Decomposition and Planning with Language Models
• 2312.08935 — 2312.08935 Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
• 2402.03300 — 2402.03300 DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• 2405.16444 — 2405.16444 CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
• 2407.21787 — 2407.21787 Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• 2408.03314 — 2408.03314 Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• 2408.06292 — 2408.06292 The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• 2409.15254 — 2409.15254 Archon: An Architecture Search Framework for Inference-Time Techniques
• 2410.02089 — 2410.02089 RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
• 2410.15625 — 2410.15625 Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces
• 2501.14723 — 2501.14723 CodeMonkeys: Scaling Test-Time Compute for Software Engineering
• 2502.17578 — 2502.17578 How Do Large Language Monkeys Get Their Power (Laws)?
• 2503.04412 — 2503.04412 Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search
• 2503.14476 — 2503.14476 DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• 2503.14499 — 2503.14499 Measuring AI Ability to Complete Long Software Tasks
• 2504.04736 — 2504.04736 Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use
• 2506.05745 — 2506.05745 SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models
• 2506.06266 — 2506.06266 Cartridges: Lightweight and general-purpose long context representations via self-study
• 2506.18203 — 2506.18203 Shrinking the Generation-Verification Gap with Weak Verifiers
• 2508.20033 — 2508.20033 DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
• 2510.04374 — 2510.04374 GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

Echo's additions (2026-08-07) — abstracts captured

• 2201.11903 — 2201.11903 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• 2303.11366 — 2303.11366 Reflexion: Language Agents with Verbal Reinforcement Learning
• 2305.10601 — 2305.10601 Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• 2305.16291 — 2305.16291 Voyager: An Open-Ended Embodied Agent with Large Language Models
• 2310.06770 — 2310.06770 SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• 2311.12983 — 2311.12983 GAIA: a benchmark for General AI Assistants
• 2401.01335 — 2401.01335 Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
• 2403.09629 — 2403.09629 Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking