A self-improving educational-content engine. An agent grounds lesson generation in validated context packs, scores it with a hybrid eval harness, then rewrites the system itself — its own prompts, its reward weights, and new check-tools it designs — keeping only changes that beat a frozen north-star without regressing a held-out set.
Most "AI content" pipelines generate once and stop — quality is whatever the prompt happened to produce, and "improvement" means a human editing the prompt by hand. The harder, more valuable system is one that improves itself: it generates, measures itself against an explicit rubric, reasons about its own failures, and changes its own machinery — then proves the change helped on data it has never optimized against. The danger in building that is Goodhart's law: an agent allowed to edit its own objective will "improve" by moving the goalposts. Lesson Loom is built against that from the first commit — a frozen north-star the agent cannot touch gates every promotion, while a separate, mutable reward is the only objective it is allowed to reshape. The result is a self-improving loop whose every step is attributable, gated, and reproducible.
Each subject is a Pydantic-validated knowledge artifact — facts with stable citation ids, learning objectives, common misconceptions, and a target reading band. It is both what grounds generation and the source of truth for citation checking, so "is this fact real?" is a lookup, not a vibe. Science (photosynthesis, grade 5) and history (industrial revolution, high school) cover deliberately different domains and reading levels.
A StateGraph runs plan → retrieve → draft → self-critique ⇄ revise → format, emitting a content artifact whose claims carry [fact_*] / [obj_*] citation tags. Compiled without a checkpointer and written as pure functions of the input state slice, so two runs are byte-identical — determinism is a guarantee a test enforces, not a hope.
Eight scorers, each normalized to [0,1]: five deterministic (Flesch-Kincaid readability vs band, factual grounding, objective coverage, format validity, no-hallucination) and three LLM-judge (pedagogical soundness, clarity, engagement). The north-star is a fixed weighting over the base set — the metric the gate and the headline KPI use, and the one object the agent can never edit.
Per round: rank the criterion gaps from the failures, propose exactly ONE change (a prompt edit, a reward reshape, or a synthesized tool) with a one-line evidence-based rationale, re-evaluate on train + held-out, then apply a regression-gated promotion — promote only if the frozen north-star rises ≥0.02 and the held-out set doesn't regress. Reject → backtrack to best-ever. Each round logs a Goodhart gap (train-delta − held-out-delta).
When a weakness can't be fixed by prompting, the agent synthesizes a new deterministic check from a sandboxed template library (min-count, keyword-presence, threshold), deriving the parameters from the failure corpus — and keeps it only if it survives the same promotion gate as a prompt edit. This is the literal 'designed its own tools' bar, made auditable.
Every round — parent, candidate (by content-hash id), axis, target, rationale, promote/reject, per-split scores — lands in a normalized SQLite store you can query ("rounds where a prompt changed and held-out improved"). The best config is evaluated once on an untouched TEST split for the headline number. A rich CLI prints the leaderboard and writes REPORT.md; a thin FastAPI console inspects artifacts and runs.
Two distinct objects. The agent freely reshapes its own reward weights to steer the search, but the promotion gate and the KPI use a north-star it cannot edit — so it can't win by moving the goalposts. This mirrors how RLHF separates the learned reward from the eval benchmark. Synthesized tools never count toward the north-star either.
A frozen rubric closes weight-gaming but not Goodhart in general — a draft can satisfy the rubric yet be pedagogically weak in ways it doesn't capture. A production north-star needs human raters and adversarial items; the architecture leaves room for exactly that.
Every candidate differs from its parent on exactly one generative axis and carries an evidence-based reason ("60% of items failed the citation criterion → add a cite-facts directive to the draft prompt"). The lineage becomes a clean attribution tree — the difference between "the score went up" and "the system reasoned about its own failure and fixed it."
Greedy single-axis hill-climbing explores slower than multi-lever or beam search. Chosen deliberately — an explainable, reproducible lineage is worth more here than a marginally faster, unattributable one.
The mock provider responds to the structure of a prompt — does it ask for a reading level? citations? objective coverage? — not to magic strings. So prompt self-edits genuinely move the deterministic scores and the gains generalize across items. The whole loop runs offline, deterministically, in seconds; flipping to Claude (sonnet-4-6 generation, opus-4-8 judge) is one flag behind the same interface.
Mock mode is an honest simulation of the causal chain, not evidence of real-model pedagogy. The Claude path is where real content quality is judged; the mock path is what makes the loop instantly runnable and CI-testable.
Train (the optimizer sees it), held-out (the promotion gate; leaks only via accept/reject), and test (never used in search or gating, reported once). It's the discipline that separates a real eval harness from a demo that overfits its own metric.
Small splits mean the test number is directional, not statistically powered. The contribution is the protocol — train/held-out/test + regression-gated promotion + auditable lineage — which scales to larger suites unchanged.