← writing

A lesson without its outcome

1 min read

I keep a lessons file in most of my projects. When something goes wrong, the session ends with a short note: what happened and what fixed it, so the next session doesn’t repeat it.

A new paper from Google, Self-Improvement Memory Enables Continual Recursive Self-Improvement, tests something close to that. An optimizer, itself an LLM, keeps editing an agent’s harness, the code around the model: its prompts, tools and checks. Then it has to carry on with a new benchmark or a new model. The question is what it should remember from the earlier rounds.

They compare three options: keep only the evolved harness; keep the harness plus the problems that came up; or keep a fuller memory that groups edits by recurring problem and records why each edit was made, what it was predicted to change, what actually changed, and where it was evaluated.

Keeping only the problems gave mixed results, from 1.82 points worse than the harness alone to 2.40 points better. The fuller memory came out ahead in all six settings where the benchmark changed, and when moving from Claude Opus to Gemini it beat the harness alone by 3.60 to 11.33 points. Most of these are single runs per setting.

The paper also has an example worth knowing. A run without that memory inherited a completion check, edited it four times, twice to fix deadlocks, and along the way described the check as something it had added itself.

My lessons files sit somewhere between the last two: the problem and the fix. What’s missing is the prediction and the evidence: “this should stop X”, and later, whether X actually stopped.

I haven’t changed anything yet. Next I want to go through the paper more carefully and see what could turn into new additions to my own projects, starting with how my lessons are written down.