We ask our agents to summarize their own memory so they don’t blow past the context window. A paper this year from UC Irvine argues that this summarizing is a slow leak. Every time an agent compresses what it knows, it loses a little information, and that loss compounds across turns.

The idea that organizes the whole thing is old. In 1959, Shannon formalized rate-distortion theory: given a budget of bits, there is an optimal amount of information you can keep, and below what the task actually needs it is impossible not to make mistakes. Deciding what to keep and what to drop from a model’s memory is exactly that problem.

One problem in four disguises

The paper’s first move is to notice that four separate research communities are all solving the same rate-distortion decision without citing each other:

KV-cache compression   drop or quantize attention keys/values
Prompt compression     prune or rewrite tokens before the model reads them
Bounded architectures  squeeze the past into a fixed-size recurrent state
Agent memory           summarize and consolidate a trajectory

Each takes the history H of everything the model has read and produces a smaller representation Z that fits a budget B, then runs on Z instead of the original. Stated or not, every method is minimizing the same thing: keep Z small while giving up as little downstream accuracy as possible.

Two properties decide whether a scheme fails.

Reversibility. Can content that was dropped be re-derived if a later query needs it? A retrieval-backed store can go get it. A summary cannot: once the text is gone, it is gone.

Query-conditioning. Does the compactor get to see the query before it decides what to keep? If it decides blind, it has to spread its budget across every question it might be asked. The paper puts a number on that cost. It is H(Q), the entropy of the query. Deciding what to forget before you know what you will be asked charges you exactly that many extra bits.

The experiment that matters

Most benchmarks compress a context once and measure accuracy once. That single-turn view is blind to the regime agents actually live in, where memory is compacted again and again over a long task.

The paper’s Prediction 4 is what happens under repeated compaction. An irreversible operator (summarize) sees its end-task error grow super-linearly in the number of compaction events. A reversible operator (retrieve) stays flat. And here is the trap: under a single-turn test the two look identical. They only separate when compaction repeats.

So I reproduced it.

A small proof of concept

The setup is deliberately simple. An agent holds 40 facts, each a room and its access code. It compacts its working memory 25 times, under two operators:

• irreversible (summarize) re-summarizes the previous, already-lossy summary under a word budget. Any code it drops is gone for good, and the loss compounds.
• reversible (retrieve) keeps the facts verbatim in an external store and fetches them on demand.

After each compaction event I measure recall objectively. A fact counts only if its exact code still survives in the text the agent would read. Nothing is hardcoded into the number:

def recall(working_memory_text, facts):
    """Fraction of facts whose exact code still appears in the text."""
    hits = sum(1 for c in facts.values()
               if re.search(rf"\b{c}\b", working_memory_text))
    return hits / len(facts)

def run_irreversible(facts, events, word_budget, summarize):
    wm = facts_as_text(facts) + "\n" + FILLER
    curve = [recall(wm, facts)]              # before any compaction
    for _ in range(events):
        wm = summarize(wm, word_budget) + "\n" + FILLER   # agent keeps working
        curve.append(recall(wm, facts))      # loss compounds on the prev summary
    return curve

The reversible operator never overwrites the store, so nothing compounds and recall stays near its retrieval ceiling. The irreversible one feeds each summary back into the next, so every drop is permanent.

Here is the result. Watch where the single-pass test would stop looking, and where the two operators actually diverge:

At one compaction both operators recall about 95%. Your unit test is green, both strategies look fine. Twenty-five compactions later the reversible operator still holds 95% while the summarizing one has fallen to 45%. It quietly forgot more than half of what it knew, and no single-turn evaluation would have caught it.

Run it yourself

The full experiment, the two operators, the objective recall measurement, and the code that renders the chart above, is on GitHub in my dojo repo: moisesvw/dojo/agents/rate-distortion-memory.

It has three backends. By default it runs a deterministic model of the mechanism, which needs no API key and is fully reproducible. This is the same honesty the paper applies to its own reference experiment: the absolute numbers illustrate the trend, and the recall measurement is real. The same script accepts a live LLM if you give it a key:

git clone https://github.com/moisesvw/dojo
cd dojo/agents/rate-distortion-memory

python3 compaction_experiment.py --backend sim         # deterministic, no key
python3 compaction_experiment.py --backend openai --model gpt-4o-mini
python3 compaction_experiment.py --backend anthropic   # claude-haiku-4-5
python3 build_video.py                                  # renders the chart above

The takeaway

The most useful way to read this paper is not as a survey of compression tricks. It is a single claim with a price tag. If your agent decides what to forget before it knows the question, it pays H(Q). If it forgets irreversibly and then forgets again on top of that, the error compounds instead of staying local to one turn.

That reframes a design choice most agent stacks make by default. Summarizing the scratchpad to save tokens feels free because the next turn still works. The cost only shows up many compactions later, as facts that silently stopped being there. A reversible, retrieval-backed memory pays storage and bandwidth up front and keeps the door open. Which is the better trade depends on your task, but it should be a decision, not an accident.

Paper: What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents (Colaco and Lahjouji, UC Irvine, 2026).