Benchmarks

• Only the Perpetuity row is measured by us. Every other row is that vendor's self-reported number under its own conditions.
• Readers differ. Ours is stated on every row. Most vendors do not state theirs. A stronger reader lifts any memory system's score, by up to 22.6 points on identical memory in our own axis.
• Exclusions differ. Ours: all 500 questions, abstentions included, empty answers scored as misses. Some reports drop questions or do not state the variant.
• Judges differ. Ours is the benchmark's official GPT-4o judge, always. Several rows judge themselves with their own model family.
• Supermemory posted its ~99, the field's top score, as a parody of leaderboards that omit cost and latency. This page reports accuracy and cost together.
• Every token figure here is in Claude's tokenizer, the unit the bill is in. Our internal capture counter runs about 0.7 of it.
• Correction, 1 October 2026. The coding costs on this site were first priced with the previous Opus model's rates. The sessions ran on Claude Opus 5.5, which is cheaper, most of all for cache reads, and the side without Perpetuity is mostly cache reads. At the correct list prices the saving over 120 prompts is 52%, where we had published 65%, and over 60 prompts 48%, where we had published 63%. Token counts, context sizes and recall scores did not change.
• The coding cost is measured on sessions run twice at the same time, with and without Perpetuity, on the same model, Claude Opus 5.5, with Claude Code on its default 1M-token window. Each session is priced in dollars at that model's list prices ($4 input, $20 output, $0.20 cache read and $8 cache write per million tokens), every call counted, as Claude Code's own cost readout totals it. 120 prompts: $50.24 against $104.71, 52% less, and 64% less in prompts 61 to 90. 60 prompts: $26.40 against $50.93, 48% less. Two 30-prompt pairs came out 11% and 35% less. On three other projects, 200 prompts each, summed from each prompt's own usage at the same list prices, a Rust command-line tool came out 38% less, a Go command-line tool 27% less and a Flask application 4% less, which we count as a tie. These are the runs after the fixes we made from our first runs, and in them both sides ended with every test passing. On the Go and Flask pairs Claude Code on its own wrote more tests to reach about the same coverage. Seven pairs of runs show the direction, not a precise average.
• On a smaller window the cost gap narrows. The same 120 prompts with Claude Code capped at a 200k-token window compacted 17 times and cost $57.05, so Perpetuity's $50.24 is 12% less, and Perpetuity cost more over the first 60 prompts. On the same 16 recall questions, graded blind in one packet, Perpetuity answered 15.5, the default-window run 14 and the 200k-window run 12.5. One run per setup.
• Claude Code compacted its history once in the 120-prompt session, during prompt 82, at a 965k-token context. Its cost per prompt fell after that, because the history it threw away is no longer paid for. Asked 16 pre-registered questions about work from across the session and graded blind, it answered 14 and Perpetuity answered 15.5, with the difference in the first half of the session. In the 60-prompt session the result was 13 of 13 against 11. On the two 200-prompt pairs on other projects, asked the same way and graded blind, the Flask session went to Claude Code on its own, 14.5 against 11, the Go session to Perpetuity, 12 against 10, and the Rust session to Perpetuity, 13 against 12, with Claude Code compacting its history once, four times and four times. Pooled over the five runs, 64.5 of 77 against 61.5: 84% against 80%. Where Perpetuity lost points it mostly declined to state a detail of the code (a hash formula, a column list, a weight) while keeping the names, values and messages right. Where Claude Code lost them it had no memory of the earliest work before a compaction. The Perpetuity side starts every prompt at about 46k tokens per call, from prompt 10 to prompt 120, while the other side reached 965k before compacting. Within one prompt its context rises with that prompt's own file reads and tool calls, up to 165k in a 25-call refactor, and starts again at 46k on the next prompt.
• The 192-session chart is an estimate. The recorded usage of 192 of our own Claude Code sessions is priced as it happened, each call at the list prices of the model it ran on (mostly Opus 4.8 and Fable 5). The same sessions through Perpetuity are modeled with a 10k-token frame and 1.3 times the calls, output equal. The model was checked against an earlier 30-prompt pair: within 2% at the settings that pair ran, 11% off at 1.3 times the calls. It gives about 60% less, and 3 sessions cost more. On Claude Opus 5.5, whose cache reads are cheaper, the saving is smaller, as the measured sessions show. 183 of the 192 sessions run past 400k tokens of context. Short sessions with small contexts save little or nothing.
• The reader has recall tools: a search over the whole memory, the rest of this turn's retrieval below the cut, an exact ledger by category, the calendar dates of matching moments, an association walk and a table of contents. A tool result is appended to that turn's context on top of the frame. On the final record the tools ran on 248 of 500 questions, 1.72 rounds per question on average, and half the questions were answered in one round.
• The frame is measured on every question of the 94.0 run with the reader's tokenizer: 3.6k tokens at the median, 4.3k at the ninetieth percentile. Earlier runs did not log it.
• The pooled row asks each question against one memory holding all 500 users' histories, with the asking user's own items given priority in the ranking: an item from another user's history scores at two thirds. Without that priority the frame fills with other users' text and the score falls to about 83, measured on 161 of the questions. The priority is a dial between a private memory and a communal one, and the frame stops changing once it is past 1.5. It is a test of retrieval at that size, not a mode users get yet. Perpetuity ships one memory per agent. The benchmark also shares 4,396 of its 18,325 sessions across its 500 users, so a hit from another user's history is often the same text under another name, which makes this pool easier than 500 truly different users would be.
• One full-context turn of the same question's history is 165k tokens. Against it a question through Perpetuity bills about 9k at the median, about the same in dollars as a history kept entirely in cache. The saving over a session comes from the history growing every turn while the frame does not.

• Supermemory ASMR 99 Self-declared parody: a real score posted to show that leaderboard numbers without cost and latency are meaningless ("not fit for production", their words). Multi-agent search with LLM reranking. No retraction changelog.
• AgentMemory 96.2 Opus 4.6 reader stated, official judge, harness published. Self-flagged: 46 tuning iterations on the test set, no held-out split.
• Chronos 95.6 PwC paper. Iterative tool-calling retrieval loop. The reader is stated in the paper.
• OmegaMax 95.4 95.4 is a category-averaged best run (raw 93.2). GPT-4.1 is reader and judge (identical-model self-judging, unofficial). Eight iterations on the test set targeting failure modes. No harness published.
• Mastra OM 94.9 Best of three readers (gpt-5-mini) and a category-unweighted average. 84.2 with GPT-4o. Harness published. About 30k tokens of context per answer, their own figure.
• Mem0 94.4 Closed platform number (top-200, about 7k tokens). Reader unstated at claim level. A third-party open-source config measures about 32%. On LoCoMo its harness drops 22.5% of questions.
• Backboard 93.4 gpt-4.1 reader. Primary judge gpt-4o-mini, not the official judge. The whole 115k history sent as one message. No latency published. Per-question outputs not visible in the repo.
• ByteRover 92.8 Excludes the 30 adversarial questions. Gemini 3.1 Pro reader judged by Gemini 3 Flash (in-family, unofficial). LLM curation at write.
• Hindsight 91.4 Gemini-3-Pro reader, judged with its own model family on custom prompts. The number drifted 90 to 91.4 to 94.6 without a changelog.
• Memoria 88.8 Self-reported. Best of three readers, judged by a model that is also one of the readers. The referenced results file is not in the public repo.
• TiMem 79 Paper. GPT-4o reader. Judge id unstated.
• Zep 63.8 Third-party roundup figure. Zep's own paper reports 71.2 with a gpt-4o-2024-11-20 reader.
• Full 115k context 59 GPT-4o given the whole history, no memory system. Our reproduction 59. The paper's figure 60.6. Mastra's independent replication 60.2.
• No memory 5.1 Our measured floor: the reader with nothing.

Every row uses the official judge. The first reads the whole history and the third reads human-picked evidence, as reference points. The five rows from 69.8 to 92.4 read the same Perpetuity memory, and only the reader or its prompt changes: the reader alone spans 22.6 points. The last row is the shipped configuration, with the frame sized per question. At the same reader, our selection beats the full 115k history it was distilled from (69.8 against 60.6).

• GPT-4o, the full 115k history, no memory60.6
• GPT-4o + Perpetuity retrieval69.8
• GPT-4o + human-picked evidence, the paper's ceiling87
• Sonnet 5 + Perpetuity88.6
• Sonnet 5 + thinking + Perpetuity90.6
• Opus 5 + thinking + Perpetuity, citation protocol on91.2
• Opus 5 + thinking + Perpetuity, the production prompt92.4
• Opus 5 + thinking + Perpetuity, the measured frame94

The 92.4 run re-judged. The official judge is always the headline. Every alternative lands within 0.2 below or 1.2 above it. Our reader is Anthropic and the official judge is OpenAI, so the grader never shares a family with the graded.

• GPT-4o-2024-08-06, the official judge92.4
• GPT-5.6 Sol92.2
• Sonnet 5 (our reader's vendor, never crowned)92.6
• Gemini 3.1 Pro93.6
• Gemini 3.7 Flash93.6

Each benchmark question is sent bare, with no answering instructions. The only system prompt is the production memory prompt, and on the one axis where a prompt could game the judge, ours points the other way.

Ours, the production prompt

Confidence without a source is a signal to say "I don't recall", not an answer.

Mem0's LoCoMo harness

NEVER say "not specified"... COMMIT AND ANSWER

Retrieval recall on the 470 questions that have evidence, zero model calls on the path: 97.4% of questions have a gold evidence session in the frame the reader sees first, 89.8% have every one of them. One recall_more call away, 99.1% and 96.4%. With the evidence this often in reach, the gap to 94.0 most likely lies in how the reader puts it together, not in retrieval.

The control is plain BM25 over the same corpora, one document per turn, no model, at the same cut. This benchmark's questions name their targets, so on them it is level with ours. We then rewrote all 500 questions once so that names became descriptions (the share of a question's words found in its evidence fell from 0.744 to 0.49) and ran both again. They differ only on the low-overlap quarter, and our claim over lexical retrieval stops there.

On the full frame BM25 reaches 100% and 98.1% on the original questions. On questions that repeat the evidence's words, it ranks the evidence higher.

These memory systems run without a model, as ours does, and we found no result for them on this benchmark. A row needs a number from the same test, so they are named here instead of scored.

• Holographic (Hermes Agent provider): model-less, SQLite full-text plus an algebraic layer.
• Vestige: model-less by default, BM25 over FTS5.
• OKF Agent Memory: model-less, local BM25 over hierarchical index files.
• minirag: model-less, BM25 over markdown.
• Files with grep, CLAUDE.md-style memory files: model-less, the simplest form.

• LongMemEval (ICLR 2025)
• Mastra OM research
• Hindsight paper
• Supermemory research
• Mem0 research
• Mem0 paper (latency)
• Zep paper
• AgentMemory
• Chronos
• OmegaMax benchmarks
• ByteRover
• Backboard
• Penfield Labs benchmark audit
• LongMemEval-V2

Perpetuity's runs: lme-chain3-record-final-z2-cap2k-500 (the 94.0 record, verified), lme-chain3-plain-opus-reason-500dry-clean (the clean 92.8 re-run of the old configuration), lme-pool57m-lambda15-z2-cap2k-500 (the pooled row, verified), lme-chain3-corpus.recall-row-served (retrieval recall on the served frame), bm25-control.turn and bm25-control.paraphrased.turn (the lexical control, original and rewritten questions), lme-chain3-corpus.recall-row-paraphrased (our retrieval on the rewritten questions). Per-question outputs and run metadata are kept per run and available on request.