LongMemEval-S · 500 questions · 2026-09-02

Benchmarks

Where Perpetuity's measured result sits among published memory systems, with every caveat stated.

92.4

one user's history per question, the standard setting

93.0

one 59M-token memory, 500 users at once. A first.

99.1%

of the evidence reaches the reader, every message kept. 96.8% get all of it. No model on the path.

zero exclusions
~2k-token frame
recall tools on
no model in retrieval
Opus 5 with native thinking, the production prompt
GPT-4o-2024-08-06, the official judge

Efficiencywhat 200 real coding sessions billed, with and without Perpetuity

what each session billedthe same session through Perpetuity$31,666 billed, $5,075 through Perpetuity, 84% less

$1$10$100$1,000

smallest bill200 sessions, sorted by bill, log scalelargest

Every other system sends the thread history and adds its memory on top, 7k to 30k tokens where published, so what a turn costs grows with the thread, and the bill grows faster, since every turn re-sends it. Perpetuity replaces the thread history with the frame. Over these 200 sessions that is 84% fewer billed tokens, 5.6× in the median session, 16.4× for the longest, none cost more. The scale is logarithmic, so an equal saving is an equal gap. Priced with prompt caching on both sides. The baseline is what those sessions actually billed, 96.7% of it cache reads. Pricing it as if nothing were cached would overstate the saving about six times.

The frame targets about 2k tokens on a specific question. When the top of the ranking is not isolated, the sign of a broad question, the cap widens, up to 12k. The runs behind 92.4 and 93.0 did not log the frame size per question, so the line draws the target. The reader also has recall tools. When it calls one, the result is appended to that turn's context on top of the frame, per need.

Retrievalaccuracy on LongMemEval-S, 500 questions

Perpetuity
measured, 462/500, zero exclusions
92.4
Supermemory ASMR
self-declared parody
99
AgentMemory
tuned on the test set
96.2
Chronos
paper, iterative retrieval loop
95.6
OmegaMax
self-judged, best run
95.4
Mastra OM
best of 3 readers
94.9
Mem0
closed platform, reader unstated
94.4
Backboard
unofficial judge
93.4
ByteRover
excludes 30 questions
92.8
Hindsight
self-judged
91.4
Memoria
self-reported
88.8
TiMem
GPT-4o reader
79
Zep
third-party roundup
63.8
Full 115k context
anchor
59
No memory
anchor
5.1

Only the Perpetuity row is measured by us: Opus 5 with native thinking, the production prompt, GPT-4o-2024-08-06, the official judge, nothing excluded. Every other row is that vendor's self-reported number under its own conditions. The line under each name is the short form of its caveat. The full form is in the receipts.

How the number was earned

The same 500 questions at every rung. The first two are where an agent stands without memory. The rest is one memory, measured as it was built. Every point past 88.6 came from the reader, none from tuning.

5.1

No memory

The reader with nothing. This is the floor.

78 to 83

Band retrieval

The retrieval campaign, first half of 2026. Sparse bands, no dense vectors at query time.

88.6

The architecture

Chain graph and dynamic ~2k contexts, cascade deleted. Sonnet 5, thinking off.

91.2

Opus 5

Same memory, stronger reader. Every point past 88.6 came from the reader, none from tuning.

93.0

59M tokens, one memory

The same 500 questions retrieved from all 500 histories at once. Paired against 92.4: +20 / −17, p≈0.74.

Full context

GPT-4o given the whole 115k-token history. The paper's own baseline lands near 60.

87.0

First full-500 record

Needed a second-model cascade and 12k-token contexts.

90.6

Thinking on

Same memory, zero code changes. The reader's native thinking switched on.

92.4

The production prompt

The answering protocol removed. Paired against 91.2: +17 / −11, p≈0.64. What users get is what was scored.

5.1
No memory
The reader with nothing. This is the floor.
59
Full context
GPT-4o given the whole 115k-token history. The paper's own baseline lands near 60.
78 to 83
Band retrieval
The retrieval campaign, first half of 2026. Sparse bands, no dense vectors at query time.
87.0
First full-500 record
Needed a second-model cascade and 12k-token contexts.
88.6
The architecture
Chain graph and dynamic ~2k contexts, cascade deleted. Sonnet 5, thinking off.
90.6
Thinking on
Same memory, zero code changes. The reader's native thinking switched on.
91.2
Opus 5
Same memory, stronger reader. Every point past 88.6 came from the reader, none from tuning.
92.4
The production prompt
The answering protocol removed. Paired against 91.2: +17 / −11, p≈0.64. What users get is what was scored.
93.0
59M tokens, one memory
The same 500 questions retrieved from all 500 histories at once. Paired against 92.4: +20 / −17, p≈0.74.

Receipts

The reader is the model that answers. The judge scores. The memory is what is under test.

Why these numbers do not strictly compare

Only the Perpetuity row is measured by us. Every other row is that vendor's self-reported number under its own conditions.
Readers differ. Ours is stated on every row. Most vendors do not state theirs. A stronger reader lifts any memory system's score, by up to 21 points on identical memory in our own axis.
Exclusions differ. Ours: all 500 questions, abstentions included, empty answers scored as misses. Some reports drop questions or do not state the variant.
Judges differ. Ours is the benchmark's official GPT-4o judge, always. Several rows judge themselves with their own model family.
The field's top score agrees. Supermemory posted its ~99 as a parody of leaderboards that omit cost and latency. This page reports accuracy and cost together.
The token saving is priced with prompt caching on both sides. The baseline is the recorded bill of 200 real coding sessions, 96.7% of its input cache reads. The memory side is modeled: system prompt, a 2k frame and the live turn, with the system prompt cached across the session and the frame within a turn. With the memory side uncached the same sessions show a 38.5% saving and 39 of 200 cost more. Equal output and equal task success from the frame are assumed.
The reader has recall tools: a search over the whole memory, this turn's full retrieved set before the cap, and an exact count by category. A tool result is appended to that turn's context on top of the frame, so an answer that used a tool saw more than the frame. How often the tools fired was not logged in the dry runs.
Frame size was not logged per question in the 92.4 and 93.0 runs. The cap is 2k tokens on a specific question and widens to at most 12k on a broad one. The efficiency chart draws the 2k target, not a per-question measurement.

Every row's caveat, in full

Supermemory ASMR 99 Self-declared parody: a real score posted to show that leaderboard numbers without cost and latency are meaningless ("not fit for production", their words). Multi-agent search with LLM reranking. No retraction changelog.
AgentMemory 96.2 Opus 4.6 reader stated, official judge, harness published. Self-flagged: 46 tuning iterations on the test set, no held-out split.
Chronos 95.6 PwC paper. Iterative tool-calling retrieval loop. The reader is stated in the paper.
OmegaMax 95.4 95.4 is a category-averaged best run (raw 93.2). GPT-4.1 is reader and judge (identical-model self-judging, unofficial). Eight iterations on the test set targeting failure modes. No harness published.
Mastra OM 94.9 Best of three readers (gpt-5-mini) and a category-unweighted average. 84.2 with GPT-4o. Open harness, credit due. About 30k tokens of context per answer, their own figure.
Mem0 94.4 Closed platform number (top-200, about 7k tokens). Reader unstated at claim level. A third-party open-source config measures about 32%. On LoCoMo its harness drops 22.5% of questions.
Backboard 93.4 gpt-4.1 reader. Primary judge gpt-4o-mini, not the official judge. The whole 115k history sent as one message. No latency published. Per-question outputs not visible in the repo.
ByteRover 92.8 Excludes the 30 adversarial questions. Gemini 3.1 Pro reader judged by Gemini 3 Flash (in-family, unofficial). LLM curation at write.
Hindsight 91.4 Gemini-3-Pro reader, judged with its own model family on custom prompts. The number drifted 90 to 91.4 to 94.6 without a changelog.
Memoria 88.8 Self-reported. Best of three readers, judged by a model that is also one of the readers. The referenced results file is not in the public repo.
TiMem 79 Paper. GPT-4o reader. Judge id unstated.
Zep 63.8 Third-party roundup figure. Zep's own paper reports 71.2 with a gpt-4o-2024-11-20 reader, the cleanest disclosure on the table.
Full 115k context 59 GPT-4o given the whole history, no memory system. Our reproduction 59. The paper's figure 60.6. Mastra's independent replication 60.2.
No memory 5.1 Our measured floor: the reader with nothing.

One memory, seven readers

Every row shares byte-identical memory, a ~2k-token frame per answer plus whatever the reader pulls through the recall tools, and the official judge. Only the reader changes. The reader spans 21.4 points on identical memory, and our 2k selection beats the full 115k history it was distilled from at the same reader (69.8 against 60.6).

GPT-4o, the full 115k history, no memory60.6
GPT-4o + Perpetuity retrieval (~2k)69.8
GPT-4o + human-picked evidence, the paper's ceiling87
Sonnet 5 + Perpetuity88.6
Sonnet 5 + thinking + Perpetuity90.6
Opus 5 + thinking + Perpetuity, citation protocol on91.2
Opus 5 + thinking + Perpetuity, the production prompt92.4

Same answers, five judges

The 92.4 run re-judged. The official judge is always the headline. Every alternative lands within 0.2 below or 1.2 above it. Our reader is Anthropic and the official judge is OpenAI, so the grader never shares a family with the graded.

GPT-4o-2024-08-06, the official judge92.4
GPT-5.6 Sol92.2
Sonnet 5 (our reader's vendor, never crowned)92.6
Gemini 3.1 Pro93.6
Gemini 3.7 Flash93.6

What the reader was told

Each benchmark question is sent bare, with no answering instructions. The only system prompt is the production memory prompt, and on the one axis where a prompt could game the judge, ours points the other way.

Ours, the production prompt: Confidence without a source is a signal to say "I don't recall", not an answer.
Mem0's LoCoMo harness: NEVER say "not specified"... COMMIT AND ANSWER

Evidence in the frame

Retrieval recall on the same 500 questions, zero model calls on the path: 99.1% of questions have a gold evidence session in the ~2k-token frame the reader gets, 96.8% have every one of them. The gap to 92.4 is the reader's assembly, not retrieval.

Sources and runs

Perpetuity's runs: lme-chain3-*-500dry (2026-08-28 to 09-02), lme-pool57m-opus-reason-500dry (the 59M row), lme-chain3-corpus.recall-row (retrieval recall). Per-question outputs and run metadata are committed per run.