LongMemEval-S · 500 questions · 2026-09-02

Standings

Where Perpetuity's measured result sits among published memory systems, with the caveats stated instead of hidden.

92.4

one user's history per question, the standard setting

one 59M-token memory, 500 users at once. A first.

99.1%

of the evidence reaches the reader, every message kept. 96.8% get all of it. No model on the path.

zero exclusions
~2k-token frame
no model in retrieval
Opus 5 with native thinking, the production prompt
GPT-4o-2024-08-06, the official judge

Efficiencycontext sent per turn, as the history grows

Every other system sends the history and adds its memory on top, so what a turn costs grows with the history, and the bill grows faster, since every turn re-sends it. Perpetuity replaces the history with the frame. Measured across 200 real coding sessions: 84% fewer billed tokens, 5.6× in the median session, 16.4× for the longest, none cost more.

Retrievalaccuracy on LongMemEval-S, 500 questions

Perpetuity
measured, 462/500, zero exclusions
92.4
Supermemory ASMR
self-declared parody
99
AgentMemory
tuned on the test set
96.2
Chronos
paper, iterative retrieval loop
95.6
OmegaMax
self-judged, best run
95.4
Mastra OM
best of 3 readers
94.9
Mem0
closed platform, reader unstated
94.4
Backboard
unofficial judge
93.4
ByteRover
excludes 30 questions
92.8
Hindsight
self-judged
91.4
Memoria
self-reported
88.8
TiMem
GPT-4o reader
79
Zep
third-party roundup
63.8
Full 115k context
anchor
59
No memory
anchor
5.1

Only the Perpetuity row is measured by us: Opus 5 with native thinking, the production prompt, GPT-4o-2024-08-06, the official judge, nothing excluded. Every other row is that vendor's self-reported number under its own conditions. The line under each name is the short form of its caveat; the full form is in the receipts.

How the number was earned

The same 500 questions at every rung. The first two are where an agent stands without memory; the rest is one memory, measured as it was built. Every point past 88.6 came from the reader, none from tuning.

5.1

No memory

The floor: the reader with nothing.

78 to 83

Band retrieval

The retrieval campaign, first half of 2026. Sparse bands, no dense vectors at query time.

88.6

The architecture

Chain graph and dynamic ~2k contexts, cascade deleted. Sonnet 5, thinking off.

91.2

Opus 5

Same memory, stronger reader. Every point past 88.6 came from the reader, none from tuning.

93.0

59M tokens, one memory

The same 500 questions retrieved from all 500 histories at once. Paired against 92.4: +20 / −17, p≈0.74.

Full context

GPT-4o given the whole 115k-token history. The paper's own baseline lands near 60.

87.0

First full-500 record

Needed a second-model cascade and 12k-token contexts.

90.6

Thinking on

Same memory, zero code changes. The reader's native thinking switched on.

92.4

The production prompt

The answering protocol removed. Paired against 91.2: +17 / −11, p≈0.64. What users get is what was scored.

5.1
No memory
The floor: the reader with nothing.
59
Full context
GPT-4o given the whole 115k-token history. The paper's own baseline lands near 60.
78 to 83
Band retrieval
The retrieval campaign, first half of 2026. Sparse bands, no dense vectors at query time.
87.0
First full-500 record
Needed a second-model cascade and 12k-token contexts.
88.6
The architecture
Chain graph and dynamic ~2k contexts, cascade deleted. Sonnet 5, thinking off.
90.6
Thinking on
Same memory, zero code changes. The reader's native thinking switched on.
91.2
Opus 5
Same memory, stronger reader. Every point past 88.6 came from the reader, none from tuning.
92.4
The production prompt
The answering protocol removed. Paired against 91.2: +17 / −11, p≈0.64. What users get is what was scored.
93.0
59M tokens, one memory
The same 500 questions retrieved from all 500 histories at once. Paired against 92.4: +20 / −17, p≈0.74.

Receipts

The reader is the model that answers. The judge scores. The memory is what is under test.

Why these numbers do not strictly compare

Only the Perpetuity row is measured by us. Every other row is that vendor's self-reported number under its own conditions.
Readers differ. Ours is stated on every row; most vendors do not state theirs. A stronger reader lifts any memory system's score, by up to 21 points on identical memory in our own axis.
Exclusions differ. Ours: all 500 questions, abstentions included, empty answers scored as misses. Some reports drop questions or do not state the variant.
Judges differ. Ours is the benchmark's official GPT-4o judge, always. Several rows judge themselves with their own model family.
The field's top score agrees: Supermemory posted its ~99 as a parody of leaderboards that omit cost and latency. This page reports accuracy, cost and time together.

Every row's caveat, in full

Supermemory ASMR 99 Self-declared parody: a real score posted to show that leaderboard numbers without cost and latency are meaningless ("not fit for production", their words). Multi-agent search with LLM reranking. No retraction changelog.
AgentMemory 96.2 Opus 4.6 reader stated, official judge, harness published. Self-flagged: 46 tuning iterations on the test set, no held-out split.
Chronos 95.6 PwC paper. Iterative tool-calling retrieval loop; the reader is stated in the paper.
OmegaMax 95.4 95.4 is a category-averaged best run (raw 93.2). GPT-4.1 is reader and judge (identical-model self-judging, unofficial). Eight iterations on the test set targeting failure modes. No harness published.
Mastra OM 94.9 Best of three readers (gpt-5-mini) and a category-unweighted average; 84.2 with GPT-4o. Open harness, credit due. About 30k tokens of context per answer, their own figure.
Mem0 94.4 Closed platform number (top-200, about 7k tokens). Reader unstated at claim level. A third-party open-source config measures about 32%. On LoCoMo its harness drops 22.5% of questions.
Backboard 93.4 gpt-4.1 reader; primary judge gpt-4o-mini, not the official judge. The whole 115k history sent as one message. No latency; per-question outputs not visible in the repo.
ByteRover 92.8 Excludes the 30 adversarial questions. Gemini 3.1 Pro reader judged by Gemini 3 Flash (in-family, unofficial). LLM curation at write.
Hindsight 91.4 Gemini-3-Pro reader; judged with its own model family on custom prompts. The number drifted 90 to 91.4 to 94.6 without a changelog.
Memoria 88.8 Self-reported. Best of three readers, judged by a model that is also one of the readers. The referenced results file is not in the public repo.
TiMem 79 Paper. GPT-4o reader; judge id unstated.
Zep 63.8 Third-party roundup figure. Zep's own paper reports 71.2 with a gpt-4o-2024-11-20 reader, the cleanest disclosure on the table.
Full 115k context 59 GPT-4o given the whole history, no memory system. Our reproduction 59; the paper's figure 60.6; Mastra's independent replication 60.2.
No memory 5.1 Our measured floor: the reader with nothing.

One memory, seven readers

Every row shares byte-identical memory, ~2k tokens per answer, and the official judge. Only the reader changes. The reader spans 21.4 points on identical memory, and our 2k selection beats the full 115k history it was distilled from at the same reader (69.8 against 60.6).

GPT-4o, the full 115k history, no memory60.6
GPT-4o + Perpetuity retrieval (~2k)69.8
GPT-4o + human-picked evidence, the paper's ceiling87
Sonnet 5 + Perpetuity88.6
Sonnet 5 + thinking + Perpetuity90.6
Opus 5 + thinking + Perpetuity, citation protocol on91.2
Opus 5 + thinking + Perpetuity, the production prompt92.4

Same answers, five judges

The 92.4 run re-judged. The official judge is always the headline; every alternative lands within 0.2 below or 1.2 above it. Our reader is Anthropic and the official judge is OpenAI, so the grader never shares a family with the graded.

GPT-4o-2024-08-06, the official judge92.4
GPT-5.6 Sol92.2
Sonnet 5 (our reader's vendor, never crowned)92.6
Gemini 3.1 Pro93.6
Gemini 3.7 Flash93.6

What the reader was told

Each benchmark question is sent bare, with no answering instructions. The only system prompt is the production memory prompt, and on the one axis where a prompt could game the judge, ours points the other way.

Ours, the production prompt: Confidence without a source is a signal to say "I don't recall", not an answer.
Mem0's LoCoMo harness: NEVER say "not specified"... COMMIT AND ANSWER

Evidence in the frame

Retrieval recall on the same 500 questions, zero model calls on the path: 99.1% of questions have a gold evidence session in the ~2k-token frame the reader gets, 96.8% have every one of them. The gap to 92.4 is the reader's assembly, not retrieval.

Sources and runs

Perpetuity's runs: lme-chain3-*-500dry (2026-08-28 to 09-02), lme-pool57m-opus-reason-500dry (the 59M row), lme-chain3-corpus.recall-row (retrieval recall). Per-question outputs and run metadata are committed per run.