Skip to Content
remimem

Benchmark

How much context remimem saves and whether it finds the right code, measured on real commits, with the method and its limits.

The repository ships a benchmark anyone can rerun with one command and no API key. Every number on this page comes from it.

Last measured

2026-11-15, remimem 9e63d69, Apple M5 (10 cores, 16 GB), macOS, Node.js 24.20.0, embedding model Xenova/all-MiniLM-L6-v2. The README always carries the latest results file, generated by the same code.

Headline

On the 36 held-out test tasks, get_context at a 4,000-token budget against a simulated grep-and-read agent:

Baselineremimem
Tokens per task27,5293,827 (86% fewer)
Tokens per task, charging remimem's incomplete answers at the cap too27,5296,854 (75% fewer)
Recall@10 (share of a task's changed files in the top ten)0.790.93
File recall (share of changed files in the answer)60%93%
File recall, counting only files shown with a full body60%92%
Symbol recall57%74%

The project's targets were at least 40% fewer tokens and recall@10 of at least 0.80.

Method

Repositories. Three permissively licensed projects, one per language, pinned to exact commits: hono (TypeScript, 99,000 lines, MIT), httpx (Python, 33,000 lines, BSD-3-Clause), and echo (Go, 50,000 lines, MIT).

Tasks from real history. For each, the 36 most recent commits that look like a fix or a feature, touch at most eight files, and modify at least one existing non-test source file. The task text is the commit message. The ground truth is the source files the commit changed, and the symbols whose lines it changed.

No peeking. Each task is indexed at the commit before it, so the answer is not in the index.

Held out. One task in three is in the test split. Settings were tuned on the other 72 tasks only, and this page reports the test split.

The baseline. It takes up to eight keywords from the task, ranks files by how many of them each contains, and reads whole files in that order until it has read every ground-truth file or used 40,000 tokens. A task it does not finish counts as the full 40,000.

All strategies

Test split, 36 tasks:

StrategyTokens per taskFile recallFull body onlySymbol recallRecall@10MRRLatency p50 / p95
Baseline, 40,000-token cap27,52960%60%57%0.790.405 / 9 ms
search_code at 2,0001,89583%80%73%0.850.6014 / 28 ms
search_code at 4,0003,35385%85%74%0.850.6018 / 30 ms
search_code at 8,0003,91385%85%77%0.850.6020 / 30 ms
get_context at 2,0001,94387%81%67%0.930.6115 / 28 ms
get_context at 4,0003,82793%92%74%0.930.6120 / 30 ms
get_context at 8,0006,21893%93%78%0.930.6127 / 43 ms

What each part contributes

Ablations of get_context at 4,000 tokens, test split:

VariantFile recallFull body onlySymbol recallRecall@10
Full93%92%74%0.93
Keyword only80%77%68%0.88
Vector only79%76%50%0.85
No graph expansion92%92%67%0.93
No signature compression90%90%67%0.93
No name-match links93%92%71%0.93
With memories93%90%74%0.93

Neither keyword nor vector search alone comes close to the two together. The call graph adds symbols more than files. Name-match links make no measurable difference either way; the evidence does not show that they help.

Per repository

RepositoryBaseline tokensremimem tokensBaseline recall@10remimem recall@10Baseline file recallremimem file recall
hono27,6223,8660.671.0058%100%
httpx28,0723,7730.710.8371%83%
echo26,8923,8421.000.9650%96%

httpx is the weakest: two of its tasks are total misses, where the commit message names nothing found in the changed files (for example "Reintroduce URLTypes.").

Embedding models

ModelFile recallRecall@10MRRp95 latencyIndex all three repositories
Xenova/all-MiniLM-L6-v2 (default)93%0.930.6130 ms62 s
jinaai/jina-embeddings-v2-base-code92%0.930.6583 ms324 s

Memories

Each repository gets 60 memories made from older commit subjects, unlinked, so they must match the task on their own. With memories on, 2.4 are attached per task; 25% of them come from a commit that touched a ground-truth file, and 54% of tasks that have such a memory get at least one. Memory costs about two points of full-body file recall (90% against 92%), because it takes part of the budget.

What the numbers do not show

  • Tasks are commit messages. They are written after the change and often name what changed, so they are easier than a bug report written beforehand.
  • The test split is small. 36 tasks: one task moves a percentage by about three points.
  • The baseline is a simulation, not a real agent, which might search more cleverly or give up sooner.
  • One test task was inspected early. httpx cabd1c095e was looked at while the harness was being debugged, before tuning. Without it (35 tasks) every headline figure is unchanged after rounding.
  • Results vary slightly between CPU types, because the quantized model rounds differently.

Reproduce it

From a clone of the repository:

npm install
npx . bench

It clones the three repositories into ~/.cache/remimem/bench (about 150 MB), indexes each task's parent commit, and runs every strategy: 5 to 10 minutes. It writes bench/results/<date>-<commit>.json with every task's raw result. To check your run against the published one, within 0.05 on recall figures and 15% on tokens:

npx . bench --compare bench/results/2026-11-15-9e63d69.json

--limit 3 runs a quick sample, --tune adds the tuning configurations on the training split, and --model tries another embedding model. See the CLI reference.

On this page