Benchmark
How much context remimem saves and whether it finds the right code, measured on real commits, with the method and its limits.
The repository ships a benchmark anyone can rerun with one command and no API key. Every number on this page comes from it.
Last measured
2026-11-15, remimem 9e63d69, Apple M5 (10 cores, 16 GB), macOS, Node.js 24.20.0, embedding model Xenova/all-MiniLM-L6-v2. The README always carries the latest results file, generated by the same code.
Headline
On the 36 held-out test tasks, get_context at a 4,000-token budget against a simulated grep-and-read agent:
| Baseline | remimem | |
|---|---|---|
| Tokens per task | 27,529 | 3,827 (86% fewer) |
| Tokens per task, charging remimem's incomplete answers at the cap too | 27,529 | 6,854 (75% fewer) |
| Recall@10 (share of a task's changed files in the top ten) | 0.79 | 0.93 |
| File recall (share of changed files in the answer) | 60% | 93% |
| File recall, counting only files shown with a full body | 60% | 92% |
| Symbol recall | 57% | 74% |
The project's targets were at least 40% fewer tokens and recall@10 of at least 0.80.
Method
Repositories. Three permissively licensed projects, one per language, pinned to exact commits: hono (TypeScript, 99,000 lines, MIT), httpx (Python, 33,000 lines, BSD-3-Clause), and echo (Go, 50,000 lines, MIT).
Tasks from real history. For each, the 36 most recent commits that look like a fix or a feature, touch at most eight files, and modify at least one existing non-test source file. The task text is the commit message. The ground truth is the source files the commit changed, and the symbols whose lines it changed.
No peeking. Each task is indexed at the commit before it, so the answer is not in the index.
Held out. One task in three is in the test split. Settings were tuned on the other 72 tasks only, and this page reports the test split.
The baseline. It takes up to eight keywords from the task, ranks files by how many of them each contains, and reads whole files in that order until it has read every ground-truth file or used 40,000 tokens. A task it does not finish counts as the full 40,000.
All strategies
Test split, 36 tasks:
| Strategy | Tokens per task | File recall | Full body only | Symbol recall | Recall@10 | MRR | Latency p50 / p95 |
|---|---|---|---|---|---|---|---|
| Baseline, 40,000-token cap | 27,529 | 60% | 60% | 57% | 0.79 | 0.40 | 5 / 9 ms |
search_code at 2,000 | 1,895 | 83% | 80% | 73% | 0.85 | 0.60 | 14 / 28 ms |
search_code at 4,000 | 3,353 | 85% | 85% | 74% | 0.85 | 0.60 | 18 / 30 ms |
search_code at 8,000 | 3,913 | 85% | 85% | 77% | 0.85 | 0.60 | 20 / 30 ms |
get_context at 2,000 | 1,943 | 87% | 81% | 67% | 0.93 | 0.61 | 15 / 28 ms |
get_context at 4,000 | 3,827 | 93% | 92% | 74% | 0.93 | 0.61 | 20 / 30 ms |
get_context at 8,000 | 6,218 | 93% | 93% | 78% | 0.93 | 0.61 | 27 / 43 ms |
What each part contributes
Ablations of get_context at 4,000 tokens, test split:
| Variant | File recall | Full body only | Symbol recall | Recall@10 |
|---|---|---|---|---|
| Full | 93% | 92% | 74% | 0.93 |
| Keyword only | 80% | 77% | 68% | 0.88 |
| Vector only | 79% | 76% | 50% | 0.85 |
| No graph expansion | 92% | 92% | 67% | 0.93 |
| No signature compression | 90% | 90% | 67% | 0.93 |
| No name-match links | 93% | 92% | 71% | 0.93 |
| With memories | 93% | 90% | 74% | 0.93 |
Neither keyword nor vector search alone comes close to the two together. The call graph adds symbols more than files. Name-match links make no measurable difference either way; the evidence does not show that they help.
Per repository
| Repository | Baseline tokens | remimem tokens | Baseline recall@10 | remimem recall@10 | Baseline file recall | remimem file recall |
|---|---|---|---|---|---|---|
| hono | 27,622 | 3,866 | 0.67 | 1.00 | 58% | 100% |
| httpx | 28,072 | 3,773 | 0.71 | 0.83 | 71% | 83% |
| echo | 26,892 | 3,842 | 1.00 | 0.96 | 50% | 96% |
httpx is the weakest: two of its tasks are total misses, where the commit message names nothing found in the changed files (for example "Reintroduce URLTypes.").
Embedding models
| Model | File recall | Recall@10 | MRR | p95 latency | Index all three repositories |
|---|---|---|---|---|---|
Xenova/all-MiniLM-L6-v2 (default) | 93% | 0.93 | 0.61 | 30 ms | 62 s |
jinaai/jina-embeddings-v2-base-code | 92% | 0.93 | 0.65 | 83 ms | 324 s |
Memories
Each repository gets 60 memories made from older commit subjects, unlinked, so they must match the task on their own. With memories on, 2.4 are attached per task; 25% of them come from a commit that touched a ground-truth file, and 54% of tasks that have such a memory get at least one. Memory costs about two points of full-body file recall (90% against 92%), because it takes part of the budget.
What the numbers do not show
- Tasks are commit messages. They are written after the change and often name what changed, so they are easier than a bug report written beforehand.
- The test split is small. 36 tasks: one task moves a percentage by about three points.
- The baseline is a simulation, not a real agent, which might search more cleverly or give up sooner.
- One test task was inspected early. httpx
cabd1c095ewas looked at while the harness was being debugged, before tuning. Without it (35 tasks) every headline figure is unchanged after rounding. - Results vary slightly between CPU types, because the quantized model rounds differently.
Reproduce it
From a clone of the repository:
npm install
npx . benchIt clones the three repositories into ~/.cache/remimem/bench (about 150 MB), indexes each task's parent commit, and runs every strategy: 5 to 10 minutes. It writes bench/results/<date>-<commit>.json with every task's raw result. To check your run against the published one, within 0.05 on recall figures and 15% on tokens:
npx . bench --compare bench/results/2026-11-15-9e63d69.json--limit 3 runs a quick sample, --tune adds the tuning configurations on the training split, and --model tries another embedding model. See the CLI reference.