Skip to content
139.5k
Log inGet started

BlogConcepts

Graph memory benchmark: LOCOMO and LongMemEval

· 3 minute read

Graphify is a code knowledge graph for AI coding agents, and the same graph doubles as agent memory. A graph memory benchmark asks whether that structure retrieves the right facts better than the alternatives. Open-source Graphify publishes results on two public long-conversation benchmarks, along with the harness used to run them, so the numbers can be checked rather than taken on trust.

I don't trust a benchmark I can't rerun, so this one comes with the harness. This post walks through the results, where Graphify leads and where it does not, and a quick way to measure token savings on your own repository.

What do LOCOMO and LongMemEval measure?Link to What do LOCOMO and LongMemEval measure?

Both test memory over long histories. LOCOMO holds very long multi-session conversations with questions whose answers sit far back in the history. LongMemEval tests recall, temporal reasoning and knowledge updates across sessions. Recall@10 checks whether the right evidence is in the top ten results; QA accuracy checks the final answer.

How did open-source Graphify score?Link to How did open-source Graphify score?

On LOCOMO with 300 questions, recall@10 was 0.497, against 0.048 and 0.149 for the two memory-layer systems tested. QA accuracy was 45.3%: behind one memory layer at 49.7% and ahead of the other at 27.3%. On LongMemEval-S with 50 questions, QA accuracy was 76%, tied with dense RAG. Building the index used zero LLM credits.

| Benchmark | Metric | Open-source Graphify | Others | | - | - | - | - | | LOCOMO (n=300) | recall@10 | 0.497 | memory layers: 0.149, 0.048 | | LOCOMO (n=300) | QA accuracy | 45.3% | memory layers: 49.7%, 27.3% | | LongMemEval-S (n=50) | QA accuracy | 76% | tied with dense RAG |

Source: the benchmark table in the open-source Graphify README and BENCHMARKS.md.

Why does retrieval lead while QA accuracy does not?Link to Why does retrieval lead while QA accuracy does not?

Retrieval and answering are separate steps. The graph plus hybrid retrieval puts the right evidence in the top ten far more often, which is what recall@10 measures. Turning that evidence into a correct answer also depends on the answering prompt and model. On LOCOMO QA, a memory-layer system scored higher, and the table shows that rather than hiding it.

How was the graph memory benchmark run?Link to How was the graph memory benchmark run?

Every system ran as an adapter in one harness, with one shared model (Kimi K2.6), the same token budgets and the same grader. The judge was blind-validated against a second independent judge at 90.6% agreement, Cohen's kappa 0.81. Sample sizes are in the table, and BENCHMARKS.md lists the commands.

Small samples deserve care. Fifty LongMemEval questions are enough to show a tie, not to separate close systems. Read the results as a direction, then run the harness on your own data.

How do you measure token savings on your own repository?Link to How do you measure token savings on your own repository?

Run graphify benchmark in a built repository. It compares the tokens needed to read the whole corpus with the tokens a graph query returns for a fixed set of sample questions. It is a quick estimate of context savings, not an accuracy test, and the ratio grows with repository size.

bash
uv tool install graphifyy
graphify update .
graphify benchmark

On itsdangerous:

text
  Corpus:          18,600 words → ~24,800 tokens (naive)
  Graph:           372 nodes, 695 edges
  Avg query cost:  ~3,205 tokens
  Reduction:       7.7x fewer tokens per query

On Flask:

text
  Corpus:          117,050 words → ~156,066 tokens (naive)
  Graph:           2,341 nodes, 3,753 edges
  Avg query cost:  ~1,234 tokens
  Reduction:       126.5x fewer tokens per query

Try itLink to Try it

Index a repository in Graphify and point your agent at the hosted MCP server; the hosted MCP docs cover setup. To rerun the numbers yourself, uv tool install graphifyy and follow BENCHMARKS.md. Related reading: knowledge graph vs RAG and the knowledge graph glossary entry.

FAQLink to FAQ

Is Graphify best on every metric?Link to Is Graphify best on every metric?

No. It leads LOCOMO retrieval recall, ties on LongMemEval-S, and trails one memory-layer system on LOCOMO QA accuracy.

Can I reproduce the numbers?Link to Can I reproduce the numbers?

Yes. BENCHMARKS.md in the open-source repository documents the harness, adapters, judge and commands.

Does graphify benchmark measure answer quality?Link to Does graphify benchmark measure answer quality?

No. It estimates tokens saved per query on sample questions. Use the LOCOMO and LongMemEval harness for accuracy.

Try Graphify now.

Try for free