There is a benchmark called LongMemEval that measures how well AI agents remember. Various libraries have taken it on, and until now, the highest score among well-known open-source libraries was MemPalace’s 96.6%.
MemKraft v1.0.1 scored 98.0%.
It currently ranks first among open-source libraries on a major benchmark for agent long-term memory.
LongMemEval measures the ability to answer questions from conversation histories that build up the way real people’s conversations do. It looks at the quality of memory, not just retrieval.
Here is the test setup.
• Dataset: LongMemEval oracle subset, 50 questions
• Evaluation: LLM-as-judge (claude-sonnet-4.6) — scoring based on meaning, not simple string matching
• Model: claude-sonnet-4.6
• Measurement: semantic majority vote (three samples per question, followed by a meaning-based majority vote)
There is a reason I used LLM-as-judge. Many answers were correct in meaning but marked wrong because the wording differed. String matching gives a lower score than it should. I used the same evaluation method MemPalace presented in its paper.
Here are the results by category.
| Category | Score |
| ------------- | ----- |
| Temporal reasoning | 100% |
| Single session (user) | 100% |
| Preference understanding | 100% |
| Knowledge updates | 100% |
| Multi-session synthesis | 92.3% |
| Single session (assistant) | 83.3% |
A comparison with other libraries makes the position clear.
| Library | Score | Notes |
| ------------------- | ----- | -------------------- |
| MemKraft v1.0.1 | 98.0% | Markdown, zero-infra |
| MemPalace | 96.6% | Requires ChromaDB |
| MEMENTO (Microsoft) | 90.8% | LLM self-compression |
| Mem0 | — | Cloud SaaS |
| Letta | — | Requires infrastructure |
| LangMem | — | Depends on LangChain |
MemKraft runs on Markdown files. No external database, no external API calls, no framework dependencies. It has the highest score and the lightest installation.
There is still room to improve in the multi-session and assistant memory categories. I plan to bring those scores up in the next update.
───
The thread running through these updates is this. From “a system that remembers” to “a system that improves itself.” Here are the key ideas behind each version.
v0.9.0: Memory needs room for failure, too
It started with watching agents repeat the same mistakes. They had memories, but no record of “why that happened.”
I added an Incident Memory Layer. incident_record, incident_rca, runbook_match. Not just what was done, but what went wrong, kept in the same memory space.
v0.9.1: A decision is an event, too
Link decisions in chronological order, and a pattern emerges. I encoded YongKeun Park’s What / Why / How principles as they were.
decision_record, decision_link, and evidence_first — an API that looks up past evidence before taking action. I put the principle of investigating before acting into code.
v0.9.2 M1: A prompt is an entity, too
I was inspired by mizchi’s post on empirical prompt tuning (“It Works Ridiculously Well”). Prompts are registered as things that accumulate a history, like people, and every tuning pass automatically records “what changed and why.”
MemKraft still makes no internal LLM calls. It only records; judgment happens outside.
v0.9.2 M2: The past is evidence for the present
prompt_evidence — automatically retrieves what worked in similar attempts, just before tuning.
convergence_check — declares convergence when three measures stabilize: accuracy, step count, and elapsed time.
It puts into code the feeling of an experienced colleague telling you, “That’s good enough.”
v1.0.0: The loop is closed
register → eval → evidence → convergence_check. A complete self-improvement loop is now in place.
No new storage, no new deps, no new LLM calls. 12 APIs, 731 tests, Production/Stable.
Bitemporal memory × empirical tuning: the first self-improvement ledger for AI agents.
v1.0.1: What a second look reveals
As soon as I shipped the stable release, I ran another E2E pass. Zero P0 issues (crashes/data loss). Five P1 fixes. 741 tests. 12 minutes, 13 seconds.
───
The theme running through all six updates from v0.9.0 to v1.0.1 was “using memory as a self-improvement ledger.”
From what to remember, to why we should remember. And then, whether that memory makes the next action better.
MIT-licensed forever. Use it however you like. Stars, forks, and PRs are all welcome!
seojoonkim/memkraftUltimate zero-dependency compound knowledge system for AI agents. Auto-extract, classify, search, and maintain memory in plain Markdown.GitHub · github.comOpen
