I hadn’t posted an update in the rabbit hole since announcing MemKraft’s 98.0% on LongMemEval a few months ago. I connected MemKraft to my personal agent and used it every day, fixing problems one by one. Before I knew it, it was at v3. I thought remembering a lot and finding things well would be enough. But as the memories grew, harder questions emerged.
In v1.0.2, I first made the performance reproducible. I added evaluation tools to the package so anyone could rerun LongMemEval’s data preparation, retrieval, answering, and scoring. I also split retrieval into exact phrasing, semantic expansion, temporal cues, and combined ranking. In v1.0.3, I stored long meeting notes in overlapping chunks. This kept the one sentence we needed from getting buried in thousands of characters, and made looser retrieval kick in only when exact retrieval failed.
Soon, the volume of stored information became a problem in itself. My MEMORY.md once grew to 153KB. v1.1.0 could import memories, move older entries to an archive layer, summarize them to a set size, and check the system’s overall health. v1.1.1 watched for file changes and scheduled cleanup. The important point is that it did not discard old memories. It kept the foundation for remembering a lot, while separating what was needed often now from what should be kept for the long term.
In v2.0, the shape of the problem changed. Finding individual notes was not enough to explain relationships. Even with both the fact that A worked with B and the fact that B was in charge of C stored, it often got questions connecting all three wrong. So I added a local SQLite knowledge graph. It linked people, organizations, and projects, and let retrieval follow those links across multiple steps.
v2.1 improved relationship extraction to handle Korean particles and word order. In v2.2, when the person in charge changed from the CEO to the CTO, it closed the validity period of the old role. It also made multiple passes through exact retrieval, relationship expansion, and chronological order, depending on the question type. v2.3 went further. BM25 accounted for document length and rare words, while RRF safely combined retrieval rankings with different scales. This was also when I added chains of cause and effect, consolidation to clean up duplicate memories and expired facts, and separate paths for questions about time and counts.
I learned one thing along the way. A system that retrieves well also retrieves wrong information well. If the person in charge has changed but a note from three months ago ranks first, that is more dangerous than finding nothing. Relationships and time were not decorative extras for retrieval. To answer a question about the present, the system had to know who a fact was connected to and how long it had remained true.
In v2.6, I separated memories into episodes, general knowledge, and procedures, and recommended storage tiers based on activity and importance. When different values for the same subject were valid at the same time, the system flagged a conflict. Once there were more memories, what mattered was less whether to store them and more what kind of memories they were and whether they could coexist.
There were failures too. In v2.7.3, I experimented with local embedding retrieval, which turns sentences into numerical vectors. The goal was to find sentences with similar meanings even when their wording differed. But testing showed roughly 13 times longer retrieval times with no meaningful improvement in accuracy. In v2.7.4, I removed it from the default path and made it optional.
After that, v2.8 and v2.9 focused on making the existing retrieval work properly and quickly rather than adding flashier methods. I built indexes and read caches so documents would not have to be reread every time, and used an inverted index to scan only relevant candidates. I also checked that the results matched the previous results, so speed would not come at the cost of missing memories.
ReasoningBank, introduced in v2.7.1, began with another problem. The agent tried the same task using an approach that had failed the month before. It had stored only the outcome and discarded the process that led there. ReasoningBank kept thoughts, actions, outcomes, and lessons together as a single trajectory.
In v2.9.2, it could record this with a single call and find similar past tasks. In v2.10 and v2.11, it brought back a short account of relevant failures and successes before a task began, but treated that content as untrusted quotations. Sentences in memory must not give orders to the next agent.
The groundwork for v3 began in v2.12. Memory Gym evaluates retrieval changes with tests that mix relationships, time, conflicts, and rare cues, and blocks a release if it does not meet the threshold. Provenance records which document and passage a memory came from, and why() lets you trace the evidence back.
The candidate feature in v2.13 holds unreviewed information temporarily rather than immediately promoting it to fact. The session overlay lets the agent use something it has just heard within the same conversation, without leaking it into other conversations or long-term memory. The resolver dry-run previews duplicates, revisions, and conflicts before anything is saved.
v3.0 brought these pieces together into a memory lifecycle. It covers the whole process: information arrives, is verified, becomes a current fact, grows outdated, is cleaned up, and is used again.
v3.0.1 made it possible to audit updates to current facts and the disposal of candidate memories. It also allowed sums and durations to be calculated only when the evidence was clear. v3.0.2 fixed a problem in limited context windows where dates were cut off before the body text, leading to incorrect time calculations. It also measured differences in date, time, and number formatting separately, distinguishing actual wrong answers from scoring discrepancies.
In v3.0.3, I ran a total of 1,260 experiments across 28 tasks and 9 sets of materials to see whether ReasoningBank’s natural-language hints reliably reduced model calls. The result was gate FAIL. Natural-language lessons still had to be interpreted, and accuracy and latency were not consistent. This led me to conclude that there needed to be a permission boundary between memory and execution.
The deterministic execution I built in response does not execute sentences from memory. It checks provenance and file integrity, and processes a request locally only when it matches one of six explicitly allowed procedure syntaxes exactly. It does not run arbitrary code or shell commands. If anything is ambiguous, it fails closed and calls the existing model path exactly once.
Of the fixed set of 28 cases, it handled 24 locally and passed only 4 to the model. Accuracy stayed at 28/28, while calls fell from 28 to 4, a reduction of 85.714%. In a sequential experiment with an actual provider, 305.406 seconds became 12.894 seconds. But this was a single observation that included a long delay, so I do not claim it as typical performance.
The full test run had 2,240 passed and 3 skipped, and an independent review found 0 Critical or Important issues. Existing Markdown, schema v1, and public APIs all remain compatible.
The moment an agent’s memory becomes the basis for its next action, the memory system is also a permission system. On top of the ability to remember a lot, we need to build connections and time, provenance and verification, learning from failure and safe execution, layer by layer.
What I want to build is an agent that remembers a lot, makes accurate connections, explains its evidence, and puts what it remembers to proper use in its next action.
MemKraft is available under the MIT license. Stars, forks, and PRs are all welcome.
seojoonkim/memkraftUltimate zero-dependency compound knowledge system for AI agents. Auto-extract, classify, search, and maintain memory in plain Markdown.GitHub · github.comOpen
