s-batman

8 topics next for Agentic Memory, plus one that keeps you Safe

Steven Batchelor-Manning — Mon, 06 Jul 2026 00:00:00 GMT

The six universal patterns in the 19 systems I've been digging through are the spine of this series. Write-time quality investment. Provenance travelling with the fact. RRF-fused hybrid retrieval. Tiered storage with heat-based promotion. Explicit context budgets. Tools-not-injections at the agent surface. Each earned its own article because three or more systems arrived at the same discipline independently. The universal patterns stop here. From there, the emerging ones begin. This article is about eight patterns that sit below the threshold, living in one or maybe two systems, where the implementation is so clearly addressing a real problem that it's hard to believe they'll stay rare. Seven appear in exactly one system. One appears in two and is one adoption away from graduating. The first gets the deepest treatment because it's the most important. Being early means some of these won't universalise. Being late is worse: you build a system in 2026 that ships without storage-layer authorisation because a synthesis called it a one-off quirk, and you get a prompt-injection demo on Hacker News in 2027. --- ## The storage engine is the only authority that holds Last week's article argued the agent should hold the tools. Give it a SQL tool and the question becomes immediate: what stops it querying the table it shouldn't? The naive approaches are well-trodden and all wrong. Prompt-level instruction: "You are not permitted to read the conversations table." Works against a cooperative model. Fails against any model that misreads the instruction, decides the prohibition doesn't apply to its current task, or is operating under a jailbreak. Application-level filter: the `sql_query` tool wraps the model's SQL, parses it for forbidden table references, rejects the query before passing it to the database. Works against a naive model. Fails against any model that quotes table names, uses `ATTACH`, exploits a parse mismatch between the wrapper and the database, or routes through a view defined elsewhere. ORM-mediated access only: the agent never touches raw SQL, it calls typed methods. Works until the system grows a debugging tool, an analytics tool, a migration tool, or a "let the agent join two tables" feature, at which point the rule breaks and the brokenness goes unnoticed. Put plainly: none of these are enforcement. They're hints. A system that wants real sub-agent isolation has to ask the storage engine itself to refuse the read, not the layer above it. This is the discipline capability-based security imposes on operating system resources: the kernel says no, not the application. It's also the discipline PostgreSQL's row-level security imposes on multi-tenant SaaS: the policy lives in the database, not the ORM, because "the engineer who built the new microservice forgot to apply the tenant filter" is a bug class that only a storage-engine policy can rule out. second-brain is the corpus's first worked example. Its scoped database has three layers, each necessary, none sufficient on its own. A fresh in-memory SQLite connection per scoped agent. The constructor opens `:memory:` and attaches the real database in read only mode. The agent's connection has no real main schema, there is nothing to read except what the next layer puts there. TEMP VIEWs in the main schema redirecting to allowed tables in the source. The view definitions come from the agent's profile and can encode column-level redaction or row-level filters. A view name validator prevents injection of malicious table names through the profile. A SQLite C-API authorizer hook denying any read of the source schema that doesn't pass through a view. The hook fires for every read SQLite is about to perform, before the optimiser even resolves the table name. A read inside a view body is allowed, the view is the scope filter. A top-level read is checked against the set of view names the scope created. Any read against the source schema directly is denied without exception. The LLM can write any SQL it wants. It can quote table names, use `UNION`, use `ATTACH`, use SQLite's full grammar. It cannot succeed in reading a row from a table that isn't in the allowlist, because the authorizer intercepts the read at the C layer before the table name is resolved. The application-level filter is bypassable. The storage-engine authorisation is not. About 50 lines of stdlib Python. No extra dependency, no extra process, no schema migration. Per-query overhead is one C callback per read, invisible against the cost of the query itself. The pattern composes cleanly with everything else: the agent's tool registry can still be feature-gated, the view definitions can still encode column-level redaction, the conversation history can still be redacted at write time. None of those layers is weakened by the storage-layer enforcement. They're reinforced by it. The framing matters. Every other security pattern in the 19 systems is a hint. The storage engine is the authority. --- ## Three more worth naming ### Async cleanup race protection (llm-wiki) Background memory maintenance is fundamentally racy with foreground user actions. A sweep that started against project A and finishes against project B has corrupted both. It's moved A's decisions into B's review store, or marked B's items resolved against A's data. This is a bug class that's easy to introduce, hard to detect, and impossible to recover from cleanly because the corrupted state looks legitimate. llm-wiki runs a two-stage background review loop and, at every yield point, re-checks two race-protection signals: an abort signal fired by the project-switch handler, and a path comparison against the current project in the UI store. Either check fails, the sweep returns mid-flight without applying decisions. The project-switch handshake on the queue side completes the picture: it flushes the active project's state to disk before clearing memory, reverts processing items back to pending, aborts both the in-flight LLM call and the in-flight sweep judgment, and only then writes to the paused project's path. The meta-pattern is the takeaway: deterministic where you can, LLM where you must, abortable everywhere. The two-stage structure keeps the LLM out of cases that simple existence checks can handle. The race protection keeps both stages abortable. The combination is a template for any background memory maintenance loop in any system. ### The auto-degraded no-op constructor (graymatter) Library API design has a recurring tension. The simplest "hello world" wants the library to just work, one line to construct, one to call. The most defensible production posture wants it to fail loudly at construction with a structured error the caller can't ignore. graymatter picks fail-silent, but with a discipline that turns the trade productive. The constructor never returns an error. If init fails, bbolt is locked, the data directory is unwritable, the vector store can't be opened, it logs to stderr and returns a degraded Memory whose methods are all no-ops. Production callers verify through `Healthy()` before trusting the handle. The library is `go get`-able, importable in three lines, and works in the demo. `Healthy()` is the production discipline tax. The pattern makes the library safe to embed in agent harnesses that have their own startup ceremony. An agent harness that calls `graymatter.New(...)` during boot, ignores the error path because there is no error path, and continues, gets a working memory layer in the happy path and a no-memory fallback when the data directory is read-only. Either way the harness boots. That's a specific kind of defensive composition that fail-loud constructors can't offer without explicit error handling at every embedding site. ### Shadow-mode duplicate detection (mem9) Every system that ships duplicate suppression has to pick a cosine similarity threshold. Above 0.95 is almost certainly a duplicate. Below 0.7 is almost certainly not. The space between is contested, and the right cut depends on the embedding model, the domain, the query distribution, and the cost of false positives versus false negatives in this particular system. The temptation is overwhelming to pick one based on intuition and ship. mem9 doesn't. It runs the duplicate-detection query for every fact, records the cosine score in a Prometheus histogram, and takes no action. The threshold is deferred until production data justifies it. "Ship the observation, not the heuristic." The same logic applies to every thresholded decision in every memory system. Rerank threshold. Recall confidence cutoff. Tier-promotion heat gate. Insight-merge similarity gate. Most systems in the 19 ship these values guessed. mem9 ships with the value deferred. The discipline is rare and the result is better. --- ## Four more, tighter **Per-tenant physical database isolation** (mem9). Instead of a `WHERE tenant_id = ?` filter on a shared store, mem9 provisions a separate TiDB cluster per tenant through TiDB Zero. Isolation is storage-engine-side. The application can't accidentally query across tenants because there is no shared store to query. It's a coarser-grained version of the same end-state as storage-layer authorisation: isolation enforced at the engine, not the application. The infrastructure cost that historically made this impractical is gone. TiDB Zero auto-provisions. Neon does the same for PostgreSQL. Cloudflare D1 does the same for SQLite. **Source-turn decoration with explicit context budget** (mem9). A retrieved memory is a string. "User prefers Postgres." Correct, terse, impossible to ground without context. mem9 attaches the originating conversation turns as decoration, scored against the query and capped by a budget triple: minimum score, per-memory limit, total limit. The agent reading "User prefers Postgres" gets the turn where the user said "we tried MongoDB but the joins killed us so I switched to Postgres last quarter." No second tool call required. The grounding is in the result. The prerequisites are already universal: provenance plus hybrid retrieval. Most systems in the 19 could implement this in two days. **purpose.md as a fourth file** (llm-wiki). The Karpathy LLM Wiki pattern has three canonical files: raw sources, the wiki working set, and schema.md for structural rules. llm-wiki adds a fourth: purpose.md, filled in by the user, inlined into every LLM call the system makes. Every ingest prompt, every generation prompt, every chat retrieval reads it. The effect is a stable directional prior that conditions every downstream behaviour. The LLM is going to read the system prompt anyway. Adding the user's intent costs nothing and lifts everything. The absence from most other systems is harder to explain than its presence in llm-wiki. **AGENTS.md as authoritative agent contract** (Tolaria, OpenContext). Most repositories with an `AGENTS.md` or `CLAUDE.md` treat it as a hint file. Tolaria and OpenContext treat it as a contract, backing every binding clause with a mechanical check that fails the build if the agent violates it. "Do not skip pre-commit hooks" isn't a polite request, it's a rule the CI enforces. "Test coverage must stay above threshold" isn't a guideline, it's a bar the test runner aborts on. A hint can be ignored. A contract backed by a check cannot. Two systems already do it. One more and it graduates. --- ## Which ones universalise next Storage-layer authorisation will universalise first. This is the most confident prediction in the source analysis. Every memory system that gives a sub-agent SQL access is one prompt injection away from a confidentiality breach without storage-layer enforcement. The infrastructure is already in place. SQLite has had `set_authorizer` since the early 2000s. PostgreSQL RLS is mainstream. LanceDB and ClickHouse have policy hooks of their own. The barrier isn't technical, it's awareness. second-brain has provided the worked example. The next generation of managed APIs will copy the discipline because the alternative is indefensible. Source-turn decoration will universalise second. The prerequisites are universal already. The implementation is two queries plus a budget. The agent-side information gain is large enough that whoever ships it first in a managed API is going to be visibly better at grounding answers in source dialogue than whoever ships it second. The pressure to copy is high. graymatter has the source per fact. supermemory has lineage. Hindsight has full conversational provenance. Any of these is one PR away from the pattern. The auto-degraded no-op constructor will universalise third, and in a different language. Go's cultural conditions make the pattern safe. The next adopter is unlikely to be Python, the culture is too eager about exceptions, but might be Rust. It's a library-API design choice, not a memory choice, and it will spread wherever "demo aesthetic plus production discipline" is the right trade. Shadow-mode deployment is the dark horse. Technically trivial, culturally hard. If a second system instruments a threshold before gating on it, the pattern graduates immediately, and the third and fourth follow within a release cycle because the engineering ergonomics are unanswerable once demonstrated. The remaining four are each conditional on a specific deployment shape becoming more common. Race protection universalises when more systems grow multi-conversation parallelism. Per-tenant physical isolation universalises when regulated enterprise customers start asking for it. purpose.md universalises when the Karpathy LLM Wiki paradigm gets its third or fourth implementation. AGENTS.md as contract universalises when the "agents are colleagues" framing becomes the dominant harness metaphor. None is implausible. None is certain. --- The unifying thread across all eight is the same position: the application layer is not a trustworthy isolation boundary. Storage-layer authorisation is the per-agent expression. Per-tenant physical databases are the per-tenant expression. AGENTS.md as contract is the per-agent-behaviour expression. The next iteration of this corpus will, in this prediction, contain a new universal pattern by that name. Last week's article made the case that the agent should hold the tools and decide what to retrieve. This one is about what holds when the agent turns those tools somewhere you didn't expect. The six universal patterns are consensus. The eight here are the leading indicators of where consensus moves next. Storage-layer authorisation is the leading edge of the leading edge, and the cost of missing it is the kind of cost that shows up on Hacker News. Part 10 of 10 in the agent memory series. Each article stands alone. --- Have thoughts? Share and discuss: [Share on X](https://x.com/S_BatMan/status/2074236054840697057) | [Discuss on X](https://x.com/S_BatMan/status/2074236054840697057) _Previous_: [2026-06-02-llm-memory-research-09](/posts/2026-06-02-llm-memory-research-09/)

You don't have a context problem. You have a harness problem.

Steven Batchelor-Manning — Sat, 04 Jul 2026 00:00:00 GMT

The harness effect on real workloads runs 5 to 40 percentage points. The model effect between frontier models on the same benchmark runs 5 to 15 percentage points. The scaffold matters more than what's inside it. The cleanest single data point on this: same model, different harness, sixteen points apart. Cursor on Claude Opus 4.6 scores 93. Claude Code on Claude Opus 4.6 scores 77. The harness, the system prompt, the tool surface, the way the context is sliced before it reaches the model, all of that is in that sixteen points. None of it is fixed by buying a bigger window. This is part three of a three-article series on context. The first covered why the advertised window is bigger than the work-window. The second covered what it costs per active user to keep that window open. This one covers what the field has settled on as the engineering answer to both problems: context engineering, the discipline of shaping what reaches the model so the model can do the work. ## What context engineering is The term has a vendor source. Anthropic's late-2025 article on context engineering for agents put it into mainstream engineering discourse, and their `compact_20260112` API made it operational. The mechanism is simple enough to fit on a slide: the conversation approaches a configurable threshold (default 150K), the API summarises older context, the summarised version replaces the original. Claude Code auto-compacts at 50 percent context usage. The same Workaccount2 HN workaround from June 2025, the one where users manually pasted their own summaries when the conversation got long, is now a product feature. Put plainly: the harness is now a first-class engineering surface, and the work most teams used to do in the model is moving into the prompt, the tools, the subagent structure, and the cache policy. Three things follow from that shift. The first is that the field has a vocabulary for the discipline, and the vocabulary is "context engineering" rather than "prompt engineering" or "RAG". The second is that the harness is where most of the engineering effort is going, not into the model. The third is that the discipline has a measurable effect, and the measurement is bigger than the inter-model gap on most tasks. ## How big the effect actually is The harness effect, measured across published benchmarks, runs 5 to 40 percentage points from scaffold alone. The model effect between frontier models on the same benchmark runs 5 to 15 percentage points. Both ranges are real. Both come from the same benchmark families. Neither is a guess. The Cursor 93 versus Claude Code 77 result from Article 1 is the cleanest single data point. Same model, same tasks, sixteen points apart. The harness, the system prompt, the tool surface, the way the context is sliced before it reaches the model, all of that is in that sixteen points. Anthropic reports similar patterns internally on their own compact-enabled tooling: same model, different scaffold, ten-point swings are routine, twenty-point swings are achievable, forty-point swings require deliberate redesign. LOCA-bench (arXiv 2602.07962, ICML 2026) reports a 10 percentage point improvement from context engineering alone, which lands inside the harness-effect range and above most inter-model gaps on the same benchmark. The benchmark isolates the eight mitigation strategies this article covers and measures each in isolation. Compaction plus tool-result clearing plus memory tools produces the largest combined gains among the tested strategies, with programmatic tool calling as the strongest single-strategy finding. The pattern LOCA-bench reveals is that the harness isn't a wrapper around the model. The harness is the system, and the model is one component of it. The practitioner consensus is that harness quality matters as much as the model you put in it. Cursor employs people whose full-time job is to rewrite system prompts and tool descriptions every time a new model ships. That's the operation in 2026. The system prompt and tool surface are not fixed code. They're living engineering artefacts that get re-tuned for every model release. The same dynamic shows up across the agent framework ecosystem: Claude Code, Cursor, Codex CLI, Windsurf, Cline. The teams shipping production agents treat the harness as a versioned, tested, deployed artefact. The model is the dependency. The implication is uncomfortable. Most of what gets attributed to the model is the model plus the harness plus the system prompt plus the way the developer chose to fill the context. None of those is fixed by buying a bigger window. All of them are within the developer's control. ## What the field is shipping Eight strategies have converged across the production systems in 2026. They are not mutually exclusive. Most systems ship three or four of them at once. Context compaction sits at the top of the list. Anthropic's compact_20260112 is the canonical reference, but the pattern is now standard across Claude Code, OpenAI's ChatGPT auto-summarise, Cursor's long-session mode, and most agent frameworks that run for more than a few hundred turns. The discipline is to summarise earlier context before it crowds out later context, and to do it at a configurable threshold. Compaction isn't free at the point it runs. Memory-tiering systems that lean on summarisation as their main budget mechanism report paying 20 or more LLM calls in a single interaction to keep their summaries current. The harness that adopts compaction has to budget for the compaction, not just for the context it saves. Structured prompt templates, the second strategy, lock the context shape. The system prompt is structured. The tool descriptions are structured. The user messages get a wrapper. The variability of natural-language prompting gets bounded by templates that the system itself can parse. LOCA-bench (arXiv 2602.07962, ICML 2026) treats structured prompting as a separate experimental condition because the effect is large enough to be worth isolating. Just-in-time retrieval, the third strategy, is RAG. RAG with a reranker. RAG past 500K tokens is mathematically dominant, by enough that the long-context choice only makes sense for the slice of traffic RAG can't service. The strategy isn't new. What's new is treating it as one strategy among several rather than as a competitor to long context. They're complementary, and the 17x cost ratio from Article 2 is the reason RAG handles the 80 percent while long context handles the 20 percent. The cheapest version of this strategy doesn't need a reranker at all. Returning identifiers and short previews instead of full records, then fetching only what the agent asks for, turns 15,000 tokens of unread matches into roughly 450 tokens of signal, with the full record paid for only when it's actually used. Across a long session that difference is the gap between compacting every few turns and not needing to compact at all. Tool-result clearing, the fourth strategy, is the discipline of removing tool outputs from context after the model has processed them. Claude Code implements this for shell and file-read tool outputs. LOCA-bench's context-reset strategy tests it explicitly. The mechanism is simple, the effect is noticeable: tool outputs are often kilobytes of structured data the model needed once but doesn't need to keep in the active context. Programmatic tool calling, the fifth strategy, is the strongest finding in the LOCA-bench paper. Writing code that orchestrates tool calls, rather than calling tools directly, substantially reduces the intermediate context the model has to carry, and it improves orchestration accuracy at the same time. The model writes a Python script, runs the script, reads the script's output. The tool calls don't appear in the active context. The intermediate state stays in the script. The reduction in context pressure is the largest single lever LOCA-bench measured. Subagents with context firewalls, the sixth strategy, are how Claude Code's Agent Teams handle multi-step work. Each subagent operates in its own context window with its own worktree. The parent agent sees the subagent's summary, not its full trace. If one context window rots, you use many small ones instead. The pattern is operationally similar to processes with isolated address spaces. The benefit is that context pressure is per-subagent, not per-conversation. Summarisation mid-conversation, the seventh strategy, sits between compaction and tool-result clearing. The discipline is to summarise a tool's output or a sub-task's result before it goes back into the parent context. It's compaction applied at finer granularity, and it's most useful in agent loops where each step's output would otherwise compound. Attention steering and calibration, the eighth strategy, is the most architectural. Differential Transformer, the Microsoft and Tsinghua paper (arXiv 2410.05258), reframes attention as the difference of two softmax maps, which cancels noise and promotes sparse patterns. The paper reports 30 percent less hallucination in QA and robustness to order permutation in in-context learning. The mechanism directly addresses the attention noise that causes context rot. The takeaway for the harness engineer is that some of the rot can be steered away at the model level, not just managed at the prompt level. The eighth strategy is sometimes called "recite then reason", and it isn't quite the same thing. Recite-then-reason asks the model to restate the relevant material from its context before answering. It's a prompting discipline, not an architectural one, and it's worth keeping in the toolkit but not as the headline strategy. Eight strategies, eight different points on the cost-versus-complexity curve. The practitioner who ships all eight is shipping a system, not a wrapper. The practitioner who ships three is shipping a wrapper. The difference is the engineering budget, not the model. ## What still doesn't work The mitigations are real. The gap they close is real. The gap they don't close is also real, and it's worth naming. NIAH benchmarks still systematically overestimate usable context compared to MRCR v2. The HELMET paper (ICLR 2025) confirms NIAH does not predict downstream task performance: GPT-5.5 can score 96 percent on NIAH-2 at 1M tokens and still fall well short on multi-needle MRCR at 128K. Vendors quote the high NIAH number. Production agents hit the MRCR floor. The two numbers live on the same vendor slide. The intra-family gap is also still large. Claude Sonnet 4.5 to Claude Opus 4.6 was an 18.5 percent to 76 percent jump on MRCR v2 at 1M, the widest intra-family gap of 2026. Opus 4.7, the newest of the family, sits at 32.2 percent on the same benchmark at 1M, well above Sonnet 4.5 but well below Opus 4.6. The lesson: the model gets better, the gap to the leader sometimes narrows and sometimes widens, but the gap to the work-window the model can actually use remains. The harness has to keep doing the engineering work the model can't. Put plainly: the harness is closing the gap the model can't. The work-window is smaller than the advertised window. The harness makes the work-window larger. It does not make the work-window equal to the advertised window. Nothing does. ## What this looks like at the source-code level The eight strategies aren't abstractions. They show up as code in the systems shipping in 2026, and the mechanisms are worth knowing about up front because they explain what production looks like. vLLM prefix caching, the same mechanism from Article 2, is the harness-level implementation of cache reuse across requests. Block-level hashing, fixed-size blocks, LRU free queue. The harness decides which blocks survive across requests. The model just sees the cache hits as part of its context. Anthropic's compact_20260112 is the harness-level implementation of compaction. The conversation approaches 150K (default), the API summarises older context, the summarised version replaces the original. The harness decides when to compact. The model just sees the compacted context as the new state. Subagents with isolated worktrees are the harness-level implementation of context firewalls. Each subagent runs in its own context window. The parent agent sees the summary. The harness decides the partitioning. The model just sees its own context. Self-guiding tool responses are the harness-level implementation of the cheapest strategy in this list. Every tool response closes with a hint about what to call next, a follow-up identifier, a suggested next tool, a warning about what the result implies. The agent learns the API by using it rather than by reading documentation for it. The harness decides what the hint says. The model just follows the thread it's handed. Four mechanisms, four different harness-level decisions, four different places where the engineering effort goes. The model is the constant. The harness is where the system actually lives. ## Where this leaves you If you've been filling 200K-token windows and wondering why the agent loses the thread, the answer is almost never that you needed 400K. It's that the harness isn't doing the engineering work the model can't. The advertised number is the size of the door. The number that does work is the size of the room the model can see when it's actually doing the work. The harness is what shapes that room. The eight strategies above are what the field has converged on. Pick three to start. Compaction, tool-result clearing, structured prompts. Those three shippable in a weekend. The rest follow when the first three stop closing the gap. The shame's in not knowing which trade-off you're making. The harness matters more than the model. The cost is asymmetric. The work-window is smaller than the slide says. The discipline is context engineering, and the engineering is in the harness, not in the model. The field has known this for two years and said it loudly for one. The shift from "buy a bigger window" to "build a better harness" is the working consensus of 2026. The model is the constant. The work is in everything around it.

Long context isn't a capability. It's a standing charge.

Steven Batchelor-Manning — Wed, 01 Jul 2026 00:00:00 GMT

One active user on Llama 3 70B at 128K context takes 40GB of GPU memory. That's half an H100. Per token, the cache footprint runs 320KB in BF16. Every additional user you put on that GPU costs you another half-H100-worth of memory before a single forward pass has run. This is part two of a three-article series on context. The first covered why the advertised window is bigger than the work-window. This one covers what it costs you per active user to keep that window open. The third covers what the harness around the model looks like in 2026 once you've accepted the cost. ## The numbers the slides don't show The headline context window is sold in tokens. The thing that actually costs money is bytes of GPU memory. The two are different, and the conversion factor varies by roughly 40x across models shipping today. Put plainly: a 1M context window doesn't cost the same on every model, and the spread is the story. Take four production models at the same advertised window. Llama 3 70B at 128K context costs 40GB per active user, half an H100. Per token, the key-value cache runs 320KB in BF16. The formula is the same one every transformer paper uses: 2 times the number of KV heads, times the head dimension, times the number of layers, times 2 bytes per element. DeepSeek V3.2 at 1M context costs 83.9GiB per active user. Per token, the cache runs about 87,000 bytes. That's Multi-Latent Attention doing some work, but it's still nearly 22GB per million tokens, and the cache scales linearly with the window. DeepSeek V4-Pro at 1M context costs 9.62GiB per active user. Per token, the cache runs about 9,800 bytes. CSA plus HCA plus SWA get the footprint down to roughly 10 percent of V3.2's. Same vendor, same family, roughly one tenth the memory per token. Qwen 3.6-35B-A3B at 128K context costs 2.5GB per active user. Per token, the cache runs about 20,480 bytes. Hybrid Gated DeltaNet with only 10 attention layers. The KV cache is mostly gone, replaced by a different mechanism that doesn't pay the per-token tax. The same advertised 128K window costs anywhere from 2.5GB to 40GB per active user, depending on which model you picked. Same input, same vendor marketing slide, 16x spread. The architecture choice is the cost. ## Why the cache scales the way it does The KV cache is the model's working memory for everything you've already sent. Every token the model reads stays in cache so it can be attended to on every subsequent decode step. At 1K tokens, the cache is small enough to ignore. At 100K, it's a budget item. At 1M, it's the binding constraint. The binding constraint shifts during inference. During the prefill phase, when the model reads your input, it's compute-bound. The H100 is busy doing matrix multiplies and the cache is being written. During the decode phase, when the model is generating tokens one at a time, the bottleneck moves to memory bandwidth. The model has to read the entire cache to attend to every token it generates, and the cache is now 40GB or 80GB or whatever the model has accumulated. The numbers from published studies put the cache read at between 68 percent and 94 percent of decode latency at 500K context. The GPU is mostly waiting on its own memory, not on the model. The bottleneck isn't the model thinking. The bottleneck is the model looking at everything you sent it, on every token, no matter how far back the relevant material sits. This is the thing that makes long context expensive even when the input tokens themselves are cheap. The cost is in the cache reads, and the cache reads scale linearly with the window. Double the window, double the per-decode-step memory traffic. Quadruple the window, quadruple it. There's no engineering reason for the model to read tokens from the start of your prompt when it's generating the last token, but that's how attention works. It reads everything, every time. ## What the per-call cost actually looks like The published per-call numbers, mid 2026, are all over the place. DeepSeek V4 Pro at 256K context runs about $0.445 per call. GPT-5.5 at the same context runs about $1.28 per call. GPT-5.4 Pro at the same context runs about $7.68 per call. That's the same window, same input length, three vendors, a 17x spread on the dollar number. The headline figure for this article sits in that spread: 200K tokens on Opus at $1.00 per call. The same task on a 4K RAG pipeline runs about $0.06. 17x is the per-call ratio between reading the full window and retrieving only what the answer needs. The architectural reasons for the spread sit in the next section. The cost figure is the symptom. The architecture is the cause. ## What the architecture choices actually do The five-orders-of-magnitude spread in cache footprint comes from four architectural moves, and they're worth knowing about up front because they explain why some 1M windows cost 17x less than others. Multi-Head Attention, the original, has the worst case. One KV head per query head. Eight heads, eight KV tensors per layer, 320KB per token on Llama 3 70B. The Shazeer 2019 paper that introduced MQA cut that by sharing a single KV head across all query heads. GQA, the Ainslie 2023 paper (arXiv 2305.13245), generalised the trick to a small fixed number of shared heads. MLA, the DeepSeek-V2 paper (arXiv 2405.04434), compressed the keys and values into a single low-rank latent vector. The paper reports a 93.3 percent KV reduction relative to standard MHA. The mechanism is what's behind DeepSeek V3.2's 87,000 bytes per token at 1M. Without MLA's 93.3 percent compression (inverting to roughly 15x the cache size), that number would be around 1.3MB per token. Sliding Window Attention, from Mistral 7B (arXiv 2310.06825), capped the cache by only attending to a window of W equals 4096 tokens. The model can't see anything older than the window, so the cache stops growing. The trade-off is the model also can't use anything older than the window. W is the upper bound on effective context for the model, no matter how big the advertised window is. Hybrid architectures, the Qwen 3.6 family being the cleanest example, replace most of the attention layers with a different mechanism. DeltaNet and similar state-space variants don't keep a per-token KV cache at all. They keep running state. The cache size stops depending on token count and depends on the model's hidden dimensions instead. That's the move that gets Qwen's per-token cost down to 20,480 bytes. A sixth move worth knowing about is native sparse attention at decode time. NSA, the DeepSeek paper from early 2025 (arXiv 2502.11089), does coarse-grained token compression plus fine-grained token selection plus sliding windows, all natively trainable, all hardware-aligned. The reported numbers match full attention quality at a fraction of the cache reads. MoBA, the Moonshot paper (arXiv 2502.13189), takes a different route: block-level MoE routing with parameterless top-k gating, production-deployed in Kimi's long-context requests. Neither of these is a free lunch, but both are evidence that the architectural menu is widening, not narrowing, in 2026. Five moves, five different cost profiles. Picking a model is picking an architecture. Picking an architecture is picking a cost. ## Where the cache economics get interesting The cache cost isn't just a function of the model. It's a function of the cache hit rate, the time the cache sits between reads, and what's backing the storage. Anthropic's prompt caching breaks even at 1.4 reads per write on the 5-minute tier. Below 1.4, the cache costs more than it saves. Above it, every additional read is free. The 1-hour tier needs a 67 percent cache hit rate before it pays for itself. Most production agents don't hit 67 percent on the 1-hour tier. They do on the 5-minute tier. Pick the tier the workload actually supports. Tutti's SSD-backed KV cache is roughly 100x cheaper per GB than DRAM-backed cache. The published numbers from their evaluation are 78.3 percent time-to-first-token reduction and 27 percent cost reduction versus LMCache-SSD. The trade-off is latency. SSD is slower than HBM. For prefill-heavy workloads, that trade-off makes sense. For decode-heavy workloads at 500K context, it doesn't. Tutti is a fit for the right shape of workload, not a universal win. The 90 percent compression cliff is the binding constraint on all of this. Compression ratios below 90 percent start killing retrieval accuracy. TurboQuant's tbq3_0 in llama.cpp hits 81 percent KV memory reduction at FP16-equivalent perplexity. That's below the 90 percent cliff and it works because they also preserve the attention pattern, not just the cache size. The lesson: KV cache compression is a mechanism design problem, not a number-tweaking problem. Below 90 percent, you're trading accuracy for memory, and the trade gets worse the further below 90 percent you push. None of this is free further up the stack either. Memory systems that build their own summarisation tiers on top of a model's context window report paying 20 or more LLM calls in a single interaction just to keep those tiers current. The cache moves cut the GPU bill. The orchestration layer sitting above it can still spend the saving back. ## What the kernels actually look like The architecture choice only matters if the kernels can run it. The 2026 state of the art for KV cache attention has three names worth knowing, and the difference between them is the difference between a research paper and a production system. FlashAttention 4, from the Dao-AILab group, is the current reference. It's written in CuTeDSL, targets Hopper and Blackwell specifically, and uses async warp-specialised kernels to hit 1.3x over FlashAttention 3 on the H100. The big change is memory: O(n) versus O(n squared) for the standard attention implementation. Install is one line, pip install flash-attn-4. FlashAttention 2.5 added paged KV cache support that's compatible with vLLM's PagedAttention. FlashAttention 3 hits 660 TFLOPS on the H800. The progression is the same pattern the rest of the field follows: each generation buys a real speedup, and the speedup is paid for by more careful memory access, not by smarter maths. FlashMLA, from DeepSeek, is the production reference for MLA-style attention at decode time. The KV cache format is FP8: 656 bytes per token, made up of 512 bytes of quantised NoPE in float8_e4m3, 16 bytes of scale factors, and 128 bytes of RoPE in BF16. The sparse indices are page-aware. The decode loop is `get_mla_metadata` once, `flash_mla_with_kvcache` per layer. That's the shape of the code that lets DeepSeek V3.2 ship 1M context at a cost most MHA models can't match at 128K. vLLM prefix caching is the production reference for cache reuse across requests. The mechanism is block-level hashing: a hash covers parent hash, block tokens, and extra hashes for LoRA IDs, multimodal input hashes, and cache salts. The block size is fixed and small enough to keep collision risk manageable. xxhash is faster than sha256 but warns about collision risk in multi-tenant environments. There's an LRU free queue. None of it is glamorous, all of it is load-bearing. The three names together explain why long context is a viable product category at all in 2026. Without the kernels, the architectural moves are theoretical. With them, the cost ratios land where the architecture said they would. ## What this means if you're shipping The architectural argument is settled. RAG is dominant past 500K tokens, and it's dominant because the cost ratio is dominant. 17x isn't a quality preference, it's a budget line. Past 500K, RAG with a reranker is cheaper than long context by enough that the long-context choice only makes sense if RAG can't get you the answer. The corollary is less obvious. The 17x number compounds. If you're running 1000 active users on a 200K window on Opus, you're spending $1,000 per turn of your entire user base. The same 1000 users on a 4K RAG pipeline costs $60 per turn. The 17x is per turn, not per session. Across a session of 20 turns, the same answer costs you $20 on long context versus $1.20 on RAG. The annualised number on a million-turn-per-day production system is the difference between a line item and a balance-sheet event. The 4K figure on the RAG side isn't arbitrary. It's what a two-step retrieval interface gets you. Ten full matches at 1,500 tokens each is 15,000 tokens injected into context whether the agent uses them or not. Identifiers and short previews for the same ten matches run about 450 tokens, and only the records the agent actually needs get fetched in full after that. Across 20 recall steps in a session, the difference compounds to around 200,000 tokens saved. That's not a model choice or an architecture choice. It's a retrieval-interface decision, and it costs nothing to adopt. The decision isn't "use long context" or "use RAG". It's "use long context for the queries where the answer can't be retrieved, and use RAG for the queries where it can". Most production systems are somewhere around 80 percent RAG-friendly, 20 percent genuinely long-context. The 80 percent should not be paying for the 20 percent's window. The shame's in not knowing which trade-off you're making. The 17x is real. The architecture moves are real. The compression cliff is real. The decision is which slice of your traffic pays the long-context tax, and whether that slice is the one that genuinely needs it. The third article in this series covers what the harness around the model looks like when the cost is this asymmetric and the model isn't going to get cheaper. The short version: the work is moving into the prompt, into the tools, into the subagent structure, and into the cache policy. The model is the constant. Everything around it is what's changing. The unit of engineering isn't the model. It's the harness. That's where the rest of the series goes from here.

You bought a 1M context window. You got 50x less than you paid for.

Steven Batchelor-Manning — Sun, 14 Jun 2026 00:00:00 GMT

Every vendor's headline context number is a lie. Not a small lie. A 50 to 100 times lie, depending on what the model is being asked to do. The architecture accepts the input. The model does not read it. The headline is uncomfortable. Vendors quote window sizes that are 2 to 8 times larger than what the model can actually use for retrieval, and 50 to 100 times larger than what it can use for reasoning. Both numbers come from the same body of benchmark work. Neither is a guess. The same vendor's newer model on the same multi-needle benchmark can score four times higher than the model it replaces, at the same advertised window. A 10M-token window can lose to a 2M-token window on comprehension of a single book. This article opens a three-part run on context: weight, cost, and management. Subsequent articles cover what the work-window actually is, what it costs you per active user, and how the systems being shipped in 2026 organise themselves around the gap. ## What advertised context actually means Every frontier vendor publishes a context window number. Gemini 3 Pro: 1M. Claude Opus 4.6: 1M. GPT-5: 400K. Llama 4 Scout: 10M, the largest of any production model. These numbers are real in one narrow sense. The architecture accepts that many tokens as input. The model does not break when you hand it a prompt that long. What the architecture accepts and what the model can use are two different quantities. The first is the size of the door. The second is how much of what's in the room the model can actually see when it's asked to do work. The way to find the second number is to test the model on tasks that require using material from across the window, then watch where the score falls off. That's what the recent wave of long-context benchmarks is doing. MRCR v2 puts eight needles in the haystack and asks the model to recall them in order. NoLiMa asks the model to reason across passages where the keyword overlap has been deliberately stripped out, so retrieval by similarity can't carry it. HELMET tests downstream task performance at 128K. Fiction.LiveBench gives the model full books and asks comprehension questions that only work if the model tracked what was in the middle. Each of these is a different lens. None of them is the vendor's needle-in-a-haystack test, and that's the point. ## What the numbers actually look like The per-model picture, as of mid 2026, is uneven in a way that should embarrass the field. Anthropic's MRCR v2 8-needle test at 1M tokens shows Sonnet 4.5 at 18.5 percent and Opus 4.6 at 76 percent. Same vendor, same benchmark, same advertised window. The newer generation is over four times better at the task the window was sold to do. If the window were the thing that mattered, those two numbers would be close. They are not. Llama 4 Scout advertises 10M tokens and scores 15.6 percent on Fiction.LiveBench at 128K. Gemini 2.5 Pro on the same test scores 90.6 percent. Scout has 80 times the advertised window of older Gemini generations and a fraction of their effective context on the harder tests. The ratio of advertised to effective context, on this benchmark, is the worst of any model shipping in 2026. The ofox.ai benchmark set, which is the most cited practitioner-facing comparison right now, shows the same spread. Gemini 3.1 Pro Deep Think hits 99 percent on NIAH-2 single-needle at 1M. Most of the other models cluster much lower on the harder tests at the same length. The single-needle number is what vendors put in slides. The harder tests are what production agents hit when the user pastes in a 400-page document and asks a question about page 312. The summarising claim, drawn from the same source material: advertised context windows are typically 2 to 8 times the effective context for multi-hop work, and 50 to 100 times the effective context for reasoning tasks. Both ends of that range are real. Both come from the same benchmark families. ## Why the framing matters more than the number Different vendors describe the same problem in different ways, and the framing they pick tells you how seriously they're taking it. Anthropic uses the phrase context rot. Their September 2025 article on effective context engineering for agents put the term into mainstream engineering discourse. The framing is front-footed. The vendor is naming a problem they say their newer models handle better, and pointing to the difference between advertised and effective as the gap they're closing. Sonnet 4.5 to Opus 4.6 is the proof point. DeepMind prefers effective context. Same underlying phenomenon, more neutral language. They publish effective-length numbers on specific tests rather than claiming the window is fully usable. OpenAI leans on needle-in-a-haystack. NIAH is the most generous of the long-context tests. It puts a single isolated fact in a haystack and asks the model to recall it. The model doesn't have to use information across the window, only from one position. Vendor benchmark numbers that look like 96 percent recall at 1M are usually NIAH-2 single-needle. They don't predict how the same model will do on a comprehension question that spans the document. Meta advertises a 10M window on Scout and provides no specific effective-length numbers. The marketing is the message. The asymmetry is worth holding. When the vendor is naming the gap, the gap is being worked on. When the vendor is showing only the most generous benchmark, the gap is being hidden. ## What the test results actually say MRCR v2 multi-needle is the test most often cited as the honest one. It asks the model to recall eight pieces of information from across the window and reproduce them in order. The order requirement is what kills naive retrieval. Even a model that finds every needle can fail the test if it can't recover the sequence. NoLiMa is the reasoning test. It strips literal keyword overlap from the question and from the supporting passages, so the model has to do semantic inference rather than pattern-matching. At 64K context, NoLiMa scores are noticeably lower than at 4K, even on the best models. The drop is the gap between retrieval and reasoning. It's the gap between finding the right passage and being able to use it once found. HELMET tests downstream task performance at 128K. RAG, in-context learning, re-ranking, summarisation. The scores on HELMET are uniformly lower than the vendor headline numbers. The drop is the same shape as the drop on NoLiMa. More tokens, less useful per token. Fiction.LiveBench is the test that's hardest to argue with. The model gets a real book. The questions are about events and relationships in the book. There's no clever prompting that fixes a model that lost track of what happened on page 40 by the time it gets to page 200. Llama 4 Scout at 15.6 percent on Fiction.LiveBench at 128K is the data point that anchors the whole conversation. A 10M-token window doesn't help if the model can't answer questions about a book at one percent of that length. The clearest single comparison across these tests is GPT-5.5 on NIAH-2 single-needle at 1M versus GPT-5.5 on MRCR v2 multi-needle at 128K. The first number is 96 percent. The second falls well short. Same model, same vendor, same marketing page. The single-needle number is what fills the slide. The multi-needle number is what fills the production incident log. ## What the Chroma study actually showed The empirical work on the gap that matters most is the Chroma Research context rot study, published July 14 2025. It tested 18 models from four vendors on four experiment types. The headline is that there is no cliff. Degradation is monotonic from the shortest contexts onward. The rot is continuous, not a step. The four experiments matter because they isolate different variables. The first looked at needle-question similarity and found that lower cosine similarity between the embedded question and the embedded needle predicts a faster degradation rate. The second looked at distractors and found they lower aggregate accuracy by about one percentage point across all models and haystacks combined. That number is small, and it surprised people who assumed distractors were the problem. The third looked at needle-haystack similarity and concluded it is not the controlling variable. The fourth looked at haystack structure and found that shuffling the haystack destroys performance more than adding distractor needles. The structure of the surrounding text matters more than the count of irrelevant chunks. Put plainly: the model is not bad at ignoring irrelevant text. It's bad at using relevant text once there's enough of it. The failure is on the use side, not the filter side. ## Where the work actually sits The result that most changes how I think about agent architecture is a side-finding from the same body of work. When the same model is run through different harnesses, the gap between harnesses is bigger than the gap between models on most tasks. | Harness | Model | Score | Delta | |---|---|---:|---:| | Cursor | Claude Opus 4.6 | 93% | — | | Claude Code | Claude Opus 4.6 | 77% | −16pp | Cursor on Claude Opus 4.6 scores 93 percent. Claude Code on Claude Opus 4.6 scores 77 percent. Same model, same tasks, sixteen points apart. The harness, the system prompt, the tool surface, the way the context is sliced before it reaches the model, all of that is in that sixteen points. The implication is uncomfortable. Most of what gets attributed to the model is the model plus the harness plus the system prompt plus the way the developer chose to fill the context. None of those is fixed by buying a bigger window. All of them are within the developer's control. The pattern across all this is consistent enough to state plainly. The advertised number is the size of the door. The number that does work is the size of the room the model can see when it's actually doing the work. The two are different. The difference is not small. The difference is not closing on its own. Buying a bigger window does not close the gap. Building a better harness, slicing context more deliberately, and choosing what to put in front of the model based on the task, those close the gap. Once the gap is real, three things follow for anyone building an agent in 2026. The first is that the choice of window is downstream of the choice of test. A team that picks a model based on the NIAH-2 single-needle number is going to ship a system that breaks on the multi-needle and reasoning tests the model can't pass. The benchmark that gets cited in the procurement meeting is the benchmark that should be the most suspect, not the most reassuring. The second is that the cost of context is not the cost of the input. It's the cost of every retrieval step, every turn, every rerank call, multiplied by whatever fraction of the window the model can actually attend to. Doubling the window doesn't double the useful work. It dilutes it. The third is that the way the context is sliced before it reaches the model is part of the model, in every practical sense. The Cursor 93 percent versus Claude Code 77 percent isn't an edge case. It's the central case. Two teams, same model, sixteen points apart, all in the harness and the system prompt and the context curation. That's where the engineering goes from here. That's the argument this series opens with. The rest of the run is about what each of those levers looks like in practice, what it costs, and how the 19 systems I've been studying approach it. If you've been filling 200K-token windows and wondering why the agent loses the thread, the answer is almost never that you needed 400K. The shame's in still treating the advertised number as the engineering number after the benchmark families have made clear it isn't.

Your memory system does not need to decide what the agent sees. The agent does.

Steven Batchelor-Manning — Tue, 02 Jun 2026 00:00:00 GMT

I've been down the rabbit hole on how memory reaches the model for a while now. The assumption I kept running into, before this research, was that the hard part of agent memory was retrieval quality. Get the right chunks, the thinking goes, and the rest is plumbing. That assumption is half right. Retrieval quality matters. But the mechanism that delivers those chunks to the model matters more, and the field has quietly reversed its position on it without most people noticing. The mature systems have all converged on the same shape: the agent is given memory tools, and the agent decides when and how to use them. The middleware is no longer a predictor of what the agent needs. It's a service the agent calls. This is not a small change. It happened almost without comment. And if you're still building injection-style memory in 2026, you're working harder than you need to. --- ## What injection looks like Early RAG was injective. The pattern was simple enough to fit on a slide: 1. Embed every document in the corpus. 2. On every user turn, embed the query. 3. Run a top-K nearest-neighbour search. 4. Concatenate the K results into the system prompt. 5. Send the resulting blob to the model. The model never asks for context. The middleware decides what's relevant. The model sees the result as a fait accompli, prepended above the user's actual question. This is automatic context injection. It was the dominant shape of RAG from roughly 2023 to 2024 and it's still what most teams mean when they say "we added RAG" without further qualification. It works, up to a point. The point where it stops working is the point where the agent needs to do something the middleware didn't predict. Feel the difference concretely. Suppose the user has a memory store containing notes about a deployment incident from the prior week, a long-standing preference for terse responses, a half-finished design document about an authentication refactor, and a transcript of yesterday's standup mentioning that the auth work has been deprioritised. The user asks: "Where are we on the auth refactor?" Under injection, the middleware embeds the query, searches, and concatenates the top-K chunks. The model gets back a chunk from the incident postmortem about an auth-related rollback, a chunk from the design doc, a chunk from the standup mentioning deprioritisation, and five other chunks that happened to share the word "authentication" with varying degrees of relevance. The model reads all eight. It answers. The user sees the answer. Nobody sees the eight chunks. Under tools, the agent reads the question, decides it needs to check memory, calls a search tool with a query it composed itself, gets back identifiers and previews, reads the previews, decides which records are relevant, and fetches only those. The trace shows every step. The agent retrieved two records, not eight. It chose them. The cost reflects the choice. The difference isn't retrieval quality. The same chunks exist in both traces. The difference is agency: who decided what the model sees. --- ## Four failure modes of injection ### 1. The irrelevant-chunk tax Injection pays retrieval cost on every turn whether or not memory was relevant. If the user asks "what time is it?", the middleware still embeds the query, still searches the store, still concatenates K chunks into the prompt. The model still processes them. None of that work was necessary. On a system doing thousands of agent turns per day, the waste is real and it compounds. ### 2. The wrong-K problem Top-K retrieval returns the K chunks most similar to the query embedding. But similarity to the query isn't the same as relevance to the task. The middleware can't tell the difference because it doesn't understand the task. It understands the query string. The model could tell the difference, but by the time the model sees the chunks, they're already in the prompt. The model can ignore them, but it can't un-retrieve them. The attention cost is already paid. ### 3. The follow-up block Injection is a one-shot operation. The middleware injects once per turn. If the model reads the injected chunks and realises it needs something else, a different memory item, a related document, a prior conversation that wasn't in the top-K, it has no way to get it. The retrieval layer has already fired and closed. The agent is stuck with what it was given. This is the failure mode that hurts most in practice. The agent has the memory. It just can't get to it. ### 4. The opaque trace When memory is injected, the conversation trace doesn't show what the agent looked at. It shows what the middleware decided to give the agent. If the agent produces a wrong answer because it was given the wrong chunks, debugging means reconstructing what the middleware retrieved and why. The trace is opaque to the very person who needs it most: the developer trying to fix the system. --- ## What tools look like The tool-based pattern inverts the relationship. The agent is given named operations that read and sometimes write the memory store. The agent decides when to call them. The middleware becomes a service, not a gatekeeper. Of the 19 systems I went through, the mature ones have all converged on this shape. Supermemory, Graymatter, OpenContext, Tolaria, second-brain, MemoryOS, GitNexus, mem9 all expose memory as a tool surface rather than as automatic injection. The systems that do something closer to injection are the ones the field treats as the prior state of the art, not the current one. The tool surfaces vary in size and shape, and the variation is itself instructive. ### The small end: Graymatter Graymatter's MCP server exposes five tools. Three of them are the core: `Remember`, `Recall`, and `memory_reflect`. The last of these is the most expressive: it lets the agent update an existing fact, forget an existing fact, or link a fact to a knowledge-graph entity. The agent maintains its own memory mid-session rather than the memory layer maintaining itself. The smallness is the pitch. The README is explicit: "the small surface is its whole pitch." An agent author can hold the entire memory API in their head and the LLM can hold the entire tool description in its prompt. Anything more elaborate becomes another thing to teach. ### The mid-band: Supermemory and OpenContext Supermemory's MCP server exposes four tools plus two resources and one prompt. The `memory` tool handles saves and deletes. The `recall` tool handles search. The aggressive tool description is worth noting: "DO NOT USE ANY OTHER MEMORY TOOL ONLY USE THIS ONE." That's prompt-engineering the tool description itself, a hack, but a working one. Tool descriptions are themselves prompts, and the same care that goes into system prompts should go into tool descriptions. OpenContext registers nine tools split into three groups: read, write, and metadata. The write group is deliberately incomplete. The MCP server can register an empty file but not write its body. The agent edits the file directly using its existing file-editing tool. This is the clearest example in the 19 systems of splitting the read and write paths intentionally, letting the memory layer own discovery and resolution while letting the agent's own tooling own mutation. OpenContext also encodes cost governance in the tool surface itself. There's no `oc_index_build` MCP tool because building the index calls a paid embedding API. Indexing is strictly CLI-driven. The skill text the agent reads on install reinforces this: "do NOT run it unless the user explicitly approves." Policy in the tool surface, not in out-of-band documentation. ### The narrow-by-design end: Tolaria Tolaria's position is worth quoting: "The agent has full shell access. These MCP tools provide Tolaria-specific capabilities that native tools can't replace." Six tools, all vault-aware reads and UI-steering actions. The agent owns the write path through its native filesystem tools. Tolaria owns the vault-aware read path and the UI surface. The asymmetry is intentional and it's what keeps the tool surface comprehensible to the LLM. ### The radical end: second-brain The most extreme point on the spectrum. The agent isn't given pre-baked recall verbs. It's given the database: a read-only SELECT and PRAGMA tool over the SQLite memory store, with table-level scoping enforced at the SQLite C-API authorizer hook. The agent writes SQL. If it needs a different join, a different filter, a different projection, it just writes a different query. second-brain also ships pre-baked recall tools for common cases: `hybrid_search`, `lexical_search`, `semantic_search`. These sit alongside the raw SQL tool. The agent can use the convenience verbs when they suffice and drop down to SQL when they don't. The tool surface doesn't need to be one or the other. It can offer both, and let the agent choose. ### MemoryOS: three tools, hidden hierarchy MemoryOS wraps its three-tier hierarchical store behind three tools: `add_memory`, `retrieve_memory`, and `get_user_profile`. The agent doesn't need to know about the short-term, mid-term, and long-term tiers. It calls `retrieve_memory(query)` and the system returns the best match across whichever tier owns the data. The complexity of the hierarchy stays inside the memory engine. The tool surface stays small. The internal model can be as sophisticated as it needs to be, heat-gated promotion, dialogue-chain reconstruction, parallel two-stage retrieval, without any of that sophistication leaking into the agent's tool descriptions. If the engine wants to reorganise its tiers behind the scenes, the agent doesn't have to know. ### mem9: same surface, many wrappers mem9 publishes a REST API and surfaces it through plugins for multiple agent frameworks. The plugin shapes are all over the place, the Claude Code plugin is bash hooks that curl the REST API, the Codex plugin is Node hooks with client-side conversation parsing, but the underlying agent surface is the same five or so operations: `store`, `search`, `get`, `update`, `remove`, plus an `ingest` for whole-conversation handoff. The same tool surface drives very different agent integrations because the surface itself is small enough to wrap many ways. The Claude Code shell hooks are 80 lines because there isn't much to wrap. Memory logic stays in the server. Plugins stay thin. --- ## The inversion: oh-my-kiro One system inverts the pattern in a way that's worth understanding. oh-my-kiro doesn't give the agent memory tools. Instead, it interposes on the agent's existing tool calls. When the agent calls a file-editing tool, oh-my-kiro's hook system intercepts the call, extracts the relevant context, and stores it in the memory layer. The agent never explicitly asks for memory. The memory layer learns from what the agent does. This isn't injection. The middleware isn't predicting what the agent needs and prepending it. It's observing what the agent does and recording it. Injection is push. oh-my-kiro's hooks are pull-by-observation. Both have a place, and a mature agent system might well combine them: tools for active recall, hooks for passive capture. --- ## The highest-leverage refinement Across the tool-based systems, one pattern costs almost nothing to adopt: self-guiding tool responses. Every tool response ends with a hint about what to do next. GitNexus does this explicitly: every tool response includes a `Next:` line suggesting the most likely follow-up call. The agent learns the API through use rather than through documentation. The trace becomes self-documenting. A developer reading it can see not just what the agent called but what the system suggested the agent call next. GitNexus has seven or more tools grouped by process: hybrid search, 360-degree symbol context, impact analysis, even a Cypher escape hatch for graph queries. A surface that size could easily overwhelm an LLM that's never seen it before. The `Next:` hints are what make it navigable. The agent calls a tool, reads the response, and the response tells it what to consider next. No documentation required. The cost is one line per tool response. The benefit is that the agent stops guessing about API shape and starts following the system's own understanding of what comes next. --- ## The two-step rhythm The natural structural fit for tool-based memory is two-step retrieval. Search returns identifiers and short previews. A separate GetByID call fetches the full record when needed. mem9's MemoryRepo interface is built around this pattern. MemoryOS wraps it behind a single `retrieve_memory` call but the internal implementation is the same: search first, then fetch. The numbers make the case plainly. Ten matches at 1,500 tokens each is 15,000 tokens injected into context whether the agent uses them or not. Two-step retrieval returns 10 identifiers and short previews at roughly 450 tokens total, then fetches only the records the agent actually needs. Across 20 recall steps in a session, that difference compounds to around 200,000 tokens saved. No architectural change to the memory store. No additional LLM calls. No information loss. It's a retrieval interface decision. --- ## What the shift implies If your memory system still auto-injects, you're working too hard. You're predicting what the agent needs without the agent's input. You're paying retrieval cost on every turn whether or not memory was relevant. You're blocking the agent from following up on partial results. You're making the trace opaque to your own debugging. You're doing the agent's job for it. The agent should ask. Give it the tools and trust it to use them. Make the tools' responses guide the next call. Make the search-then-read split the default rhythm. Then get out of the way and let the trace tell you, after the fact, what the agent actually needed, which is the question you should have been asking all along.

You are not running out of tokens. You are wasting them. Here is the difference.

Steven Batchelor-Manning — Sat, 30 May 2026 00:00:00 GMT

I have been down the rabbit hole on context budget management for a while now. The assumption I kept running into, before I started this research, was that longer context windows had more or less retired the problem. If you can fit 200,000 tokens in a single call, the argument goes, you stop worrying about what goes in. That assumption is wrong. It is not even close to right. The finding that keeps surfacing across the 19 systems I looked at is that bigger windows intensify the budget problem rather than dissolve it. A 200K-token window does not pay equal attention to all 200K tokens. Performance degrades long before the window fills. The degradation is non-uniform: material in the middle of a long context is reliably attended to less than material at the edges. And the agent's actual task occupies a fixed slice of the window regardless of how large the window is, which means everything else is overhead competing for the same attention budget. The systems that handle this well have converged on six mechanisms. None of them are exotic. Several are embarrassingly simple. But the ones that skip them pay for it. --- ## The six mechanisms ### 1. Compaction passes The most visible mechanism. You take a long conversation or a large memory segment, summarise it, and replace the original with the summary. MemoryOS does this at the segment level: its segment summariser fires when a conversation segment grows past a threshold, collapsing it to a compact representation before it can crowd out working context. The Karpathy-pattern wikis (purpose.md, overview.md) do a version of this at the knowledge level: the wiki is the compacted form of everything the agent has learned about a topic, maintained across sessions. The trade-off is information loss. Compaction is a lossy operation by definition. The summary captures what the summariser judged relevant at the time of compaction. If the agent later needs a detail that was not judged relevant, it is gone. This is not a reason to avoid compaction, but it is a reason not to treat it as the only mechanism. There is a second cost that is easy to miss. Compaction is not free at runtime. MemoryOS can pay 20 or more LLM calls in a single interaction to maintain its segment summaries. For systems with high interaction frequency, that is a real operational cost. ### 2. Result-preview truncation Rather than returning full memory content on every retrieval, return a short preview and let the agent decide whether to fetch the full record. supermemory exposes snippet-length controls that let callers tune how much text comes back per result. mem9 goes further: it decorates source turns with three environment variables (MEM9_SOURCE_TURN_MIN_SCORE, MEM9_SOURCE_TURN_PER_MEMORY_LIMIT, MEM9_SOURCE_TURN_TOTAL_LIMIT) that give operators precise control over how many source turns appear and at what minimum relevance score. The trade-off is an extra tool call. If the agent needs the full content, it has to ask for it explicitly. For most retrieval patterns this is the right trade-off: the agent gets enough signal to decide whether the record is relevant before paying the token cost of reading it in full. ### 3. Two-step retrieval A specific and important variant of preview truncation. Search returns identifiers and short previews. A separate GetByID call fetches the full record when needed. mem9's MemoryRepo interface is built around this pattern: search and fetch are distinct operations with distinct token footprints. The numbers make the case plainly. Ten matches at 1,500 tokens each is 15,000 tokens injected into context whether the agent uses them or not. Two-step retrieval returns 10 identifiers and short previews at roughly 450 tokens total, then fetches only the records the agent actually needs. Across 20 recall steps in a session, that difference compounds to around 200,000 tokens saved. This is the cheapest discipline you can adopt. It requires no architectural change to the memory store, no additional LLM calls, and no information loss. It is a retrieval interface decision. ### 4. Decompose-then-recall Rather than sending the full user query to the retrieval layer, decompose it into sub-queries first. SimpleMem's intent-aware retrieval planner breaks incoming queries into atomic retrieval intents before hitting the memory store. GitNexus does something similar with its query tool decomposition: complex queries are split into targeted sub-queries, each of which retrieves a focused slice of the memory graph. The benefit is precision. A decomposed query retrieves less irrelevant material, which means less noise in context. The trade-off is latency: decomposition adds a planning step before retrieval begins. For interactive agents this matters. For batch or background agents it usually does not. ### 5. Tiered storage as budget filter If you have already built a tiered memory architecture (the subject of last week's piece), you get budget filtering as a side effect. supermemory's three-tier model means that hot, frequently-accessed material lives in a tier that returns compact, high-signal results. Cold material is in a tier that is not queried by default. Hindsight's observation tier works the same way: raw observations are not injected into context directly; they are promoted to higher tiers before they become retrieval candidates. The trade-off is recall completeness. Material that has not been promoted may be relevant but will not surface in a standard retrieval pass. This is the same trade-off as compaction, but the failure mode is different: instead of losing information through summarisation, you lose it through demotion. ### 6. Self-guiding tool responses The least discussed mechanism, and one of the more interesting ones. Rather than leaving the agent to decide what to do after a tool call, the tool response itself includes a hint about what to do next. GitNexus appends a `---\n**Next:**` block to tool responses, suggesting follow-up actions. mem9 decorates source turns with structured metadata that guides the agent's next retrieval step. The effect is that the agent spends fewer tokens on planning between tool calls. The tool response carries enough structure to make the next step obvious. The trade-off is prompt-engineering effort: writing good self-guiding responses requires knowing in advance what the agent is likely to need next, which is not always possible. --- ## The Tolaria limit case Tolaria is worth looking at separately because it represents the logical endpoint of budget discipline taken to its extreme. ADR-0009 documents the decision to remove embeddings entirely from the system. Tolaria uses substring-only search. No vector index, no semantic retrieval, no embedding calls. The reasoning is direct: the cheapest token is the one you never retrieve in the first place. Embedding-based retrieval returns semantically similar results, which means it returns results the agent did not explicitly ask for. Some of those results are useful. Many are not. All of them cost tokens. Tolaria's position is that the cost of irrelevant-but-similar results, compounded across a session, exceeds the benefit of semantic recall for its use case. Whether that trade-off holds for your system depends on what your system is for. For systems where queries are precise and structured (code navigation, document lookup by identifier), Tolaria's position is defensible. For systems where queries are vague and exploratory, removing embeddings breaks recall in ways that are hard to recover from. The value of the Tolaria case is not that you should copy it. It is that it makes the cost of semantic retrieval visible in a way that most systems do not. --- ## The case against compaction-only systems Several of the 19 systems rely on compaction as their primary or only budget mechanism. The failure modes are worth naming. The first is that summarisation loses details that were not judged relevant at compaction time but become relevant later. This is not a hypothetical: it is the standard failure mode of any lossy compression scheme applied to information whose future relevance is unknown. The second is that compaction is a hot-path cost. MemoryOS paying 20+ LLM calls per interaction is not unusual for compaction-heavy systems. At scale, that cost is not negligible. The third, and most subtle, is that compaction without an escape hatch is slow forgetting. If the only way to reduce context size is to summarise, and summaries are lossy, then the system is continuously discarding information with no way to recover it. Two-step retrieval, tiered storage, and result-preview truncation all preserve the original record. Compaction does not. None of this means compaction is wrong. It means compaction alone is not enough. --- ## Recency weighting and the persistent queue Two mechanisms that do not fit neatly into the six categories above are worth noting. graymatter uses RRF fusion with recency at half-weight. This is not a budget mechanism in the strict sense, but it functions as one: by down-weighting older material in retrieval rankings, it reduces the probability that stale, low-signal records crowd out recent, high-signal ones. The effect is soft tiering through ranking weights rather than explicit tier promotion. llm-wiki's 540-line ingest queue state machine takes a different approach. The queue serialises ingest operations and applies a four-signal relevance ranker before anything enters the memory store. Budget control happens at write time rather than read time. Material that does not clear the relevance threshold is not stored, which means it cannot be retrieved and cannot consume context. This is indirect budget control, but it is durable: the savings compound across every future session. --- ## What the well-designed systems have in common Looking across the 19 systems, the ones that handle context budgets well share a few properties. They treat retrieval as a two-step operation rather than a one-step injection. They return previews before full records. They preserve original records rather than replacing them with summaries. They give operators control over retrieval volume through explicit parameters rather than hardcoded defaults. And they think about budget at write time as well as read time. The ones that handle it poorly tend to rely on a single mechanism, usually compaction, and treat the context window as a buffer to be filled rather than a resource to be managed. The closing position from the research is simple. Bigger windows demand more discipline, not less. Not because filling them is wrong in principle, but because filling them with the wrong material costs more than leaving the room empty. --- *Next week: the shift from memory-as-injection to memory-as-tools, how the 19 systems handle the boundary between what gets pushed into context automatically and what the agent has to ask for explicitly.*

Storage is cheap. Attention is expensive. Are you using the system that exploits the difference?

Steven Batchelor-Manning — Tue, 26 May 2026 00:00:00 GMT

## The asymmetry that makes tiering pay back If you change one thing about how you design agent memory, change this: stop treating storage cost and attention cost as if they're the same thing. I've been going deep on this across 19 systems, and the finding that keeps surfacing is that the ones that handle memory well aren't necessarily the ones with the most sophisticated retrieval. They're the ones that figured out which memories belong in the prompt at all. That's a different problem, and it has a different solution. Storage is cheap. Attention is expensive. A 128k-context model reading 110k of mediocre context isn't, empirically, a better agent than the same model reading 8k of carefully selected context. The research on this is consistent: retrieval quality degrades as context fills with noise, and the degradation isn't linear. The model doesn't simply ignore the irrelevant material. It processes it, and the processing crowds out the signal. Storage doesn't have this property. A fact sitting in a database costs nothing to keep. The cost only arrives when you retrieve it and load it into the prompt. Which means the question isn't "should I store this?" but "should I retrieve this, and if so, when?" That reframing is where tiering comes from. Different memory items have different access patterns. Some things an agent needs every turn: the user's name, their current project, their stated preferences. Some things it needs often but not always: recent decisions, open questions, episodic facts from the last few sessions. Some things it should be able to find when needed but should never burden every turn: meeting notes from three months ago, completed tasks, raw transcripts, one-off reference material. A flat store treats all three categories identically. Hot items pay the search cost of cold items. Cold items inflate the prompt with noise. There's no mechanism for items to graduate as they become more relevant, and no mechanism for items to age out as they become less relevant. The OS analogy is exact: CPUs have L1, L2, L3, RAM, SSD, and disk not because bytes are different but because frequency of access varies by orders of magnitude. Seven of the 19 systems I went through had already built explicit tiering before I started looking. Two of them are worth understanding in detail. --- ## MemoryOS as reference implementation MemoryOS is the clearest implementation of tiered memory in the 19 systems. Three tiers, each with a distinct data shape, a distinct latency budget, and a distinct role. The short-term tier is a Python deque with a maximum length of 10 QA pairs. No embeddings, no search index, no heat tracking. It's pure conversational raw material, the last ten exchanges, available at microsecond latency. When the deque fills, the oldest pair drains into the mid-term tier. The mid-term tier holds up to 2000 sessions. Each session carries a summary, an embedding, a keyword set, and heat counters. The tier is indexed with Faiss and searched by cosine similarity. When the tier reaches capacity, the coldest sessions are evicted. When a session gets hot enough, it's promoted to the long-term tier. The long-term tier is a 90-dimension psychology and alignment schema, two knowledge-base deques, one for user facts and one for assistant facts, each capped at 100 entries. This is the persistent layer, the one that survives across sessions and carries the durable model of the user. The heat formula that governs promotion is twelve lines of Python. Three signals: visit frequency, an LFU analogue; interaction depth, a proxy for topical engagement; and recency decay, exponential with a 24-hour half-life. A segment crosses the promotion threshold at 5.0. After promotion, the visit and interaction counters reset to zero, and heat collapses back to roughly 1.0. The design decision that's easy to miss: heat gates promotion, not retrieval. The retriever is purely semantic, cosine similarity over the mid-term embeddings. Heat is a background signal that decides whether a segment should graduate to the long-term tier. The two concerns are decoupled, and that decoupling matters more than the formula itself. What MemoryOS leaves on the table is worth naming. The coefficients are hardcoded at 1.0, there's no mechanism for learning weights from actual usage patterns. There are no demotion paths; once something reaches the long-term tier, it stays. And the formula optimises for frequency over importance. A critical but rare fact, a partner's name, a medical condition, a hard constraint, may never cross the promotion threshold if it only surfaces once. Once the mid-term tier evicts the segment, the fact is gone. --- ## Hindsight's theory-derived tiers Hindsight arrives at the same three-tier structure from a completely different starting point. Where MemoryOS draws on OS cache theory, Hindsight draws on cognitive science. The three tiers are World, Experience, and Observations. World holds objective claims about the universe, ground truth, always-on, long-lived. Experience holds first-person actions of the system itself, the episodic record. Observations hold consolidated beliefs derived from World and Experience facts, carrying source memory IDs, a proof count, and a history field that tracks how the belief has evolved. All three tiers live in the same database table, differentiated by a fact-type discriminator. Partial HNSW indexes are built per fact type. The schema is unified; the access patterns aren't. Promotion in Hindsight isn't counter-driven. It's batched LLM-driven consolidation. When a new fact is written, it's enqueued into an async operations table. A background worker fetches the new facts alongside existing overlapping observations, builds a batch prompt, and asks the model for creates, updates, and deletes. Source memories are stamped with a consolidated-at timestamp to prevent reprocessing. Every new fact, regardless of how frequently it's been accessed, gets considered for promotion to the Observations tier. That's the key difference from MemoryOS. By tying upper-tier promotion to consolidation rather than heat, Hindsight avoids the blind spot for critical-but-rare facts. A single consolidation pass considers every new fact. Frequency is irrelevant to whether something graduates. Put plainly: two systems, different first principles, different implementation languages, different target use cases, and both land on three tiers with raw material at the bottom, a working layer in the middle, and a synthesised persistent layer at the top. Both use async promotion. Both carry provenance back to lower tiers. That's what convergent evolution looks like in software architecture, and it's the strongest signal I know of that a pattern is load-bearing. --- ## supermemory's genre-conditioned tiering supermemory operates in the managed API deployment shape, which changes the implementation without changing the architecture. The three tiers are static profile, dynamic profile, and document and chunk store. The static profile holds stable long-term facts, the hot tier. It's returned as a static array from the profile endpoint, cached at the edge, with a latency budget of roughly 50ms. The dynamic profile holds recent and episodic context, the warm tier. Many entries carry a forgetAfter field that sets a TTL. The document and chunk store is the cold tier, queried via search endpoints when needed. Tier assignment happens at write time. An extraction LLM classifies each incoming memory with an isStatic boolean and optionally a forgetAfter value. The classification is enforced by a closed extraction prompt, uniform across all consumers. The managed API deployment shape enables three things that an in-process designer can't directly copy but should understand. Cold-tier data can sit on cheaper hardware, object storage for raw bytes, a standard relational store for metadata and chunks, with the hot profile cached separately at the edge. The hot tier gets its own endpoint with its own SLA, separate from the search path. And the extraction prompt is centralised, which means tier assignment is consistent in a way that per-agent classification rarely is. The trade-offs are real. A remote hot tier is only fast if the network is fast. The agent can't override the engine's tier classification. The extraction prompt is a black box. But the architectural pattern, hot tier as its own endpoint, cold tier on cheaper hardware, tier assignment at write time, is worth copying even if the deployment shape isn't. --- ## Promotion vs. fixed typology mem9 is the contrast case that clarifies what tiering isn't. mem9 has a memory-type column with three values: pinned, insight, and digest. Pinned memories are assigned by explicit content-write paths, manually created, protected from LLM reconciliation. Insights are assigned by every LLM-extracted write, mutable, versionable, supersedable. Both participate in the same hybrid recall with the same RRF scoring. The type field is a write-protection flag, not a retrieval filter and not a tier signal. This is typology, not tiering. The distinction matters because the two are easy to conflate. Typology describes governance: who can mutate this memory, under what conditions. Tiering describes access patterns: how frequently is this memory needed, and what storage representation best serves that frequency. A system can have both. supermemory's isStatic is a tier signal, while a separate isInference flag is closer to a governance class. But conflating them produces the most ambiguity in practice. If you find yourself adding a type field to your memory rows, the question to ask is whether the field describes an access pattern or a governance class. If it's an access pattern, you're building tiering. If it's a governance class, you're building typology. Both are useful. They're not the same thing. --- ## What flat costs you Four concrete things follow from running a flat memory store. The hot path pays the cold path's search cost. Every retrieval scans the same index over the same items. The agent looking up the user's name pays the same search cost as the agent looking up a meeting note from six months ago. At small scale this is invisible. At scale it's a latency problem. The cold path inflates the prompt with noise. Retrieval returns semantically near items, which includes standing context, superseded facts, and material that's factually correct but irrelevant to the current turn. The model processes all of it. The signal-to-noise ratio in the context window degrades as the store grows. There's no mechanism for items to graduate. Memory isn't static. A fleeting episodic fact from early in a relationship may, over time, become a durable signal about the user's preferences or constraints. A flat store gives no machinery for noticing that transition. The item stays in the same representation it was written in, regardless of how its relevance has changed. There's no mechanism for items to age out. Demotion isn't deletion. A flat store that wants to remove stale material must delete it. A tiered store can move it to a colder representation, still findable, no longer burdening the hot path. The flat store forces a binary choice that the tiered store doesn't. --- ## The closing position 18 of the 19 systems implement tiering, gesture at it, or have explicit recommendations for it. The exception is mem9, which has a typology layer solving a different problem and would benefit from tiering on top of it. If you're building agent memory from scratch, the progression the 19 systems point to is this. Identify what the agent needs every turn, that's your hot tier. Identify what it needs often but not always, that's your warm tier. Identify what it should find when needed but never burden every turn, that's your cold tier. Pick a promotion mechanism: heat-based like MemoryOS, LLM-judgement like Hindsight, or extraction-time classification like supermemory. Pick a demotion mechanism: time decay, TTL, or freshness categories. Keep promotion and retrieval scoring separate at first, the decoupling is easier to add than to unpick. Track provenance from upper tiers back to lower tiers, so you can always answer the question of where a synthesised belief came from. The 19 systems aren't unanimous on much. On this, they are: a single flat memory store is the wrong default for any non-trivial agent memory system. Storage is cheap. Attention is expensive. Build the system that exploits the difference. --- *Next week: context budget management, how the 19 systems handle the constraint that the context window is finite, and what the ones that handle it well have in common.*

Almost every serious memory system made the same retrieval decision. Here's why.

Steven Batchelor-Manning — Thu, 21 May 2026 00:00:00 GMT

Ask a flat-vector retrieval system to find the note that mentions the string `idx_memory_units_text_search`. The embedding model has no privileged representation for an arbitrary identifier. The tokeniser splits it into pieces, the encoder averages those pieces into a vector that looks much like every other identifier-shaped vector, and the note may or may not surface in the top fifty results. A keyword search returns it instantly. Now ask a keyword search to find notes about "authentication" when every relevant note uses the word "login". Exact-match is exact-match by construction. Stemming and stop-word removal help at the margins; they do not bridge the semantic gap. The note is not found. These two failure modes are not edge cases. They are the default failure modes of single-index retrieval, and they are why almost every system in the 19 I went deep on has either abandoned flat-vector search or wrapped it behind something else. The something else is almost always the same thing. Hybrid retrieval -- running both lexical and semantic search and fusing their ranked lists -- has become the consensus architecture for serious memory systems. The fusion algorithm is Reciprocal Rank Fusion, almost universally at the constant `k=60`. The corpus disagrees about nearly everything else: what to extract from a conversation, when to forget, whether memory is files or rows, whether the agent should drive retrieval or be handed results. On the question of how to order a candidate list, it has converged so completely that `k=60`, inherited from a 2009 information-retrieval paper, has become a magic number copied without comment from one implementation to the next. ## Why the two lanes are not interchangeable Dense retrieval and lexical retrieval fail in opposite directions, which is why combining them works. Dense retrieval handles semantic variation well -- "login" and "authentication" land near each other in the vector space -- but struggles with exact identifiers, rare tokens, and anything the embedding model has no privileged representation for. Lexical retrieval handles exact terms, identifiers, and rare strings well but cannot bridge synonyms or paraphrase. The failure modes are complementary. Running both and fusing the results covers the ground neither covers alone. The fusion step matters because you cannot simply concatenate the two ranked lists. A document that ranks first in the vector lane and fifteenth in the keyword lane should score differently from one that ranks first in both. Reciprocal Rank Fusion handles this by converting each rank into a score of `1 / (k + rank)` and summing across lanes. The constant `k` controls how much weight goes to top-ranked items versus the rest of the list. At `k=60`, a rank-1 result scores `1/61` and a rank-60 result scores `1/120` -- a 2x difference. The algorithm is rank-based rather than score-based, which means it is robust to the different score distributions produced by different retrieval strategies. A cosine similarity of 0.87 and a TF-IDF score of 14.3 are not directly comparable; their ranks are. ## graymatter as the reference implementation If the corpus has a single canonical worked example of hybrid retrieval done cleanly, it is graymatter. The implementation is short, the choices are explicit, and the whole flow fits on a screen. Three rankings are produced independently. Vector ranking: cosine similarity between the query embedding and each stored fact's embedding. Keyword ranking: a TF-IDF-style score -- term frequency multiplied by a log-IDF factor, summed across query terms, divided by term count to dampen long facts. Not full BM25, but in the same family. Recency ranking: an exponential decay score, `exp(-lambda x age_hours)`, with a default half-life of 30 days. Each ranking produces a map from fact ID to rank. The fusion step sums `1/(60+rank)` across all three lanes for each fact, sorts descending, and returns the top results. Three lanes. One fusion. The recency lane is the interesting addition -- it means a fact that is semantically relevant and keyword-matched but six months old will score lower than a fresher fact with similar relevance. The decay is tunable; the discipline is that recency is a first-class signal rather than a post-hoc filter. ## GitNexus: lanes compose without limit GitNexus generalises the pattern. Where graymatter has three lanes, GitNexus has five separate full-text indexes -- one per content type -- merged by score summation, plus a BM25 lane and a dense vector lane fused with RRF. The `group_query` function runs cross-repository RRF, fusing results from multiple repositories into a single ranked list. The architecture documentation records `RRF_K=60` as the standard constant, with Elasticsearch and Pinecone running in parallel as the retrieval backends. The lesson from GitNexus is that lanes compose. You do not need to redesign the fusion step when you add a new lane. You add the lane, produce a ranked list from it, and hand it to the same `1/(60+rank)` summation. The algorithm absorbs the new signal without modification. This is why the corpus has converged on it: it is not just correct, it is extensible. ## Hindsight: RRF as a stage in a longer pipeline Hindsight is the most sophisticated retrieval architecture in the corpus. It runs four retrieval strategies in parallel -- dense vector, keyword, temporal, and graph-based -- fuses them with RRF, and then passes the fused list to a cross-encoder reranker. The cross-encoder reads the query and each candidate document together and produces a relevance score that is more accurate than any of the individual lane scores, at the cost of being more expensive to compute. Running it over the full candidate set would be prohibitive; running it over the top-N from RRF is tractable. The pipeline is: four parallel lanes, RRF fusion, cross-encoder rerank, final ranked list. Each stage narrows the candidate set so the next stage can be more expensive and more accurate. RRF is not the end of the pipeline; it is the merge stage that makes the expensive final stage feasible. Hindsight also does something the other systems do not: it preserves per-strategy scores and per-strategy ranks on the result struct alongside the fused score. The trace machinery lets an operator inspect a problematic recall and see exactly which retriever found which item, what its rank was in each lane, what the RRF score was, and what the cross-encoder did. This is unusual. Most systems return only the final result list. Hindsight's debuggability advantage is largely a consequence of this one decision -- keeping the per-lane data rather than discarding it after fusion. ## The provenance gap most systems leave open llm-wiki illustrates the cost of not keeping per-lane data. The fusion arithmetic is correct: `RRF_K = 60` is set at `src/lib/search.ts:53` with a comment citing the Cormack 2009 paper. But the result struct's `score` field is overwritten with the RRF score, discarding the original token score and vector score. The downstream rendering code receives a single number whose provenance has been lost. A user looking at a high-relevance result cannot ask whether it ranked highly because of semantic similarity or keyword match. This is the same pattern that appeared in the provenance piece: the cost of not having the data is deferred but not avoided. When you want to add a cross-encoder reranker, it needs the per-lane signals. When you want to build a UI that explains why a result was retrieved, it needs the per-lane signals. When you want to A/B test a new lane, you need the old per-lane numbers to compare against. The fused score is a local optimum for ordering. The per-lane scores are the substrate everything else is built on. The discipline is worth stating plainly: one score for ranking, one score per lane for explainability, and they live in different fields. ## The outlier worth taking seriously The strongest dissent in the corpus is not a variation on hybrid retrieval -- it is Tolaria, which removed embeddings entirely. Tolaria is a Markdown vault manager. Earlier in its life it shipped a semantic indexer; ADR-0009 removed it. The reasoning: the operational complexity of shipping a Go binary, code-signing it, auto-installing it, and surfacing its index status in the UI was not justified by the search-quality benefit for the specific workflow. The replacement is plain substring search over title and content, with title matches ranked above content matches. Crucially, Tolaria does not pretend that substring search is as good as hybrid retrieval. ADR-0009 is explicit: the AI agent provides an alternative for exploratory and semantic queries. The semantic-retrieval intelligence is shifted out of the system entirely and into the agent's reasoning. The agent can read manifest files, reason about which folders are relevant, and read full notes. This works because Tolaria expects to be paired with a capable agent whose context window is the retrieval budget. This is a real architectural position. The embedded-search-engine model assumes the system is responsible for finding relevant content. The agent-as-retriever model assumes the agent is responsible and the system's job is to surface structure the agent can navigate. The two models have different cost profiles, different deployment shapes, and different failure modes. Tolaria's position wins when the corpus is small enough to fit into the agent's context window in summary form, when the agent is capable enough to navigate structure intelligently, and when the operational cost of running an embedding pipeline is not justified by the query volume. It loses when the corpus is large, when queries are latency-sensitive, or when the agent cannot be trusted to navigate structure reliably. The honest framing: hybrid retrieval is the right default for systems that do serious retrieval. Tolaria's position is the right default for systems where the agent is the retrieval engine. Knowing which one you are building is the first decision. ## What the consensus actually says The corpus has converged on hybrid retrieval with RRF at `k=60` because the algorithm is correct, robust, and extensible. It is correct because it covers the complementary failure modes of dense and lexical retrieval. It is robust because it operates on ranks rather than raw scores, making it insensitive to the different score distributions produced by different retrieval strategies. It is extensible because lanes compose: adding a new signal means adding a new lane, not redesigning the fusion. The variation in the corpus is in what the lanes are, how many there are, and what gets layered on top of the fused list. graymatter shows the clean three-lane reference. GitNexus shows that lanes compose without limit. Hindsight shows that RRF is the merge stage of a longer pipeline, with a cross-encoder reranker sitting above it. mem9 shows that a managed-API system can hide the embedding step behind the database and amortise the cost across tenants. Tolaria shows that the entire hybrid-retrieval edifice rests on an assumption -- that retrieval is the system's job -- and that the assumption is challengeable. The single piece of practical advice from the corpus compressed into one sentence: use RRF at `k=60`, keep the per-lane scores on the result struct, and spend your engineering budget on the lanes rather than on the fusion. The algorithm has been right for long enough that you can trust it. The lanes are where the leverage is. The provenance is where the debugging is. The next piece covers tiered storage -- the pattern that separates systems that keep everything in one flat store from the ones that have learned to match the storage medium to the access pattern. That piece is coming up.

A fact without provenance is an island. Why every memory must carry its origin.

Steven Batchelor-Manning — Mon, 18 May 2026 00:00:00 GMT

Sit down with a flat-RAG system that has been running in production for a year and try to ask it four questions. "Why did you say that?" The model produced a confident answer about the customer's contract renewal date, but the trail back to the source clause does not exist. "Re-validate this claim -- the source changed." The contract was amended last week. Every fact derived from it is suspect. You cannot find them because they do not know which document they came from. "Decide between these two contradicting facts." One says Alex works at Google, another says Stripe. Neither carries a confidence score or a source timestamp. Recency of the write has nothing to do with recency of the evidence. "Attribute this hallucination." The model said the meeting was on Thursday. There was no meeting on Thursday. You cannot tell whether the LLM invented the date or whether bad data in memory misled it. These four failures share a single cause. They are each a missing column. A source identifier, a capture timestamp, a confidence score, a response citation log. Each costs a few bytes at write time. The cost of not having them is unbounded: every claim is unauditable, every contradiction is unresolvable, every hallucination is untraceable. Across 19 systems the pattern is consistent. The strongest implementations treat provenance the way a court treats evidence -- every claim arrives with an unbroken chain of custody or it does not arrive at all. The weakest carry none, and pay for it every time something goes wrong in production. This piece walks the six levels of provenance the corpus has surfaced, the three tiers of implementation maturity separating them, and the honest cost of building this right from the start. ## Six levels, one discipline Reading 19 systems back to back, six distinct levels of provenance separate themselves out. They are not a hierarchy; they are orthogonal. A system can have source provenance with no causal provenance. It can have versioning without confidence scoring. The strongest implementations cover all six. None of the weakest cover any. Identity answers "which fact, exactly?" OpenContext mints UUIDs for every piece of context at ingest time and exposes them as a citation scheme the agent can use directly in responses. The identifier survives renames, moves, and reorganisations because it is bound to the content, not the path. mem9 carries a stable memory ID on every row plus an explicit version counter with If-Match concurrency protection. Without identity provenance you cannot even name what you are talking about when things go wrong. Source answers "where did it come from?" Hindsight records source type and identifier on every observation, so the system can answer which conversation or document produced a given fact. mem9 carries source, agent ID, and session ID on every row. Supermemory stores document IDs alongside memories. Without source provenance you cannot cascade updates when a source changes, because you do not know which facts depend on it. Causal answers "which agent step used this?" Hindsight captures every retrieved-and-used fact in an observation tier with full retrieval context -- which retriever surfaced it, what rank it achieved, and whether the agent actually consumed it. Moraine treats every trace step as its own provenance record, making the agent's entire execution recoverable as a sequence of source-addressable events. Without causal provenance you cannot distinguish "the system retrieved this fact" from "the agent used this fact to produce that response." Capture confidence answers "how sure were we when we wrote it down?" Graphify marks every edge in its knowledge graph with three-level capture confidence: CONFIRMED for deterministic extractions, LIKELY for high-confidence LLM inferences, and AMBIGUOUS for uncertain claims. The AMBIGUOUS edges surface as "knowledge gaps" for human review rather than being silently treated as fact. Hindsight carries a confidence score on every observation that decays over time through its freshness lifecycle -- fresh observations are trusted, stale ones are down-weighted or retired. mem9 runs near-duplicate detection in shadow mode first, recording scores without acting on them until the engineer has calibrated the threshold from real data. Without capture confidence, uncertain facts and certain facts are retrieved with equal weight, and the agent cannot discriminate between a solid claim and an educated guess. Versioned answers "what did we believe before?" Supermemory treats memory as a versioned DAG with typed edges -- updates, extends, derives -- giving every belief commit history. mem9 splits the write path: in-place mutation for human edits, append-and-archive for LLM-driven rewrites where the new content semantically replaces the old. Tolaria lets Git carry the version history entirely, treating one-line diffs as a first-class user-facing artefact. Without versioned provenance you cannot rewind to what the system believed on Tuesday, because old beliefs are deleted rather than archived. Reciprocal answers "what other facts share this origin?" llm-wiki weighs source overlap above direct linking in its four-signal relevance graph -- two pages that came from the same raw document are presumed more strongly related than two pages with a direct wikilink, because the LLM is unreliable at cross-linking but the sources frontmatter is mechanically maintained. EdgeQuake accumulates source IDs on entities and relationships across the corpus, so repeated mentions of the same entity from different sources strengthen its provenance weight. second-brain carries source as part of its lexical-index composite key, letting a single document be indexed from multiple pipelines independently. Without reciprocal provenance you cannot answer "show me everything in this system that came from the same conversation" or "what else did we learn from that document?" The six are orthogonal but they reinforce each other. Source provenance is useless without identity (you need to name the fact before you can trace it). Versioning is weaker without confidence (you know the lineage of edits but not how sure the system was about any of them). Reciprocal queries depend on source being present first. The systems that carry all six are the ones whose memory does not silently rot. ## Three tiers of implementation maturity The corpus separates into three tiers based on where provenance lives and what it can do at read time. Tier 1 is no-provenance RAG, the starting point for most teams. Flat vector stores with content and an embedding. No source column, no confidence score, no version history. When a fact is retrieved, you get text and a similarity score. You cannot trace where it came from, how sure the system was about it, or whether it has been superseded. Every system that started here has moved toward Tier 2 over time. Tier 2 carries provenance on the row. Source ID, confidence, version -- all present as columns alongside the fact. mem9 sits here with source, agent ID, session ID, and version on every row. Supermemory's versioned DAG is Tier 2 structure. Graphify's three-level edge confidence is Tier 2 discipline. The provenance is available for queries, but it does not automatically decorate retrieval results. You have to write the query that uses it. Tier 3 decorates read-time results with provenance context without the caller having to ask for it. Hindsight is the reference here -- every retrieved fact arrives with its source type, confidence score, freshness state, and per-retriever ranking already attached. The agent consuming the result sees provenance as part of the fact, not as a separate lookup. mem9's source-turn decoration sits halfway between Tier 2 and Tier 3, grafting read-time context onto a Tier 2 schema. The migration is one-way. No system starts at Tier 3 and decides to remove provenance discipline. The columns prove their value as soon as they are present. If you are designing a memory system today, the question is not "do I need provenance?" but "which tier do I want to start at, knowing that every tier above the one I pick is harder to reach later than to bake in now?" ## The cautionary case: computed and discarded Understand-Anything illustrates what happens when the substrate for confidence exists but the column to record it does not. The system distinguishes deterministic edges (resolved by a project scanner from source files) from inferred edges (guessed by an LLM during semantic analysis). That information is real and meaningful -- a structural import edge deserves higher confidence than a non-code inference. But both are stored as weight 0.7, identical in the graph. The confidence signal is computed at write time and discarded before persistence. Adding a confidence field would be a small change with a large information-quality payoff. The system already knows which edges are solid and which are guesses. It just does not record the distinction where it matters -- on the row that gets retrieved later, when the agent needs to discriminate between them. This pattern appears across multiple systems in the corpus: information is available at write time, cheap to capture, and then lost because nobody added the column. ## The honest cost of building this right Provenance costs bytes. A source ID, a confidence score, a version counter -- each adds a few fields per row. At scale that matters, but it matters far less than the cost of not having them when something goes wrong in production and you cannot trace why the system produced a wrong answer. The compute cost is lower than expected. Confidence scoring typically requires one additional LLM call at write time (or a deterministic heuristic for structural facts), which is amortised across every subsequent read. Source tracking costs nothing beyond recording an identifier that already exists. Versioning costs one extra column or one append per write, not a full snapshot. Hindsight's observation tier adds storage proportional to the number of retrieved-and-used facts, but only records what the agent actually consumed, not everything it saw. The operational cost is the real question. Tier 3 decoration means more data flowing on every retrieval, which increases context-window usage and response latency slightly. The systems that ship this handle it by keeping decorations concise -- a source type enum, a confidence float, a freshness state -- rather than full provenance trees inline. The agent gets enough to discriminate without drowning in metadata. ## What the strongest implementations share The exemplars across the corpus deserve restating because no two solve the same slice of the problem: OpenContext mints UUIDs that survive any filesystem reorganisation and exposes them as a citation scheme the agent can use directly. mem9 carries source, agent ID, session ID on every row, version on every update, and decorates search results with source-turn context governed by an explicit budget. Supermemory treats memory as a versioned DAG with typed edges, giving every belief commit history. Hindsight captures every retrieved-and-used fact in an observation tier with full source provenance, evolution history, and per-retriever ranking. Graphify marks every edge with three-level capture confidence, surfacing uncertain edges for human review rather than treating them as fact. The unifying observation is plain: provenance is not metadata, it is part of the fact. The systems that treat it that way are the ones whose memory does not silently rot, whose contradictions can be adjudicated, whose hallucinations can be traced, and whose belief history can be rewound to any point in the past without having anticipated the need. The systems that do not are the ones whose users learn eventually that the memory they were trusting was an island all along. Provenance is the cheapest insurance you can buy at write time. The discipline is to buy it before you find out you needed it. The next piece walks hybrid retrieval and RRF, the pattern that lets you combine vector and keyword signals without one drowning out the other -- which matters most when provenance tells you a fact is solid but relevance alone would bury it. That piece is coming up.

Pay at write time, read for free. The one Agentic Memory move that compounds across every retrieval

Steven Batchelor-Manning — Thu, 14 May 2026 00:00:00 GMT

If you change one thing about how you build agent memory in 2026, change how much of the work you do at write time. After 19 systems deep on this, no other architectural decision compounds the way this one does. The systems with the cleanest recall behaviour all spend disproportionate compute when a fact enters the store. The systems that don't are paying interest on that decision forever, on every read, by an agent that no longer has the original context to reason from. The case is built on an asymmetry so obvious it tends to slide off the eye. Every fact is written once and read many times. In real agent traffic the ratio is typically two to four orders of magnitude. A second of LLM work paid at write time, divided across thousands of subsequent reads, is a rounding error per read. A second of work skipped at write time has to be re-derived on every read, by an agent navigating around the rough edge instead of past it. That arithmetic forces the design conclusion. The corpus has already done the forcing for the field. Let's look at what "write time" actually means, the six forms the pattern takes, the convergence evidence that's hard to argue with, and the honest costs of going this route. ## What write time actually is Write time is the moment a fact enters the persisted store. The user message that gets retained. The document dropped into the ingest queue. The clipped page that lands on disk. The tool result the agent decides to keep. The background sweep that promotes raw observations into synthesised beliefs. What write time isn't is the user-perceived response. The agent has already replied from the previous state of memory, the new state will be visible to the next query. That asynchrony is what makes the cost tolerable. Write-time work runs behind the response, not in front of it. Hindsight is sharp about this discipline. Extraction and entity resolution run synchronously, but the consolidation pass that turns facts into observations is deliberately deferred to a background sweep so retain latency stays low. mem9's reconcile phase issues an LLM call for every batch of new facts, then hands the result to a background goroutine, not the agent's response. Supermemory's document-to-chunk-to-memory pipeline does the heavy LLM work entirely off the API request path. The systems that mix write-time work into the response path suffer for it. The systems that hold the discipline are the ones that ship. ## The six forms the pattern takes Across the 19 systems, write-time investment has surfaced in six recognisable forms. Most mature systems run several of them at once. The ones that run several are the ones with the cleanest behaviour under load. Online dedup-and-synthesis is the first form, and the highest-leverage. When a fact arrives, the system queries the existing store for candidates that overlap, then issues a single batch LLM call that emits per-fact actions: add, update, delete, or no-op. SimpleMem's `add_memories` is the textbook version. mem9's reconcile is the same pattern at scale. The store never accumulates near-duplicates that have to be filtered or re-ranked on every later read. A subtler benefit is that synthesis surfaces contradictions that flat-write systems never even detect. When "user likes React" arrives followed later by "user has switched to Vue", Hindsight's consolidation refines the observation to capture the journey rather than overwriting, so the memory records preference as it evolved, not just the latest state. Atomisation is the second form. Break a statement into the smallest individually retrievable propositions before embedding. A wall of text recovered as a single chunk is opaque to ranking. A paragraph atomised into half a dozen short, self-contained claims gives the retriever something to actually discriminate against. LLM-Wiki's two-step ingest is the cleanest expression: step one produces a structured analysis that names the entities, concepts, claims, and relations. Step two writes the actual wiki pages, each one functionally an atomised proposition. The Louvain community detection running over the graph only makes sense because the units are atoms. Run it over arbitrary chunks and the community structure means nothing. Multi-step ingest is the third form. Once you accept that ingest doesn't have to be a single LLM pass, the pipelines fan out into richer compositions, deterministic where possible, LLM-where-necessary. Understand-Anything is the most extreme example in the corpus. Six of its nine agents follow the same internal structure: phase one writes and runs a deterministic helper script (Tree-sitter for structure, a Node script for fan-in and fan-out metrics), phase two reads the JSON output and applies LLM judgment. The LLM is explicitly told *"Do NOT re-run file discovery commands or re-count lines, trust the script's results entirely"*. OpenKB's compilation pipeline is four steps built around prompt-cache reuse, the cache amortising the document context across many fan-out calls. A multi-step pipeline that's naïvely implemented is brutally expensive. A multi-step pipeline that's built around the cache is cheaper per document than the single-shot equivalent. Provenance metadata is the fourth form, and the simplest to implement. Every entry carries the source it came from, the timestamp it was captured at, optionally the confidence the system had at capture, and optionally the citations that justify it. Hindsight is the most rigorous expression: every fact carries the journalist's interrogation, what, when, where, who, why, plus typed kind and category, plus source memory ids, plus a consolidated-at timestamp. Supermemory carries provenance at every layer of its three-tier object model, memories are immutable nodes in a versioned DAG connected by typed updates, extends, and derives edges. mem9 carries source, agent id, and session id on every row, plus a versioning column that supports an append-and-archive transaction so the previous version is never lost. Confidence scoring is the fifth form. Each fact gets a number or a state telling downstream readers how much to trust it. Hindsight uses a freshness lifecycle, observations move from fresh to confirmed as corroborating evidence arrives. Supermemory exposes a relative version distance from the primary memory so clients can render a temporal slider over a memory's history without re-querying. The hard part isn't producing the score, it's calibrating it. The deployment pattern that makes this shippable is shadow-mode: ship the heuristic dark first, collect the score distribution against real traffic, decide the threshold from the data, not from intuition. Type tagging is the sixth form. Every atom is labelled with what kind of thing it is, concept, entity, claim, relation, event, conversation. Tolaria's frontmatter-as-type convention shows that even a convention-only type system delivers most of the value, you don't need schema validation to get the benefits of typed retrieval. The label gives the retriever a second axis to filter on, which gives the agent a way to ask for "the concepts on this topic" rather than "the chunks that match this query". Six forms. They aren't a checklist where you pick three. They reinforce each other. ## Why the forms compose A subtle observation that doesn't come out of any single system but is visible across the corpus: the forms are mutually reinforcing. Atomisation is more useful when atoms are typed, because the type tells the retriever what kind of atom it has. Type tagging is more useful when atoms have provenance, because the type plus the source lets the retriever filter on both axes. Provenance is more useful when atoms have confidence, because confidence tells the retriever how much to trust the provenance. Confidence is more useful when the system performs online dedup, because dedup folds many low-confidence corroborating sources into a single high-confidence fact with multiple source ids. And the multi-step pipeline that does all of the above is more useful than the single-shot pipeline that does one of them, because the steps can hand structured intermediate results between each other rather than hand prose summaries. Hindsight illustrates this composition the most cleanly. The atom is the extracted fact. The fact is typed by kind and type. It carries six-dimensional provenance via what, when, where, who, why, plus event date and mentioned-at. It carries confidence implicitly via the freshness state on its derived observations. It's reconciled into observations via a background batch consolidation pass. The whole thing runs through a multi-step ingest pipeline that separates extraction, entity resolution, embedding, and consolidation. All six forms compose, and the composition is what makes the downstream observation tier legible to the reflection agent at all. The inverse is also visible. Skipping one form weakens the others. A typed corpus without atomisation gives you typed wall-of-text. An atomised corpus without types gives you a flood of equivalent units the retriever can't discriminate between. Provenance without confidence tells you where a fact came from but not how much to trust it. Dedup without provenance loses the audit trail of which sources fed which fact. The decision isn't whether to invest at write time, it's how many of the six forms to compose, and the corpus suggests the honest answer is "all of them, eventually". ## The convergence is the evidence Anyone can find a clean architecture in a single mature system. The interesting question is whether independent systems, starting from different assumptions, end up in the same place. On write-time investment, they do. The Karpathy LLM Wiki pattern is the clearest convergence. Karpathy's original gist did the work in a single LLM pass per document. Three independent implementations in the corpus, LLM-Wiki, OpenKB, and Understand-Anything's knowledge-graph mode, all moved away from that single-pass design within their first major iteration. LLM-Wiki landed on the two-step chain-of-thought pattern explicitly motivated by the observation that single-shot generation forgets to link to existing content. OpenKB landed on a four-step pattern explicitly motivated by prompt-cache reuse. Understand-Anything landed on a six-of-nine two-phase pattern explicitly motivated by the observation that LLMs are slow at counting and wrong at line numbers. Three different starting points, same architectural conclusion. That kind of convergence is the strongest possible evidence that the pattern is load-bearing, not stylistic. The negative evidence is just as sharp. Systems that started without write-time investment have, at some maturation point, added it. The shift goes one way. No mature memory system in the corpus has reverted from "do work at write time" to "do work at read time". The systems that designed for it from the start, Hindsight, Supermemory, Tolaria, carry their architecture forward gracefully. The systems that didn't are paying interest on the architectural debt forever. Plainly: if you're starting a new memory system in 2026 and you skip write-time investment, you are choosing to repeat a journey the field has already finished and learned from. ## The honest costs This isn't free. Anyone who's shipped a memory system will recognise the costs. Latency goes up at the write path. Multi-step ingest pipelines take seconds, sometimes tens of seconds, per document. Supermemory's 10,000-docs-per-hour throughput is bottlenecked by extraction LLM cost at roughly three to five LLM calls per document. mem9's reconcile is a synchronous LLM call on the ingest path, even with batching it dominates the wall-clock cost of writing. OpenKB's multi-step compilation runs five LLM calls per document even with the prompt cache reused across all of them. Token spend goes up. Online dedup costs tokens because the LLM needs to see the candidate existing facts. Atomisation costs tokens because the LLM has to be asked to split rather than summarise. Multi-step pipelines cost tokens at every step. Prompt-cache reuse blunts the marginal cost but doesn't eliminate it. Engineering complexity goes up. A pipeline with five steps has more failure modes than a pipeline with one. mem9's extraction prompt has three fallback strategies for malformed JSON, including recovery from a known flattened-fact corruption pattern. Understand-Anything's merge script has explicit logic for recovering nodes the analysis script dropped, remapping unknown node types, restoring dropped dangling edges. Every multi-step pipeline accretes this kind of defensive code. The trade-off is real. It's also, on every honest reckoning across the corpus, lopsided. The reads outnumber the writes by orders of magnitude. The user-perceived latency is on the read path, not the write path. The agent's reasoning budget is consumed at read time. The hallucinations are produced at read time. Every dimension along which "less work" sounds appealing turns out, on inspection, to be a dimension along which write-time investment buys read-time relief at favourable rates. A rule of thumb the corpus suggests: if a write-time pipeline costs three to five LLM calls per document, but the document will be read hundreds of times across its lifetime, the per-read amortised cost of the write work is far below the cost of running a single additional LLM call at read time to compensate for what wasn't done at write. The arithmetic is forgiving in a way that almost no other architectural decision in the field is. ## The adoption order the corpus implies For a team retrofitting an existing flat-RAG system, the order that pays back fastest at each step is something like this. Provenance first, it's the cheapest to add and the prerequisite for almost everything else. Types second, also cheap, and it unlocks faceted retrieval immediately. Multi-step ingest third, once provenance and types are in place, refactoring the single-shot extraction into two steps becomes tractable. Atomisation fourth, partially a consequence of multi-step ingest, but worth making explicit. Online dedup fifth, the most invasive form because it requires the ingest pipeline to read the existing store before deciding what to write. Confidence scoring sixth, shipped in shadow mode first, calibrated from data, acted on later. This isn't the only order that works. It's the one that pays back fastest at each step. Provenance enables debugging immediately. Types enable faceted retrieval immediately. Multi-step ingest enables better extraction immediately. Each step yields a visible improvement and lays groundwork for the next. If you're picking up an existing memory system that isn't getting the read-time quality you want, the single most useful question is the one this corpus is built around: of every piece of work the system is doing on every read, could this have been paid once at write time instead? The answer will be "yes" more often than is comfortable. The next piece is on confidence and provenance, the substrate that makes everything in this piece auditable rather than mysterious. If the write-time pattern in this piece is the engine, confidence and provenance are the instrumentation. Hindsight's freshness lifecycle, Supermemory's versioned DAG, and OpenContext's per-fact source trails are three different shapes of the same underlying argument, that a memory system without provenance is one you can't debug, and a memory system without confidence is one you can't calibrate. That piece is next.

Six deployment shapes for agent memory. Did Supermemory get it right?

Steven Batchelor-Manning — Mon, 11 May 2026 00:00:00 GMT

Most agent-memory comparisons argue about the wrong axis. After 19 systems deep on this, what's clear is the architecture everyone debates publicly isn't the thing that decides how a memory system actually feels to use. The deployment shape is. Same architecture in two different shapes is two completely different products. Six shapes have surfaced across the field. Most teams pick one as primary. The loudest of those six right now is the managed API service, and the loudest exemplar of that shape is Supermemory, which has done a competent job of building exactly what the shape demands. Whether that's the right shape to have built for is a different question, and one I think the corpus implies an honest answer to. Let's look at the six, then at where Supermemory sits in them. ## The question operators actually answer first Most published comparisons of memory systems classify them by paradigm. Is the engine flat vector RAG, knowledge-graph augmented, progressive compression, multi-index hybrid, LLM-as-retriever, trace-as-memory, a Karpathy-style wiki, or filesystem-native? That's a useful question. The previous piece in this series was dedicated to it. It's not the question an operator answers first when picking a memory system. The question they answer first is shaped like this. Do I want to call an HTTPS endpoint, link a library against my agent, point a desktop app at a folder, run a CLI from a script, or install a skill into the agent I already use? That's the deployment-shape question, and it's orthogonal to paradigm. The same paradigm, multi-index hybrid retrieval, has been built and shipped four ways across the 19 systems. As a managed API service (Supermemory at api.supermemory.ai). As a self-hostable open-source server with a managed endpoint on the side (mem9 at api.mem9.ai). As an in-process Go library that doubles as an MCP server (graymatter). As a filesystem-native context store with an MCP front door (OpenContext). Four products. One paradigm. To the operator they feel like four different categories of thing. That's the gap I want to dig into. Why deployment shape dominates the daily experience, what the six shapes actually look like, and whether the most prominent bet in the most prominent shape, Supermemory's bet on the managed API, is the bet operators picking now should be making. ## The six shapes the field has settled into Six deployment shapes have emerged across the 19 systems. Each one forces a particular operational shape, a particular trust boundary, and a particular set of trade-offs the operator has to live with from day one. Managed API service. A hosted endpoint the agent calls over HTTPS. The vendor runs the storage, the embedding pipeline, the upgrades, the on-call rotation. The operator sets a base URL and an API key. Supermemory and mem9's hosted endpoint sit here. In-process library. A package the operator's code links against. The memory engine runs inside the same OS process as the agent. Storage is local, typically a single embedded database file. SimpleMem and graymatter sit here. Filesystem-native plus MCP. The user's actual filesystem is the canonical store. Markdown files on disk are the artefact, a small index is a derivative, and an MCP server fronts the system to whichever agent the user is running. OpenContext, Tolaria, second-brain. Desktop application. A complete user-facing program that includes the memory layer rather than exposing it as a separate component. The user launches the app, sees a UI, interacts with memory through the app's chrome. llm-wiki, Tolaria again, Memex. CLI tool. A command-line program the operator runs from a shell or script. The artefact is a binary that takes arguments, performs work, returns a result. OpenKB, Graphify, GitNexus. Skill or hook framework. Memory expressed as agent-side artefacts: skill definitions, slash commands, hooks that fire between turns. There's no separate process. The memory is in the shape of the agent's behaviour. oh-my-kiro, Understand-Anything. Six shapes. Most teams pick one as primary. A small number genuinely span more than one, and those are the most interesting cases in the corpus. ## So, did Supermemory get it right? Supermemory is the loudest exemplar of the managed API shape, and the most useful test of whether that shape is a good bet for the operator picking now. The honest answer is shaped: yes for the audience they're built for, with caveats that the audience either doesn't care about or hasn't hit yet. What Supermemory got right is real. The engineering is genuinely competent. The Cloudflare Workers + Durable Objects + Hyperdrive-PostgreSQL stack is well-chosen for the workload. The TypeScript and Python SDKs plus the integrations with Vercel AI SDK, Mastra, LangChain, LangGraph, OpenAI Agents SDK, Agno, VoltAgent, Cartesia, and Pipecat make Supermemory genuinely drop-in for an agent runtime that already exists. The Memory Router pattern, an OpenAI-compatible reverse proxy that injects memories into prompts and harvests memories from completions transparently, is the slickest integration story in the corpus. The connector ecosystem (Google Drive, Gmail, Notion, OneDrive, GitHub, web crawler) and the multi-modal ingestion (PDF, image OCR, video transcription, AST-aware code chunking) are weeks of work the operator doesn't have to do. If you're a team building a consumer product where memory is a feature and you don't want to operate it, Supermemory is the most credible answer in the corpus. None of that is in dispute. What gives me pause is structural rather than technical. Three things, none of which Supermemory's team can fix without re-platforming. The trust boundary places the user's most intimate context inside someone else's perimeter. Memory contains what the user has asked, what they've been told, what they care about, and the agent's accumulated model of them. A managed API places that data inside the vendor's trust boundary, which is reasonable for a fraction of users and structurally unacceptable for another fraction. For privacy-sensitive deployments, regulated industries, or single-developer power users who refuse to ship their context off the local machine, Supermemory's shape isn't a tradeoff to weigh, it's a non-starter. That's not a critique of Supermemory's posture, which is reassuring. It's a critique of the shape they bet on, which constrains who they can serve. Costs scale with usage in ways the operator can't architecturally prevent. Supermemory's MCP error pathway includes "402, Memory limit reached. Upgrade at supermemory.ai", and the platform tracks per-organisation document limits with overage billing. For an agent that calls memory on every turn (which is the recommended pattern for an agent that actually uses memory well), the per-call cost compounds quickly. The operator can budget but can't architecturally cap. An in-process library has no such failure mode, the cost is the cost of a few embedded-database reads and a few cosine distances, in the agent's own process, billed to nobody. The engine is opaque. Supermemory's repo contains a 1,464-line zod-openapi schema documenting the wire contract in considerable detail, versioning, soft-deletion, the relation enum, the static/dynamic profile split. From the schemas you can reverse-engineer most of what the engine does. You can't reverse-engineer how. The embedding model, the chunking heuristics, the extraction prompts, the reranker, none of those are inspectable, and when the engine misbehaves on the operator's specific corpus the only recourse is escalating to support and waiting. That's the price of the genre, not a Supermemory-specific failing, but it's a price the corpus shows you don't have to pay if you pick a different shape. Set against the other 18 systems, this looks less like Supermemory got it wrong and more like Supermemory committed hard to one shape on the matrix and is now constrained by the cell they're in. If the audience they're serving is comfortable with the trust boundary and the cost model, the engineering they've built on top of those decisions is genuinely good. If the audience they're serving isn't, no amount of engineering rescues a shape mismatch. The most honest answer to the title's question is: Supermemory got their bet right for their audience, and the operator picking now should ask whether they're in that audience before defaulting to the loudest option in the room. mem9 is worth holding up as the counter-example. It ships the same paradigm as Supermemory (multi-index hybrid) in the same shape (managed API at api.mem9.ai), and also ships as a self-hostable Go binary the operator can run on their own infrastructure. The operator who wants to start managed and migrate to self-hosted later doesn't have to re-platform, they change a base URL and a credential. That's a structurally less constrained bet than Supermemory's, and it's the bet I expect the second-generation systems in this corpus to default to. ## Why the shape often dominates the paradigm The argument, plainly. Switching paradigm within a shape is a re-implementation. Switching shape is a re-platform. If you start with Supermemory (managed API) and want to move to graymatter (in-process library), you're not just changing your memory engine. You're changing your data-residency posture, your billing model, your operational on-call shape, your trust boundary, and your dependency graph. These are infrastructure concerns, and infrastructure changes are expensive. By contrast, swapping a flat-vector-RAG engine for a multi-index-hybrid engine within the same shape is mostly a matter of changing the call site and re-tuning the recall path. Same trust model, same operational shape, same billing posture. Different recipe inside. The shape constrains the paradigm in practice too. A managed API is structurally suited to multi-index hybrid (the most popular paradigm in this column by a wide margin) and structurally awkward for filesystem-native (which presupposes the user's filesystem). A skill framework can't ship a heavyweight engine because it has no engine, the host agent does the work. The shape choice rules out a substantial fraction of the paradigm space before the operator gets to choose paradigm. And the operational surface is the daily experience. Once a system is deployed, the operator interacts with the shape, not the paradigm. They manage subscriptions and API keys, or embedded database files and library versions, or a vault folder and an MCP configuration, or desktop application updates, or shell scripts and batch jobs, or skill files and hook scripts. Whichever shape they've chosen, the daily texture of their work is shaped by it. The paradigm is a property of the engine that mostly matters at recall time. The shape is a property of everything else. ## Three trade-offs the shape forces, every time The trade-offs that come with each shape compound across years. Three are worth naming because they show up in every shape and resolve differently in each. Trust delegation. Where does the user's context live? A managed API places that context inside the vendor's trust boundary. Supermemory's marketing is reassuring, the encryption posture is reasonable, the Cloudflare-native architecture is competent. None of that is the same as running the engine on hardware you own. For some users this trade is fine, they already trust OpenAI with far more. For others it's a non-starter. An in-process library or a filesystem-native shape keeps the data on the operator's machine. A skill framework keeps the data wherever the host agent already keeps it. The trade-off resolves differently per shape, and you can't move it later without re-platforming. Operational ownership. A managed API outsources operations to the vendor. An in-process library puts the operator on call. A filesystem-native shape splits the responsibility, the application owns its index, the user owns the vault, and reconciliation between them is the engineering. A desktop application owns its chrome and asks the user to handle the rest. A CLI is operational only when invoked. A skill framework piggybacks on the host agent's operations. None of these are wrong. They're different bets about who's awake at 3am when something breaks. Distribution shape. A managed API ships through a `npm install` or a `pip install` plus an API key. An in-process library ships through the same channels but with no key. A filesystem-native system ships as an installer plus a folder. A desktop application ships as a code-signed platform binary with auto-update. A CLI ships through a package manager. A skill framework ships through the host agent's plugin protocol. The cost of a release is wildly different across the six. Supermemory pushes a Cloudflare Worker in seconds. Tolaria ships a calendar-versioned `YYYY.M.D` Tauri build to three operating systems with notarisation. Different shapes, different release cadences, different feedback loops with users. These three trade-offs aren't abstract. They're the texture of working with a memory system day to day. The shape locks them in early, and the paradigm choice rides on top. ## The systems that span more than one shape A handful of systems in the corpus genuinely occupy more than one shape, and they're the most instructive cases because they show what it actually costs to do so. mem9 is the corpus's first system that's both a fully open self-hostable engine and a managed API endpoint at the same time. The README states the position with unusual precision. Switching between the hosted endpoint and the self-hosted server is "a base-URL and credential change, not a plugin rewrite". Three architectural properties combine in mem9 that no other system in the corpus exhibits together: a managed externally-callable API, a multi-backend storage abstraction spanning TiDB, PostgreSQL, and a third backend called db9, and auto-provisioning of a fresh database per tenant. Each property is a direct consequence of being both self-hostable and managed. The cost is real, multi-backend abstraction, per-tenant provisioning, spend-limit middleware, control-plane and data-plane separation. The cost is also finite, and once paid the engine spans two cells of the matrix at once. second-brain spans three shapes in a single Python codebase. It's filesystem-native (SQLite plus the user's folder), a desktop runtime (terminal REPL), and a skill framework (a Telegram bot acting as a hosted agent surface). One process, three surfaces, and a SQLite authorizer hook gating per-agent reads at the C layer to make multi-shape multi-agent isolation safe. It's not the cleanest system in the corpus, but it's the most ambitious about deployment-shape porosity. graymatter ships a single ~10MB Go binary that becomes an in-process library, an MCP server, an HTTP server, a CLI, or a TUI dashboard depending on how you invoke it. Same engine, five surfaces. The library API is sixteen public symbols, of which three cover ninety-five percent of use. The whole production surface fits on one screen. This is the cleanest single-binary expression of the deployment-shape question I've seen. These multi-shape systems are still rare. But they're not anomalies. They're the leading edge of a porosity that's been latent in the field since the beginning, and they suggest the binary "managed-or-library" framing is probably the wrong one. A more useful question is, what is the shape of the engine, and which deployment surfaces are exposed? ## The honest checklist for picking now If you're picking a memory system for a real project today, the corpus implies you should commit to shape before you commit to paradigm. The honest version of the choice is something like this. Will the user's context leave the user's machine? If no, the managed API shape is out. This is a substantial fraction of single-developer power users and most regulated industries, and the constraint is binary, not negotiable. Does the operator want to take operational responsibility for the engine? If no, the managed API is in and the in-process library and filesystem-native shapes are out, or significantly harder. The team's capacity to be on call is the cap. Will the agent call memory on every turn? If yes, the CLI tool shape is out, latency is too high. The in-process library is the strongest candidate, since the recall path is microseconds plus embedding time inside the agent's own process. Is the user a non-developer? If yes, the desktop application shape is the only one that fits without re-skilling them. CLIs and MCP configurations are non-starters for non-technical users. Is the agent the host's agent (Claude Code, Cursor, Kiro CLI), and does the host expose a hook-and-skill surface? If yes, the skill or hook framework shape offers the lightest deployment, the host does most of the work. Is the substrate the user's existing notes folder, and do they expect to keep editing it directly? If yes, the filesystem-native shape is the only one that respects the constraint. Bolting a database on top would break the user's relationship with their own files. Once the shape is fixed, the paradigm question becomes manageable. There are still real choices to make inside any column, multi-index hybrid versus knowledge-graph augmented versus Karpathy LLM Wiki, but the choice is bounded by what fits inside the shape you've already committed to. The corpus has 19 systems in 15 populated cells of the shape-by-paradigm matrix. The job is to pick the right cell and then pick from the systems in it, not to pick a paradigm in the abstract and discover later that no exemplar in your shape actually implements it. ## Where this leaves the field The corpus's existing porosity, mem9 spans two cells, second-brain spans three, graymatter blurs the library/CLI/MCP boundaries, suggests the boundaries between shapes are conventions rather than constraints. The systems built next will probably commit less to a single shape and more to a deployment surface that exposes multiple shapes from one engine, the way mem9 ships one Go server that runs both as `api.mem9.ai` and as `make build`. That's not the canonical shape story this corpus inherited from the field's first generation, where almost every system was clearly one thing. It's plausibly the shape story of the second generation, where the engine is the same and the deployment surface is what changes. For the operator picking now, the practical advice is short. Pick the shape because it fits the constraint that's hardest to move later, where the data lives, who's on call, who the user is. Pick the paradigm inside that shape from what's available there. Be prepared to discover that the most popular paradigm in the abstract isn't the most popular paradigm in your column, and that's fine, popularity in the abstract isn't a constraint your project actually has to satisfy. The next piece is on write-time investment, the highest-ROI design decision across the 19 systems, and the one move that compounds at every subsequent read. It's the principle that pulls SimpleMem's online synthesis, Hindsight's async consolidation, and LLM-Wiki's two-step ingest into the same pattern. If you've ever wondered why the systems with the cleanest recall behaviour all spend disproportionate compute at write time, that piece is the answer.

Eight Agentic Memory Paradigms

Steven Batchelor-Manning — Thu, 07 May 2026 00:00:00 GMT

The first piece in this series landed on the one thing 19 open-source agent-memory systems agree on, and it was a negative. Flat vector RAG, on its own, isn't enough. Every team that started there ended up adding something. The agreement on what to add ends there. (Need a catchup? The first piece is here: [LINK_TO_PIECE_1]) This piece is what's behind that "what to add" disagreement. Once you actually look at what the 19 added, the surface variety resolves into a small, sharp set of architectural commitments. Eight of them. Eight paradigms. Not eight implementations of the same recipe. Eight different bets about what memory fundamentally is. That distinction matters more than it sounds. Most of what reads as fragmentation across the agent-memory field is actually a small number of incompatible commitments dressed up in different vocabularies. Once you can name the eight, the field stops looking chaotic and starts looking like a design space with known edges. Let's look at the eight, then at what they're really arguing about. ## The eight, in one sentence each Flat vector RAG with structured extras (Flat-RAG). Embed everything, retrieve top-k, prepend to prompt. Then layer something on top, because nobody ships it bare. SimpleMem and early Memex are the textbook examples. Knowledge-graph augmented (Graph). Memory is a typed graph of entities and typed edges. Recall is traversal, not similarity. Graphify, EdgeQuake, GitNexus. Progressive compression (Prog-Compression). Memory is a hierarchy of increasingly summarised representations, with heat-gated promotion between tiers. The verbose original fades; the dense distillation persists. MemoryOS is the textbook implementation. Multi-index hybrid search (MI-Hybrid). Several indexes running in parallel, fused at recall time, almost always with Reciprocal Rank Fusion at k=60. Hindsight, supermemory, mem9, graymatter. LLM-as-retriever (LLM-Retriever). Skip the vector store. Give the model a hierarchical map of the documents and let it navigate. Memex evolved into this. OpenKB ships it. Supermemory's rewrite mode runs on it. Trace-as-memory (Trace). The agent's own execution history is the memory. Not the user's documents. Moraine is the purest case. Hindsight's observation tier is a hybrid component. Karpathy LLM Wiki (Wiki). Plain Markdown, wikilinks, frontmatter, an index file as the catalogue, the user as the ultimate curator. Three independent teams rebuilt this in 2025. Understand-Anything reads it. OpenKB writes it. LLM-Wiki does both as a desktop app. Filesystem-native context store (FS-Native). The file is the artefact. The database is a derivative cache. When in doubt, the disk wins. OpenContext, Tolaria, second-brain. Eight commitments. Read them again as commitments rather than as feature lists. (Trace) says memory is what the agent did. (Prog-Compression) says memory is the densest faithful summary of what's been observed. Those aren't different settings on the same dial. They're different answers to "what is the system trying to remember?" ## What the paradigms are actually arguing about The temptation is to read the eight as a tier list and pick the one with the best benchmark. That's a category error. Each paradigm is a commitment about three things at once. What kind of question the agent will ask. What shape the answer should take. Where the cost of getting from one to the other should be paid. A coding agent asking "what breaks if I change this function's return signature?" wants structural intelligence. Cosine similarity over chunks of source code is a poor substitute for a typed graph of call edges. (Graph) is the natural fit, and a (Flat-RAG) system trying to answer the same question will lose to a graph system every time. A research agent reading a 400-page report wants navigation. It wants to descend into the section that matters and ignore the rest. (LLM-Retriever) over a hierarchical map of the document beats (Flat-RAG)'s top-k cosine search, because the question shape is "find the right region, then read carefully" rather than "find the most similar 800-token window". A personal assistant accumulating context over years wants compression. Without it the store grows linearly in tokens until retrieval drowns. (Prog-Compression)'s tiered hierarchy is the answer. A (Flat-RAG) system at year three is a system whose retrieval is mostly stale chunks pretending to be relevant. An observability-minded team operating agents in production wants the trace. (Trace) turns "what did this agent do last Tuesday at 3pm" into a literal query. Most systems with rich logging can't answer that question because the logs aren't memory; they're text dumps the agent itself can't see. The paradigm-fits-question framing is the most useful single move you can make on this material. Once it clicks, the eight stop being competitors and start being tools, each with a clean problem shape it's right for and a set of problem shapes it's wrong for. ## Some of these don't compose The mature systems in the corpus increasingly hybridise. Hindsight is primarily (MI-Hybrid) with a (Trace) observation tier and a (Graph) entity sub-component. Supermemory is (MI-Hybrid) with a versioned-DAG schema that's (Graph)-adjacent and an (LLM-Retriever) rewrite mode. LLM-Wiki is (Wiki) with (Graph) community detection and opt-in (MI-Hybrid) vector retrieval. OpenContext is (FS-Native) with (MI-Hybrid) retrieval mechanics layered on top. That's the optimistic reading. The pessimistic reading, which is also the accurate one, is that composition isn't free. Take (Prog-Compression) and (Trace) together. (Prog-Compression) says the verbose original fades; only the densest representation persists. (Trace) says the verbose trace is the memory; compressing it is destroying the artefact. You cannot make both your primary mode without an internal contradiction. You can run them as separate stores with separate query paths, which is what Hindsight does. But the choice of which gets first call on retrieval is an architectural decision that propagates through the whole stack. Or take (Wiki) and (Flat-RAG) together. The (Wiki) paradigm makes a deliberate philosophical refusal: memory should be a persistent compounding artefact a human can read, not a re-derivation from chunks on every query. Bolting a vector store onto a wiki to "improve retrieval" doesn't compose them; it turns the wiki into chunked text the LLM happens to have access to and loses the compounding property that made the wiki worth building. (Graph) and (FS-Native) compose more cleanly, because filesystem-native storage and graph-as-derivative-cache aren't fighting over the same role. The graph is a query optimisation; the file is the source of truth. But even there, the synthesis question (what does the graph do when the file changes?) is an open problem across every system in the corpus that's tried it. Put plainly: The eight paradigms compose where their commitments don't collide. Where the commitments collide, composition is a deliberate engineering choice that costs something on both sides. The systems that have done this well have done it by being explicit about which paradigm is primary and which is secondary, and by accepting that the secondary's strengths are partly muted in the bargain. ## Five angles on why this matters Hindsight is the clearest case for paradigm-as-strategy rather than paradigm-as-style. Its TEMPR retrieval system doesn't just run multiple indexes; it has a deliberate routing policy that decides which paradigm to lean on for a given query. Temporal queries go to the trace tier. Relational queries go to the entity tier. Semantic queries go to vector RRF. The routing is the system. Strip the routing out and you have four indexes that disagree about the answer. EdgeQuake makes the case from the cost side. Its multi-pass gleaning extractor catches 15-25% more entities than a single-pass extractor and dominates the system's compute bill by an order of magnitude over retrieval. (Graph) isn't expensive at recall time. It's expensive at write time, in a way that makes (Flat-RAG)'s "embed and forget" feel almost free by comparison. The teams that picked (Graph) picked it knowing this. The teams that drift into it accidentally end up surprised by their LLM bill. MemoryOS makes the case from the long-tail side. Its three-tier STM/MTM/LPM hierarchy with the heat-gated promotion formula isn't a clever optimisation on top of flat-RAG. It's a different commitment about what storage is for. STM holds raw turns. MTM holds segment summaries promoted by access pressure. LPM holds the user profile that survives indefinitely. A system without that hierarchy at year three is a system whose retrieval has quietly become noise. Moraine makes the case from the observability side, and from the most extreme end of the spectrum. There's no embedding model. No vector database. No LLM in the operational dataplane at all. ClickHouse, materialised views, BM25 over the verbatim trace. The argument is that for its specific problem, "what did the agent do, when, with what tokens, and what was the outcome", structured trace storage is the better trade than any retrieval-by-similarity mechanism. The paradigm fits the question. Understand-Anything, OpenKB, and LLM-Wiki together make the case from the curation side. Three teams, working independently, built variants of the same Karpathy sketch in the same year. Plain Markdown. Wikilinks. An index file. The compounding artefact, with the human as the eventual arbiter of staleness and the LLM as the heavy lifter of synthesis. Convergent design across three independent teams is rare in this field. When it happens, it's worth taking seriously as a signal that the underlying commitment is sound. Five systems, five paradigms, same underlying point. Each one is a deliberate answer to a question the other four were asking less directly. ## What to do if you're picking now If you're building agent memory in 2026, the choice of paradigm is more consequential than the choice of database, the choice of embedding model, or the choice of retrieval algorithm. The paradigm sets which of those choices matter. Characterise the questions your agent will ask before you characterise the documents it will read. The wrong paradigm with the right corpus retrieves the wrong things. The right paradigm with a worse corpus still works for the questions it was designed for. Map question shape to paradigm and pick a primary. Plan for one secondary if the question shape is genuinely mixed. Don't plan for three. Be honest about which paradigm's weaknesses you're taking on. Knowledge-graph systems are the most expensive to build and the hardest to maintain when the source changes. Progressive compression loses information by design. Multi-index hybrid systems need an explainable fusion stage or they become folklore. Karpathy-wiki systems push curation work back onto the user. Filesystem-native systems push synthesis back onto the agent. Trace-as-memory captures actions but not facts. None of these is disqualifying. All of them are worth knowing about up front. Take the deployment shape as seriously as the paradigm. A managed API and an in-process library implementing the same paradigm feel like completely different products to the team using them. The choice between them is often more consequential than the paradigm itself, because deployment dictates who owns the data, who runs the infrastructure, and what the trust model is. Pick the deployment shape first. Pick the paradigm inside it. ## Where the field is going The longer-horizon read of the eight paradigms is that the equilibrium probably isn't eight. It's two or three composite paradigms that absorb three or four of the others each. Three candidates are already visible in the corpus. Hybrid retrieval with provenance and an observation tier, which is (MI-Hybrid) plus (Trace) plus the write-time-investment habit from (Flat-RAG). Hindsight is the closest current expression. Karpathy wiki on a filesystem-native substrate, which is (Wiki) plus (FS-Native) with (LLM-Retriever) as a query mode. LLM-Wiki and OpenKB are moving towards this. Knowledge-graph systems for structurally rich domains, which is (Graph) with (MI-Hybrid) mechanics for fuzzy entry-point selection. EdgeQuake, GitNexus, and Graphify are converging on this shape. The pure single-paradigm systems aren't going to disappear. SimpleMem-style (Flat-RAG) is the right answer when the budget is small and the question shape is genuinely flat. Moraine-style (Trace) is the right answer when the question is about what the agent did rather than what the world contains. The point isn't that the paradigms are converging. The point is that the loudest paradigm right now isn't necessarily the right paradigm in six months, and the problem has been the same all along. If there's a single piece of advice that falls out of the eight, it's this. Pick the paradigm because it fits the problem. Not because it's the loudest paradigm in the room. Not because the benchmark on the dataset that doesn't match your domain says so. Not because the framework you already use happens to ship with one. The shape of the question your agent will ask is the only thing that should drive the answer. Next up I'm writing about the patterns that show up across most of the eight. Provenance. Hybrid retrieval. Write-time investment. The handful of moves that are universal even when the paradigm choices aren't. If this was useful, share it with someone in your network who's building agent memory right now, or who's about to. The conversations I keep ending up in with engineers and tech leads working on this stuff are the best part of writing the series, and the only way more of them happen is if pieces like this find the right people.

What 19 agent-memory systems all agree on (and it's a negative)

Steven Batchelor-Manning — Sat, 02 May 2026 00:00:00 GMT

Over the last few months I've continued down the rabbit hole of agentic memory systems, digging deep into open-source repos and research papers. It's clear the current solutions out there agree on one thing, and it's what not to do. Vector RAG, on its own, is not enough. From the 19 systems, every single one that started with flat vector RAG ended up adding something on top of it. Every one. The additions are all over the place. Some added knowledge graphs. Some added trace logs. Some added a hierarchical wiki the LLM navigates by reading files. Three teams independently rebuilt Andrej Karpathy's late 2025 LLM wiki sketch and called it memory. Two teams ripped the vector store out partway through and replaced it with ripgrep over a structured folder tree. The variety of what people added is noticeable. The total agreement that something must be added is the most useful single finding I've spotted. That's the consensus. Let's look at why, and where it's coming from. The question every team is trying to answer Digging through the systems, it's clear they're all working towards one question. How do you give a language model usable, durable, agent-friendly memory beyond what fits in a context window? That's the field. Everything else, the embeddings, the graphs, the wikis, is a particular bet on how to answer it. Each project's README, in its preferred dialect, is a restatement of that one sentence. That's where the alignment stops. From there, the myriad of solutions begin. What the disagreement actually looks like Eight paradigms. Sharp boundaries. Real engineering trade-offs that pull in incompatible directions. Most teams take a primary position on one paradigm and a secondary position on another. A few hybridise across three. Nobody's converged on a winner. The eight, one sentence each. Flat vector RAG with structured extras. Embed everything, retrieve top-k, prepend to prompt, then add a layer or three of cleverness on top. SimpleMem and Memex started here. Knowledge-graph augmented. Memory as a typed graph of entities and relationships, retrieved by traversal plus vector ranking. Graphify, EdgeQuake, GitNexus. Progressive compression. Memory as a hierarchy of increasingly summarised representations, with heat-gated promotion between tiers. MemoryOS is the textbook implementation. SimpleMem hybridises into this from paradigm 1. Multi-index hybrid search. Multiple indexes, one fusion stage, almost always Reciprocal Rank Fusion at k=60. Hindsight, Supermemory, Graymatter, OpenContext, mem9. LLM-as-retriever. Forget the vector store, give the LLM a hierarchical map of the documents, have it navigate to the answer. Memex evolved into this. OpenKB ships it. Supermemory's rewrite mode runs on it. Trace-as-memory. Memory is the agent's own execution history, not the user's data. Moraine in its purest form. Hindsight's observation tier as a hybrid component. Karpathy LLM wiki. Plain Markdown, wikilinks, frontmatter, an index file as the catalogue, the user as the ultimate curator of staleness. Three independent implementations. Understand-Anything reads it. OpenKB writes it. LLM-Wiki does both as a desktop app. Filesystem-native context store. The file is the artefact. The database is a derivative cache. When in doubt, the disk wins. OpenContext, Tolaria, second-brain. The boundaries between these aren't stylistic. They're different commitments about what memory fundamentally is. Paradigm 6 says memory is what the agent did. Paradigm 3 says memory is the densest faithful summary of what's been observed. You can't make both your primary mode without an internal contradiction. You can compose them, and a few systems do, but composition forces a choice about which gets first call on retrieval, and that choice has architectural consequences that propagate through the whole stack. That's the framing that, more than any other, explains why the field looks the way it does. The thing everyone agrees on Of the 19 systems, every one that started with flat vector RAG as its primary mechanism has ended up adding something on top of it. The additions aren't all the same. The agreement that something must be added is universal. Put plainly: No serious system believes that "embed everything, retrieve top-k by cosine, prepend to prompt" is sufficient as a memory architecture for an agent. That sentence is worth being careful with. It's not a claim that vector search is useless. Vector search shows up in roughly three quarters of the 19 systems. It's the substrate. It's just no longer treated as the solution. Hindsight is the clearest case. Its documentation enumerates four pain points of pure vector RAG and then architects the rest of the system around answering each one. Pure semantic similarity loses temporal signal. Facts get disconnected. Agents need to consolidate. Reasoning style is bank-specific. Hindsight's four headline subsystems map almost directly onto those four pain points. The system is, in the words of its own deep-dive, a complete answer to "what does an agent memory system look like if you take the limitations of RAG seriously?" EdgeQuake makes the same case with a different illustration. Imagine the query "How did Sarah Chen's research on neural networks influence the work of her colleagues at Quantum Dynamics Lab?" A flat vector store returns disconnected chunks. One mentioning Sarah Chen. One about neural networks. One about Quantum Lab. The system can't follow the influence chain across documents because the chain was never indexed. What was indexed was the similarity of each chunk's embedding to the query's embedding. Similarity isn't relationship. SimpleMem makes the case in benchmark numbers. Its paper compares the cost-versus-quality frontier of four memory approaches on LoCoMo. Pure vector retrieval is the bottom of the chart on F1 and the middle on cost. The systems that beat it all do something more than retrieve top-k. SimpleMem's contribution is intent-aware retrieval planning plus online write-time synthesis. A-Mem's is iterative LLM reasoning loops. Mem0's is per-write LLM extraction. Every method that beat naive vector RAG did so by adding a step. Karpathy makes the case from the user's side rather than the agent's. RAG re-derives knowledge from raw chunks on every query, with no accumulation. A wiki, by contrast, is a persistent compounding artefact. Cross-references already there. Contradictions already flagged. Synthesis already reflecting everything that's been read. The maintenance burden, which historically killed personal wikis, is the part the LLM does for free. This isn't a paradigm-1 system that's added things. It's a deliberate refusal to be a paradigm-1 system in the first place. Moraine makes the case from the most extreme end of "the LLM is never invoked during ingestion or retrieval". There's no embedding model, no vector database, no LLM runtime, no message queue, no Python-managed daemon in the operational dataplane. Just ClickHouse, materialised views, and BM25 over the verbatim trace. Moraine isn't arguing that vector search is wrong. It's arguing that for its specific problem, BM25 over a normalised event table is the better trade. Five teams, same negative argument from five angles. The argument isn't that flat vector RAG is useless. It's that on its own, it leaves too many of the relevant retrieval shapes unsupported. Temporal queries. Relationship queries. Multi-hop queries. The kinds of question that need provenance, navigation, or structured filters. The kinds of question that need the system to know something it never observed but should have inferred. Even the systems that do run flat vector RAG as their primary mechanism stack things on top. SimpleMem layers intent-aware retrieval planning and online write-time synthesis on the cosine search. Memex, which started in the same camp, ended up replacing the vector store entirely with file-system-level tools and an LLM-as-search-engine pattern over structured P.A.R.A. organisation. The final shape of Memex isn't a vector RAG system that happens to use ripgrep. It's a system that abandoned vector RAG once the team had actually used it. The pattern's consistent enough to state plainly. Every one of the 19 systems that started with flat vector RAG ended up adding something. None of them judged the original recipe sufficient on its own. That's the most unambiguous finding from the 19 deep-dives. It's the closest thing the field has to settled engineering. And the finding's negative rather than positive. The field knows what doesn't work. It doesn't yet know what does. Why no convergence If everyone agrees that flat vector RAG isn't enough, why hasn't the field converged on a single replacement? Because the engineering trade-offs are real and they pull in incompatible directions. Six of them are worth naming. Write cost versus read cost. A team that pays heavily at write time, like SimpleMem with online synthesis or Hindsight with async consolidation or LLM-Wiki with two-step ingest, gets a cleaner store that reads cheaply. A team that pays at read time, like Moraine with BM25 over the verbatim trace or OpenKB with LLM tree-walks over PageIndex, gets a write path that scales linearly in raw bytes and a read path whose cost depends on query shape. The rough rule of thumb across the 19 is: if the data will be read more than five times, pay at write time; otherwise, pay at read time. The threshold's different for every team. Substrate ownership. Where does the data live? In a database the operator runs (PostgreSQL, SQLite, ClickHouse). In the user's filesystem (OpenContext, Tolaria, OpenKB). In a vendor's cloud (Supermemory, mem9's hosted endpoint). Or in the agent's harness logs on disk (Moraine). Each substrate has implications for backup, portability, and operational burden. The widest gap is between Tolaria's position, the file is the artefact and the database is a derivative cache, and Supermemory's position, in which the substrate is a closed PostgreSQL behind a Cloudflare Workers API the developer can never directly inspect. These aren't just different engineering decisions. They're different philosophies about who owns the user's memory. Agent shape. The shape of the agent that consumes memory matters more than the abstract quality of the retrieval algorithm. A coding agent operating over a single repository wants structural intelligence: blast radius, call chains, community membership. Vector similarity is a poor substitute for "what breaks if I change this function's return type". A research agent reading long PDFs wants hierarchical navigation. A personal-life-recording agent wants a condensed user profile that grows over time, plus the ability to navigate back to the original entries by date. The right paradigm depends on the shape of the question the agent will ask, not on any abstract quality of the retrieval algorithm. Operator burden. An in-process library running against an embedded SQLite file imposes near-zero operational burden. A managed API service imposes near-zero configuration burden but high vendor-lock-in burden. A self-hosted PostgreSQL with pgvector and a worker process imposes meaningful infrastructure burden but offers full transparency. A filesystem-native store imposes near-zero burden of either kind but pushes synthesis work back onto the agent. Different teams have different operator profiles. Update semantics. When a fact stops being true, what happens? Flat vector RAG handles updates badly. A new chunk supersedes an old one only if retrieval happens to favour it, and the old chunk lingers as a ghost. Knowledge-graph systems handle updates with explicit edge manipulation, but the maintenance question (what happens to derived edges when a source document changes?) is largely unsolved across the 19. Versioned-DAG systems handle updates by appending new nodes that point to old ones, which solves the lineage question but trades it for a garbage-collection one. Filesystem-native systems handle updates by letting the user edit the file, which is correct but pushes the synthesis question back onto the human. None of the 19 has a clean cross-paradigm answer to "the user said this last month and now it isn't true". Every system has a partial answer it isn't quite happy with. Provenance. Does every fact in memory carry where it came from? The mature consensus is yes. Every system that started without provenance has added it. But adding provenance to a system that didn't have it from the start is genuinely hard. The chunk has an embedding. The embedding has a topive-k rank in the result. What it doesn't have, unless someone built it, is a pointer back to the conversation turn or document section that produced it. Retrofitting this onto a flat vector RAG store is a meaningful project. The fact that every mature system has done it is the strongest evidence that it has to be done. These six trade-offs aren't solved problems. They're the live edges of the design space. Different teams resolve them differently because their constraints are different. That's why the field looks fragmented from the outside, and why on closer reading the apparent fragmentation is mostly informed eclecticism rather than confusion. What this means if you're building memory now If you're building agent memory in 2026, the landscape gives you no winning blueprint. What it does give you is a clear set of known-bad starting points and a clear set of known-valuable additions, parameterised by the trade-offs above. The practical advice that falls out of 19 deep-dives is short. Don't ship flat vector RAG and call it a memory system. It's the right substrate for many systems but it isn't the answer on its own. Every team that started there ended up adding something. The question isn't whether you'll add something but what you'll add. The three most common additions are hybrid lexical plus semantic retrieval fused with RRF, provenance metadata travelling with every fact, and write-time synthesis (online dedup, atomisation, structured extraction). If your system has none of those, the weekend's project is to pick the most relevant one and ship it. Match the paradigm to the problem. The right architecture depends on the shape of the question your agent will ask. A coding agent should consider knowledge-graph augmentation. A research agent reading long documents should consider LLM-as-retriever over hierarchical structure. A personal assistant accumulating context over years should consider progressive compression with tiered storage. There's no shame in hybridising. Most mature systems do. The shame's in not knowing which trade-off you're making. Be honest about the weakness you're taking on. Knowledge-graph systems are the most expensive to build and the hardest to maintain under source change. Progressive-compression systems lose information by design. Multi-index hybrid systems need an explainable fusion stage. None of these is disqualifying. All of them are worth knowing about up front. Expect to invest at write time. The highest-ROI design decision across the 19 is paying compute up front to make the store cleaner. SimpleMem's online synthesis. Hindsight's async consolidation. LLM-Wiki's two-step ingest. These aren't all the same mechanism but they're all the same principle. The work you do at write time pays back at every subsequent read. Build for the agent's behaviour, not just for the agent's prompt. oh-my-kiro's central design claim, "if a constraint can be expressed in code, it shouldn't be enforced with words", is more broadly true than its specific implementation. Memory systems that rely on the LLM to behave a certain way will see those behaviours decay across long sessions. The systems with lasting behavioural properties get them by making the substrate enforce them. Take the deployment shape as seriously as the paradigm. A managed API and an in-process library can implement the same paradigm and feel like completely different products. The choice between them is often the most consequential decision the team makes. It dictates who owns the data, who runs the infrastructure, and what the trust model is. None of those is downstream of the retrieval algorithm. Where the field is Not in convergence. Not in fragmentation. In what the literature on emerging engineering disciplines calls informed eclecticism. A community that's agreed on the problem, agreed on the constraints, agreed on what doesn't work, and is working through the design space without a single dominant solution emerging. That's not a bad place to be. It's the necessary phase before convergence. Database systems went through it in the 1970s. Mobile OSes went through it in the late 2000s. Agent memory's in that phase right now. The eight paradigms across the 19 systems aren't all going to survive. Six months from now the scene will be different again, new approaches, refined approaches arriving constantly will continue to probe the solution space. If the field has any single piece of received wisdom right now, it's the one this piece opened with. Flat vector RAG, on its own, isn't enough. Every team that started there ended up adding something. The agreement on what to add ends there. The agreement on the need to add is total, and that's the foundation everything else sits on. Over the coming few weeks I will dig into the topics mentioned in this high level and cover some of what I've found on this magical mystery tour of Agentic memory systems.