How do you even test whether an AI remembers?
The field's standard evaluation is LoCoMo — a benchmark of roughly 1,540 questions spanning four categories, purpose-built to test recall across multiple separate conversation sessions rather than within one continuous chat.
That distinction is the entire point of the benchmark: a system with a merely long context window can look like it remembers as long as everything fits in one conversation, but LoCoMo specifically tests whether a fact survives being retrieved from an earlier, distinct session — closer to how an actual ongoing relationship with a person works.
What happens if you just feed the whole conversation history back in every time?
This is the naive baseline every memory architecture gets compared against, and the research is fairly blunt about its limits. One head-to-head comparison of ten different approaches — spanning published research baselines, open-source tools, retrieval-augmented generation, full raw-context reloading, and dedicated memory systems — found full-context approaches averaging around 26,000 tokens per conversation just to answer a query, versus roughly 7,000 tokens for a more selective, retrieval-based memory approach on the same benchmark.
That's not a marginal difference — it's the gap between an approach that's economically sustainable at scale and one that isn't.
Do these systems actually work reliably, or just better than nothing?
The honest answer is "meaningfully better than nothing, but not solved." Published comparisons on LoCoMo show retrieval-based and graph-based memory architectures consistently ahead of full-context reloading on both accuracy and cost — but the same body of work also documents real failure modes: privacy risks in what gets stored, memory contamination where incorrect facts get incorporated and persist, and inconsistent performance on harder multi-hop questions that require combining several remembered facts rather than recalling one in isolation.
That combination — clearly better than the naive alternative, clearly not a finished problem — is a more accurate picture than either "AI has real memory now" or "AI can't remember anything," and it's worth holding onto that nuance rather than overclaiming in either direction.
What should you actually expect from an AI tool that claims to have memory?
Based on where the published research currently stands:
- > It should be able to recall specific facts from previous, separate sessions — not just within one long conversation — since that's the entire distinction the field's benchmarks are built to test.
- > It should update, not just accumulate. A system that never resolves contradictions — your job title changed, your preference changed — is closer to a growing pile of old notes than genuine memory.
- > Perfect recall on complex, multi-step questions isn't yet a reasonable bar. Current published work still shows measurable failure rates on harder reasoning-over-memory tasks, even for the strongest systems.
For the architecture behind all of this, see how AI memory actually works; for why the "just use a longer context window" alternative degrades, see context rot.
Key takeaways
- > LoCoMo, roughly 1,540 questions, is the benchmark that tests recall across separate sessions.
- > Reloading the full history costs around 26,000 tokens per conversation versus roughly 7,000 for selective retrieval.
- > Retrieval and graph-based memory beat full-context reloading on both accuracy and cost.
- > Known failure modes remain: contaminated facts, and weak multi-hop recall.
- > Real memory updates and resolves contradictions; it does not just accumulate.
Sources
- > Chhikara, P., et al. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arxiv.org/pdf/2504.19413
- > Mem0 (2026). State of AI Agent Memory 2026 benchmark report. mem0.ai/blog/state-of-ai-agent-memory-2026