Benchmark
LongMemEval-S Results
The standard benchmark for AI memory retrieval quality. We publish our full results, methodology, and raw data.
Overall accuracy on LongMemEval-S, running gpt-4o-mini
Edging ahead of Supermemory (81.6%) who require more expensive answer models. 10x cheaper for you per query, enabling you to build better systems for lower cost and swap between answer models as needed with no degradation in performance.
Category Breakdown
LongMemEval-S tests six categories of memory retrieval. Each measures a different aspect of how well your memory system understands, stores, and retrieves information.
Overall
Combined accuracy across all six categories
Single-Session User
Recall facts and details the user has shared in conversation
Single-Session Assistant
Recall information the AI agent itself provided earlier
Single-Session Preference
Recall the user's stated preferences, choices, and decisions
Knowledge Update
Track when information changed without backups or version history, and present the latest fact
Temporal Reasoning
Answer questions that depend on when things happened
Multi-Session
Combine knowledge across multiple conversations to answer a question. The most important metric for agentic AI
Methodology
What is LongMemEval-S?
LongMemEval-S (Short) is a standardised benchmark for evaluating AI memory systems. It tests how accurately a system can store, update, and retrieve information from long-running conversations across six categories.
How we tested
Each system was given identical conversation histories and asked the same retrieval questions. m3mory used gpt-4o-mini for all LLM calls. All competitors used GPT-4o for their answers, a significantly more expensive model.
The cost advantage
m3mory achieves 83.6% accuracy on gpt-4o-mini at roughly 1/10th the per-query cost of competitors running GPT-4o. Our proprietary retrieval pipeline does the heavy lifting, not the LLM.
Where we excel
Single-session user recall (97.1%), single-session preference (86.7%), temporal reasoning (80.5%), and multi-session (79.7%) are our strongest categories. We beat Supermemory overall (83.6% vs 81.6%) and on preference, temporal reasoning, and multi-session. Every single category above 79%. These map directly to the most common real-world use cases. It doesn't matter what model you choose to use, our system still delivers.
Active development
We are continuously improving our retrieval pipeline. New modules are being added regularly to improve accuracy across all categories.