An example: which came first?
The paper opens with a very clear example. A user asks: "Which did I get first, the Samsung Galaxy S22 or the Dell XPS 13?". A classic memory system searches and hands the agent two notes: the user owns both devices, and they bought the Galaxy S22 on February 20th. There's no date for the laptop. With that, the only thing the agent can answer is "I don't know".
Now give the same agent memory tools. It no longer waits to be handed anything: it searches the conversations itself and finds two messages. The Dell was pre-ordered on January 28th but arrived on February 25th; the Galaxy came on the 20th. The correct answer is the Galaxy S22, and the trap was the pre-order date.
That difference, receiving versus searching, is the whole paper.
The problem with passive retrieval
Agents no longer live in a single conversation: they stay with us for weeks, at work, while we study or in our daily lives, and for that they have to remember us. The hard part is that what they know about a user comes in many kinds, and every future question needs a different memory, so a fixed way of storing and querying memory falls short.
So far, memory has been improved from two sides:
Finer categories and vector, graph, hybrid or hierarchical stores: Mem0 keeps compact records, Zep builds a temporal knowledge graph, MemOS treats memory as a system resource and MemoryOS splits it into short, mid and long term.
Searches that reflect on what they found, personalized queries for each user and better reranking of results, sometimes trained with reinforcement learning (Search-R1, DeepRetrieval, RMM).
But in almost all of them, accessing memory is a system function, a pipeline designed in advance. No matter how you store memory, as a list, a graph, vectors or a hierarchy, in the end it's reduced to a chunk of context that someone else picks and hands to the agent. If that chunk is incomplete, the agent can't go and look for more. It is, as the authors put it, a consumer of pre-selected evidence.
Memory as an action space
The proposal is to flip that around: memory becomes a structured action space. The agent gets memory tools and decides three things:
- Whether it needs memory for this question.
- Which level of detail to consult.
- Whether what it has found is enough to answer.
They call it active memory navigation, and the difference from other work that also trains memory policies is where the focus lies: not on what to store or when to delete, but on how to use memory at answer time, picking the level and refining based on what it finds along the way.
The memory pyramid
NapMem builds a pyramid for each user with four levels. They all tell the same story, each at a different level of abstraction, like a map you can see as a whole or zoom into a single street.
1. Raw conversations
The base stores every message as it is, with its content, who said it, when, the session and a unique identifier to look it up exactly. This layer only grows: it's never rewritten. It's the highest-fidelity evidence, the one the agent checks when it needs the fine detail.
2. Memory records
The second level is compact sentences that strip away the chatter and keep what's useful. There are four types: facts, events, instructions and preferences. Each record carries its identifier, type, text, date and the identifiers of the messages it came from; that link lets you go from a summarized memory back to the original conversation.
They're created in two steps. First, after a few turns of conversation (the batch grows from 1 to 2 to 4 turns and then stays at 5), the model extracts new records from the recent messages. Then it compares them with what it already had, using hybrid search, and chooses what to do with each one: store it, skip it if it's redundant, update an old one if something changed, or merge them if they describe the same thing. When updating or merging, the union of sources and timestamps is kept.
3. Topic tracks
A record is an isolated fact, but many things about a person evolve over weeks: a project, a hobby, a relationship. Each topic track is a Markdown file with metadata, a summary, the story of that topic and an evidence section linking the records that back it up. There are at most 20 tracks, and the agent decides whether to update one, merge two or start a new one. Writes are validated before they're committed; if the format fails, they're rolled back.
4. The user profile
At the top sits the profile: a global summary of who the user is, their long-term preferences and how they like to interact. It's capped at 2,000 characters, so it gets revised instead of growing. And it can only use what the topic tracks say: if there's no evidence, the field stays empty rather than making something up.
How it's built
The pyramid is built from the bottom up, little by little, replaying the user's sessions in chronological order. Each new session is added to the conversations, records are extracted from it, and the changes flow upward: tracks are revised before a new session if records are pending, at date boundaries, or when more than 20 new records have piled up; the profile, after the tracks are updated or when there are more than 50.
The two lower levels are structured JSONL records with vector and keyword indexes, to search by meaning and to fetch something exact by its identifier. The two upper levels are Markdown files that are read as they are. And everything stays linked: from the profile to the tracks, from the tracks to the records and from the records to the messages.
The five tools
To move through the pyramid, the agent has five tools, each designed for a level:
| Tool | Input | What it does |
|---|---|---|
search_conversation | free text | Searches raw messages and returns the top 5 snippets |
search_records | free text | Searches the records and returns the top 5 |
get_conversation | list of message ids | Fetches exact messages by identifier |
get_records | list of record ids | Fetches exact records by identifier |
read_file | file name | Reads a topic track or profile.md |
The two search tools are hybrid: they mix keywords and vectors with reciprocal rank fusion (RRF, with k = 60), on top of Qwen3-Embedding-0.6B embeddings. The exact-access tools use identifiers returned by earlier calls, and the file reader gives access to the more abstract levels.
Navigating both ways
With that, navigation can go top-down (start from the profile or a track, narrow down with the records and drop to the conversation to verify the evidence) or bottom-up (search for a detail and, if it doesn't show up, climb to a summary). At each step the agent looks at what it has found and decides its next action, or whether to stop and answer. Formally, at step t it picks an action given the question, the pyramid and what it has seen so far, and the trajectory ends with the answer.
The example from figure 2 of the paper: "What console does Nate own?". The agent first reads Nate-Entertainment.md, which mentions Cyberpunk 2077 and Xenoblade Chronicles. It searches the conversations with search_conversation("Nintendo Switch PS5 Xbox console") and finds that Nate is a big Nintendo fan. Then it fetches the exact record with get_records: he's playing Xenoblade Chronicles, a fantasy RPG. It reasons and answers: a Nintendo Switch. Three steps across three different levels.
There's a limit: at most four tool steps per question; going over counts as a wrong answer. And the prompt is clear: don't guess anything about the user; if two memories contradict each other, instruction records take priority; use raw conversations when you need exact evidence; and stop searching as soon as you have enough evidence, without chasing marginal details. The final answer is JSON with the answer and the reason.
Learning to navigate: the rubric and GRPO
Having tools isn't enough: you have to know how to use them. That's why they train the agent with reinforcement learning. Each example is a question that needs memory, along with that user's pyramid; the agent answers with its chain of calls and only gets a score at the end, for the outcome. That score trains both the answer and every navigation decision that led to it.
The score follows a rubric with three binary criteria: valid format (F), correct answer (C) and memory use (U):
| Case | F | C | U | Score |
|---|---|---|---|---|
| Right, using memory | 1 | 1 | 1 | +1 |
| Right, without memory | 1 | 1 | 0 | 0 |
| Wrong after checking memory | 1 | 0 | 1 | −0.5 |
| Wrong without checking | 1 | 0 | 0 | −1 |
| Broken format (or over budget) | 0 | – | – | −1 |
So searching and being right is rewarded, and making things up is punished harder. Since every training question needs memory, getting it right without checking memory doesn't score.
The algorithm is GRPO (Group Relative Policy Optimization). For each question, four attempts are generated, and each attempt's advantage is its score minus the group's average: attempts above the average are reinforced and the ones below are penalized. There's no need for a separate model to judge every step: the group itself is the reference. The advantage applies to every token of the attempt, including the tool-call tokens, with the usual clipped objective and a KL penalty against the starting model.
If you want to reproduce it, the hyperparameters from the appendix: Qwen3.5-9B as the base model, batch size 8, groups of 4 attempts, learning rate 1×10⁻⁶, KL coefficient 0.001, no entropy term, bfloat16, 5 epochs and up to 5 assistant turns, with the verl framework on NVIDIA H20 GPUs.
Training and evaluation
They train on questions from two benchmarks, LoCoMo and PersonaMem-v2, splitting by user (60% for training, 20% for validation and 20% for testing) so no one shows up on both sides. The third, LongMemEval, is never seen during training: it's there to check whether what was learned works outside familiar ground. LoCoMo and LongMemEval measure factual memory (recalling something from long ago, changes over time, reasoning across sessions); PersonaMem-v2, implicit preferences. The test sets have 1,315 LoCoMo questions, 100 LongMemEval questions and 911 PersonaMem-v2 questions.
They compare it against five memory systems: Mem0 (flat vector memory), Zep (temporal graph), MemOS (a memory operating system), MemoryOS (hierarchical memory) and AgeMem (agentic memory trained with RL), all on the same Qwen3.5-9B. NapMem and AgeMem get at most four tool steps and each search returns up to 5 items; non-agentic systems are allowed up to 20 per search.
For open-ended questions the judge is Claude Opus 4.6. They checked its reliability on 100 answers labeled by three people: it agrees with the human majority 99% of the time (precision 1.0, recall 0.98, Cohen's κ 0.98). The only disagreement was a "0 times" the judge marked wrong when the expected answer was that it was never mentioned. All results are averaged over three seeds.
The results
Table 1 of the paper (L-J is the judge's score; the average combines the LoCoMo and LongMemEval judge scores with PersonaMem-v2 accuracy):
| System | LoCoMo F1 | LoCoMo L-J | LongMemEval F1 | LongMemEval L-J | PersonaMem-v2 | Avg. |
|---|---|---|---|---|---|---|
| Mem0 | 41.19 | 60.86 | 53.86 | 78.00 | 38.89 | 59.25 |
| Zep | 36.09 | 52.21 | 52.61 | 73.33 | 36.33 | 53.96 |
| MemOS | 35.22 | 55.82 | 39.16 | 54.33 | 37.58 | 49.24 |
| MemoryOS | 31.40 | 44.28 | 22.68 | 23.67 | 35.82 | 34.59 |
| AgeMem | 38.00 | 45.02 | 37.83 | 51.33 | 24.19 | 40.18 |
| NapMem-122B (untrained) | 35.39 | 51.69 | 48.86 | 70.00 | 41.38 | 54.36 |
| NapMem-397B (untrained) | 38.31 | 54.42 | 53.85 | 78.33 | 46.10 | 59.85 |
| NapMem-9B w/ RL | 41.28 | 59.92 | 57.41 | 80.33 | 47.97 | 62.74 |
Trained NapMem gets the best average, 62.74, versus 59.25 for the best rival, Mem0. On LongMemEval, the test it never saw in training, it reaches 80.33, the best of them all; on PersonaMem-v2, 47.97, also in the lead.
On LoCoMo it doesn't win everything, and that's worth saying: it has the best F1 (41.28 vs 41.19), but on the judge's score Mem0 is one point ahead (60.86 vs 59.92). In other words, on factual memory over long conversations it's on par with the best; where it pulls ahead is personalization and data it never saw during training.
Trained 9B vs 397B
The most striking part of the table is the two rows in the middle. They tried the same pyramid and the same tools with much bigger models, untrained: Qwen3.5-122B and Qwen3.5-397B. The biggest averages 59.85; the small 9B model, trained, 62.74. Learning how to use memory is worth more than making the model forty times bigger.
When memory isn't needed
What if the question doesn't need memory at all? To check that training doesn't turn it into an agent that checks memory for everything, they gave it three unrelated tasks (science with GPQA-Diamond, function calling with BFCL-v3 and visual search with V*Bench) with a random pyramid and every memory tool switched on:
| Benchmark | Metric | Base model | NapMem-9B w/o RL | NapMem-9B w/ RL |
|---|---|---|---|---|
| GPQA-Diamond | Accuracy | 53.03 | 55.74 | 57.58 |
| GPQA-Diamond | Memory calls | – | 34.51% | 6.90% |
| BFCL-v3 | Task / tool / exec | 95.00 / 97.50 / 93.50 | 93.50 / 97.50 / 94.00 | 95.00 / 97.50 / 94.50 |
| BFCL-v3 | Memory calls | – | 0% | 0% |
| V*Bench | Answer | 84.82 | 89.53 | 91.10 |
| V*Bench | Memory calls | – | 0% | 0% |
Untrained, the model checked memory on 34.51% of the science questions for no reason; trained, on just 6.90%. On the other two tasks, never. And it doesn't lose ground: it even improves a bit in science and in the visual test. Reinforcement learning calibrates when to use memory; it doesn't just use it more.
The cost: space and time
A user's memory never stops growing, so a small footprint matters. Across the three benchmarks, NapMem takes up 4.83 GiB, less than almost everyone despite storing four levels:
| System | PersonaMem-v2 | LongMemEval | LoCoMo | Total |
|---|---|---|---|---|
| NapMem | 0.69 | 4.01 | 0.12 | 4.83 |
| Mem0 | 1.40 | 8.90 | 0.14 | 10.44 (×2.16) |
| MemOS | 3.61 | 18.85 | 0.64 | 23.10 (×4.78) |
| Zep | 3.50 | 10.00 | 0.60 | 14.10 (×2.92) |
| MemoryOS | 1.98 | 4.30 | 0.40 | 6.68 (×1.38) |
| AgeMem | 0.34 | 2.56 | 0.09 | 2.99 (×0.62) |
Only AgeMem uses less, with its light flat memory, but it performs far worse (40.18 on average). Size alone doesn't tell the whole story.
And response time. On a hundred PersonaMem-v2 questions, latency grows almost in a straight line with the tokens the model generates: the more it writes, the longer it takes. Passive systems receive lots of text and reason over all of it; NapMem searches for what it needs and stops as soon as it has evidence, so it writes less and answers sooner, even though it uses tools.
Which piece matters
To see which part gives the edge, they removed pieces one at a time (judge score on LoCoMo and LongMemEval, accuracy on PersonaMem-v2):
| Variant | LoCoMo | LongMemEval | PersonaMem-v2 | Avg. |
|---|---|---|---|---|
| Full NapMem | 59.92 | 80.33 | 47.97 | 62.74 |
| Without reinforcement learning | 52.09 | 72.33 | 20.75 | 48.39 |
| Without navigation (passive retrieval) | 48.37 | 68.00 | 45.88 | 54.08 |
| Records-only tools | 39.77 | 60.00 | 35.02 | 44.93 |
| Without upper levels (no tracks or profile) | 49.13 | 75.00 | 38.20 | 54.11 |
All three pieces add up. Navigating beats receiving, even with the same memory available. The summaries on top complement the detail below. And using memory well is learned: without reinforcement learning, with the same pyramid and the same tools, the average drops to 48.39, and on PersonaMem-v2 it collapses from 47.97 to 20.75. Explaining it in the prompt isn't enough.
How it navigates
They looked at the first tool each model reaches for. The 9B models almost always start with the records (92.16% untrained, 88.84% trained). The big ones, on the other hand, start by reading files in almost half of the cases (44.38% for 122B and 46.60% for 397B). One possible explanation from the authors: big models digest a long summary better, while small ones prefer short, targeted evidence.
And reinforcement learning changes how it searches. Untrained, the 9B model made 3.97 calls per question; trained, 2.15, and it's still more accurate: out of every hundred results the tools return, 34.92 are useful evidence, compared with 20.66 before. It doesn't stick to a single level: 79.63% of questions still use several (80.39% before training). It has learned when to stop.
One success, one failure
The success. Question: "What state did Nate visit?". search_conversation("Nate visit state travel") turns up nothing useful; search_records(...) returns irrelevant or vague records. So the agent reads the track Nate-Identity-Interests.md, and there it is: Nate traveled to Tampa beach on November 11, 2022. Answer: Florida. Climbing to a more abstract level rescued what direct search couldn't find.
The failure. Someone asks for calm ideas for the weekend. The agent finds a record: the user thought about booking salsa for a charity gala because it had worked the year before. It finds another: they get energized at packed events but need time alone afterward. And it picks the option that assumes they love going salsa dancing every weekend. It turned a one-off event into a stable preference: the memories were relevant, but not strong enough for that conclusion. The authors flag it as future work: calibrating how much each memory actually proves.
Limitations
The authors point out three themselves:
- The benchmarks don't capture every real form of personalization or open-ended conversation.
- Privacy and forgetting, two key requirements for any system that remembers things about people, aren't fully addressed.
- Model size: how the memory-access strategy changes with scale still needs a proper study.
A curiosity: in the implementation appendix another name slips through, "PyraNav", most likely an earlier name for the method.
Wrap-up
NapMem stores the user's memory in a four-level pyramid (conversations, records, topic tracks and profile) and trains the agent with reinforcement learning to move through it with five tools, instead of receiving what someone else picks. That way, a 9B model beats an untrained 397B one, searches less, gets more right and almost never checks memory when it doesn't need to.
An agent's memory works better as a place it moves through than as a chunk of context it's handed.
What would you want your assistant to remember about you?
Sources
- "From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space", by Yue Xu, Yutao Sun, Yihao Liu, Mengyu Zhou, Jiayi Qiao, Lu Ma, Kai Tang, Wenjie Wang, Xiaoxi Jiang and Guanjun Jiang (Qwen, Alibaba; ShanghaiTech; Zhejiang; Peking; NUS), arXiv 2607.05794, July 7, 2026.
- On other approaches to agent memory: my posts on Quarq Agent and LongMemEval and on Karpathy's LLM Wiki.
- The video, on my channels: English and Spanish.
