Quarq Agent: 98.2% on LongMemEval with an agent that remembers

The exam: LongMemEval

LongMemEval measures the long-term memory of chat assistants. Its S version has 500 questions, and before each one the system reads a haystack of about fifty conversation sessions; the answer is hidden in one of them. In total, around 57 million tokens, roughly 115,000 per question.

There are six types of questions. Three look at a single session: something the user said, something the assistant replied, and a preference mentioned in passing. The other three are harder: combining facts from several sessions, knowing the current version of something that changed, and reasoning about dates ("how long ago did that happen?").

So why not put the whole conversation in the context and be done? Because it does not work. In the article's table, GPT-4o with the full history in front of it gets only 60.2% right; given only the sessions that contain the answer (the "oracle"), 82.4%. Remembering is not reading everything: it is finding what matters.

The results

Quarq Agent gets 491 of the 500 questions right: it misses only nine. That is 98.2%. And it does it with memory on your own machine: a local FAISS vector index and a layered memory architecture.

By type, the three single-session categories reach 100%. Knowledge updates reach 98.7%, with a single miss. And the two hardest, multi-session and temporal reasoning, land at 97%, with four misses each.

In the leaderboard they publish, second place goes to Mastra (with GPT-5 mini), at 94.87%, followed by Hindsight (with Gemini 3 Pro), at 91.4%. Several systems using GPT-4o land between 81.6% and 86%. Quarq is 3.3 points ahead.

The article highlights three things. First, its generator (GPT-4.1) is less capable than the newer models many rivals use. Also, temporal reasoning is the hardest category, and its protocol targets the most common failure: confusing when a fact was stored with when it happened. And finally, everything runs locally, while nearly every other system depends on cloud vector databases.

A grain of salt

The comparison is worth taking with a grain of salt. Each team builds its own evaluation, with its own judge and models: Quarq grades the answers with GPT-5. And in the article's own charts, some rivals' numbers do not quite match its table: Mastra shows up at 94.37% in one and 94.87% in the other, and Supermemory at 84.6% versus 81.6%.

They themselves warn that the result is provisional: it may change as uncertain questions are rerun and fixes are added.

Memory in three layers

Inside, the memory has three layers:

1. Semantic: facts and interaction history, what happened, when, and who was involved.
2. Procedural: behavioral and operational rules.
3. Episodic: conversational experiences, with their context.

Three models instead of one

Most memory systems use a single language model for everything: retrieval, generation and memory management. It is like a one-man band, playing every instrument at once and none of them quite right. Quarq splits those jobs into three roles, each with its own model:

Searching well: HyDE and hybrid retrieval

The planner uses HyDE (Hypothetical Document Embeddings): instead of searching with the question as it is, it imagines what a document answering it would look like, and searches with that. It is like a police sketch: you do not have a photo of the suspect, but a rough drawing helps you find them in a crowd.

Each retrieval pass combines four pieces: vector similarity, keyword search, metadata filters and temporal validation. Being similar in meaning is not enough: the memory also has to contain what matters and fit in time.

And there are two modes. The precise mode uses a similarity threshold of 0.38, for specific facts. The deep mode lowers it to 0.28, for broader recall. It is the difference between fishing with a rod and casting a net.

Self-correcting retrieval

What if what it finds is not enough? When the first search does not give enough evidence to answer confidently, the generator can ask for another. The system then runs new searches with the deep threshold, merges the new memories with what it already had, and rewrites the answer under stricter rules against hallucination.

The idea behind it is simple: the model can ask for better evidence instead of making up what is missing.

The Temporal Truth Protocol

Reasoning about time is one of the hardest problems in long-term memory. Quarq separates four dates: the storage date (when the memory was saved), the event date (when it actually happened), the narrative date (the benchmark's simulated timeline) and relative dates, like "last week", which are resolved at retrieval time.

Think of an old photo you put in a new box today: the box says 2026, but the photo is from your childhood. If someone asks how long ago that summer was, the answer cannot come from the label on the box. That is why the rules are hard-coded: bracketed timestamps are not event dates; a date is never inferred from neighboring memories; two unrelated memories are never assumed to be about the same thing; and "how long ago" is always computed from the actual event.

Quantitative fidelity

Numbers are a similar story: memory benchmarks punish sloppy arithmetic. For totals, durations, prices, quantities and counts, Quarq identifies four things: who acts, which action is measured, on what event or object, and how exact it needs to be. And it throws out numbers that are merely nearby or on the same topic: it only adds up the values that meet every condition.

An example of mine: if you ask how much you spent on books, what your brother spent does not count, and neither does the bookshelf.

Learning without making you wait

So you do not notice any delay, the agent learns in the background, after every response, with three safeguards: at most four learning tasks at once, persistent retries that wait longer and longer between attempts, and synchronization barriers during evaluation.

And this is how it is evaluated: all memories are wiped; the haystack sessions are fed in chunks of eight messages, with learning enabled; pending learning is allowed to finish; the question is asked with learning disabled, and GPT-5 decides whether the answer is right or wrong.

Every piece came from a failure

The first version of Quarq used flat text files, loaded whole into the context on every request. It worked poorly, but it taught them something: a single model cannot optimize retrieval quality, answer quality and cost all at once. That led to four versions: 0.1, with flat files and a single model; 0.2, with vector retrieval and a better memory structure; 0.3, with the three models, the memory editor and the temporal protocol; and 0.4, with self-correcting retrieval, released as open source and ready for production.

And every piece came from a specific failure. HyDE, from seeing that single-vector search missed relevant memories. The temporal protocol, from hallucinations like answering "three weeks ago" when the answer was "yesterday". And the second pass, from models confidently making up answers instead of admitting they lacked context.

Try it yourself

Along with the agent, they released the program they used to run the benchmark, quarqlabs/benchmarks. It is kept separate: it talks to the agent over HTTP, like any other client, instead of importing its code, which makes the evaluation easier to reproduce and inspect on your own. It includes the cleaned dataset, evaluation runs that resume where they left off, a log of the retrieved context, isolated memory for each question, parallel evaluation and cost guidance. The goal: reproducible results instead of opaque leaderboard claims.

To try it, these are the commands from the article:

git clone https://github.com/quarqlabs/agent-oss
cd agent-oss
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# .env: OPENAI_API_KEY=sk-...  USER_ID=local_user  AGENT_ID=my_agent
python agent.py                      # the agent in the terminal
uvicorn main:app --reload            # or as a server
curl -X POST http://127.0.0.1:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What do you remember about me?"}'
python run_dataset_evals.py          # the full evaluation
EVAL_WORKERS=4 python run_dataset_evals_parallel.py   # in parallel

A heads-up if you go looking for it: the repository is now called quarqlabs/argus ("Argus Agent"). Its documentation warns that a full LongMemEval-S run has cost about $2,500, roughly five per question, so better start with just a few. Everything is released under the Apache 2.0 license.

Wrap-up

Quarq Agent reaches 98.2% on LongMemEval with a deliberately simple architecture: three specialized models, local hybrid retrieval, strict rules for dates and numbers, and a second search when evidence is missing.

What would you want your agent to remember?

Sources

• "98.2% on LongMemEval: How We Built an Open-Source Agent That Actually Remembers", by Quarq (@quarqlabs), June 1, 2026.
• The code: quarqlabs/argus (formerly agent-oss) and the benchmark runner, quarqlabs/benchmarks.
• On another kind of memory for agents: my post on Karpathy's LLM Wiki.
• The video, on my channels: English and Spanish.