Better memory helped a personal agent find the right rules—and still produced unsupported health assumptions
On 349 synthetic health and daily-living tasks, a memory-linked retrieval system reached 86.25% accuracy versus 79.94% for a leading memory baseline. Its remaining failures show why retrieving the right document is not enough when an agent misstates what it knows about a person.
The short version
- What happened
- ColaBench contains 349 synthetic tasks across 10 fictional profiles and a frozen knowledge base of 13,033 real public health and daily-living documents.
- Who it affects
- A personal agent that remembers sensitive history could save someone from repeating context, but a misclassified medicine, outdated condition or lost uncertainty can make a sourced answer personally wrong.
- What remains unproven
- This is an unreviewed preprint using 10 synthetic profiles; it does not measure outcomes for patients, carers or other real users.
Keep in mind
This is an unreviewed preprint using 10 synthetic profiles; it does not measure outcomes for patients, carers or other real users.
The benchmark construction, answering and evaluation pipelines rely heavily on language models. Accuracy was not independently calibrated by clinicians or a representative human panel.
All methods shared one answering model and a frozen knowledge base. Results may not transfer to other models, live web sources or domains outside health and everyday information.
The 111-task component ablations cover three selected profiles, and exploratory subgroup intervals are wide.
Several authors list company affiliations, including DataArc Tech Ltd.; no separate funding or competing-interest statement is reported.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
The test joined conversation memory to documents the agent had not seen
A personal agent may need two kinds of evidence at once: something a user disclosed weeks earlier and a rule found today. The researchers built 10 fictional profiles whose public conversation histories contain personal conditions, preferences and earlier document access. They then created 349 health and daily-living tasks backed by a frozen collection of 13,033 documents from sources including MedlinePlus, openFDA and the US National Cancer Institute. The personal histories are synthetic; the documents are real public information.
Five systems received the same current request, knowledge index and DeepSeek v4.1 Flash answering model. The difference was how they stored and retrieved history. MemKB first assembled a user-state summary, linked remembered conditions to documents associated with earlier conversations, and fused those candidates with ordinary semantic and keyword search. A GLM-5.3 Flash evaluator checked each final answer against a reference, required conclusions and the evidence actually shown to the system.[1]
Linking memory to knowledge improved the benchmark result
MemKB passed 86.25% of the 349 tasks, compared with 79.94% for Mem0 plus the same hybrid document retrieval. The paired gain was 6.30 percentage points; a profile-cluster bootstrap placed its 95% interval between 3.19 and 9.09 points. MemKB also used 4.07 interaction rounds per task on average, against 4.44 for that baseline. Removing the state summary or the memory-to-document links reduced accuracy in the smaller 111-task ablation set.
The improvement was not simply a matter of reading more reference material. When both systems accessed the reference document, MemKB scored 90.44% and the Mem0 baseline 86.03%; when both actually read it, the figures were 90.74% and 87.96%. The authors interpret the remaining gap as evidence that organising known personal conditions helps the agent apply retrieved rules. Because a model graded the answers, human calibration is still needed before treating those percentages as dependable real-world accuracy.[1]
The distinction that matters
Where a memory-guided answer can succeed—or fail
The benchmark traces both the personal premise and the external rule behind each required conclusion.
Remember
Recover a disclosed condition from earlier conversations
Preserve who said it, when and what remains unknown
Retrieve
Find the document whose rule matches that condition
Expose the evidence needed for the current request
Apply
Compare the rule with the person's known circumstances
Keep unsupported premises conditional instead of turning them into facts
A remembered detail can become a false fact during reasoning
The paper's most useful warning appears in the 48 failed MemKB answers. Seventeen were primarily classed as unsupported personal assumptions and 12 as rule misapplication. In one synthetic case, the record said only that a person used prescription pain medicine; the system converted that into opioid use when applying naloxone guidance. In another, the system read heat-risk guidance but still recommended a foot bath. Better retrieval did not prevent the agent from changing an unknown condition or misusing a visible rule.
For people, that is the difference between an assistant recalling context and a system making a personal judgment. The benchmark does not test live web changes, real users, clinical safety or privacy, and several authors list company affiliations. Independent human grading, prospective tests with consented users and an audit trail that shows which disclosure supports each conclusion would change the assessment. Until then, the result supports traceable memory evaluation—not autonomous health advice.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring long-term memory: compare the finding and its evidence limits.
Can an AI agent safely change a clinical record?
A 43-task benchmark found that long-term memory helped a GPT-4.1-mini agent use FHIR tools, but the strongest held-out result was only 60.6% task success. This was a resettable test server—not a live hospital record system.
Read the next reportExplore the full source and evidence record
What this means for people
- A personal agent that remembers sensitive history could save someone from repeating context, but a misclassified medicine, outdated condition or lost uncertainty can make a sourced answer personally wrong.
Global context
Long-term-memory agents are being designed across languages and services, but personal histories and public guidance change over time. Any deployment would need local privacy, consent, record-correction and professional-advice safeguards beyond benchmark accuracy.
What the evidence does not yet show
- This is an unreviewed preprint using 10 synthetic profiles; it does not measure outcomes for patients, carers or other real users.
- The benchmark construction, answering and evaluation pipelines rely heavily on language models. Accuracy was not independently calibrated by clinicians or a representative human panel.
- All methods shared one answering model and a frozen knowledge base. Results may not transfer to other models, live web sources or domains outside health and everyday information.
- The 111-task component ablations cover three selected profiles, and exploratory subgroup intervals are wide.
- Several authors list company affiliations, including DataArc Tech Ltd.; no separate funding or competing-interest statement is reported.
What to watch next
- Independent human and domain-expert grading of the benchmark
- Prospective tests with consented users and changing real-world records
- Auditable provenance for every personal condition used in an answer
- Methods that preserve unknown or outdated information instead of inferring a convenient fact
Living evidence record
Impact record IAI-1D6XVVP
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
23 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 23 October 2026
Research question, topics and regions
Whether a personal agent can use facts disclosed across earlier conversations to retrieve new rules and apply them without inventing personal conditions
Topics: Personal agents · Long-term memory · Health information · Retrieval · Evidence grounding
Regions: China · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.