Evaluate AI Agent Memory With Reproducible Tasks
“The agent remembered” is not a measurement. A useful evaluation fixes the source corpus, task set, expected answer, allowed evidence, system version, and grading rule before the run.
Test a correction, a citation, and a refusal
Run exact retrieval against five labelled synthetic records. This is deterministic JavaScript, not model inference, embeddings, or a deployed memory service. All fixture values, including the simulated restricted record, are public demo data.
Ready. The current support window should cite handbook-v2.
Newest version wins only with an intact correction reference. A restricted current record never falls back to an older public value. Missing evidence, unknown roles, and conflicts abstain. Sources do not grant action permissions. Real access controls must run on a trusted server.
Inspect all five synthetic source records
See a failing stale-answer test
The starting answer deliberately uses the superseded fact. Grade it, then replace it with the current answer from the lookup.
Candidate has not been graded.
Run correction, citation, access, and abstention fixtures
Fixtures have not run yet.
Run node rick-memory-eval.js with Node.js 22+. The standalone source contains the corpus, retrieval rules, and PASS/FAIL fixtures. No dependencies, credentials, network, or file writes. Results describe these synthetic cases only; they are not measured customer or product outcomes.
The score above is a conservative release heuristic: 55% answer accuracy, 35% source-citation coverage, and a penalty for stale-memory use. It is not a universal benchmark and does not prove business impact.
Build a task set that can fail
Include at least these categories:
- Exact retrieval: current price, approved owner, active configuration, or last verified state.
- Temporal state: distinguish what was true last month from what is true now.
- Correction: prefer the superseding record while preserving the earlier record as history.
- Conflict: surface two credible sources instead of averaging them into a false answer.
- Abstention: refuse when the memory does not contain enough evidence.
- Source citation: link the record or event used for the answer.
- Privacy: exclude a secret or restricted record from an unauthorized task.
- Action boundary: retrieve relevant context without claiming the context grants permission.
Version everything needed to reproduce a run
Record the corpus snapshot or commit, memory schema, retrieval settings, embedding model if used, agent/model version, task-set version, expected answers, grading rule, run time, and environment. If a model change improves the result, you should be able to tell whether it improved retrieval, reasoning, or only presentation.
Metrics that should not be blended away
| Metric | Why it matters | Failure example |
|---|---|---|
| Answer accuracy | The retrieved and synthesized answer is correct | Correct customer, wrong plan status |
| Citation precision | The cited record actually supports the claim | Link points to an old receipt |
| Stale-memory rate | Superseded facts are not treated as current | Old price reused after correction |
| Abstention quality | Missing evidence produces an honest stop | Agent invents an approval |
| Retrieval latency | Memory fits the task’s response budget | Correct answer arrives after the workflow times out |
| Privacy leakage | Unauthorized data stays unavailable | Private note enters a public draft |
Compare against simple baselines
Test a curated-file baseline before adding a vector database. Test exact filters before semantic retrieval. Test a smaller authoritative corpus before indexing everything. A more complex system earns its operational cost only when it improves a declared task without weakening freshness, auditability, or privacy.
Release rule
Do not release an agent into higher-impact actions because its average score improved. Critical tasks—permission checks, current financial facts, destructive actions, identity, and private data—need individual pass conditions. One critical stale-memory or privacy failure can block the release even when the composite score looks healthy.