Evaluate AI Agent Memory With Reproducible Tasks
“The agent remembered” is not a measurement. A useful evaluation fixes the source corpus, task set, expected answer, allowed evidence, system version, and grading rule before the run.
The score above is a conservative release heuristic: 55% answer accuracy, 35% source-citation coverage, and a penalty for stale-memory use. It is not a universal benchmark and does not prove business impact.
Build a task set that can fail
Include at least these categories:
- Exact retrieval: current price, approved owner, active configuration, or last verified state.
- Temporal state: distinguish what was true last month from what is true now.
- Correction: prefer the superseding record while preserving the earlier record as history.
- Conflict: surface two credible sources instead of averaging them into a false answer.
- Abstention: refuse when the memory does not contain enough evidence.
- Source citation: link the record or event used for the answer.
- Privacy: exclude a secret or restricted record from an unauthorized task.
- Action boundary: retrieve relevant context without claiming the context grants permission.
Version everything needed to reproduce a run
Record the corpus snapshot or commit, memory schema, retrieval settings, embedding model if used, agent/model version, task-set version, expected answers, grading rule, run time, and environment. If a model change improves the result, you should be able to tell whether it improved retrieval, reasoning, or only presentation.
Metrics that should not be blended away
| Metric | Why it matters | Failure example |
|---|---|---|
| Answer accuracy | The retrieved and synthesized answer is correct | Correct customer, wrong plan status |
| Citation precision | The cited record actually supports the claim | Link points to an old receipt |
| Stale-memory rate | Superseded facts are not treated as current | Old price reused after correction |
| Abstention quality | Missing evidence produces an honest stop | Agent invents an approval |
| Retrieval latency | Memory fits the task’s response budget | Correct answer arrives after the workflow times out |
| Privacy leakage | Unauthorized data stays unavailable | Private note enters a public draft |
Compare against simple baselines
Test a curated-file baseline before adding a vector database. Test exact filters before semantic retrieval. Test a smaller authoritative corpus before indexing everything. A more complex system earns its operational cost only when it improves a declared task without weakening freshness, auditability, or privacy.
Release rule
Do not release an agent into higher-impact actions because its average score improved. Critical tasks—permission checks, current financial facts, destructive actions, identity, and private data—need individual pass conditions. One critical stale-memory or privacy failure can block the release even when the composite score looks healthy.