· 4 min read
The More Your AI Remembers, the Harder Memory Becomes
As AI agents accumulate years of data, they risk becoming rigid or inaccurate. Effective memory requires context, expiration, and the ability to adapt to change.
Understanding AI systems through research and practice.
Exciton Research brings together Tian Shi’s work in machine learning, retrieval, and production AI systems. Current research focuses on AI agents: how to establish what they actually accomplished, and how to make evaluations reliable as agents and tasks change.
Evaluation of AI agents
In detailAn agent can report that the work is done when the record says otherwise. The question is which observations establish what the agent changed, and whether that measurement still holds when the agent, the judge, or the task changes.
- 01
Which observations establish each requirement, and which important facts stay unobserved.
- 02
Reliability across repeated trials, with cost beside accuracy, not a single lucky success.
- 03
Whether an automated judge still grades fairly after the agent is upgraded.
- 04
A proposed study: under a fixed budget, can an evaluator choose which extra evidence to inspect and compare two agents more accurately?
Working survey, October 2026. The comparison study is proposed. No new experiments have been run.
Read the evaluation essayFrom the blog
All posts· 3 min read
Automatic AI Agent Evaluation: What to Check with Code, Reference Data, and LLM Judges
Learn how to evaluate AI agents by combining automated code checks, reference data, and LLM judges to measure task success, accuracy, and safety effectively.
· 1 min read
A Checklist for Evaluating AI Agents
Evaluate AI agent performance with a structured checklist covering task success, safety, tool use, and error recovery to ensure reliable, secure deployment.