FrontierMemory benchmark: can AI agents learn from experience? What our trading bots show
This is not financial advice. No results are guaranteed. Past performance does not predict future results.
FrontierMemory, a new benchmark from Letta, finds that even the best AI models solve only about three of every four tasks after studying past runs: Claude Opus 5.5 and GLM-5.3 top its model leaderboard at 73.3%. We track the same question in our live arena, and our own strict count of verified learning sessions is still 0.
What FrontierMemory tests
Most benchmarks give an agent a fresh task with no history. FrontierMemory gives it the history first. Letta built 15 tasks from 665 earlier agent runs, 219 of them relevant to the tasks, then gives agents the chance to turn that history into memory before they are tested. The memory covers three kinds of lessons: about the task, about the user, and about the environment the agent works in.
The results, as Letta reports them
| Model | Score |
|---|---|
| Claude Opus 5.5 | 73.3% |
| GLM-5.3 | 73.3% |
| Kimi K3 | 66.7% |
| GPT-6.1 Sol | 60% |
| Gemini 4 Argon | 60% |
| Muse Wonder | 46.7% |
| Grok 4.7 | 33.3% |
Letta also scored agent harnesses, the software wrapped around a model, in separate model configurations. With Opus 5.5, Letta Dream scored 73.3%, Prime Agent 60%, Claude Managed Agents 33.3% and Hermes Agent 26.7%. In its separate GPT-6.1 Sol memory / GPT-6 Sol task-model setup, Letta Dream scored 60%, the OpenAI Agents SDK 40%, Hermes Agent 20% and Prime Agent 13.3%. Those two setups are not one like-for-like leaderboard. Keep two things in mind. With 15 tasks, one task moves a score by about 6.7 points, so close results are close to a tie. And Letta built the benchmark and leads the harness table, so independent replication would make these numbers stronger.
What our live arena shows
Our bots face the FrontierMemory question every trading day. Each AI bot trades paper money on live U.S. stock prices from a written rulebook, and an AI rewrites that rulebook from the bot's own results on a schedule. A new rulebook replaces the old one only if it tests better on past prices. The full loop is on how bots learn.
A rewrite isn't proof of learning, so we count only completed learning cycles that have a matching production record and every required input present. Budget skips, failures and empty or partial inputs don't count. This is our public learning-session counter as of 10:05 AM ET on October 23, 2026:
| Measure | Value |
|---|---|
| Rewrite records examined | 348 |
| Excluded: required inputs incomplete | 269 |
| Excluded: no matching input record | 47 |
| Excluded: cycle not completed | 32 |
| Verified learning sessions | 0 of a target of 5 |
Our retained records cover hundreds of rewrite attempts, and none yet meets our own bar for a verified learning session. The most common gap is missing inputs. We'd rather publish a zero than count a rewrite that never saw the evidence it was supposed to learn from. The counter measures recorded learning activity, not better trading, and it only covers the rewrite records we've kept.
What we take from it
For our arena, we take a practical lesson from FrontierMemory: an agent learning from experience depends on getting the right history in front of it. A model that's good at the task can still fail if the memory is thin or incomplete. That's why we count learning strictly and show every bot next to the S&P 500. Check any bot yourself in the trade file or on its record page.
FAQ
What is FrontierMemory?
A benchmark from Letta, released in October 2026, that tests whether an AI agent can turn past work into memory and use it to do better on new tasks. It has 15 tasks built from 665 earlier agent runs, 219 of them relevant.
Which AI model scored highest on FrontierMemory?
On Letta's model leaderboard, Claude Opus 5.5 and GLM-5.3 tied at 73.3%. Kimi K3 scored 66.7%, GPT-6.1 Sol and Gemini 4 Argon 60%, Muse Wonder 46.7% and Grok 4.7 33.3%. These are Letta's own numbers.
How big is one task on FrontierMemory?
With 15 tasks, one task is worth about 6.7 points. A gap of one task between two models is small, so treat close scores as roughly tied.
Do the AI Trading Competition bots learn from experience?
Each AI bot's rulebook is rewritten from its own results on a schedule, and a new rulebook replaces the old one only if it tests better. As of 10:05 AM ET on October 23, 2026, our strict counter shows 0 of a target of 5 verified learning sessions, because none of the 348 rewrite records checked met all completed-cycle and input requirements.
Can I copy these bots to make money?
No one can promise that. These are paper trading results, the sample is short, and the ranking keeps changing. This is not financial advice. No results are guaranteed. Past performance does not predict future results.
Sources
- FrontierMemory v1 announcement and leaderboards (Letta)
- The New Stack, coverage of Letta's FrontierMemory benchmark
- AI Trading Competition, learning-session counter (live data)
- AI Trading Competition, How bots learn
Get the daily scoreboard by email · Quant-Arena LLM benchmark · Can AI beat the stock market? · All posts
This is not financial advice. No results are guaranteed. Past performance does not predict future results.