Agents that must discover the rules
Interactive benchmarks so far test a deductive mode: the agent is told how the world works and is scored on carrying out a task in it. Deployment also needs the inductive mode, where the agent works out how the world works from what its actions do.
Deductive
The transition rules arrive in the prompt. Success measures how well the agent applies them.
Inductive
Nothing about the dynamics is given. The agent has to probe, read the feedback and build the rules itself.
Extremely long horizons
Episodes of hundreds to thousands of steps, where strategic coherence and accumulated error decide the outcome, where most suites stop under fifty.
Active exploration
No fully specified success criterion to execute against. The agent probes, reacts to feedback and revises its own strategy as it goes.
Inductive reasoning from interaction
Latent transition laws are inferred from real-time experience, which is what lets an agent anticipate outcomes and plan instead of react.
- A new perspective for agent evaluation. Agents are scored on what they can discover about an environment's dynamics through long-horizon, active interaction, rather than on following rules they are handed.
- Reliable and scalable evaluation. Four environments built on four structural primitives, each with an unbounded task space; OdysseyArena-Lite fixes a standard suite for efficient, reproducible comparison and OdysseyArena-Challenge stress-tests stability over extreme horizons.
- Extensive evaluation and insights. Frontier and open-weight LLMs, with an analysis that locates the bottleneck in inductive discovery: exploration, memory and action loops, not planning.
Four primitives, four environments
Every environment is a transition function the agent cannot see. Each one isolates a different kind of structure to be discovered: Boolean couplings, factors under noise, periodic cycles, and a dependency graph. All four run as lightweight APIs, with no simulator or container to deploy.
Trajectories follow the demonstrations in the paper; the prices, news and package names are schematic. Use the arrow keys or Play to step through.
Two suites from one task space
Each environment generates tasks from a bounded parameter range. A task is a structural configuration that stays fixed for the episode plus a pre-computed temporal trajectory, so every model faces exactly the same world and every instance is verified solvable.
OdysseyArena-Lite
The suite behind every result on this page: a tractable parameter range, run at high throughput, with horizons long enough that nothing short of rule induction gets through.
- Tasks
- Step budget
- Protocol
OdysseyArena-Challenge
The empirical limits of the same parameter ranges, for failure modes that never surface under a standard budget: credit assignment over very long histories and compounding planning errors.
- Tasks
- Horizon
- Use
- Optional; not part of the main evaluation
From a configuration to a verdict
A task is written once as a configuration, generated and validated automatically, then run through one interaction loop: the agent reads the observation and its working memory, emits an action, and the environment's step logic answers.
1Configuration
Entity counts, dependency depth, noise, periods and thresholds, set by hand per difficulty.
difficulty control2Task generator
Samples the latent rules and the fixed temporal trajectory, then checks a solution exists.
automatic validation3Config files
One deterministic instance per file; identical for every model and every run.
init()load the instancestep()apply the action, return the observationeval()check the terminal stateAgainst other agent benchmarks
Whether a benchmark requires inducing rules, how many steps a task takes (short under fifty, long up to a hundred, extra long beyond), and what has to be deployed to run it.
| Benchmark | Inductive | Horizon | Deploy |
|---|
How far frontier models get
Proprietary and open-weight LLMs on OdysseyArena-Lite, four runs per task, against human participants who solved the same instances by trial and error.
Human participants outperform every evaluated model on Avg@4 in Turn On Lights, AI Trading and Repo System. Gemini 3 Pro Preview leads on Energy Dispatch.
Given the latent rules in the prompt, frontier models reach near-perfect success on Turn On Lights. Without them the same models fall to a fraction. The bottleneck is discovering the dynamics, not executing them.
Energy Dispatch needs about twenty steps of observation to pin down an efficiency cycle. Nearly every model fails every task there, and scale alone does not change that.
Leaderboard
Success rate per environment; profit rate for AI Trading. Click a column to sort.
The same models with the rules given
Each model run twice on the same tasks: as in the benchmark, and with the hidden transition rules written into the prompt.
Where the runs go wrong
Failed trajectories were read by hand and the behaviour measured over every run: what a model does with more steps, how often it repeats a rejected action, and which interventions move the number.
More steps stop helping
Cumulative share of tasks solved by each step, or profit rate for AI Trading. Most success curves flatten after an early exploratory phase; a longer budget does not substitute for a model of the dynamics. The weakest models track or trail a random policy.
Action loops
The share of a model's steps spent immediately repeating an action the environment just rejected, against its success rate. Looping is inductive stagnation made visible: the model does not turn the failed trial into a revised rule.
What went wrong in the failed runs
Every failed trajectory of Gemini 3 Pro Preview, read and labelled by hand; one trajectory can carry several modes. Planning is almost never the problem.
More reasoning helps
gpt-oss-120b at three reasoning budgets. Higher effort improves Turn On Lights and AI Trading. Repo System dips at medium effort; Energy Dispatch stays at zero.
A longer memory does not
Repo System with more or fewer turns of history in the prompt. A window of about a hundred turns does best; the full history does worse.
A worked example does not either
One complete solution in the prompt leaves Repo System success where it was. Eliciting induction takes more than a demonstration.
Paper, code and benchmark
Paper
The full description of the four environments, the two suites and the analysis, on arXiv.
Code
The environments, the task generators and the evaluation harness.
Benchmark
The OdysseyArena-Lite and OdysseyArena-Challenge task instances, in the code repository.
Leaderboard
Every evaluated model on the four environments, sortable.
BibTeX
@article{xu2026odysseyarena,
title={OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions},
author={Xu, Fangzhi and Yan, Hang and Sun, Qiushi and Wu, Jinyang and Huang, Zixian and Huang, Muye and Gong, Jingyang and Ding, Zichen and Cheng, Kanzhi and Wang, Yian and Che, Xinyu and Sun, Zeyi and Zhang, Jian and Yin, Zhangyue and Luo, Haoran and Kao, Ben and Lin, Qika},
journal={arXiv preprint arXiv:2602.05843},
year={2026}
}