Finding the Right Fit: Harness × Model Agent Trajectories

Paper · Code (GitHub)

This dataset accompanies the paper Finding the Right Fit: Model–Harness Interactions across Agent Tasks. It contains 6,204 scored agent runs from 66 model–harness configurations on three command-line agent benchmarks, with the full model–tool conversation of every run in one common format, plus a per-task results table with reward, cost, token usage and agent time.

⚠️ For analysis, not for training. This dataset is released for analyzing agent behavior, comparing harnesses and other research on agent evaluation. Please do not use it to train, fine-tune or distill models. The benchmarks behind it ask the same: the ALE authors do not permit training on their data, and Terminal-Bench tasks carry a canary string that marks them as not for training corpora.

Configurations

• Harnesses: OpenHands (Software Agent SDK 1.44.1), DeepSeek Harness 0.1.1-rc.2, PI 0.84.4, openJiuwen 0.1.18, and the native pairings Codex 0.150.1 with GPT-6 Astra and Claude Code 2.1.251 with Claude Opus 5.
• Models (via OpenRouter, pinned to first-party providers): Claude Opus 5, GPT-6 Astra, GLM-5.3, Kimi K3, DeepSeek V4 Pro.
• Benchmarks: TUA-Bench (120 tasks), ALE-CLI (99 local-Docker tasks), Terminal-Bench 4 (63-task non-H100 subset).
• One run per task. Each record is the final scored run of its task; runs repeated because of infrastructure failures are replaced by their rerun. Unresolved outcomes score 0.

The code that ran these configurations, converted the logs and produced the paper's figures is on GitHub.

Files

results/task_level.tsv (.parquet)        one row per scored run (6,204 rows)
results/config_cost_summary.csv          one row per configuration (66 rows)
trajectories/<benchmark>/<harness>/<model>.jsonl.gz   trajectories, one file per configuration
viewer/<harness>/<benchmark>.parquet     the same trajectories in Parquet, one subset per harness

<benchmark> is tua-bench, ale-cli or terminal-bench-4; <harness> is openhands, deepseek-harness, pi, openjiuwen, codex or claude-code.

Trajectory record format

Each line of a .jsonl.gz file is one run:

Of the 6,204 runs, 6,067 include the full conversation. The other 137, mostly runs whose environment never started or whose native log was lost, carry results only (trajectory_available: false). Some harnesses do not log everything: Codex reasoning is encrypted, so its reasoning field is empty, and Claude Code and PI do not record their system prompts.

Very long strings are truncated (first 25,000 and last 5,000 characters kept), images are replaced by a placeholder, and secrets and host paths are removed. Verifier files, test outputs and benchmark reference data are not included.

Loading

from datasets import load_dataset

results = load_dataset("yixuanli97/finding-the-right-fit", "results", split="train")
runs = load_dataset("yixuanli97/finding-the-right-fit", "openjiuwen", split="train")  # one subset per harness
print(runs[0]["task_id"], runs[0]["reward"], len(runs[0]["messages"]))

License and terms

Released under CC BY-NC 4.0, because TUA-Bench is licensed CC BY-NC 4.0; ALE-CLI data is CC BY 4.0. Please also respect the licenses and terms of the three benchmarks and of the harnesses: use the data for analysis and research, do not use it to train, fine-tune or distill models, and do not expose it to agents that are being evaluated on these benchmarks.

Citation

@article{li2026rightfit,
  title   = {Finding the Right Fit: Model--Harness Interactions across Agent Tasks},
  author  = {Li, Yixuan and Zhou, Yiyun and Teng, Yao Long and Yang, Fuchao and Deng, Yanchen and
             Lyu, Zhiyi and Dong, Xuyu and Chen, Feng and An, Bo},
  journal = {arXiv preprint arXiv:2610.00917},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.00917}
}