OpenRSI Index opens long-horizon benchmark for model-development agents

OpenRSI Index Preview v0.1 packages research-agent evaluation around an execution loop rather than a fixed question set. RSI Harness starts a persistent Work container where the agent edits the task workspace, while the task's private tests remain outside that environment. The same workspace survives across attempts, allowing the agent to change code, run experiments and respond to earlier evaluation feedback without receiving the hidden test suite.

On each submission, the harness pauses Work and snapshots its filesystem. It then starts a fresh Judge from that snapshot, injects the private tests, runs the evaluator, records the output and reward, and removes the Judge before the agent continues. The separation is intended to keep the agent's iterative research state persistent while making each scored attempt start from a clean judging environment.

The public catalogue spans pre-training, post-training, vision, systems, agents and robotics. Smaller examples include speeding up a sparse-attention prefill operator, choosing interaction context for a browser-action model and redistributing decoder width under a fixed parameter budget. The tasks constrain what may change and define their own evaluation path instead of reducing every research problem to the same coding benchmark.

At the other end, the preview labels three model-development workloads as signature tasks. The site budgets 18,432 H100-hours per run for its Marin optimizer-scaling ladder, 20,864 H100-hours for a Qwen post-training task and 5,815 H100-hours for its GPIC image-generation task. OpenRSI also publishes agent results for those workloads, but the site calls them initial preview samples and says they may still be adjusted.

The OpenRSI-Index repository is published under Apache-2.0 and contains RSI Harness alongside task definitions and supporting material. Current main was at commit 15802c426faf12798765a3e2acfcbb8f2b9d2f87 when this story was verified on 25 September. GitHub exposes no Release objects for the repository, so the site's Preview v0.1 label should not be treated as an immutable software release or tag.

RSI Harness documents a local Docker backend plus LSF and Slurm cluster adapters. Task setup records resources and network policy before execution, while Work and Judge resource requirements are kept separate. That makes the execution mechanics inspectable, but anyone comparing results still needs to pin the exact repository state and the task-specific environment rather than assuming the preview is fixed.

The harness can be exercised on a Linux machine with Docker, and its documentation includes CPU-capable local task paths. Full reproduction of the signature workloads is a different proposition: the published compute budgets run into thousands or tens of thousands of H100-hours, and some tasks depend on large-model training infrastructure or external agent access.

That creates two distinct reproducibility questions. Researchers can inspect and test the Work-to-Judge lifecycle, snapshotting, private-test boundary and scheduling behaviour without recreating every flagship experiment. Reproducing the reported improvements on the largest tasks requires substantially more compute and a pinned version of task definitions that the project still describes as preview material.