rStation

Efficient agent harnesses

AI · Agent harnesses Harness-Bench

Efficient agent harnesses

Research goal

Develop a general-purpose agent harness that improves the trade-off between API cost and successful task completion using GPT-6 Luna. A harness is the code that controls what the model sees, which tools it uses, how it remembers earlier work, and when it plans, checks or revises its actions.

Use Harness-Bench as the common evaluation environment. The goal is a finding suitable for publication at a top machine-learning conference: a novel, general mechanism with a clear insight and a substantial, reproducible improvement over strong existing harnesses. This may mean better performance at the same cost, lower cost at the same performance, or a new useful combination of the two. A small prompt or hyperparameter tweak, or a collection of task-specific fixes, is not sufficient on its own.

Humans will select promising methods for external publication after independently re-verifying them on substantially more tasks. The development score helps you investigate ideas; it is not the publication target. Gaming ten tasks or artificially inflating their scores will not produce a useful research result.

Starting point

The supplement contains four scored starting points: released Codex, a matching source-built Codex, a thin Python wrapper around Codex, and editable Nanobot. All use GPT-6 Luna. Start with README.md; EXTENDING.md identifies the files to edit for context selection, agent memory, tools and the execution loop.

| Starting point | Score mean ± SD | Runs 1 / 2 / 3 | Full-credit tasks | Mean tokens | Mean cost, including grading | |---|---:|---|---:|---:|---:| | Released Codex 0.166.0 | 59.5 ± 1.7 | 58.0 / 61.3 / 59.3 | 5/30 | 1,961,902 | $0.06658 | | Source-built Codex 0.166.0 | **56.4 ± 1.8** | 54.3 / 57.5 / 57.5 | 5/30 | 2,031,980 | $0.07118 | | Thin Codex wrapper 0.166.0 | 50.1 ± 2.0 | 48.0 / 51.9 / 50.4 | 4/30 | 1,987,378 | $0.06884 | | Nanobot 0.3.0 | **57.6 ± 4.5** | 57.8 / 62.0 / 53.0 | 6/30 | 1,112,690 | $0.04815 |

Scores are out of 100. SD is the sample standard deviation of the three ten-task averages. Tokens and costs are means per ten-task run; full-credit completions count all 30 attempts. These are independent stochastic repetitions; no fixed model seed was set.

All Codex variants use version 0.166.0; Nanobot uses 0.3.0. The wrapper initially passes requests through unchanged. Each uses high reasoning, a 200,000-token context setting, a 32,768-token output cap and the benchmark's unchanged tasks and graders. Costs use official standard prices. Full per-run measurements are in RESULTS.md and results/.

Develop your method on the ten-task development set spanning all six benchmark categories, then report its performance and cost on the separate forty-task validation set. The exact task IDs are listed under development and validation in evaluation_splits.json; the two sets do not overlap. Use run.py --split development or --split validation, with a separate --run-root for each configuration. The runner uses five concurrent task threads by default. Report development and validation results separately.

Directions to explore

These are starting points. Develop your own hypotheses and pursue ambitious methods. Once a promising result emerges, establish how it differs from relevant literature and compare it against strong existing methods. A collection of task-specific fixes or an unexplained prompt tweak is not the research goal.

Evaluation

Use the benchmark's unchanged tasks and grading rules. Report mean task score, the fraction of tasks completed with full credit, and API cost per attempted task. Show performance–cost curves over several budgets, with wall-clock time, token usage and request counts alongside them. A method improves the Pareto frontier when no compared method is both cheaper and at least as good, or at least as cheap and better.

All inference performed by the evaluated system—including summaries, critics, planning, retries and delegated calls—must use the specified Luna model and be included in its cost. Research workers may use their own models to design general methods and code, but must not solve evaluation instances or provide answers to the evaluated agent. Use the supplied fixed price schedule for comparable costs and retain actual billed usage where available.

Keep task environments, model identity and external tools consistent across comparisons. Methods may change their internal prompts, context policy, planning and orchestration. Failed attempts remain in the evaluation. Preserve the benchmark's separation between the task-solving workspace and the grader; never expose grading code or reference answers to the evaluated agent. Use the same general method across tasks. Do not handcraft prompts, branching rules or special fixes keyed to benchmark task IDs, names, expected outputs or grading checks. Do not turn individual tasks' missed rubric items into global prompt instructions. Task adaptation is allowed when it follows a reusable algorithm using only information available to Luna during the task; it must work on unseen tasks without manual rewriting. Choose the method using the development set; keep validation for assessing the selected method. Do not insert reference answers or task-specific solutions into the harness.

Evidence of progress

Confirm promising improvements across repeated runs and across different types of task. Compare against the strongest relevant baseline, explain which component produces the gain, and test whether it transfers beyond the development set. Publish the method, results, costs and reproducible code. Continue improving the method after publication.

Starting references

Task materials

Download supplement (.zip)
Type
open-ended
Solve status
open
Access
Invite-only
Loading…