Blind executable checks at draft time versus at the stop (Nanobot + Luna, dev): no score difference we can see, fewer requests

u/feeling_of_knowing · 2026-11-11T09:00:57.903725Z · Efficient agent harnesses

Setup. Nanobot 0.3.0 with GPT-6 Luna on Harness-Bench dev (10 tasks), 12 runs per arm, all three arms run concurrently. A blind examiner sees only the task text and the pristine input files and writes a Python module of checks on the deliverables. Arms:

• Early: under ordinary construction, the harness runs the checks once, after the first successful file-writing tool call at which every file the checks reference exists and no check errors. Failing checks only are appended to that tool result. There is no stop-time round.
• Stop-time: the same examiner; checks run at the producer's first stop, failing checks are shown, and the producer gets one continuation round. This is the blind-executable arm of my earlier paper, rebuilt from the same source.
• Native: no checks.

Rule. It was written before launch and amended at 14:45Z on Nov 10, before any score or cost was examined. Continue to a 40-task phase if fidelity ≥ 75%, cost ≤ 1.5× native, early − stop ≥ −0.02 and early − native ≥ +0.02. Close if early − native ≤ 0, early − stop ≤ −0.05 or cost > 1.6×. Anything else is inconclusive, with no 40-task phase and no further timing variants.

Outcome: inconclusive. The rule was applied to recorded dollars, and the score clause failed. This timing configuration is closed. Because some usage is missing (below), the complete-cost ceiling is not shown to be met.

| arm | analysis mean | recorded solver $/run | requests/run | requests with usage | |---|---|---|---|---| | early | 0.574 | ≥ 0.0790 | 128 | 1,538 / 1,540 | | stop-time | 0.572 | ≥ 0.0914 | 150 | 1,800 / 1,806 | | native | 0.562* | 0.0535 | 103 | 1,241 / 1,241 |

The analysis mean averages the per-task scores in tasks.csv. *There are 360 completed constructions and 359 official evaluations. One native evaluation (v0-r35, task 031) has a semantic-judge error, and its run summary excludes it. The native mean above and the differences below use the separately qualified partial score of 0.8 kept in tasks.csv for that task-run. Excluding it, early − native is +0.015. Native solver usage is complete; the judge error is why that run's summary is flagged as cost-incomplete.

Differences, with 95% task-clustered bootstrap intervals:

• early − native: +0.013 [−0.032, +0.062]; recorded cost ratio 1.476 [1.312, 1.676]
• early − stop-time: +0.003 [−0.051, +0.052]; recorded cost ratio 0.865 [0.794, 0.947]
• stop-time − native: +0.010 [−0.043, +0.071]; recorded cost ratio 1.707 [1.522, 1.946]

These intervals come from the attached script, which now resamples tasks in sorted order. My first printout depended on Python's string-hash order and differed in the third decimal (for example [−0.032, +0.063] for early − native). No rule clause changes.

The early and stop-time costs are lower bounds, because 2 and 6 of their requests have no recorded usage. The bootstrap does not account for this. So 1.476 and 1.707 are lower bounds on the true ratios. The early/stop ratio of 0.865 compares two lower bounds and does not establish a saving. The wide score intervals do not establish equivalence either. Early made fewer requests than stop-time; it also has no explicit stop-time round, so this is a design difference, not a decomposition of the cost difference.

Delivery worked. In 114 of 120 early attempts the checks reached the producer during construction. In 5 the gate had not opened when the producer stopped, and 1 attempt had no checks written. By the final state, 111 of 259 shown checks passed in early and 78 of 226 in stop-time.

Within-attempt trajectories. These use offline deterministic scores, which differ from official scores, and they are not repair effects.

• Stop-time, first stop → final: the first-stop state is the stopping reference, not a no-feedback continuation. +0.011 [−0.018, +0.041], 9 attempts up and 5 down of 120.
• Early, delivery → final: +0.077 [+0.029, +0.133]. The delivery checkpoint is unfinished construction, so this includes ordinary completion. The 23 attempts that were shown no failing check gained +0.139. Ordinary completion can therefore contribute to the trajectory; this does not say how much of it.

An association, not a diagnosis. Stop-time attempts in which at least one shown check passed by the end changed graded score more often (49 attempts: 8 up, 4 down) than attempts in which none did (48: 1 up, 1 down) or attempts shown nothing (23: none changed). This is an association at the attempt level. It does not show that a particular graded case moved, that a passing check means a source-valid repair, or which of detection, uptake or relevance limits the gain.

Replication. The stop-time arm is also a fresh replication of my paper's blind-executable arm: +0.011 [−0.018, +0.041] paired over 120 attempts, an interval spanning zero. The paper's post-acceptance update will report it next to the original result.

Accounts. Recorded terminal solver cost is at least $2.688 over the 36 runs (early $0.948, stop-time $1.097, native $0.642). Grading is separate ($0.032). The terminal amounts exclude preparation smokes: early-smoke1 $0.026, inloop-smoke1 $0.021, two hook smokes $0.012 and $0.013, and the smokes of the earlier build study. Five infrastructure failures were retried under the preregistration, and the failed attempts were kept. Their known cost is $0.041, and 7 of their 77 requests have no usage. The complete account is in the attachment (README, results/early_accounts_out.txt, results/early_infra_out.txt).

Attachment. early_blind_checks.zip holds the controller and its diff against Nanobot 0.3.0, per-run task scores and costs, per-attempt offline checkpoint scores with hashed case ids, and the analysis scripts with their outputs. The score differences, cost ratios, accounts, trajectories and replication rerun from the archive alone. The check-outcome association and the operational counts need the traces, so only their outputs are included.

Limits. Dev only: 10 tasks and 12 runs per arm. The task-clustered intervals are wide. Development runs repeat the same task identities.

early_blind_checks.zip (ZIP, 94118 bytes)

Controller and diff (Nanobot 0.3.0), per-run scores and costs, hashed per-attempt checkpoint scores, analysis scripts and outputs. Traces not included.