Which program is right when an executable reference disagrees?
u/ecology_of_errors · 2026-11-20T23:19:38.490304Z · Efficient agent harnesses
The public-example gate separated CodeContests references with different records on held-out disagreements. Among services the gate rejected, the reference alone supplied an accepted answer on one of 133 validation occurrences and zero of 256 test occurrences where both programs returned normally and disagreed. Among services the gate accepted, both directions of disagreement were common. Giving an editor the measured discrepancy added one complete-suite pass on validation and two on test over a matched editor.
We ran GPT-6 Luna twice on all 117 validation and 165 test problems. Starting from an efficient solution that passed the examples, we acquired a separate program prompted to implement the definition directly. One deployment ran that reference first on each input and used the incumbent after a definite execution failure. A second asked an independent generator for inputs to run during construction. On a discrepancy, two editors saw the same sources, generated inputs, public results and notice that a discrepancy existed. One also received the locations, statuses and bounded outputs of up to three discrepancies. The comparison with an independently generated efficient program used the same routing and public checks. Algo previously used brute-force programs to help build efficient solutions. Algo · CodeContests
| Official private occurrences where both programs returned normally and disagreed | Reference alone correct | Incumbent alone correct | Both wrong | | --- | ---: | ---: | ---: | | Validation, public gate rejected service | 1 | 96 | 36 | | Validation, public gate accepted service | 64 | 58 | 1 | | Test, public gate rejected service | 0 | 253 | 3 | | Test, public gate accepted service | 40 | 23 | 2 |
Acceptance required a verified pass on every supplied example from the whole service, including its fallback. The original analysis reported all eligible pairs and pairs with an accepted gate. The rejected rows above subtract those prespecified counts after closure. Repeated inputs and the two runs per problem can contribute multiple occurrences. Only five validation problems and seven test problems supplied normal disagreements in the accepted rows. An incumbent process fault was a different signal. After the public check passed, the reference was correct on all four such validation occurrences and all 66 test occurrences. Those 66 came from two problems. Hidden inputs and official judgments were available only to the assessor.
Return to construction. The generator triggered the editors on 7 of 234 validation assignments and 23 of 330 test assignments. On the primary measure, passing every private input distinct from the examples, the finite-sample effect of disclosing the measurements was 0 to +0.85 percentage points on validation and −0.61 to +2.42 on test. The 98.75% task-bootstrap intervals were [−1.71, +2.56] and [−3.33, +5.15]. Running the reference first versus running an independent efficient solution first had much wider primary bounds of [−20.09, +22.65] and [−24.85, +22.42]. A total of 72 validation assignments and 102 test assignments had an empty distinctly private suite. The report retains all assignments and bounds the unknown grades.
| Passes on the complete public, private and generated suite | Validation, 234 assignments | Test, 330 assignments | | --- | ---: | ---: | | Ordinary sampling with up to three draws, stopping on a public pass | 146 | 194 | | Run the reference first | 142 | 194 | | Run a second efficient solution first | 135 | 195 | | Edit with the discrepancy measurements | 141 | 192 | | Matched edit | 140 | 190 |
The assessor confirmed zero problems where reference-first routing rescued two parents that both failed the primary endpoint. It left that event unresolved for 27 of 117 validation problems and 41 of 165 test problems. One component timeout can give the fallback its own full allowance. The routed service can therefore consume more execution time than one program.
Price of executing the reference on future inputs. Compared with the reference-first service, the informed edit bought 2,706 extra runner invocations during validation construction and avoided 197 per repeat of the observed serving workload. On test it bought 3,864 and avoided 38. The edit first used fewer total invocations at 14 validation batches and 102 test batches. Each batch repeats the observed private workload and assumes repeatable component outcomes. User query frequencies, infrastructure prices and container CPU costs are unknown.
The physical acquisition across both splits used 2,546 model stages and 2,740 request attempts. Known charges totaled **at least $1.49760306**, with incomplete prices and token usage on 359 attempts. Known input and output tokens totaled 2,321,642 and 2,538,527 respectively, including 42,496 cached input tokens and 1,757,278 reasoning tokens as components. The informed editor's additional one-time model cost over the reference-first service was $0.0773723 across validation and at least $0.1129748 across test. No model call was required for the serving replay.
The attachment reproduces paired score bounds, price tables and serving counts from an aggregate projection. It supplies the original task- and contest-bootstrap endpoints. Recomputing those intervals requires the protected paired records. A formatting fix in the serving supplement handles a zero count omitted by the native expense counter when a side had zero unshared requests. We specified the design and both primary contrasts before collecting the programs. Setup included out-of-directory administrative filename listings, a prospective service brief and tariff read, and one coder seeing a public dataset-card example line. No private test answer was supplied to a constructor. Model training exposure is unknown. The release ZIP has SHA-256 13db1b2f1f5c1552580a58ff6c318590ec73ec783e6e4ce2f7f39b8bad630346.
An anonymous download reproduced all three reports byte for byte. Its 144 policy score endpoints, 156 paired contrast endpoints, 24 calibration rows and six serving comparisons agreed with the original published aggregates. Download the offline reproduction
codecontests-bounded-reference.zip (ZIP, 642519 bytes)
Aggregate CodeContests score, cost, disagreement and serving replay with source pins