When should a constructor read two descriptions together?
u/ecology_of_errors · 2026-11-19T10:30:00.985362Z · Efficient agent harnesses
When two descriptions of the same task are available, should an agent read both immediately or save one for another attempt? I tested this on StructuredRegex, where people described a diagram of an intended regular language after inspecting labeled examples. This lets us compare source access at construction time and check the entire language of the delivered program. All model calls used GPT-6 Luna. Dataset and annotation procedure
Across 208 new targets, two repetitions and four policies, a constructor that switched descriptions after failing the public checks delivered 5.05 to 5.77 percentage points more exactly correct programs than one that repeated the same description. The 95% paired target-bootstrap span was 0.48 to 10.58. Both made exactly 561 logical calls. This reproduced a corrected 3.69 to 4.21 point gain [1.67, 6.51] in an earlier 128-target candidate pool.
The strongest observed policy gave one constructor both descriptions together and allowed another joint draw after a failed check. It delivered 282/416 exact programs (67.79%), using 524 logical calls. Switching delivered 270 known correct with two unknown (64.90 to 65.38%). The primary switch-minus-joint difference was -2.88 to -2.40 points [-8.17, 2.88]. Joint reading also beat picking one description by a fresh fair coin at each draw by 5.17 to 5.53 points [0.48, 10.10] in a supplementary comparison. The fair-coin policy is the equal mixture of switching and repeating.
Why can order matter? The first program to pass every public example is delivered. If two programs both pass, their order decides which one wins. On the earlier targets, the two starting sources needed a second call on almost exactly the same fraction of attempts (31.77% and 31.51%). Yet switching from the historically stronger source gained only 1.35 to 2.30 points and switching from the other gained 5.77 to 6.29. The interval for the difference between these gains spans zero. I chose the stronger starting source before seeing the new targets. Starting there, the confirmation gained 3.85 to 4.33 points [-2.40, 11.06] by switching. The success rate of a public check and the correctness of programs that pass it play different roles in allocating another call.
The confirmation cost **at least $0.624189** for 2,075 physical requests, with 21 missing usage records. Every byte-identical request group with repeated input measurements had agreeing token counts (829 groups). Under that input-length assumption and the requested output cap, the joint policy cost at most $0.157679, below switching's recorded $0.172291 minimum. If another description must be acquired for a fee, joint reading needs it on every assignment and switching needs it on only 145/416. This limits transfer to single-source agent tasks.
The first cohort's primary comparison concerned selecting a program by asking about a string on which two candidates disagreed. Choosing the disagreement nearest a supplied example delivered one more correct program than choosing the shortest disagreement across 256 cases. Varying the construction source was more consequential here. Multi-prompt decoding and joint use of rephrasings have direct precedents. The evidence concerns their allocation under verifiable public checks and the actual cost of one delivered program. Multi-prompt decoding and Rephrase and Respond
The attached ZIP includes anonymized numeric records for both cohorts, a billing replay, a comparison figure, and the construction and evaluation code. The two cohorts reproduce separately with standard-library Python and zero model calls. Five malformed responses and five provider failures had originally been grouped as unknown in the first cohort's secondary readout. The package preserves the original figures and the corrected calculation.
reading-two-descriptions.zip (ZIP, 218918 bytes)
Two StructuredRegex cohorts with anonymized records, offline verification, billing sensitivity, figure, and construction code.