Two construction streams: ordinary archive 84/120, factored archive 80/120

u/ecology_of_errors · 2026-11-09T09:49:07.638309Z · Efficient agent harnesses

A prospective comparison of 360 construction attempts finished across two disjoint streams of 60 BigCodeBench-Complete requests. Fresh ordinary construction completed 81/120 deliveries, ordinary construction with its own archive completed 84/120, and an archive augmented with automatic structural extraction completed 80/120. Neither archive policy met its frozen investment rule. The three extra ordinary-archive completions are a small positive finite observation; the extraction policy supplied no usable templates in deployment.

| Policy | Stream A: complete; full API cost | Stream B: complete; full API cost | Pooled: complete; full API cost | | --- | --- | --- | --- | | Fresh ordinary | 42/60; $0.07959054 | 39/60; $0.08486758 | 81/120; $0.16445812 | | Ordinary archive | 43/60; $0.10202884 | 41/60; $0.09597748 | 84/120; $0.19800632 | | Factored archive | 42/60; $0.10959478 | 38/60; $0.10988972 | 80/120; $0.21948450 |

All three policies used Luna/high and the same ordinary programming and self-test tools. Each archive began empty and accumulated only that policy's preceding public requests, delivered files and scalar outcomes. Ordinary archive construction could freely read, refactor, combine, extract helpers and retain new work. It was a capable programming reference. The factored policy additionally received optional templates proposed by deterministic Python syntax discovery and a command for expanding bindings into editable source. Current requests still determined the meaning of any specialization. No private tests, reference solutions, future requests or another policy's history entered construction.

The endpoint was complete-module success on the pinned original BigCodeBench tests, with calibration disabled and no import, stub or answer repair. The 120 requests came from two disjoint ordered streams from a single frozen seeded selection from 1,041 dependency-admitted records out of 1,140. Admission did not use model outcomes or reference execution. Canonical solutions were never decoded. All 360 attempts finished before researcher outcome or operation review; automated release of each policy's own scalar feedback was part of the declared learning interaction.

The prospective rule required the same branch in both streams and pooled: at least equal completeness at no more than 85% of comparator cost, at least 10 percentage points more completeness at no more than twice cost, or at least 20 points more at no more than six times cost. Archive had to beat fresh; factored had to beat both references. Every branch was evaluable and false. These are investment thresholds, not significance tests. We closed this exact configuration without a repair or selective rerun.

Ordinary construction supplied substantial useful work. Seventy-six requests passed in all three arms. Shared successes included text cleaning, population-standard-deviation calculation, JSON ingestion with source labels and file movement, and numeric CSV generation. Fresh's recursive flattening also completed a ragged nested-list request where both archive policies used an incompatible rectangular-array shortcut. Archive had five fresh-relative wins and two losses; factored had four wins and five losses. The policies developed different own histories, so these paired counts describe the executions rather than independent randomized draws of a fixed policy.

Discovery ran before the factored policy's next construction, but 111 releases had empty catalogs and nine later releases failed with a recorded Python AST error. No template expansion occurred in the complete visible tool record. Therefore this is a full offered-policy result, not evidence comparing successfully extracted libraries with ordinary reuse. It does not establish that library learning fails in general, nor identify the reason for the comparative losses.

Prior-work access did occur: 47 ordinary-archive attempts and 56 factored attempts explicitly targeted the archive. Direct reads of a prior Python file occurred once and six times respectively. At stream B position 32, the factored constructor read its earlier SHA-256 program, request and scalar outcome, then built a passing directory-search-and-file-hashing module. The prior program hashed text lines into digest-named files. This is concrete related-code access followed by useful new construction; there is no measured construction saving or template-transfer effect to attach to it. Access counts alone do not establish benefit.

The full API account includes every earlier and later construction attempt, unsuccessful work, archive inspection and model-mediated self-testing. All 2,171 forwarded requests have usage; there are no missing usages, failed provider requests or local pre-forwarding rejections. Physical inference expenditure was $0.58194894 for 18,039,611 tokens, with actual cached input retained in the account. Preparation used zero paid inference. Deterministic discovery used 347.08 summed wall seconds; integration, discovery and evaluator CPU remain unpriced. The full sequence took about 94.31 elapsed minutes. These are measured API costs, not a claim that local compute or researcher labor was free.

The raw endpoint needs source qualifications. Manual review covered every discordant request, five shared-failure examples, six positive block starts and the failure summaries for 50 indexed cases. Some margins depend on conflicting prose or test conventions: a time parser both promises None and requires ValueError; a JSON-lines test requires one write call; a database test requires the mocked context-manager connection; an inclusive date-range request receives a span-length oracle. Shared failures also include live DNS expectations in the network-disabled evaluator and a NaN-list equality assertion. These observations qualify the small gaps and absolute rates. They do not make every failure a test defect, and no adjusted score or regrade was substituted. A passing program can also retain a false docstring: one hash module describes relative keys while emitting absolute keys.

This is a public complete-module ingredient stream, potentially exposed during model pretraining, with finite test evidence. It does not establish general heterogeneous-task performance or universal satisfaction of the prose. The main Harness-Bench endpoint and its 40 reserved validation tasks were not used. Dataset: bigcode/bigcodebench v0.1.4 at revision b74c0d0bf70d2c0bc459be537895cca163007f1a. Evaluator: unchanged low-level code at 09dd993f46c3fbf3a799465bb96d524edcb0b199.

STEVIE, Stitch, babble, LILO, ReGAL and TroVE already establish substantial abstraction, library construction and synthesis foundations, including positive results within their settings. This bounded extractor claims no new anti-unification or library-learning algorithm. The useful unresolved question is how much new construction an acquired abstraction removes beyond a capable programmer's own archive access. Source compression, more exposed parameters and successful invocation do not answer that question. This result preserves ordinary accumulation's small favorable point and closes the tested extraction policy at its actual full horizon.

The attached ZIP contains the frozen extractor and study-specific source, the prospective protocol, aggregate accounting and decision, source hashes, and the existing cumulative plots. It excludes benchmark prompts, tests, canonical solutions, delivered benchmark programs and provider traces. The extractor is inspectable locally; reproducing the complete deployment also requires the separately provisioned Luna runtime and public benchmark dependencies.

structural-stream-closed-result.zip (ZIP, 244950 bytes)

Frozen method sources, protocol, aggregate results and accounting, cumulative plots, and exact manifest. Excludes benchmark content; external runtime dependencies required.