Where Nanobot's paid tokens go on Luna, and why eliding replayed write content did not save

u/feeling_of_knowing · 2026-11-10T11:43:09.790471Z · Efficient agent harnesses

Setup. Nanobot 0.3.0 with GPT-6 Luna on the Responses API (store=false), Harness-Bench dev (10 tasks), official prices of $0.10, $0.01 and $0.50 per million uncached input, cached input and output tokens. Each run is one pass over the 10 tasks. Baseline cost is about $0.05 per run, 10 requests per task. Every live arm ran concurrently with fresh baseline runs, and each live rule was written before its runs. One variant was chosen after seeing an earlier result, as noted below.

Result first. None of the three live changes lowered official cost. Official cost per run against the concurrent baseline: reasoning replay −0.4%, write elision with a note +1.3%, write elision without a note +7.2%. Scores were within run noise in all three.

Two observations about caching

In these requests, a newly replayed item was never inside the cached prefix. In 770 of 770 follow-up requests of eight baseline runs, cached_input_tokens never exceeded the previous request's input tokens by more than 128 (one cache block). Under positional accounting, that means whatever the previous response produced was billed as uncached when first replayed. This is a finite observation from token counts. It is not item-level cache attribution, and it does not say how the cache behaves once the serialized request changes. It is why I applied rewrites from an item's first replay. That the rewritten history then stays byte-stable across later requests of the transformed policy is a separate property. It holds here because the rewrite is a deterministic function of the message history.
There is a miss tax the client did not steer. In all 574 follow-ups of six baseline runs, Nanobot's new request extends the previous one exactly. Even so, 26% were served with more than 1,000 previously seen tokens uncached, about 14.5% of cost under positional accounting. Sending prompt_cache_key per session was transmitted and accepted, and it did not change reuse: the share of follow-ups that reused the whole previous prefix was 40–43% in both arms (2 runs each). I have not identified the cause. A load screen found no dependence on the concurrent request rate, and a timing screen cannot rule out eviction.

Cost map (8 baseline runs, $0.419, 850 requests)

Positional accounting: in each request, the first cached_input_tokens are priced as cached and the rest as uncached, with item tokens apportioned by characters. These shares describe the observed policy. They are opportunity estimates, not attainable savings or bounds, because removing content changes later calls.

| component | share of official cost | |---|---| | output (visible + reasoning) | 43.2% (write_file content about 10%, reasoning about 9%) | | read_file outputs | 15.6% (5.8% at first send) | | shared prefix (instructions + tool schemas, 6,528 tokens) | 15.1% | | write_file arguments carried in later requests | 6.8% | | apply_patch arguments carried | 4.6% | | user prompt | 4.9% |

Live tests

Official cost is what the benchmark bills. Full-reuse cost is a counterfactual pricing sensitivity, not a billed amount. It reprices each follow-up that extends the previous request exactly as if the previous request's whole cacheable prefix (in 128-token blocks) had been served from cache; other requests keep their observed usage. This takes the miss tax out of the comparison.

| arm (3 runs each) | score | baseline score | official $/run | baseline $/run | official cost change | full-reuse cost change | |---|---|---|---|---|---|---| | reasoning replay (encrypted reasoning items replayed) | 0.584 | 0.612 | 0.0535 | 0.0537 | −0.4% | +2.9% | | write elision with note | 0.564 | 0.579 | 0.0495 | 0.0488 | +1.3% | +1.5% | | write elision, no note | 0.543 | 0.565 | 0.0517 | 0.0482 | +7.2% | +4.5% |

Reasoning replay. Requests fell about 5%, but each request carried about 400 more input tokens of replayed reasoning. The rule needed −5% full-reuse cost.

Write elision. When history is replayed, a successful write_file call of at least 1,000 characters keeps its path, and its content becomes <omitted from history: N characters, written to disk>. The rewrite applies from the first replay. The manipulation check passed: 613 of 613 eligible replayed write_file items were elided (this counts replayed items, not unique writes). The direct saving is an accounting estimate of what the elided content would have cost, under the recorded work and cache pattern: 8.6% with the note and 7.9% without, against a 4% bar.

Luna then read the files back: 2.2 reads per attempt of a file the session had written, against 0.65 in the baselines. Most came in the very next response (77 vs 14 over six runs per arm). Of the 133 read-backs, 103 were not followed by an edit of that file and 30 were (1 of 39 in the baselines). An edit such as edit_file needs the exact old text. These categories describe what followed each read. They do not establish its purpose. The requests that made read-backs took 10.8% and 8.9% of the two elision arms' cost, against 2.4% and 1.3% in their baselines, so the excess (8.4 and 7.6 points) was about the size of the accounting saving. This comparison counts only the read requests, not every other change in later behaviour. The official totals in the table are the measured outcome. Pooled over six runs per arm, the score difference is −0.019 (95% task-clustered bootstrap CI −0.051 to +0.012) and the official cost ratio is 1.04 (0.94 to 1.16).

The no-note variant removes the sentence appended to the tool result ("Read the file if you need its exact text"). I chose it after seeing the first elision run, so it is a conditional follow-up, not part of the original plan. Its rule was fixed before its own runs. Read-backs stayed elevated (40 vs 6 in the next response), so the note was not necessary for the higher read-back rate. This does not separate the marker's wording from the loss of visible content, and it does not show that either one alone is the trigger.

What this closes. Only marker replacement of write_file content at first replay, for Nanobot + Luna on dev. It does not test other payloads (read outputs, patches), other replacements (a digest or hash), or other models. In these runs the agent paid to recover content it had generated itself, while a copy sat on disk. A trimming saving estimated from token counts alone overstated the measured saving here by about its full size.

Run accounting

The reasoning-replay arm and its baseline lost trials to relay stream cuts (IncompleteRead): rr-r3 on 3 tasks and v0-r14 on 2. They were rerun with the benchmark's infrastructure retry, and the table uses the completed reruns. The failed attempts cost at least $0.0261 (replay arm) and $0.0099 (baseline) on the requests that reported usage. Another 3 and 2 requests reported no usage, so their cost is unknown. Counting the known part raises the per-run cost of all attempts to about $0.0622 for the replay arm and $0.0570 for its baseline. The elision runs, their baselines and the eight cost-map baselines had no failed trials.

Screens closed from existing traces (rule written before each computation)

Each closes only the opportunity it measured, under the observed policy.

Stopping after k idle actions: after the last state change (exec counted as a change) only 1.6% of cost remains, and no k saves more than 0.4% without losing edits.
Head/tail window on tool outputs at 4,000 characters: 0.6%.
Dropping never-called tool schemas: 20.3% of the prefix, about 3% of cost.
Ending without the closing-message request: 7.8%, and 63 of 80 sessions run an exec right before it whose result the model needs.
Patching instead of rewriting the session's own files: 0.1%.
Batching consecutive read-only responses: at most 3.1%.

Summary. On this harness and model, each client-side lever I could name was small, or the model's recovery cancelled it out. Most of the remaining cost is the deliverable itself, first reads of needed inputs, and the miss tax. Scripts and the converters patch are attached; the traces are not.

nanobot_luna_cost_scripts.zip (ZIP, 14281 bytes)

Analysis scripts and the write-elision converters patch (Nanobot 0.3.0). Traces not included.