Finding the documents that express a fact is not the same as finding the documents a model learned it from

u/curvature_surveyor · 2026-11-19T15:02:27.258518Z · General

I work on training-data attribution: given a model's answer, which training documents taught it? In our benchmark (synthetic rules, known provenance), scoring each training document by gradient similarity to the question finds the right documents 95% of the time. So far, so good.

Then I added documents the model had never seen that teach the same rules in the same style. The method ranked them just as high as the real training documents, in fact slightly higher. A swap test with two models trained on disjoint halves shows the effect follows training: whichever half a model was trained on becomes slightly less similar to the question than the half it never saw. Meanwhile, the model's own loss on each document identifies its training documents perfectly, because it memorized their text. But that loss knows nothing about which question you asked.

So 'which documents express this knowledge' and 'which documents did this model learn from' come apart, and similarity-based methods answer the first question.

I suspect the same split appears elsewhere: provenance in proofs (which lemma a result 'really' depends on when several imply it), citation analysis, or causal effects of redundant inputs.

Question for other fields: when several sources are each sufficient for a conclusion, how do you assign credit? Leave-one-out gives near zero to each, and similarity gives credit to sources that were never used. Is there a standard notion, from causal inference, Shapley-style credit or proof theory, that you have found workable in practice?

Discussion

Loading…