a harness audit of gui-agent screenshot history. eccv, paper id #10. dm me for the pdf.
short version: i set out to compare four rules for deciding which past screenshots a gui agent should keep, and found they were all the same. then i found out why the comparison could not have told me anything either way. where the prompt was rendered mattered about twenty times more than any of the rules i was measuring.
a gui agent takes screenshots as it works, and that history gets expensive fast. the literature proposes rules for what to throw away: keep what looks like the instruction, re-weight by space and time, keep the most recent, or keep at random. i ran all four at a matched 20% budget with zero-shot ui-tars-1.5-7b on 23 gui-odyssey episodes, plus a full-history arm and a zero-history arm as bookends.
a tie is only interesting if the thing you removed was doing work. mine reached 24.0% step accuracy on a benchmark where a third-party reproduction reported 32.9%, and a gap that size means the harness, not the retention rule, is setting the ceiling. so i stopped comparing rules and started auditing my own setup, one variable at a time, with the template and parser held fixed.
the variable that moved was prompt placement. rendering the same prompt in the system role instead of in the user turn lifted official-evaluator accuracy from 17.4% to 39.1% on 46 identical steps. a control that changed only the ordering, keeping the user role, reached 23.9%, so ordering alone recovers about a third of the gain and the role does the rest.
with the placement fixed, the configuration reaches 45.5% at scale over 378 steps. that is the version worth asking the original question of, so i re-ran full history against zero history on it. removing every historical screenshot, verified token-exact, shifted accuracy by 1.3 points down, with a 95% confidence interval running from 12.4 points down to 9.7 points up and mcnemar p = 0.72. keeping those frames cost 2.6x peak memory and 3.0x latency.
this is a negative result and it is bounded like one. the study is single-seed, greedy decoding, one model, one benchmark. with 23 episode clusters the minimum detectable effect is 16.4 points, so a real 6.1-point history effect reported elsewhere would not show up here even if it exists. i did not detect that effect. that is different from ruling it out, and about 170 episodes would be needed to say anything stronger.
a tie between methods is not evidence that the methods are equivalent. it can just as easily mean the harness is broken in a way that swamps them, and the way to tell the difference is to show that the input you are compressing changes the answer at all. a nuisance variable nobody reports, where the prompt goes, was worth an order of magnitude more than every effect under study.