On a wide view, the search button sits to the right of the input field. On a narrow screen, it sits below it. An agent that simply repeats the first click position can therefore miss, even though the actual task has not changed. The coordinate was not wrong. Its frame of reference is out of date.
The original UI-TARS repository describes the visual operation of graphical user interfaces and dates UI-TARS-2 to 4 September 2025. The release is more than a year old and serves here as a technical foundation. We infer from it neither a current leading position nor a benchmark of our own. The interesting question is: what observation does an action need for its result to be judged at all?
A screenshot describes a situation
An image shows an interface at a particular moment. Window width, scroll position and open sections belong to that situation. The position of a button cannot be fully separated from these conditions. A position only acquires its meaning through the corresponding image and the current task.
Our own example uses a fictional library page. The person wants to find a book. The screen shows search, sign-in and a notice about events. There may be three similarly styled buttons. The largest or most colourful one is not automatically the right one. The task decides which target is relevant.
This shows a first limit of visual operation: recognising and deciding are linked. An agent must not only locate an element but also match it to its intended action. A cleanly recognised sign-in button does not help with the book search.
Image size and target position belong together
If an image is scaled for processing, coordinates have to match the frame of reference actually used. The same pair of numbers can denote different places on images of different sizes. Even an unchanged screen size does not help if an expanded notice pushes the content down.
A clear observation note therefore links the most important details. The following schema is our own; it contains no real UI-TARS output and is not an executable agent format:
Observation: search page, narrow view
Goal: trigger a search for the entered book title
Action: press the visible search button
Expectation: results or an understandable result state
Afterwards: assess the new viewThe last entry is decisive. It prevents an input that has merely been executed technically from already counting as a completed task. At first, a click only produces a transition whose effect still has to be examined.
Four possible responses to the same action
After the search is triggered, results may appear, a loading indicator may remain visible, an input warning may show up or a technical error may be reported. These states call for different next steps. Clicking again would not make sense in every case.
Results also have to fit the search task. Perhaps a similar title was suggested, or the list still belongs to the previous input. The new observation therefore once again needs the original instruction as its reference. The cycle is not merely image, click, image, but understand, act and measure the effect against the goal.
For a harmless exercise of your own, you can prepare the four states as cards that you draw yourself. Assign each card a reasoned next action. The exercise explains thinking in states without claiming a real model run or a working connection to a library system.
Good interfaces help people and agents
Clear feedback reduces uncertainty for people too. “No results” answers a different question from “Search could not be loaded”. A decorative loading effect with no recognisable end, by contrast, offers little orientation. Design can support the observation cycle by clearly distinguishing the relevant states.
Running an agent locally does not, on its own, answer every question about data flows. Likewise, a successful sequence of clicks does not yet prove reliable operation in general. Both would have to be investigated on the actual system.
The transferable idea is simpler: an old position is not a permanent meaning. As soon as an action can change the interface, the next decision needs a current reference. Only a fresh observation connects the step that was carried out with the result that was actually meant to be achieved.
Sources and status
Research and documentation as of 25 October 2026. UI-TARS repository: UI-TARS-2 update and coordinate processing.
