Accessibility Trees vs Screenshots: How AI Agents Should See an App

Vercel Labs' agent-desktop brings the accessibility-tree approach from browsers to desktop apps. What an accessibility tree is, how the snapshot-and-ref loop works, why it beats pixel guessing, and where it goes blind.

Rahul Bisht

Founder, CrawlPilot

·
Nov 12, 2026
·AI & Agents·
10 min read
·
Accessibility Trees vs Screenshots: How AI Agents Should See an App
Key takeaway
The screen is the worst thing to show a model that needs to use an app: it has to guess what each region is and where to click. Operating systems already publish a labelled tree of every button and field for screen readers, and tools like Vercel Labs' agent-desktop hand that tree to the agent with a reference for each element. The agent acts on references, not coordinates, and only falls back to pixels when the app hasn't labelled itself.

The obvious way to let a model operate an app is to show it what a person sees: take a screenshot, ask the model where the Save button is, click there. It is also the hardest version of the problem. The model has to recognise a button from its pixels, guess its exact bounds, and hope the layout hasn't moved by the time the click lands. Meanwhile the operating system already knows there is a button, that it's called Save, and that it's enabled, because screen readers need that information to work at all.

On October 31, 2026 Vercel Labs released agent-desktop, an open-source CLI that gives agents computer use on macOS, Windows and Linux by reading that accessibility information instead of screenshots. Its author described it as agent-browser for your whole computer, and the comparison is exact: it brings the approach we covered in our Vercel Agent Browser guide from web pages to native apps.

Two Ways to See the Same Window

Screenshot
  • Works on anything that draws to the screen, labelled or not
  • Needs a vision-capable model on every step
  • A moved dialog or a theme change breaks the click
  • Can't tell a disabled button from an enabled one unless it looks different
  • The model gets an image and must find the button by how it looks
  • Acts on coordinates such as x=812, y=440
Accessibility tree
  • No vision model needed to find the target
  • Survives layout shifts: the reference points at the element, not a spot
  • Exposes state: enabled, selected, value
  • Blind where the app doesn't label its controls
  • The model gets a list such as: button 'Save' (enabled), text field 'Filename'
  • Acts on a reference to the element, not a position

Verdict: Read the tree first and act on references; keep screenshots for the parts of an app that don't describe themselves.

The same "Save the report" task, seen as pixels and as an accessibility tree

An accessibility tree is the interface as the operating system describes it to assistive technology: a hierarchy of elements, each with a role (button, text field, menu item), a name, often a value, and states such as enabled or selected. Each OS has its own API for it: the Accessibility (AX) API on macOS, UI Automation on Windows and AT-SPI on Linux. agent-desktop is written in Rust and ships as a CLI plus a C library with an adapter for each (Marktechpost). Neither it nor agent-browser is a model; they are the agent's eyes and hands, and the agent you already run decides what to do.

The Loop: Snapshot, Pick a Reference, Act, Snapshot Again

Everything in this style runs one loop. The agent takes a snapshot, which returns the tree with a short reference for each element. It picks a reference and acts on it. Then it takes a new snapshot, because the action changed the interface and the old references may no longer point at anything. Step through a save in a text editor:

The snapshot-and-reference loop
snapshot -i1@e3 = button 'Save'23re-snapshot4Agentyour modelSnapshottree with refsAct on refclick @e3The appUI changes
  1. The agent asks for a snapshot of the app, usually filtered to interactive elements so the list stays short.
  2. The snapshot lists each element with a reference. The agent reads that @e3 is the enabled Save button and decides to press it.
  3. The tool performs the press on that element through the accessibility API. No coordinates are involved.
  4. The press opened a dialog, so the old references are stale. The agent snapshots again before its next action.

4 steps. Press Play to walk through them, or click any part of the diagram.

Snapshot returns the tree with references, the agent acts on one, and a fresh snapshot replaces stale references

On the command line it is short. The project's own macOS example, as quoted by Flavio Copes, uses Finder:

bash
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # look again after the change

The -i flag keeps only interactive elements, and each reference carries the ID of the snapshot it came from, so an agent can't accidentally reuse a reference from an older view. agent-browser does the same for web pages, with shorter references like @e2:

bash
agent-browser open example.com agent-browser snapshot # accessibility tree with refs agent-browser click @e2 # click by ref

Two habits make the loop reliable. Snapshot again after anything that changes the screen, and treat a stale-reference error as "look again", not as a failure. Agents that cache an old snapshot to save a call are the ones that click the wrong row after a list re-sorts.

Accessibility Action First, Mouse Last

Having a reference doesn't mean the tool moves the mouse to it. The better tools in this space try the element's own accessibility action first (press, set value, select) and only synthesize a physical click or keystroke when the element doesn't support one. Cua's driver, for example, returns the tree and a screenshot together, tries accessibility actions first and falls back to pixel clicks. The accessibility action doesn't care whether the window is partly covered or on another monitor, and it doesn't steal the user's cursor.

Before you run any of this on your own machine, it needs permission to read and drive other apps. On macOS that means granting Accessibility permission to the terminal that runs it (ai-tldr's setup notes), which is exactly as powerful as it sounds.

Where the Tree Goes Blind

The approach is only as good as the labels apps publish, and plenty of apps publish little.

The agent can find and operate this element by reference. Every condition below holds.

  1. The app exposes a treehandled

    When it's missing: Games and canvas-drawn interfaces often expose no tree at all, so there is nothing to snapshot. Fallback: screenshot plus coordinate clicks.

  2. The element has a namehandled

    When it's missing: An icon-only button shows up as 'button' with no label, and the agent can't tell it from its neighbours. Fallback: screenshot of that region, or its position in the tree.

  3. The element supports an actionhandled

    When it's missing: The control ignores the accessibility press, so the tool gets an 'action not supported' error. Fallback: synthesized mouse or keyboard input.

  4. The snapshot is freshhandled

    When it's missing: The interface changed after the snapshot and the reference now points at nothing, or at the wrong row. Fallback: snapshot again before acting.

What has to be true for an agent to operate an element through the accessibility tree. Turn one off to see what goes wrong.

Three more limits are worth knowing before you build on it:

  • No web pages. agent-desktop can't see apps that lack an accessibility tree, and it doesn't automate web pages; Vercel points people to agent-browser for those.
  • Linux differs by display server. On Linux, mouse input needs X11 or the headed mode, and clipboard access on Wayland needs wl-clipboard (same source).
  • Snapshots are private data. A snapshot of your desktop can include email subjects, chat messages and documents, and clipboard reads can include passwords. explainx lists this as the main risk. Anything that sends snapshots to a hosted model is sending that content too.

Why This Matters Beyond Desktop Apps

For anyone building web agents, this is the same lesson the browser tools learned first. A page's accessibility tree is a cleaner, smaller description of what a person can do than its raw HTML or a screenshot, and references are sturdier than selectors or coordinates. It also cuts the other way: apps and sites that label their controls properly become far easier for agents to use, which is one more reason accessibility work pays off. For scraping, where you read rather than operate, the trade-offs are different, and we compare them in AI agents vs traditional web scrapers.

Check yourself

Q1An agent clicks @e3 from a snapshot taken before a dialog opened, and gets a stale-reference error. What should it do?

Q2When does an agent need to fall back to screenshots?

Cheat sheet
Accessibility trees for agents

The idea

  • The OS describes every control for screen readers: role, name, value, state.
  • macOS uses AX, Windows uses UI Automation, Linux uses AT-SPI.
  • agent-desktop does this for native apps; agent-browser does it for web pages.

The loop

  • Snapshot with -i to list interactive elements and their refs.
  • Act on a ref, never a coordinate, when one exists.
  • Snapshot again after any change; a stale ref means look again.

Where it fails

  • Apps with no tree: games, canvas UIs.
  • Unlabelled controls such as icon-only buttons.
  • Web pages inside a desktop app; use a browser tool.

Before you run it

  • It needs OS accessibility permission to read and drive other apps.
  • Snapshots and clipboard reads can hold private data.
  • Prefer accessibility actions; synthesize input only as a fallback.

Sources