agent-desktop

Overview

agent-desktop is a CLI and C library from Vercel Labs that reads the accessibility tree of a desktop app and returns it as JSON. Each element an agent can act on gets a ref; the agent acts on a ref, reads the UI again to verify, and repeats. It targets macOS, Windows, and Linux and does not include an LLM or GUI.

Key Features

• Reads a desktop app's accessibility tree and returns snapshots as JSON, with a ref for every actionable element.
• Uses snapshot-scoped refs and re-reads the UI after actions to verify results rather than matching pixels.
• Ships as a CLI with 60 command names covering observation, interaction, keyboard, mouse, app and window, notification, clipboard, wait, session, and system commands.
• Ships as a C library called libagent_desktop_ffi for in-process calls with documented entry points, memory ownership, and threading rules.
• Supports macOS through the AX API, Windows through UI Automation, and Linux through AT-SPI.
• Can map dense apps with a shallow skeleton snapshot and drill into one region before acting.

Use Cases

• Agents that need to inspect and operate desktop applications without relying on pixel matching.
• Automating actions such as clicking, typing, selecting, scrolling, and waiting in native apps.
• In-process integrations that prefer calling a C library instead of launching the CLI for each command.
• Mapping large or dense desktop interfaces by taking a skeleton snapshot and drilling into one region.

Getting Started

• Install globally with npm: npm install -g agent-desktop; then run agent-desktop version to check the install.
• On macOS, run agent-desktop permissions --guide to grant Accessibility permission.
• Take a snapshot with agent-desktop snapshot --app Finder -i, act with agent-desktop click @s8f3k2p9:e8, then snapshot again to verify.

Deployment & Requirements

• macOS 13 or later on Intel or Apple Silicon, with Accessibility permission for the app that runs agent-desktop.
• Windows 10 1809 or later or Windows Server 2019 or later, on x64 or ARM64, in the signed-in user's interactive desktop.
• Linux: Ubuntu 24.04 or later on x86_64 or aarch64 (glibc 2.35 floor), in the user's graphical session with a running AT-SPI bus.
• Linux needs at-spi2-core.
• On macOS, TMPDIR must be owned by the current user with mode 0700; the standard macOS per-user temporary directory satisfies this requirement.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.