gui-agent

LGWM (Latent GUI World Model): an action-conditioned world model that verifies GUI agent transitions in representation space. Official code for "Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents".

A FiftyOne remote model for ScreenParser.

ScreenAgent

Windows PC automation for AI agents: a resident PC Actor (UIA structure + SendInput hands + pixel fallback) plus a zero-model vision toolkit with measured results.

Fast computer use for AI agents: event-driven settling MCP server - replaces time.sleep between clicks, waits for the UI to react and stop changing, returns the settled screenshot. No blind waits, no wasted vision turns.

Provider-neutral enhanced Qwen2.5-VL agent and evaluation runner for OSWorld

Local-first, self-learning work agent for Apple Silicon Macs: watches allowlisted apps, learns your routines, asks before every click. Runs a local vision model on loopback.

Let AI agents use your real Android phone through its accessibility service: screen tree with stable refs, taps, typing, gestures, notifications, apps. MCP server + hub + app.

【半成品 Beta】纯视觉 AI 手机社交助手 · 电脑看屏→大模型理解→adb 操控→回读验证 · 核心链路已真机跑通，其余见 README

MCP server for Microsoft OmniParser: turn screenshots into structured UI elements for GUI agents (such as Claude Desktop / Cursor / Cline)

Modular block-based GUI automation framework combining VLM vision, LLM planning, Playwright browser control, and pyautogui desktop automation. Includes food review crawler with multi-agent collaboration and MCP server.

Unofficial PyTorch reproduction for Demo2Tutorial, multimodal software tutorial generation from human experience.

Free, fully offline computer-use agent (Set-of-Marks grounding + crop-zoom super-resolution) that drives a real desktop with small local VLMs via Ollama - a local alternative to Claude Computer Use, OpenAI Operator/ChatGPT Atlas, and Project Mariner.

Self-hosted team of AI agents you message like colleagues: async delegation, a shared browser and desktop, long-term memory, a skill pool, and bots in Feishu/Slack/Telegram. TypeScript, Node 22, Apache-2.0.

Adversarial image attacks on vision-language web agents, from visual grounding to browser execution

Give your AI its own computer. Watch it work through the glass. A VS Code extension: a chat sidebar, a live view of the bot's own sandboxed Linux desktop, and an agent that works with any LLM.

Vision-based web agent that operates live websites through their own UI. 2nd place in the WebRetriever Challenge 2026. Paper: arXiv:2610.03036

SCoR: Sparse Context Reset for Long-Horizon GUI Agents (EMNLP 2026)

Computer use for Grok Build: a plugin and MCP server that let the agent see the screen, click and type in desktop apps and browsers on macOS, Windows and Linux. Accessibility-tree first, with safety hooks. Built for UI testing in the dev loop.

Android Agent Reliability Runtime A debugging and safety runtime for mobile GUI agents: detect readiness, block unsafe actions, verify progress, diagnose failures, and save reproducible traces.