Repository navigation
Add iOS Simulator control and live panel (macOS) - #136
Merged
2 commits merged intoOct 16, 2026
Merged
2 commits merged into
2 commits merged into
Conversation
Let the agent run, test and visually check iOS apps in the Xcode iOS Simulator, and let users watch and drive the simulator inside ACECode. - acecode-ios-sim: an Objective-C++ helper that loads the selected Xcode's private CoreSimulator / SimulatorKit at runtime (no link-time symbols). It describes devices, captures the framebuffer turned to the interface orientation, streams rate-limited JPEG frames, injects HID input (Indigo for Xcode <= 26, the dtuhidd digitizer for CoreSimulator >= 1155.4) and reads the accessibility tree. Framed JSON+binary protocol over stdio. - Xcode discovery without xcode-select: configured path, DEVELOPER_DIR, xcode-select, /Applications, ~/Applications and Spotlight, so copies of Xcode outside /Applications work; simctl and xcodebuild run from the chosen Xcode's own Developer directory. - Tools ios_simulator (attach, screenshot, tap, swipe, touch paths, text, key, button, open_url, launch, inspect, logs, boot, ...) and ios_simulator_build. The control tool is auto-approved but asks once per simulator before first use; the build tool keeps the permission prompt. - REST routes, a WebSocket stream, a live simulator preview tab with mouse/keyboard/paste input that opens on attach, and Settings > Tools > iOS 模拟器 for Xcode selection, device consent and stream quality. - The helper is built and packaged only for macOS (app bundle, portable zip, pkg, update zip, npm darwin packages); package checks updated. Protocol shapes follow Meta's idb (MIT, license included with the helper). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # web/scripts/i18n-en-overrides.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
为什么
iOS 同事需要让 Agent 在 Xcode iOS 模拟器里运行、测试、目视检查 App，并在 ACECode 里实时观看和操作模拟器。参照 Claude Desktop 的同类功能逆向分析后自研复刻；与它不同的是 不依赖 xcode-select：Xcode 常被绿色安装在 /Applications 之外，必须自动发现且可配置。
改了什么
acecode-ios-simhelper（Objective-C++，仅 macOS）：运行期加载所选 Xcode 的私有 CoreSimulator / SimulatorKit（无链接期符号）。设备描述、按界面方向转正的截图（1 像素 = 1 point）、限帧去重的 JPEG 实时流、HID 输入（Xcode ≤ 26 用 Indigo，CoreSimulator ≥ 1155.4 / Xcode 27 用 dtuhidd）、横竖屏换算、无障碍树。stdio 上的定长帧 JSON+二进制协议，随父进程退出。ios_simulator（list/boot/shutdown/attach/detach/screenshot/tap/swipe/touch_path/touch2_path/text/key/button/open_url/launch/terminate/list_apps/inspect/logs/hardware_keyboard）与ios_simulator_build。控制工具像 Agent Browser 一样免逐次确认，但每台模拟器首次使用前询问用户一次（记在 state.json，可在设置撤销，超时/取消一律算拒绝）；构建工具会执行工程脚本，保留普通权限确认。plan 模式只允许观察类动作。WS /ws/ios-simulator/<udid>实时流；预览区「iOS 模拟器」页签（鼠标/滚轮/键盘/粘贴交互，attach 时自动打开，标题栏有手动入口）；设置 > 工具 > iOS 模拟器（启用、Xcode 选择与自定义路径、设备授权、启动/关闭、流画质）。if(APPLE)目标；ACECode.app、便携包、PKG、更新包、npm darwin 包与 CI 校验都要求有它；Windows / Linux 不构建不打包。相关打包脚本测试与 verify-package 已同步。docs/ios-simulator.md、docs/daemon-api.md第 20 节、CLAUDE.md；openspecadd-ios-simulator。协议形状参照 Meta idb（MIT，许可证随 helper 源码附带）。验证
本机 Intel Mac，Xcode 26.6 绿色安装于
~/Downloads（xcode-select 指向 CLT），iOS 26.5 两台已启动模拟器：IosSimulator*/XcodeLocator*/SimctlParse*单测 30 项通过；真机IosSimulatorE2E.*（授权、截图附件、Home、inspect、滑动、组合键、越界拒绝、推流）通过。pnpm test、pnpm build通过。RemoteWebTcpProxy崩溃均在远程 Web 代理、WebSocket 客户端、跨进程锁、沙箱路径、TUI 渲染、通道日志与 macOS 浏览器工具注册等与本改动无关的用例。未验证：Xcode 27 上的 dtuhidd 输入通道；签名公证包（交给 CI）；真实模型 attach 自动开面板（前端单测覆盖）。后续项：helper 的 Seatbelt 沙盒、Desktop 原生零拷贝模拟器视图（Intel 上动画期间帧缓冲回读会被 GPU 同步拖慢，Apple 的
simctl io screenshot也一样）。🤖 Generated with Claude Code