Date: 2026-07-08. Engine: a pre-v0.2.0 source checkout declaring openadapt-flow 0.1.0. One task, two ways to automate it, one success check.

Task (MockMed, the bundled demo clinic app; fake data only): sign in as nurse.demo, open the first referral task, create a New Encounter of type Triage, enter a note, save.

Measured on Flow 0.1.0, 2026-07-08. The measurement used a pre-v0.2.0 development source checkout; its exact runtime HEAD was not retained. The rows were first committed in b2eec0be after parent 45f5ba8a; those two SHAs describe artifact history, not the runtime used for the measurement. Not re-measured on a later release.

MockMed re-rendered with a dark palette, which invalidates every recorded template crop:

• compiled (healing on): succeeded in 9.7s, 8 heals
• agent (as-is): succeeded in 87.4s, 23 actions, $0.6319 — close to the 25-action budget. In an earlier smoke run under the same drift the agent exhausted its budget and failed, so treat the drift rows as single observations either way (see caveats).

• Record + compile once. The demo is recorded through the Playwright demo driver and compiled into a vision-anchored bundle (openadapt-flow demo-record + compile). Recording and compiling are a one-time cost and are not included in per-run latency.
• Identical environments. Each run of either arm gets a fresh chromium browser + page against the same locally served MockMed app (app state lives entirely in the page, so a fresh page is a fresh instance).
• Same interface. Both arms drive the same PlaywrightBackend, vision-only: PNG screenshots in, pixel-coordinate clicks / typed text / key presses out. Neither arm uses DOM selectors at run time.
• Agent arm. Model claude-sonnet-5 with the computer_20251124 computer-use tool (beta header computer-use-2025-11-24), a 25-action budget, and history bounded to the last 3 screenshots. The task prompt states user intent (the numbered task above), not steps or coordinates. Every executed action returns a settled screenshot, using the same settle logic the replayer uses.
• Same success criterion. After each run, a screenshot of the final state is checked by OCR (openadapt_flow.vision.find_text): the Encounter saved — <note> banner AND the Triage — <note> encounter row must both be visible. Neither arm's self-reported success is used.
• Latency is wall-clock around the replay / agent loop only (browser and server startup excluded for both arms).
• Cost is computed from API usage token counts at list pricing ($3.00 / $15.00 per MTok input/output for claude-sonnet-5). An introductory $2/$10 rate applies through 2026-08-31, so billed cost today is about a third lower than reported. Compiled replay makes zero model calls.
• Identity-protection coverage: not captured in this results.json. The armed-coverage metric was added to the generator on 2026-07-10; future runs report how many click steps carry the pre-click identity check and list the unarmed steps (which proceed with NO identity verification — see docs/LIMITS.md).

• MockMed is a simple app. Five screens, no scrolling, no popups, high contrast, big labels. It is close to a best case for both arms; harder apps would slow and likely degrade both, plausibly at different rates.
• The agent arm has a smaller N (20 vs 100) because agent runs cost real money and minutes. Its success rate carries wider error bars.
• Model version pinned. Results describe claude-sonnet-5 with the computer_20251124 tool on 2026-07-08; newer models will differ.
• The compiled arm needs a demonstration first. The one-time record + compile step (about a minute of human demonstration) is the price of the fast replays; the agent needs only the prompt.
• Drift is n=1 per arm — an existence result, not a rate.
• Latency includes deliberate settle waits (screenshot stability polling) in both arms; a tuned production loop could shave both.
• Single machine (macOS-15.7.3-arm64-arm-64bit), local server, no network variance in the compiled arm; agent latency includes real API round trips.

openadapt-flow benchmark --n-compiled 100 --n-agent 20 --out benchmark/

Requires ANTHROPIC_API_KEY (or ~/.anthropic/api_key). The agent arm costs real money (about $5.43 at list price for 20 runs when this was generated).

An honest assessment of what the suite exercised BEFORE the multi-system benchmarks landed: mostly single-application, linear, worklist-driven form fills of roughly 5-15 recorded steps, with the fault-model studies driving a single consequential write. Real back-office work is longer, cross-system, document- and email-driven, and exception-heavy. The ap_invoice and o2c_recon benchmarks were added to close that gap; the table below states the shape of each benchmark so the difference is not overstated either (the new benchmarks actuate through the api tier and do not measure GUI perception; see each README).

Both new benchmarks run every consequential write through the real Replayer's api actuation tier after the real Standard-profile run gate admits the sealed bundle and binds a single-use authorization to its exact inputs. Every write is verified through a separate persisted-state read (read-only SQL, REST oracles, a maildir read, or a CSV re-read) and judged by a direct-file adjudicator whose expectations come from immutable source fixtures rather than the prepared worklist. This is not an independent service/failure domain. Zero model calls; healthy-path governed runs classify VERIFIED under the Section-3 transaction taxonomy. Measured headline over both benchmarks (n=3 per cell; 60 base runs): governed silent-incorrect-success 0/30 governed runs and healthy-path over-halts 0/6; naive banner-oracle silent-incorrect-success 6/30 (the collateral overwrite and the phantom write-back classes). Deterministic coverage matrix, not a sampled incidence rate. See benchmark/ap_invoice/README.md and benchmark/o2c_recon/README.md for what each does and does not prove.