Review: coder_eval — pr:207 (21 files) axis:1,2,3,4,5,6,7,8

Scope: pr:207 (21 files) axis:1,2,3,4,5,6,7,8 · branch chore/move-delegate-sdk-2 (PR #207 → main) · 0ea0d99 · 2026-10-02T05:53Z · workflow variant

Change class: complex — replaces the in-tree Node host with the public @uipath/delegate-stdio protocol, rewriting the agent's spawn/init/event/turn-finalization control flow and its config surface

The architecture is clean (10/10) and no critical defects exist, but the Delegate agent change has three high-impact problems: old Delegate task.json files no longer load, so --resume silently re-runs finished tasks; the documented token auth path fails on every published host version; and verbose mode writes the bearer token in clear text to task.log. Fix these three before you merge (9.1/10 overall).

Summary

Overall Score: 9.1 / 10 · Weakest Axis: Code Quality & Style at 8.4 / 10
Totals: 🔴 0 · 🟠 4 · 🟡 5 · 🔵 5 across 8 axes.

Blockers

1. [Axis 1] Pinned @uipath/delegate-stdio floor (^1.202.1) ignores the auth init option while env-token vars are removed, so the token auth path fails or silently uses the ambient login (src/coder_eval/agents/delegate/package.json:7) — package.json line 7 pins "@uipath/delegate-stdio": "^1.202.1". On 2026-10-01, npm view @uipath/delegate-stdio dist-tags returns latest: '1.202.1', and the only newer releases are 1.203.0-preview.*, which a caret range does not select. The PR's own docs/agents/DELEGATE.md:53 says: "version 1.202.1 and older ignore it, and then init fails with Auth required unless a saved login exists." The agent also strips the variables that 1.202.1 does read: _HOST_ENV_REMOVED (delegate_agent.py:199-207, which contains "AUTH_TOKEN", "TENANT_ID", "ORG_ID", ...). So the auth path this PR documents (DELEGATE_AUTH_TOKEN etc. sent through _auth_option(), line 592 options["auth"] = auth) cannot work on the version that npm install resolves. This affects the CI delegate-live-tests job, which runs a plain npm install in agents/delegate/, and every user who follows the error message at line 296. The code handles this with an error hint (_INIT_CONFIG_ERROR_HINT, line 132-133: "A @uipath/delegate-stdio without the auth init option ignores it") instead of a correct version floor. Fix (corrected by verification: a live test showed that 1.202.1 AND the newest 1.203.0-preview.20260929135425 both ignore options.auth, so pinning the preview does not help): keep AUTH_TOKEN/TENANT_ID/ORG_ID (and the slug names) in the host env, or set them from the DELEGATE_* values, until a host release that reads auth exists and becomes the floor. After the floor is correct, remove the hint sentence about older hosts (the project has no backward-compatibility burden).
2. [Axis 2] Typed Delegate effort field replaced by untyped sdk_options: dict[str, Any], so effort values lose validation (src/coder_eval/models/agent_config.py:419) — On main, DelegateAgentConfig had effort: str | None = Field(default=None, ...), and Pydantic v2 rejected a non-string value. The PR replaces it with line 419 sdk_options: dict[str, Any] = Field( plus _validate_sdk_options_keys at lines 454-463, which runs only unknown = sorted(set(v) - _DELEGATE_SDK_OPTION_FIELDS) and never checks the value. I validated these at PR HEAD (0ea0d99): DelegateAgentConfig.model_validate({'type':'delegate','sdk_options':{'effort':5}}) is ACCEPTED, and so are {'effort':['high']}, {'effort':{'x':1}} and {'effort':None}. delegate_agent.py:579 options.update(self.config.sdk_options) then puts the value into the init frame as-is. The SDK ignores a value it does not recognize (see the field description), so a typo such as effort: 5 or effort: [high] silently has no effect on the run. A YAML-native non-JSON value, such as an unquoted date, raises TypeError in json.dumps inside _send_command instead of a validation error. The allowed key set has exactly one entry (_DELEGATE_SDK_OPTION_FIELDS = frozenset({"effort"})), so a free-form Any dict is not needed here. Fix: type the pass-through with a nested model, e.g. class DelegateSdkOptions(BaseModel): model_config = ConfigDict(extra="forbid"); effort: str | None = None, and declare sdk_options: DelegateSdkOptions = Field(default_factory=DelegateSdkOptions). Then forward self.config.sdk_options.model_dump(exclude_none=True) at delegate_agent.py:579. This keeps the -D agent.sdk_options.effort= path and the "sdk_options" in model_fields probe in overrides.py working. It restores value typing, and extra="forbid" replaces the hand-written key validator. Keep str (not a Literal) to match the documented forward-compatibility intent. Also add a test that asserts sdk_options={'effort': 5} is rejected.
3. [Axis 4] Bearer token travels in the stdin init frame. With DELEGATE_STDIO_VERBOSE=1 the host echoes it to stderr twice, and coder_eval logs that stderr unredacted to task.log and error_log_tail. (src/coder_eval/agents/delegate_agent.py:1275) — Source: the PR moves the token out of the host env and into the init command. _build_init_options sets options["auth"] = auth (line 593), with auth["accessToken"] taken from DELEGATE_AUTH_TOKEN/AUTH_TOKEN. _spawn_and_init writes it to stdin with await self._send_command({"cmd": "init", "options": self._init_options}) (line 556). The shipped host, @uipath/delegate-stdio 1.202.1 dist/delegate_stdio.mjs (I de-obfuscated it locally), gates its trace on DELEGATE_STDIO_VERBOSE==='1'||'true'. When the trace is on, it runs log("[dispatch] Received line ("+len+" chars): "+truncate(line)) to stderr for every stdin line. The truncation cap is 50,000 chars by default, so the full init JSON is written, accessToken included. This PR documents that flag as a supported knob at docs/agents/DELEGATE.md:55: "DELEGATE_STDIO_VERBOSE=1 (trace every frame to stderr)". Sink: _drain_stderr sends every stderr line to logger.debug("delegate[stderr]: %s", text) (line 1275). _log_stderr_tail re-logs the last 20 lines at WARNING (line 1131). orchestrator.task_log_handler lowers the coder_eval logger to DEBUG for the whole task and attaches a FileHandler. As a result, the token lands in clear text in <run_dir>/.../task.log, and on error in task.json's error_log_tail (orchestrator.py:663). Those run artifacts are uploaded to shared blob storage and the evalboard. Before this PR, the token was only in the env, and the host trace printed AUTH_TOKEN=SET (n chars), so this leak path is new. Fix: keep the token value on the agent and redact it from each stderr line before _stderr_lines.append(...) and the logger.debug call, for example text.replace(token, "***"). Alternatively, remove DELEGATE_STDIO_VERBOSE from the host env when auth is sent, or warn that verbose mode logs the credential. Add a test that feeds a stderr line containing the token and asserts the token is absent from the log records. CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:N
4. [Axis 8] Removing DelegateAgentConfig.effort makes every Delegate task.json from v0.12.5 to v0.12.9 unloadable: --resume silently re-runs finished tasks, and run.json recovery drops the rows (src/coder_eval/models/agent_config.py:419) — The PR replaces the top-level field effort: str | None = Field(default=None, ...) with sdk_options: dict[str, Any] = Field( (line 419). BaseAgentConfig declares model_config = ConfigDict(validate_assignment=True, populate_by_name=True, extra="forbid") (line 120). The orchestrator writes task.json with self.result.model_dump_json(indent=2, exclude=TASK_JSON_TRANSCRIPT_EXCLUDE), with no exclude_none. So EVERY Delegate run from the released versions (Delegate first shipped in v0.12.5 and is in v0.12.5, v0.12.6 and v0.12.7) persists "effort": null in agent_config, even when the user never set it. Reproduced: I dumped DelegateAgentConfig(type='delegate') on main and validated it against the PR head's ResolvedAgentConfig. Result: effort Extra inputs are not permitted [type=extra_forbidden, input_value=None]. So EvaluationResult.model_validate_json fails on those records, with these effects. (1) orchestration/batch.py:490 _load_completed_result: except ValueError: ... treat as not-yet-complete so the task re-runs. --resume on an old Delegate run re-executes every finished task with no log, which spends cost again and can change its score and final_status. (2) recover_task_results skips each row with only a warning, so tasks_run and the pass-rate denominator shrink. (3) evaluate <run_dir> (regrade.py:66) and report (reports/helpers.py:238, report_command.py:168) fail or skip. This is the Axis 8 scoring-correctness class: a persisted task.json that mis-loads under a schema change. Cross-repo: a Python consumer in coder-eval-uipath or eval-runner that loads Delegate task.json through EvaluationResult breaks for the historical Delegate population. evalboard (TS, raw JSON) is not affected. Fix: on DelegateAgentConfig, add a @model_validator(mode="before") that pops a legacy effort key and moves a non-null value into sdk_options["effort"]. Add a regression test that loads a v0.12.7 Delegate task.json fixture through EvaluationResult.model_validate_json and through _load_completed_result. If the greenfield policy deliberately accepts the break, mark the change as BREAKING in the CHANGELOG and the PR body, and name the resume, recover and regrade consequences. Harness improvement (this cannot be a static check, because it needs a released-version fixture): add a golden task.json fixture for each built-in AgentKind, taken from the last release, and assert that it still loads. n/a

Non-blocking, but please consider before merge

1. [Axis 1] communicate cyclomatic complexity rises to D(28) (src/coder_eval/agents/delegate_agent.py:723-852) — Radon grades communicate at D(28); on main it was D(27). delegate_agent.py is not one of the hot modules, so this is Medium. One while True loop (lines 771-820) does five jobs: deadline checks, reads from the queue, frame dispatch (if mtype == "result" / "error" / "usage" / "event"), the max-turns cut and the cooperative-stop cut. The status is then derived from an if/elif chain on local flags (lines 822-827). Fix: move frame dispatch into a _dispatch_frame(msg, state, ...) helper that returns a loop-control value. Store the end status on _TurnState so the stopped_early local and the chain at lines 822-827 can go.
2. [Axis 3] The untested respawn path re-raises a marker-class init AgentConfigError as a retryable AgentCrashError (delegate_agent.py:736-745). This contradicts the PR's new "a retry cannot fix" classification. (src/coder_eval/agents/delegate_agent.py:738-745) — communicate() respawns a dead host. On failure it runs except AgentConfigError as exc: ... raise AgentCrashError(str(exc)) from exc (lines 740-745). The routed coverage reports lines 740-745 as missed. This branch decides FinalStatus: retry versus end the task. The PR now sorts init errors with _INIT_CONFIG_ERROR_MARKERS (including "expired", "401", "invalid token") and calls them errors 'a retry cannot fix'. The same expired-token message at start() is terminal, but after a mid-run respawn it is retryable. Since this PR, every host error force-kills the host (_abandon_host_and_crash), so every retry goes through this respawn. A ~1h ROPC token that expires mid-run (the CI live job's credential model) therefore gets this behavior. No test pins either outcome. Add a test with this sequence: start OK, crash a turn, then patch_exec([_line({'type':'error','message':'Auth required: token expired'})]), then call communicate(). Assert the exception type that is intended. If terminal is intended, re-raise AgentConfigError for marker-class errors.
3. [Axis 3] The rewritten Delegate wire protocol has no golden-master/full-record parity test. Delegate's exemption from golden coverage cites UNVERIFIED field guesses that this PR deletes. (tests/test_delegate_agent.py:36-38) — The PR replaces the whole frame protocol: event/usage/result frames, toolArgs/toolResult/toolStatus, and the turnUsages call count. The tests build these frames by hand (def _ev(**event): """One ``event`` frame wrapping an SDK event, as ``delegate-stdio`` writes it.""") and assert hand-picked TurnRecord fields one test at a time. No test snapshots the full EventCollector-built TurnRecord over its model_fields, so a future unmirrored field drops silently. Delegate stays in _NO_GOLDEN_COVERAGE in tests/test_agent_golden_master.py:242 with the reason "field shapes UNVERIFIED against a live backend — see delegate_agent.py". This PR removes both UNVERIFIED markers from delegate_agent.py (main has 2, PR HEAD has 0), and the live-test docstring now names these frame builders as the reference shapes. The exemption therefore points at text that no longer exists, and TestGoldenCoverage stays disabled for Delegate on a stale reason. Record one or two delegate-stdio transcripts as golden scenarios. Run them through assert_reconciliation and assert_timing_captured, and add AgentKind.DELEGATE to SCENARIOS_BY_AGENT. If you keep the exemption, rewrite its reason so that it is true.
4. [Axis 6] Init-error classifier matches bare '401'/'403' substrings, so transient failures become terminal AgentConfigError (and no test covers it) (src/coder_eval/agents/delegate_agent.py:568) — The PR adds _INIT_CONFIG_ERROR_MARKERS (lines 113-124), and line 568 tests it as if any(marker in message.lower() for marker in _INIT_CONFIG_ERROR_MARKERS): raise AgentConfigError(...). The markers include the bare substrings "401", "403" (lines 120-121) and "expired" (line 119). Nothing anchors them, so they also match digits inside port numbers, GUIDs and request ids that the host puts in transient messages. Example: the SDK spawns interop on a random free port, and connect ECONNREFUSED 127.0.0.1:54013 contains "401". So does a correlation or tenant GUID such as c7a3f401-.... Any such message is raised as AgentConfigError and routed to non-retryable AGENT_CONFIG_ERROR by isinstance (errors/categorization.py:41). The task ends at start() without the retry that execute_with_retry (orchestrator.py:1548) would otherwise give it, and the message gets the misleading auth hint _INIT_CONFIG_ERROR_HINT. If a user's TENANT_ID/ORG_ID GUID contains "401"/"403" and the host echoes it, every transient init failure for that user becomes non-retryable. Fix: match the status codes with a word-boundary regex (re.search(r"\b40[13]\b", ...)) or with a structured prefix such as "HTTP 401" / "status 403", and narrow "expired" to "token expired" / "jwt expired". Add negative test cases (port 54013, a GUID containing 401) to test_only_an_init_error_a_retry_cannot_fix_is_non_retryable. A lint rule cannot catch this. A parametrized negative test is the guard.
5. [Axis 7] Delegate get_sdk_options() persists host init options into run.json sdk_options, and reuses env as a string where consumers expect a dict (src/coder_eval/agents/delegate_agent.py:686) — On main, Delegate did not override get_sdk_options(), so it returned None. Now it returns the camelCase host init dict (delegate_agent.py:686-698), and that dict goes into EvaluationResult.sdk_options and run.json (run_record.py:176). Both public contracts still describe that field as Claude-shaped: results.py:676-679 has description="Raw SDK options dump from ClaudeAgentOptions (all fields including defaults)", and REPORT_SCHEMA.md:142 says sdk_options (raw ClaudeAgentOptions dump). The keys clash in meaning. _build_init_options sets options["env"] = environment (line 591), a string such as "alpha". The orchestrator reads the same key as a ClaudeAgentOptions environment dict (sdk_env = sdk_options.get("env"), then sdk_env.get("PATH"), orchestrator.py:1614-1616). Today an isinstance(..., dict) check stops it from failing, but any downstream run.json reader that keys on sdk_options.env now gets a string for one agent and a dict for another. The record also stores workingDirectory, an absolute host sandbox path, so sdk_options is not the same across machines or runs. Fix: either persist a non-colliding projection (for example delegate_env, and drop workingDirectory), or update the field description and REPORT_SCHEMA.md in the same change to say that the shape of sdk_options depends on the agent, and list the Delegate keys. n/a

Nits

1. [Axis 1] The auth variable names are kept in sync by hand in three places (src/coder_eval/agents/delegate_agent.py:128-134) — The same variable names appear in _AUTH_OPTION_FIELDS (lines 184-190: ("accessToken", "AUTH_TOKEN"), ...), in _HOST_ENV_REMOVED (lines 199-207) and again in the prose of _INIT_CONFIG_ERROR_HINT (line 130: "DELEGATE_AUTH_TOKEN / DELEGATE_TENANT_ID / DELEGATE_ORG_ID / DELEGATE_ORG_SLUG / DELEGATE_TENANT_SLUG"; line 131-132: "AUTH_TOKEN / TENANT_ID / ORG_ID / ORG_LOGICAL_NAME / TENANT_NAME"). If a name is added or renamed in one tuple, the hint becomes wrong. Fix: build the hint from _AUTH_OPTION_FIELDS and _HOST_ENV_REMOVED, for example ' / '.join(f'DELEGATE_{n}' for _, n in _AUTH_OPTION_FIELDS).
2. [Axis 3] The sdk_options override-guard tests hard-code the registry contents. No test proves that the new registry-derived acceptance works for an agent kind other than claude-code/delegate. (tests/test_overrides_engine.py:147) — The PR replaces the hard-coded claude-code check with _kinds_accepting_sdk_options() (src/coder_eval/orchestration/overrides.py:92-101), which reads AgentRegistry after ensure_plugins_loaded(). Three tests (tests/test_overrides_engine.py:147, :157 and tests/test_merge_characterization.py:309) assert the literal match="only supported for claude-code, delegate agents". The sibling plugin coder_eval_uipath registers delegate-sdk/studio-web configs with sdk_options. tests/test_agent_golden_master.py says that this plugin may be installed in the dev env. With it installed, the message lists more kinds and these 3 tests fail for environment reasons. Also, no test proves the generic contract, which is what lets coder_eval_uipath's _overrides_patch.py be deleted. Match on the prefix ('only supported for') or build the expected string from _kinds_accepting_sdk_options(). Add one test that registers a throwaway config class with an sdk_options field (monkeypatched registry) and asserts that -D agent.sdk_options.x is accepted for it.
3. [Axis 4] Doc claims agent shell commands 'cannot read the token', but the token stays readable through the token file left in env and through the parent coder_eval process environment (docs/agents/DELEGATE.md:53) — Line 53 says: "It also removes the host's own variable names (...) and DELEGATE_AUTH_TOKEN from the host's environment. So the agent's shell commands cannot read the token". The code docstring says the same at delegate_agent.py:208-210: "the agent's shell tools inherit the host env, so no spelling of the token may stay in it". This scrub is defense in depth, not a boundary. (1) DELEGATE_AUTH_TOKEN_FILE/AUTH_TOKEN_FILE are deliberately left in the host env, so a shell tool can cat "$DELEGATE_AUTH_TOKEN_FILE". The saved-login path leaves ~/.aria/sdk-auth.json, which holds a refresh token, readable by the same uid. (2) The coder_eval Python parent still holds DELEGATE_AUTH_TOKEN in its own environment. On Linux, including the docker driver, a same-uid descendant can read it from /proc/<coder_eval pid>/environ, because Yama restricts only PTRACE_MODE_ATTACH. (3) DELEGATE_BACKEND_URL stays in the env, although delegate_agent.py:710 notes that the full URL "can carry embedded credentials". (4) The durable LLMGW_CLIENT_SECRET stays whenever no token file is set. Fix: reword the claim as defense in depth, as CLAUDE.md does for the reference anti-cheat, and list the known gaps. Optionally also remove DELEGATE_BACKEND_URL from the host env, because it is sent as the backendUrl init option. CVSS:3.1/AV:L/AC:L/PR:L/UI:R/S:U/C:L/I:N/A:N
4. [Axis 6] Missing transitive @uipath/delegate-sdk is no longer a loud pre-spawn AgentConfigError; it surfaces as a retryable 'host exited before responding' crash (src/coder_eval/agents/delegate_agent.py:1224) — Before this PR, _resolve_sdk_entry checked that @uipath/delegate-sdk's dist/index.mjs existed before the spawn and raised AgentConfigError if it did not. Now _resolve_host_bundle (lines 247-300) checks only delegate_stdio.mjs. Line 267 checks only path.name != _HOST_BUNDLE_NAME. But the bundle does a static import { DelegateAgent, ... } from "@uipath/delegate-sdk", with the SDK kept external. A broken install, or a DELEGATE_SDK_PATH that points at a copied delegate_stdio.mjs outside its node_modules tree, passes resolution. Node then dies with ERR_MODULE_NOT_FOUND before init_ok. _read_until takes the EOF branch: reason = "Delegate host exited before responding" / raise AgentCrashError(reason) (lines 1224-1226). This is retryable AGENT_CRASH, so start() is retried for a deterministic missing-prerequisite failure. The real cause appears only in the WARNING stderr-tail log. A related gap: if the host is already dead when the init frame is written, await self._send_command({"cmd": "init", ...}) (line 556) raises a raw ConnectionResetError/BrokenPipeError. That path never logs the stderr tail. In communicate()'s respawn (line 739) the error also escapes unwrapped, because only AgentConfigError is caught there. Fix: when init hits EOF, scan the captured stderr for ERR_MODULE_NOT_FOUND / Cannot find package and raise AgentConfigError, or resolve @uipath/delegate-sdk relative to the bundle in _resolve_host_bundle. Also wrap the init _send_command failure so it logs the stderr tail and raises AgentCrashError.
5. [Axis 7] DELEGATE_SDK_PATH / DELEGATE_SDK_NODE_MODULES keep their SDK names but now point at the delegate-stdio host (src/coder_eval/agents/delegate_agent.py:259) — Line 259 is explicit = os.environ.get("DELEGATE_SDK_PATH") and line 275 is root_override = os.environ.get("DELEGATE_SDK_NODE_MODULES"). Both variables now must locate @uipath/delegate-stdio/dist/delegate_stdio.mjs. The adapter needs an extra error branch (lines 267-272: "must point at {_HOST_PACKAGE}'s dist/{_HOST_BUNDLE_NAME}, not at @uipath/delegate-sdk's dist/index.mjs") only because the variable name still says SDK, and that name invites the old value. The PR already breaks the meaning of these variables, and the project has no backward-compatibility burden. Rename them to DELEGATE_HOST_PATH / DELEGATE_HOST_NODE_MODULES, so the name matches the target and the guard for the wrong file can go. n/a

What's Missing

Parallel paths:

• 🟡 docs/TASK_DEFINITION_GUIDE.md:177-184 and :195 still say that sdk_options takes ClaudeAgentOptions keys only and that it is "Requires type: "claude-code" ... on other agent types it raises an error". The PR changes the guard in overrides.py into a registry lookup (_kinds_accepting_sdk_options), and DelegateAgentConfig now accepts sdk_options: {effort}. So the authoritative task guide now contradicts the code and docs/agents/DELEGATE.md:173. The comment block in experiments/default.yaml ("Keys must be ClaudeAgentOptions fields") has the same drift. Update both to say that the allowed keys depend on the agent type. (trigger: src/coder_eval/orchestration/overrides.py)
• 🔵 The PR renames the Delegate credential contract to DELEGATE_AUTH_TOKEN / DELEGATE_TENANT_ID / DELEGATE_ORG_ID / DELEGATE_ORG_SLUG / DELEGATE_TENANT_SLUG / DELEGATE_ENV. Nothing updates the driver: docker path to match. These names are not in the default env_passthrough (models/sandbox.py:209). The framework image does not install @uipath/delegate-stdio (DOCKER_ISOLATION.md:18 lists the agent SDKs it bakes in). docs/agents/DELEGATE.md says nothing about docker. As a result, a Delegate task under driver: docker gets no credentials and no host bundle, and nothing tells the user. Either document env_passthrough_extra plus a derived image, or reject the delegate + docker combination at start. (trigger: .env.example)

Downstream consumers:

• 🟡 The PR adds a second init-error classifier, _INIT_CONFIG_ERROR_MARKERS, which marks errors that "a retry cannot fix" as AgentConfigError. The two consumers that decide retries did not change with it. (1) The respawn branch in communicate() turns that error into AgentCrashError. (2) The substring table in errors/categorization.py has no entries for "expired", "auth required", "403" or "requires org/tenant slugs". The same expired-token or missing-slug message therefore ends the task when it happens at start(), but is retried as AGENT_CRASH when it happens mid-run. Make one classifier the source for both paths. (trigger: src/coder_eval/agents/delegate_agent.py) (restates: Axis 3: respawn re-raises marker-class init AgentConfigError as retryable AgentCrashError)

Tests:

• 🟠 The Delegate usage path is new: usage frames per call, sent before that call's tool results, with turnUsages as the call count and the max-turns cut keeping the tokens of calls under the cap. No test asserts the reconciliation invariant for it, i.e. that the token buckets summed across TurnRecord.messages equal token_usage. The tests check only the token_usage totals and num_turns. Delegate also stays exempt from golden-master coverage, and the reason given for the exemption is stale. Add assert_reconciliation over Delegate transcripts, including the cap-cut case. (trigger: tests/test_delegate_agent.py) (restates: Axis 3: no golden-master/full-record parity test; stale UNVERIFIED exemption)
• 🟠 Some tests for new code paths are missing. (1) No test rejects a non-string sdk_options.effort on DelegateAgentConfig. (2) No test feeds a host stderr line that contains the bearer token and asserts that the token stays out of task.log and error_log_tail. (3) test_only_an_init_error_a_retry_cannot_fix_is_non_retryable has no negative case where a port, GUID or request id contains 401/403. (4) No test registers a throwaway config with sdk_options to show that the registry-derived override guard accepts it. The guard tests also hard-code the literal 'claude-code, delegate' list. (trigger: tests/test_delegate_agent_config.py) (restates: Axis 2: typed effort replaced by untyped sdk_options dict)

Display & mapping dicts:

• 🟡 Delegate's get_sdk_options() now returns the camelCase init dict. Because of that, the Agent Settings table (collect_agent_settings_rows in reports/markdown.py and reports/html.py _render_agent_settings) reads that dict and no longer falls back to agent_config. The table then loses these rows: 'Plugins' (the init dict has bundledSkillsPath, not plugins), the configured 'Permission Mode' (now 'N/A'), and 'Max Turns' (the adapter enforces run_limits.max_turns but does not send maxSteps). It also gets no rows for the Delegate keys projectId, enableComputerUse, enableSkills and env. The only test asserts the Model and Effort rows. (trigger: src/coder_eval/agents/delegate_agent.py) (restates: Axis 7: get_sdk_options() persists host init options into run.json sdk_options)

Nightly pipeline:

• 🟠 The PR does not say what happens to the run records that already exist. Every Delegate task.json from v0.12.5 to v0.12.9 holds "effort": null, and the PR head can no longer load it. So --resume silently re-runs finished tasks, recover_task_results drops rows from the denominator, and evaluate/report fail on those run dirs. This also hits any Python consumer in coder-eval-uipath or eval-runner that loads them through EvaluationResult. The PR body does not mark the change as breaking. (trigger: src/coder_eval/models/agent_config.py) (restates: Axis 8: removing DelegateAgentConfig.effort makes v0.12.5-v0.12.9 Delegate task.json unloadable)
• 🟠 The PR changes the CI delegate-live-tests job to send only DELEGATE_AUTH_TOKEN/… through the auth init option, and it removes the host's own AUTH_TOKEN/TENANT_ID/ORG_ID names from the host env. No published @uipath/delegate-stdio release (1.202.1 or 1.203.0-preview) reads auth. The PR does not say that the live job, and any scheduled Delegate run that uses env tokens with no saved login, will fail at init after the merge. (trigger: .github/workflows/pr-checks.yml) (restates: Axis 1: pinned @uipath/delegate-stdio floor ignores the auth init option while env-token vars are removed)
• 🟡 The PR does not mention the coordination it needs with coder_eval_uipath. (1) The registry-derived sdk_options guard is exactly the change that _overrides_patch.py Patch 1 waits for before it is deleted. Its removal recipe keys on #181, so nobody will know to delete it on the next pin bump. Patch 1 also hard-codes core's old message. (2) HARNESS_PARITY.md now says that the built-in delegate 'replaced' delegate-sdk. But the weekly coder-eval-daily cron still runs HARNESS=delegate-sdk on the same @uipath/delegate-stdio host, and the evalboard still lists both harnesses. State whether the nightly moves to delegate and when Patch 1 goes. (trigger: docs/agents/HARNESS_PARITY.md)

Harness & Lint Improvements

Static checks (lint / type):

• [ruff] Add "C901" to [tool.ruff.lint] select and set [tool.ruff.lint.mccabe] max-complexity = 20. Mark each current offender with a # noqa: C901 debt marker, the same way PLR0915/PLR0912 are handled today. Any new function above 20 then fails make check. Prevents: Finding 'communicate cyclomatic complexity rises to D(28)' (src/coder_eval/agents/delegate_agent.py:723-852), for new god-functions. It does not catch +1 growth of a function that already has a noqa marker. CE068 below covers that case.
• [ce-lint] CE068 complexity ratchet: a @pytest.mark.lint class in tests/test_custom_lint.py. It runs radon cc (radon is already a runtime dependency) over src/coder_eval/ and compares each function's score with a checked-in baseline (tests/lint/complexity_baseline.json). It fails when a function's score goes above its baseline, and it lowers the baseline when a score goes down. Add CE068 to the ruff external list. 068 is the next free id: 067 and 062 are retired, see the runner.py comment. Prevents: The D(27) to D(28) growth of DelegateAgent.communicate (delegate_agent.py:723). The ruff C901 cap cannot catch growth in a function that already carries a noqa marker.
• [ce-lint] CE069 tests/lint/rules/ce069_no_untyped_dict_config_field.py, wired in tests/lint/runner.py ALL_RULES. In the CE009-scoped YAML input-model modules (models/agent_config.py, tasks.py, limits.py, ...), a field annotated dict[str, Any], alone or inside a union, needs # noqa: CE069 <reason>. When a field_validator only checks the dict keys against a closed frozenset, use a nested BaseModel with extra='forbid' and typed fields instead. The ClaudeCode sdk_options pass-through and claude_settings get noqa markers with reasons. Prevents: Finding 'Typed Delegate effort field replaced by untyped sdk_options: dict[str, Any]' (src/coder_eval/models/agent_config.py:419, _validate_sdk_options_keys at 454-463). The rule makes effort: 5 and effort: [high] fail validation instead of being forwarded silently, and it makes a non-JSON YAML value fail at load time instead of raising TypeError in json.dumps.
• [ce-lint] CE070 tests/lint/rules/ce070_no_bare_status_code_substring.py. The rule forbids an all-digit string literal ("401", "403", "429", "502", ...) as a substring-membership pattern: "401" in s, or any(p in s for p in X) where X is a literal list/tuple/set or a module-level constant that contains such an entry. Status codes must be matched through one shared word-boundary helper, for example re.search(r"\b40[13]\b", s) in errors/. Wire it in runner.py and the ruff external list. Prevents: Finding 'Init-error classifier matches bare 401/403 substrings' (delegate_agent.py:113-124 and :568, where 'ECONNREFUSED 127.0.0.1:54013' and a GUID that contains 401 become a terminal AgentConfigError). It also flags the same pre-existing defect in src/coder_eval/errors/categorization.py:81, 89, 94 and 108 ("401", "402", "429", "502"/"503"/"504").
• [ce-lint] CE071 tests/lint/rules/ce071_stderr_through_redactor.py. In src/coder_eval/agents/, a function that reads subprocess stderr (.stderr.readline(), .stderr.read(), or a future created from one) must pass the text through a shared redact_secrets() helper before the text goes to a logger.* call or to a buffer that is logged later (_stderr_lines.append). Stated blind spot: the rule does not follow the text across functions. Current sites: delegate_agent.py _drain_stderr, opencode_agent.py:1018 and pi_agent.py:958. Prevents: Finding 'Bearer token travels in the stdin init frame ... coder_eval logs that stderr unredacted' (delegate_agent.py:1275 logger.debug and the :1131 WARNING tail, from there to task.log and error_log_tail). CodeQL clear-text-logging does not catch it, because the secret goes into a third-party process and comes back as untainted stderr text.
• [ci-gate] Persisted-schema snapshot gate: commit EvaluationResult.model_json_schema() (it includes every ResolvedAgentConfig member) to tests/fixtures/persisted_schema.json. A test fails when a property is removed or renamed in a model that has extra='forbid' and that task.json/run.json persists, unless (a) the model has a mode='before' model_validator that names the old key, or (b) CHANGELOG.md has a BREAKING entry that names the field. The check needs only the schema, so it is static. Prevents: Finding 'Removing DelegateAgentConfig.effort makes every Delegate task.json from v0.12.5 to v0.12.9 unloadable' (src/coder_eval/models/agent_config.py:419). The removal would fail CI and force a migration validator or an explicit BREAKING note.
• [ci-gate] pyright tightening (make typecheck): change the return type of Agent.get_sdk_options() and the type of EvaluationResult.sdk_options from dict[str, Any] | None to a record tagged by AgentKind, for example a discriminated union ClaudeSdkOptionsRecord | DelegateInitOptionsRecord | ... keyed on agent. A consumer such as orchestrator.py:1614 (sdk_options.get("env").get("PATH")) must then narrow by agent before it reads a key. The CE030 doc-parity check then makes REPORT_SCHEMA.md describe each shape. Prevents: Finding 'Delegate get_sdk_options() persists host init options into run.json sdk_options, and reuses env as a string' (delegate_agent.py:686 and :591, results.py:676-679, REPORT_SCHEMA.md:142).
• [ci-gate] Make the _NO_GOLDEN_COVERAGE entries in tests/test_agent_golden_master.py structured: (reason, evidence_path, evidence_marker) instead of free prose. TestGoldenCoverage asserts that evidence_marker (for example 'UNVERIFIED') still occurs in evidence_path. When the marker is removed, the exemption fails, and the author must record scenarios or write a new, true reason. Prevents: Finding 'Delegate's exemption from golden coverage cites UNVERIFIED field guesses that this PR deletes' (tests/test_agent_golden_master.py:239-242). The PR removed both UNVERIFIED markers from delegate_agent.py, but the exemption stayed.

Harness improvements (not statically reachable):

• Delegate host contract test plus a lockfile. Commit agents/delegate/package-lock.json and use npm ci in the delegate-live-tests CI job. Add an offline test that runs on every package.json change: spawn the installed delegate_stdio.mjs with an empty HOME and no AUTH_/TENANT_/ORG_* env, send init with options.auth and an unreachable backendUrl, and assert that the failure is NOT 'Auth required' (init must get past resolveAuth). Until a host release passes this test, the adapter must keep or set the host's own env names. Why not static: Whether the third-party host honours the auth init option is runtime behaviour of an obfuscated bundle. The semver range '^1.202.1' does not show it. Only spawning the resolved version can show it. Prevents: Finding 'Pinned @uipath/delegate-stdio floor (^1.202.1) ignores the auth init option' (src/coder_eval/agents/delegate/package.json:7, delegate_agent.py:199-207 and :593).
• Central secret redaction plus a canary test. Add a logging.Filter to task_log_handler and to the error_log_tail buffer that replaces registered secret values with '***'. Each agent registers its credential values at start(). Add a parametrized test per agent that uses a fake host which echoes stdin to stderr (the same as DELEGATE_STDIO_VERBOSE=1) with a canary token. Assert that the token is absent from task.log, task.json error_log_tail and run.json sdk_options. Why not static: The leak goes through a third-party process that echoes data back. Only a runtime canary shows that a secret value reaches a file sink. CE071 only enforces that the redactor is called. Prevents: Finding 'Bearer token travels in the stdin init frame ... DELEGATE_STDIO_VERBOSE=1' (delegate_agent.py:1275, :1131, orchestrator.py:663).
• Sandbox secret-reach canary for driver: docker. Add a fixture task whose shell step tries cat "$DELEGATE_AUTH_TOKEN_FILE", reads /proc//environ, and echoes $DELEGATE_BACKEND_URL and $LLMGW_CLIENT_SECRET. The test asserts what the agent can reach, and the docs describe the scrub as defense in depth with these known gaps. Why not static: What a same-uid child process can read (env inheritance, /proc environ, token files) depends on the process tree and the container at runtime. Whether a doc claim like 'cannot read the token' is true needs semantic judgement. Prevents: Finding 'Doc claims agent shell commands cannot read the token' (docs/agents/DELEGATE.md:53, delegate_agent.py:208-210).
• Released-version golden task.json fixtures. For each built-in AgentKind, keep one task.json from the last release tag (start with Delegate v0.12.9, which persists "effort": null). Assert that it loads through EvaluationResult.model_validate_json, through orchestration/batch.py _load_completed_result, through regrade and through recover_task_results. Also change _load_completed_result so that it logs a WARNING when a task.json exists but fails validation, instead of silently re-running the task on --resume. Why not static: The test needs records produced by a released version. The schema-snapshot gate catches the field removal, but only real old records show that resume, recover and regrade still give the same rows and the same pass-rate denominator. Prevents: Finding 'Removing DelegateAgentConfig.effort makes every Delegate task.json from v0.12.5 to v0.12.9 unloadable' (src/coder_eval/models/agent_config.py:419, batch.py:490).
• Record Delegate golden-master scenarios from a live delegate-stdio 1.202.1 transcript. Add AgentKind.DELEGATE to SCENARIOS_BY_AGENT and remove the exemption. Run assert_reconciliation and assert_timing_captured, and snapshot the full EventCollector-built TurnRecord over model_fields, not hand-picked fields. Why not static: Frame field shapes and the reconciliation of token buckets can only be checked against a real event stream. The hand-built _ev frames in tests/test_delegate_agent.py pin the guesses, not the host. Prevents: Finding 'The rewritten Delegate wire protocol has no golden-master/full-record parity test' (tests/test_delegate_agent.py:36-38, tests/test_agent_golden_master.py:242).
• Retry-classification parity test. Parametrize over the init-error markers and assert that the same host init error gives the same exception class and the same retryability at start() and in the mid-run respawn in communicate() (delegate_agent.py:736-745). Add negative cases that must stay retryable: 'connect ECONNREFUSED 127.0.0.1:54013', a GUID that contains 401, 'fetch failed'. Run the result through errors/categorization.py so the final FinalStatus is the asserted value. Why not static: Retryability depends on message content at runtime and on two code paths that classify it. An AST rule could flag except AgentConfigError: raise AgentCrashError (there is only one site), but whether the downgrade is correct is a semantic decision. A parity test pins it. Prevents: Findings 'The untested respawn path re-raises a marker-class init AgentConfigError as a retryable AgentCrashError' (delegate_agent.py:738-745) and the missing negative tests for the bare-digit markers (tests/test_delegate_agent.py:247-264).
• Plugin-isolated agent registry. Add an autouse conftest fixture that pins AgentRegistry to the built-in agents (third-party entry points disabled), and add a CI matrix leg that installs coder_eval_uipath. Add one generic-contract test that registers a throwaway config class with an sdk_options field and asserts that -D agent.sdk_options.x=... is accepted for it. Why not static: The test result depends on which entry-point plugins are installed in the environment at runtime. A static check cannot see that. Prevents: Finding 'The sdk_options override-guard tests hard-code the registry contents' (tests/test_overrides_engine.py:147, :157; tests/test_merge_characterization.py:309).
• Host install preflight and fault-injection tests. In start(), resolve @uipath/delegate-sdk relative to the host bundle, or dry-import the bundle (node --input-type=module -e "await import(...)"), and raise AgentConfigError on ERR_MODULE_NOT_FOUND. Wrap the init _send_command so that BrokenPipeError/ConnectionResetError logs the stderr tail and raises AgentCrashError. Add tests with a missing transitive package and with a host that is dead before the init write. Why not static: Whether the module graph resolves depends on the node_modules tree on disk at run time. Whether the pipe is broken depends on process timing. Prevents: Finding 'Missing transitive @uipath/delegate-sdk is no longer a loud pre-spawn AgentConfigError' (delegate_agent.py:1224, :556, :739).
• No rule for the hand-synced auth names or the DELEGATE_SDK_PATH naming. Remove the pattern instead: build _INIT_CONFIG_ERROR_HINT from _AUTH_OPTION_FIELDS and _HOST_ENV_REMOVED, and rename DELEGATE_SDK_PATH/DELEGATE_SDK_NODE_MODULES to DELEGATE_HOST_PATH/DELEGATE_HOST_NODE_MODULES. Then the wrong-file guard at delegate_agent.py:267-272 can go. This is a recorded decision not to add a guard. Why not static: Whether prose duplicates a tuple, or whether a variable name matches its target, needs semantic judgement. A rule with one call site would break the 'delete before you guard' principle. Prevents: Findings 'The auth variable names are kept in sync by hand in three places' (delegate_agent.py:128-134) and 'DELEGATE_SDK_PATH / DELEGATE_SDK_NODE_MODULES keep their SDK names' (delegate_agent.py:259, :275).

Top 5 Priority Actions

1. src/coder_eval/models/agent_config.py:419: Add a mode="before" validator on DelegateAgentConfig that moves a legacy effort key into sdk_options["effort"], and add a regression test that loads a v0.12.9 Delegate task.json fixture. At the moment, extra="forbid" rejects the "effort": null field that every Delegate run from v0.12.5 to v0.12.9 recorded. As a result, --resume silently re-runs finished tasks, which can change their score and final_status, and run.json recovery drops those rows from the pass-rate denominator.
2. src/coder_eval/agents/delegate_agent.py:113-126,568,740-745: Make init-error classification give the same final_status for the same error. Match status codes with a word-boundary regex (\b40[13]\b) and narrow expired to token expired, so that a port number or GUID cannot end a task with no retry. In the respawn path, re-raise marker-class errors as AgentConfigError, not as a retryable AgentCrashError. Add negative test cases (port 54013, a GUID that contains 401) and a respawn test.
3. src/coder_eval/agents/delegate/package.json:7 and delegate_agent.py:199-207: Keep AUTH_TOKEN/TENANT_ID/ORG_ID (and the slug names) in the host env, or set them from the DELEGATE_* values. A live test showed that no published @uipath/delegate-stdio (1.202.1 or 1.203.0-preview) reads the auth init option, so the documented token path and the CI delegate-live-tests job fail. Remove the error hint at lines 128-134 only after a host release that accepts auth is the version floor.
4. src/coder_eval/agents/delegate_agent.py:1275 (and the tail at :1131): Remove the access token from each host stderr line before it goes into _stderr_lines and logger.debug, or remove DELEGATE_STDIO_VERBOSE from the host env when auth is sent. Add a test that the token does not appear in log records. With the documented verbose flag, the host echoes the token twice, and it lands in task.log and in error_log_tail, which are uploaded to shared blob storage and the evalboard.
5. src/coder_eval/models/agent_config.py:419 and delegate_agent.py:579: Replace sdk_options: dict[str, Any] with a typed DelegateSdkOptions(extra="forbid", effort: str | None) model. Forward it with model_dump(exclude_none=True) and add a test that rejects effort: 5. Also update results.py:676-679 and REPORT_SCHEMA.md:142 to say that the shape of sdk_options depends on the agent, because Delegate now writes its camelCase init dict (with a string env) into run.json.

Stats: 0 🔴 · 4 🟠 · 5 🟡 · 5 🔵 across 8 axes reviewed.