Skip to main content
By default, the verifier runs in the same container as the agent. The tests/ folder is uploaded to /tests/ at the start of the verifier phase. However, you can opt for running the verifier in a separate container, which is useful for isolated grading. This can improve the security boundary between the agent and the verifier and allow you to pre-install dependencies into the verifier image and build it ahead of time, which can reduce installation flakiness and speed up the verifier phase. It also enables trial regrading.

​
Opt in

There are two ways to opt in to a separate verifier environment:
  1. Set environment_mode = "separate" under [verifier] in task.toml.
  2. Add [verifier.environment] to task.toml.

​
Option 1: Reuse the agent’s environment configuration

Set environment_mode = "separate" under [verifier]:
Without a verifier-specific image or build definition, Harbor starts a fresh copy of the agent environment and uploads tests/ to /tests/. The agent’s filesystem changes are not inherited.

​
Option 2: Use a verifier-specific environment

Add [verifier.environment] to use a different image or resources. This implicitly enables separate mode:
This uses the same schema as [environment], including network policy. It cannot be combined with environment_mode = "shared".

​
Mode resolution

​
Image selection

Harbor selects the first available definition: Resource settings come from [verifier.environment] when provided, otherwise [environment]. An inherited agent image never overrides a verifier build definition.

​
Dedicated verifier image

Dedicated verifier images must provide /tests/test.sh (or /tests/test.bat on Windows). Harbor does not upload tests into these images at runtime.
For example, tests/Dockerfile can bundle the grading files:

​
Artifact transfer

When a separate verifier runs, Harbor copies into the verifier env:
  • /logs/artifacts/ (agent publish directory)
  • Paths listed in task-, trial-, and step-level artifacts fields
Artifacts are uploaded to their original source paths in the verifier environment, not their host destination paths. For example, { source = "/app/report.json", destination = "report.json" } is restored to /app/report.json in the verifier; destination only controls where it is saved on the host. See Artifacts. /logs/agent/ and /logs/verifier/ are not copied unless declared as artifacts — e.g. trajectory grading:

​
Multi-step tasks

Each step can override verifier mode under [steps.verifier]. Mixed shared/separate per step is supported:
Resolution: [steps.verifier].environment_mode when set; else [steps.verifier.environment] present implies "separate"; else trial-level. Network rules: Network policies. Image precedence is step image, step tests build definition, task verifier image, task tests build definition, then agent environment. A step tests/ directory without a build definition does not override a task verifier image. With the agent-environment fallback, Harbor uploads base tests, then overlays step tests. Tests are validated against each step’s effective verifier OS, so a Linux agent can be graded on Windows (and vice versa) when the matching test.sh / test.bat exists.