Generate an environment
On the Environments page, pick a harness and a task group. The task group’s held-out successful examples become the task list, one JSON object per line:task is what the agent is asked to do and expect is the known-good outcome the judge grades against. Tasks can also carry an explicit verifier tag, which the bare form is shorthand for:
expect as the reference; a different approach or format than the reference still passes. The judge is never the model under test (set JUDGE_MODEL to change it; the runner warns if you point it at the model under test). The runner comes in two forms: Belvedir Runner on Harbor runs each task in its own container and is preselected on hosted Belvedir; the in-sandbox Belvedir Runner runs the whole set in one sandbox (--preset harbor-runner or --preset belvedir from the CLI; omitting --preset picks the platform’s default, Belvedir Runner on Harbor on hosted Belvedir). To run your own agent per task, set BELVEDIR_HARNESS_CMD: a shell command that receives the task on stdin and prints the attempt.
Only harnesses that accept a task list can back an environment: the Belvedir Runner (either form), the available agent scaffolds from the catalog, or a repo of your own that reads the file at BELVEDIR_TASKS_FILE and handles that format.
Synthesize verifiers
On hosted Belvedir, the wizard offers Synthesize verifiers: each task gets a rubric of specific, checkable criteria, graded independently with partial credit instead of a single pass or fail. A synthesis whose rubrics do not check out falls back to the standard judge automatically, and the environment says so. Synthesis runs in the background for several minutes (around ten on a full task group): the environment appears on the Environments page immediately, shows its progress, and gets its Run button once the verifiers land (belvedir environments status <id> --wait from the CLI). An environment whose synthesis is marked abandoned can be generated again. Complete suites like SWE-bench bring their own tasks and are refused. The generated task set caps at 512 KB, and very long inputs or outputs are truncated per field; a task group whose examples are too long to be useful is refused with a clear message.
Run it
Environment runs are ordinary benchmark runs: same sandbox, same pricing, same results contract. Click Run on the environment, or launch it from the New Benchmark wizard with attempts and a grading rule like any other harness; see Benchmarks.Simulations
The Environments page can also run simulations: scripted episodes of an environment against an agent model you pick, without a full benchmark launch (belvedir simulations run <simulation-id> --episodes 8 from the CLI). The agent model runs on Belvedir’s account, so it must be a priced Claude, Grok, or hosted open model (see Available models; claude-opus-5 and anthropic/claude-opus-5 spellings both work), and it follows the project’s Chinese models setting; anything else is refused before the run starts.
Use your own codebase as the harness
Pick one of the project’s connected repos as the harness and the sandbox clones your real agent (private repos authenticate through your GitHub connection automatically; no token to paste), so the model is benchmarked in the exact harness it runs in production. Your repo needs a small entrypoint that reads the task file, runs each task through your agent, and writesresults.json with a numeric score. Connect repos under Project Settings → Repository Access.