Start from the catalog
The New Harness page offers one-click presets, starting with the first-party Belvedir Runner — a minimal agent loop with a pinned independent LLM judge, and the default driver for generated environments — alongside common suites (SWE-bench Lite, τ-bench, Terminal-Bench, GAIA, Aider Polyglot, lm-eval and MT-Bench, vLLM-served) and agent scaffolds: OpenClaw, Hermes Agent, OpenCode, Prime Agent, Pi, QM, and Mastra. Every prefilled field is editable before you create the harness. The Belvedir Runner and the scaffold drivers bring the agent but no task suite: you author the tasks in the harness’s Tasks field, and the sandbox exposes them to your run command as a file at$BELVEDIR_TASKS_FILE.
Runs are metered
Benchmark runs are currently free: only inference is billed. Your organization’s balance, auto reload, and card live under Organization Settings → Billing.Choose a compute type
CPU (default) is for harnesses whose agent calls a hosted model API: Anthropic, OpenAI, or your own inference endpoint (set its URL as a harness env var). GPU is for harnesses that serve a model inside the sandbox: install vLLM or Ollama in your install command and start it from your run command; the sandbox gets an NVIDIA GPU (A10G by default). All sandboxes have git, Python 3, and Node preinstalled.The harness contract
Your run command executes at the repo root with these environment variables set (plus any you configure on the harness):FRACTAL_* names are still set as aliases for the key, base URL, run id, and tasks file, so existing drivers keep working unchanged (BELVEDIR_TASK_ATTEMPTS and BELVEDIR_GRADING are newer contract variables with no aliases). If the task-set file can’t be downloaded at startup, the runner refuses to execute the harness and the run fails with a belvedir: note rather than running taskless.
Which model runs
Public benchmark drivers don’t hardcode a model. The model under test is ordinary harness env, set when you run:https://openrouter.ai/api/v1): one key covers essentially every hosted model, addressed as vendor/model ids like anthropic/claude-sonnet-5. Any other OpenAI-compatible endpoint works too — a Together-deployed fine-tune, your own vLLM — and leaving the base blank infers the provider from the model id. On the hosted platform, OpenRouter-based benchmarks need no API key at all: leave the key blank and Belvedir mints a per-run, spend-capped key and covers the model usage under the current free-benchmarks policy. Managed-key runs respect the project’s Chinese models permission: naming a Chinese-lab model on a project with it off refuses the launch. LLM-judged drivers take the judge separately as JUDGE_MODEL / JUDGE_API_BASE / JUDGE_API_KEY, so the grader stays fixed while the model under test varies. GPU drivers (lm-eval, MT-Bench) can serve the model instead of calling one: leave MODEL_API_BASE blank and MODEL (a HF id) is served in-sandbox with vLLM — the fine-tuned-model path — or set it and they evaluate the endpoint like any other suite.
Run with attempts and grading
The New Benchmark wizard (My Benchmarks → New benchmark) picks one of your benchmarks, how many times each task runs, and the grading rule —pass@1, pass@k, or mean score. The choice is stamped on the run and handed to the sandbox as BELVEDIR_TASK_ATTEMPTS / BELVEDIR_GRADING; grading itself is the harness’s job — run each task that many times and fold the attempts into the top-level score accordingly.
Send traces explicitly
The SDK does not read these variables on its own; pass them toinitialize():
Report a score
Before exiting, writeresults.json at the repo root. A top-level numeric score on a 0–1 scale is shown in the dashboard (scores are compared across models in the routing editor, so a percent scale would read as 100× better); everything else is stored verbatim. A score in (1, 100] is read as a percentage and divided by 100, with a belvedir: note appended to the run’s log tail; a negative score or one past 100 fails the run, as does exiting 0 without a numeric score. The platform records which model the run was launched with as results.model itself, so attribution never depends on the harness: a model field your harness writes is informational, and if it disagrees with the launch configuration it is preserved as results.model_reported with a note in the log tail. Runs attributed to a fine-tuned model’s name appear in that model’s View benchmarks panel on the Models page.
Benchmark a fine-tuned model
GPU harnesses that serve the model in-sandbox can also be launched from the Models page against one of your fine-tuned models. The sandbox is then given extra environment variables:MODEL (base HuggingFace id), ADAPTER_URL (a short-lived signed URL to the LoRA zip), MODEL_LABEL (the model’s name; the platform records it as results.model on the run, so attribution doesn’t depend on the harness echoing it), and HF_TOKEN for gated bases, so the harness serves base + adapter and the run shows up in that model’s View benchmarks panel.