Local and hosted open models

Agents on local or hosted open models are traced like any other: same sessions, tasks, and task groups.

1. Run a model locally

Download the Belvedir Local desktop app from the dashboard’s Local Inference page; it bundles the llama.cpp engine (with Ollama and vLLM as switchable alternatives) and walks you through picking a model that fits your machine’s memory; the weights download on first launch.

macOS blocks the app on first open (it isn’t notarized yet). Click Done, not Move to Trash, then run xattr -cr "/Applications/Belvedir Local.app" and reopen it. Approving it under System Settings → Privacy & Security → Open Anyway also works.

The app serves on 127.0.0.1:8080, llama.cpp’s standard port, adopting a llama-server you already run there. On the Ollama engine the endpoint is 127.0.0.1:11434 instead; use whichever address matches your engine in the baseURL below.

2. Point your agent at the local endpoint

The local server exposes an OpenAI-compatible API, so your agent talks to it through the OpenAI SDK, which the Belvedir SDK already instruments. This example targets the app’s default llama.cpp engine on 127.0.0.1:8080; on the Ollama engine use http://127.0.0.1:11434/v1 instead:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: "local", // required by the SDK, ignored locally
});

const res = await client.chat.completions.create({
  model: "qwen3:4b",
  messages: [{ role: "user", content: "hello" }],
});

3. Trace it as usual

Initialize the SDK and wrap agent runs in withSession as usual; see Installation. Spans arrive with the model name (for example qwen3:4b), so local, hosted and cloud-provider calls stay distinguishable in the dashboard.

Inference stays on your machine, but traces do not: prompts and completions are exported to Belvedir like any instrumented app. Only enable tracing for local workloads you’re comfortable sending.

Hosted OpenAI-compatible endpoints

Open models too big for your hardware run the same way from a hosted OpenAI-compatible endpoint (Together, Fireworks, Groq, OpenRouter, or your own vLLM server). Steps 2 and 3 apply unchanged, with the provider’s base URL and a real API key:

const client = new OpenAI({
  baseURL: "https://api.together.xyz/v1",
  apiKey: process.env.TOGETHER_API_KEY,
});

Unlike a local model, the hosted provider sees your prompts. To have Belvedir’s router serve a model on such an endpoint, register it under Cloud Inference.

Get started

Node SDK

Guides

Inference

Training

Benchmarks

Resources

Local and hosted open models

1. Run a model locally

2. Point your agent at the local endpoint

3. Trace it as usual

Hosted OpenAI-compatible endpoints

​1. Run a model locally

​2. Point your agent at the local endpoint

​3. Trace it as usual

​Hosted OpenAI-compatible endpoints

1. Run a model locally

2. Point your agent at the local endpoint

3. Trace it as usual

Hosted OpenAI-compatible endpoints