Open source · MIT licensed · OpenAI-compatible

One endpoint for your LLM farm

bosco is a load balancer and router for LLMs and embedding models spread across the machines on your network. Add GPUs, Macs and servers to the pool, and every app keeps calling a single OpenAI-compatible URL.

View on GitHub Get started →

$git clone https://github.com/AutreMachine/bosco

USERS & APPLICATIONS

Teams

Applications

AI agents

Developers

Batch jobs

boscoLLM Router · OpenAI API · Embeddings · JEV

PC · RTX 4090

DGX Spark

Mac Studio Ultra

Mac Studio Max

GPU server

LLM FARM

← Swipe to see the whole picture →

Why bosco

Grow your LLM farm. Keep one address.

High-end GPUs are expensive. Several smaller machines, plus that old RTX 3090 in the closet, can serve more clients, as long as something sits in front of them and decides who does what.

Add machines, not endpoints

Register a new box running llama.cpp, Ollama, vLLM or LM Studio from the admin console. It joins the pool immediately, with no client to reconfigure.

The right request goes first

A priority queue lets the interactive chat jump ahead of the nightly embedding batch. Requests in progress are never interrupted.

See what every GPU really does

Every request is measured: queue wait, time to first token, prefill and generation tok/s, per machine and per model.

Features

Everything you need to run a small inference cluster

Drop-in OpenAI API

Chat completions, completions and embeddings on /v1. Point any OpenAI SDK at bosco by changing one base URL.

Priority queue

high › medium › low, then arrival order. Optional aging so low-priority requests are never starved.

LLM substitution

Map a requested model to one or more available ones. Asked for a 27B that's busy? A free 14B can answer instead.

Health checks & failover

Machines are polled continuously. If one is unreachable, the request is retried on another machine serving the same model.

Named API keys

One key per app, compared in constant time, with brute-force throttling. Revoke one client without touching the others.

LAN-only admin console

Live requests, history, machines and statistics in a dependency-free web UI, restricted to private networks.

Real performance metrics

Prefill and generation tok/s, TTFT and queue wait, saved to SQLite and JSONL. Uses llama.cpp's own timings when available.

Hot reconfiguration

Add, edit, disable or remove machines live. In-flight requests finish; the pool is persisted to machines.yaml.

llama.cppOllamavLLMLM StudioMLXany /v1/chat/completions backend

Technical overview

Under the hood

A single Python process with two ports: the OpenAI API on one side, the admin console on the other. Between them, a scheduler that knows every machine, every model and every free slot.

How bosco works

A single OpenAI-compatible entry point that spreads requests across every LLM machine on your network.

USERS & APPLICATIONS

User
interactive chat
priority high

Web app
built-in assistant
priority high

Agent / RAG pipeline
chained calls
priority medium

Developer notebook
tests & prototypes
priority medium

Nightly batch
bulk embeddings
priority low

OpenAI API · HTTPS + API key · priority (X-Priority)

bosco

router & load balancer · OpenAI-compatible

1 · Authentication
API key checked,
otherwise rejected (401)

2 · Model routing
which machines serve it?
none → 404 error

3 · LLM substitution
an LLM call can trigger
another available LLM

4 · Priority queue
all busy → the request waits;
high jumps the queue

5 · Dispatch
to the free machine
with the best performance

Every response is measured (prefill and generation in tok/s, TTFT, time in queue) and shown in the admin console, on the local network.

sent to the chosen machine

PC · RTX 4090
llama.cpp
llama-3.1-8b · qwen3-14b

NVIDIA DGX Spark
vLLM
gpt-oss-120b

Mac Studio M3 Ultra
MLX · LM Studio
llama-3.3-70b

Mac Studio M2 Max
Ollama
bge-m3 (embeddings)

Multi-GPU NVIDIA server
vLLM
qwen3-235b · bge-m3

INFERENCE MACHINES · LOCAL NETWORK

Machine down → automatic failover to another machine serving the same model

← Swipe to see the whole diagram →

Routing, step by step

Each machine declares a performance score and a number of slots (max_concurrent).
No machine declares the model → 404 model_not_found. All of them offline or disabled → 503.
If substitutions exist, Bosco looks for a machine with a subtitutable model.
A machine is free → the request goes to the most performant free machine.
Otherwise it enters the queue, sorted by priority, then by arrival order.
When a slot frees up, it goes to the highest-priority request that can run on that machine, so a request for another model never blocks the queue.

Two ports, two audiences

Port	Role	Access
`8000`	OpenAI API `/v1/…`	Internet / LAN API key required
`8001`	Web admin console	LAN only private-network IP filter

Put HTTPS in front. A reverse proxy like Caddy gives you a Let's Encrypt certificate in three lines, and never forward port 8001.

Quickstart

Running in five minutes

Install from source or with Docker, declare your machines, and call bosco like you'd call OpenAI.

bash

$ git clone https://github.com/AutreMachine/bosco && cd bosco
$ pip install -r requirements.txt
$ cp config.example.yaml config/config.yaml
$ python -m llm_router --gen-key   # paste it into server.api_keys
$ python -m llm_router -c config/config.yaml

# admin console → http://<router-ip>:8001/

docker-compose.yml

services:
  llm-router:
    image: autremachine/llm-router:latest
    restart: unless-stopped
    volumes:
      - /data/llm-router/data:/data/
      - /data/llm-router/config:/config/
    ports:
      - "42000:8000"   # OpenAI-compatible API
      - "42001:8001"   # admin console (LAN only)

config.yaml · machines

machines:
  - name: gpu-4090
    url: http://192.168.1.10:8080/v1   # llama.cpp
    performance: 100
    max_concurrent: 2
    models:
      - { name: qwen3-32b, type: llm }
  - name: gpu-3090
    url: http://192.168.1.11:11434/v1  # Ollama
    performance: 70
    models:
      - { name: bge-m3, type: embedding }

python · standard OpenAI client

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.mydomain.com/v1",
    api_key="sk-router-…",
)

r = client.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={"X-Priority": "high"},
)

emb = client.embeddings.create(
    model="bge-m3", input=["text 1", "text 2"]
)

What you get back

Responses carry routing headers so you always know where a request ran:

`X-Router-Machine`	machine that served it
`X-Router-Request-Id`	ID to find it in the console
`X-Router-Queue-Wait-Ms`	time spent waiting in the queue

How performance is measured

prefill tok/s	`prompt_tokens / (first token − request sent)`
generation tok/s	`(completion_tokens − 1) / (last − first token)`
embeddings	`prompt_tokens / total duration`

Put your idle GPUs to work.

bosco is free and open source under the MIT license. Star the repo, open an issue, or send a pull request.

AutreMachine/bosco Star on GitHub