AI & Intelligence · plugin
Your models. Your hardware. Your data never leaves the building.
Local Inference connects QuoxCORE to inference servers running on your own infrastructure: Ollama, vLLM, llama.cpp or NVIDIA NIM. Every request is proxied straight to your hardware over an OpenAI-compatible API. No cloud model provider ever sees the prompt.
In plain words
What it is, where it lives, when to reach for it
- What is it
- A QuoxCORE plugin that connects your agents to model servers on your own hardware, like Ollama or vLLM.
- Where do I use it
- In the dashboard, on the Local Inference page under Settings, once the plugin is installed.
- When would I use it
- When a prompt must not leave your network, or cloud inference costs more than your own idle hardware.
- How do I use it
- Add it from the store, then enter your server's URL on the Settings Local Inference page and test the connection.
QuoxCORE is the free, self-hosted platform underneath this. What is QuoxCORE
Why local
Cloud inference means someone else's server reads every prompt.
A local inference source runs the model, the weights and the request on hardware you control. There is no marginal API cost per token, no provider outage taking your agents down with it, and no question of which jurisdiction your data passed through. The request stays inside your network from the moment an agent asks to the moment it answers.
Supported backends
Four backends, one interface.
Point Local Inference at a running server and it auto-detects the backend type by probing Ollama's own API first, then falling back to the generic OpenAI-compatible surface every other backend shares.
Ollama
Easiest setup, CPU or GPU. Auto-detected via a probe of /api/tags. Best for workstations and hobbyist use.
vLLM
High-performance serving with tensor-parallel multi-GPU support. Best for multi-user and enterprise workloads.
llama.cpp
Lightweight and flexible, CPU or GPU. Expects server mode with --host and --port. Good for experimental setups.
NVIDIA NIM
Enterprise-grade NVIDIA microservice for data centre deployments. Requires the NIM container runtime.
How it works
The request goes to your hardware, not ours.
Local Inference is a proxy, not a relay through a Quox-operated server. When an agent asks a question through a local source, the collector calls your registered server directly and streams the answer straight back.
Agent asks
An agent or chat session sends a request through QuoxCORE, same dispatch path as any other provider.
Source resolved
The registered source's baseUrl is looked up server-side; the URL was SSRF-checked when the source was registered.
Proxied to your host
The collector calls {baseUrl}/v1/chat/completions directly on your GPU machine. The request never reaches a Quox-operated server.
Logged, not silent
A VOLT run starts before the call and the tool-loop dispatcher emits model.requested / model.responded events as it runs.
→ source resolved: office-gpu (backend: ollama) → SSRF check passed: http://192.168.1.100:11434 → VOLT run started POST http://192.168.1.100:11434/v1/chat/completions ← volt: model.requested ← volt: model.responded ✓ 200 · 340ms · llama3:8b
plugins:
local-inference:
sources:
- name: office-gpu
baseUrl: http://192.168.1.100:11434
backendType: auto # or ollama / vllm / llamacpp / nim
defaultChatModel: llama3:8b
dataResidencyTag: local # Enterprise: enforced, not just a labelConfiguration
One block, one source at a time.
Register a source with its base URL and Local Inference probes it to work out the backend type. Give it a friendly name, an optional bearer token if your server needs one, and a default model, and it is ready for agents to use.
- Auto-detect, or set the backend type explicitly
- Optional bearer-token auth per source
- A data residency tag, label-only on Free and Pro today
Pairs with QuoxGPU
Local Inference tells you which model is loaded. QuoxGPU tells you how the hardware is doing.
Local Inference proxies requests and lists models per source. It does not read VRAM, temperature or utilisation for the machine running your server: that is QuoxGPU's job, a separate standalone daemon that reports GPU hardware state to the same fleet dashboard.
See QuoxGPU →Editions
Choose your edition.
Every tier is free during the public beta: nothing here is charged today. The prices shown struck through are the launch prices that apply once the beta ends.
Free
Try it against one server
Connect one inference source and see it work. Default model only, manual health checks.
Free
Free during public beta, and free after it
- ✓ 1 inference source
- ✓ Default model only
- ✓ Basic streaming
- — Embeddings
- — Automatic health checks
- — Usage metrics
- — AEE-shaped audit envelopes
Promost popular
For teams running their own GPU boxes
Unlimited sources and models, full SSE streaming, embeddings, and automatic health checks every 30 seconds.
$89 Free
Free during public beta; $89 one-time once the beta ends
- ✓ Unlimited inference sources
- ✓ All models per source
- — Full SSE streaming (scaffolded, no server route yet)
- — Local embeddings (scaffolded, no server route yet)
- — Automatic health checks (scaffolded, no server route yet)
- — Basic usage metrics (scaffolded, no server route yet)
- — GPU fleet agent, multi-node routing
Enterprise
For fleets and regulated environments
The regulated-environment design: multi-node routing, model allowlists, local-to-cloud failover and enforced data residency. These are config schema and roadmap today, not enforcement. Do not buy this tier to satisfy a residency control: nothing in the code reads the tag yet.
$699 Free
Free during public beta; $699 one-time once the beta ends
- ✓ Everything in Pro
- — GPU fleet agent (via QuoxGPU, itself a Labs surface)
- — Multi-node inference routing (designed, not built)
- — Model allowlists per team (config field, not enforced)
- — Local-to-cloud failover (config field, not enforced)
- — Data residency enforcement (a stored label, not enforced)
- — Full usage metrics + audit trail (designed, not built)
Pro and Enterprise are one-time licence purchases, activated on your own QuoxCORE instance. No subscription. Lines marked “scaffolded”, “config field” or “designed” have no enforcement path in this codebase today: the limits band below is where each of those words came from, and it names the file.
Where it stands
What's proven, what's not yet.
- Source registration and connection testing are real, SSRF-checked collector routes (POST/GET/DELETE /api/v1/local-inference/sources, POST /api/v1/local-inference/test in services/collector/server.js), called by the real Settings UI, not a localStorage-only mock
- Backend auto-detection genuinely probes your server: Ollama’s /api/tags first, then a generic /v1/models for vLLM, llama.cpp and NVIDIA NIM, exactly two probe shapes behind the four backend names
- Chat requests through a registered source proxy straight to your host’s own /v1/chat/completions and run through the same tool-loop dispatcher OpenAI-provider chat uses, starting a real VOLT run and emitting model.requested / model.responded / tool.call.executed events as they happen
- The docs page’s API reference lists thirteen endpoints under /api/plugins/local-inference (models, completions, embeddings, metrics, health, audit/envelopes, audit/volt, config); only source registration and connection testing exist as standalone collector routes today, chat runs through the shared dispatch path instead of a dedicated /completions route
- Streaming, local embeddings, scheduled health polling and a Prometheus /metrics endpoint are documented and scaffolded in the plugin manifest’s feature list, with no corresponding server route in this codebase yet
- Registered sources live in an in-memory Map on the collector process: a restart clears every configured source, there is no database table backing them
- Data residency enforcement is a label today, not an enforcement path: dataResidencyTag is stored on the source record and nothing in this codebase reads it to block a request from leaving a zone, despite Enterprise-tier copy implying otherwise
- Local-to-cloud failover, model allowlists per team, and per-source concurrency limits are schema fields in the plugin config with no runtime check applying them
- GPU hardware telemetry (VRAM, temperature, fan speed) for the machine running your inference server is a separate product, QuoxGPU, not part of this plugin: pair the two if you need fleet-wide hardware visibility alongside the model list
Run it on your own hardware
Keep every prompt inside your network.
Connect Ollama, vLLM, llama.cpp or NVIDIA NIM in minutes. Free during the public beta, activated on your own QuoxCORE instance; $89 one-time once the beta ends.
Go deeper