Local Inference

Run inference on your own hardware. Zero API costs, full data privacy.

The Local Inference plugin connects QuoxCORE to self-hosted LLM inference servers on your own infrastructure. All inference traffic stays within your network. Every request is tracked, metered, and recorded through AEE envelopes and VOLT receipts.

During the public beta, all features are available at no cost.

Overview

Local Inference acts as a bridge between QuoxCORE agents and any OpenAI-compatible inference endpoint. Point it at a running server, and the plugin auto-detects the backend type, enumerates available models, and begins routing requests.

Key capabilities:

• Auto-detection of backend type from the server's API responses
• OpenAI-compatible API surface for all supported backends
• SSE streaming for real-time token delivery
• Model switching without restarting agents or sessions
• Local embedding generation for RAG and vector workflows
• Health monitoring with configurable alerting thresholds
• Per-request AEE envelopes and VOLT audit records
• Data residency labels with policy enforcement
• Configurable local-to-cloud failover

Supported Backends

All backends expose an OpenAI-compatible /v1/chat/completions endpoint. The plugin normalises minor differences in response format across backends.

Configuration

Add a local-inference block to your QuoxCORE plugin configuration.

Single Source

plugins:
  local-inference:
    sources:
      - name: dev-gpu
        url: http://10.20.0.50:11434
        backend: ollama          # Optional. Auto-detected if omitted.
        default_model: llama3:8b
        timeout_ms: 30000
        tls: false

Multiple Sources

plugins:
  local-inference:
    sources:
      - name: ollama-cluster
        url: http://10.20.0.50:11434
        backend: ollama
        default_model: llama3:8b

      - name: vllm-prod
        url: http://10.20.0.60:8000
        backend: vllm
        default_model: mistral-7b-instruct
        gpu_ids: [0, 1]

      - name: nim-server
        url: http://10.20.0.70:8000
        backend: nim
        default_model: meta/llama3-70b-instruct

    failover:
      enabled: true
      order: [ollama-cluster, vllm-prod, nim-server]
      cloud_fallback: false      # Set true to fall back to cloud provider

    residency:
      label: "uk-datacentre-01"
      enforce: true              # Block requests that would leave this zone

Environment Variables

Model Management

Listing Models

The plugin queries each connected source and returns a unified model list.

The unified models API below is part of the planned surface described above; today the dashboard panel enumerates models from each registered source directly.

# Planned surface, not yet shipped
curl http://localhost:3001/api/plugins/local-inference/models

Response:

{
  "models": [
    {
      "id": "llama3:8b",
      "source": "ollama-cluster",
      "backend": "ollama",
      "parameters": "8B",
      "quantisation": "Q4_K_M",
      "loaded": true
    },
    {
      "id": "mistral-7b-instruct",
      "source": "vllm-prod",
      "backend": "vllm",
      "parameters": "7B",
      "quantisation": null,
      "loaded": true
    }
  ]
}

The API surface today

Honesty note (2026-08-07): an earlier version of this page documented model switching, a standalone streaming endpoint and an embeddings endpoint that have not shipped. The routes that exist today are the source registry and the connection test, on the collector:

# Register an inference source
curl -X POST http://collector:9848/api/v1/local-inference/sources \
  -H "Content-Type: application/json" \
  -d '{"name": "ollama-cluster", "base_url": "http://gpu01:11434", "type": "ollama"}'

# List registered sources
curl http://collector:9848/api/v1/local-inference/sources

# Remove a source
curl -X DELETE http://collector:9848/api/v1/local-inference/sources/<id>

# Test a source's connectivity and detect its backend type
curl -X POST http://collector:9848/api/v1/local-inference/test \
  -H "Content-Type: application/json" \
  -d '{"base_url": "http://gpu01:11434"}'

Chat requests do not use a dedicated public endpoint: they route through the collector's normal chat dispatch, which proxies to your registered source's OpenAI-compatible API.

Planned, not yet shipped

Model switching mid-session, a standalone SSE streaming endpoint, an embeddings endpoint and a metrics API are planned surfaces. They are listed here so the roadmap is visible; nothing on this page should be read as implying they work today.

Health Monitoring

The plugin polls each configured source at a regular interval (default: 30 seconds) and exposes health status through the QuoxCORE dashboard and API.

# Planned surface, not yet shipped. Today: POST /api/v1/local-inference/test
# performs an on-demand connectivity and backend check per source.
curl http://localhost:3001/api/plugins/local-inference/health

Response:

{
  "sources": [
    {
      "name": "ollama-cluster",
      "status": "healthy",
      "latency_ms": 12,
      "models_loaded": 2,
      "gpu_utilisation": 0.45,
      "last_check": "2026-03-10T14:30:00Z"
    },
    {
      "name": "vllm-prod",
      "status": "degraded",
      "latency_ms": 340,
      "models_loaded": 1,
      "gpu_utilisation": 0.92,
      "last_check": "2026-03-10T14:30:00Z",
      "warning": "GPU utilisation above 90%"
    }
  ]
}

Health states: healthy, degraded, unreachable. When a source becomes unreachable, failover policies activate if configured.

Metrics

The plugin tracks three categories of metrics for each inference source.

Token Usage

• Prompt tokens per request
• Completion tokens per request
• Cumulative token counts per model, per source, per hour

Latency

• Time to first token (TTFT) at P50, P90, and P99
• Total request duration at P50, P90, and P99
• Per-model latency breakdown

Request Tracking

• Total requests per source and model
• Success and failure counts
• Requests per minute (rolling window)
• Active concurrent requests

Metrics are available through the dashboard plugin panel and via the API:

# Planned surface, not yet shipped
curl http://localhost:3001/api/plugins/local-inference/metrics?range=24h

Audit Trail

Every inference request generates two audit artefacts.

AEE Envelopes

Each request is wrapped in an AEE (Agent Envelope Exchange) envelope containing:

• Request metadata (model, source, timestamp, session ID)
• Token counts and latency measurements
• Data residency label
• Requesting agent identity

Envelopes are stored in the QuoxCORE audit log and can be queried through the standard AEE API.

VOLT Records

Completed inference requests produce a VOLT (Verifiable Operations Ledger and Trace) record. VOLT records are append-only and tamper-evident, providing a verifiable chain of all inference activity.

Each VOLT record includes:

• Hash of the request envelope
• Hash of the response payload
• Timestamp and ordering proof
• Source and model identifiers

VOLT records support compliance workflows where organisations must demonstrate that inference ran on approved infrastructure within designated data boundaries.

Data Residency

Each source can carry a residency label (e.g. uk-datacentre-01, eu-west-rack-3). When enforce: true is set, the plugin blocks any request that would route to a source outside the permitted residency zone, including cloud failover.

Pricing Tiers

Local Inference is free during the public beta. After general availability, three tiers will apply.

API Reference

Shipped endpoints live under /api/v1/local-inference on the collector; the /api/plugins/local-inference prefix shown in planned examples arrives with those surfaces.

All endpoints require a valid QuoxCORE session token in the Authorization header.