QuoxGPU

GPU monitoring and inference backend management.

QuoxGPU is a standalone Go daemon that runs on machines with GPUs. It monitors GPU hardware, discovers local inference backends, and reports everything to QuoxCORE via the heartbeat protocol.

What It Does

QuoxGPU is a single binary that installs on a GPU host as a systemd service. Once running, it:

• Detects NVIDIA GPUs via NVML (primary) or nvidia-smi CLI (fallback)
• Reports VRAM usage, temperature, power draw, fan speed, and utilisation per GPU
• Probes inference backends — Ollama, vLLM, llama.cpp, and NVIDIA NIM
• Tracks loaded models with VRAM allocation and backend type
• Sends heartbeats to the QuoxCORE Collector every 30 seconds
• Exposes an HTTP API on port 9846 and Prometheus metrics at /metrics

The Problem

GPU infrastructure is opaque. Teams running local inference face several blind spots:

• No fleet view — Each GPU machine is a silo. Nobody knows what models are loaded where.
• VRAM is invisible — A developer loads a 70B model and consumes all VRAM on a shared machine. Others hit OOM errors with no explanation.
• Backend sprawl — Different teams run Ollama, vLLM, and llama.cpp on different machines. No single inventory.
• Temperature surprises — A GPU hits thermal throttling or fails silently. Nobody notices until inference latency spikes.
• No audit trail — In regulated environments, there is no record of which model produced which output.

QuoxGPU makes GPU infrastructure visible by placing a lightweight monitor on each host.

Who It's For

Architecture

┌──────────────────────────────┐
│         GPU Host             │
│                              │
│  ┌──────────┐  ┌──────────┐ │
│  │  NVIDIA   │  │ Ollama / │ │
│  │  GPUs     │  │ vLLM /   │ │
│  │ (NVML)    │  │ llama.cpp│ │
│  └─────┬─────┘  └─────┬────┘ │
│        │              │      │
│  ┌─────┴──────────────┴────┐ │
│  │       QuoxGPU           │ │
│  │  (port 9846)            │ │
│  └──────────┬──────────────┘ │
└─────────────┼────────────────┘
              │ POST /api/v1/heartbeat
              ▼
┌──────────────────────────────┐
│     QuoxCORE Collector       │
│        (port 9848)           │
└──────────────────────────────┘

QuoxGPU is not a QuoxAgent plugin. It is a separate binary with its own release cycle, systemd service, and install script. It shares the heartbeat protocol with QuoxAgent but operates independently. You can run both on the same host.

GPU Detection

QuoxGPU uses a two-tier detection strategy:

1. NVML (primary) — The NVIDIA Management Library accessed via go-nvml. Uses dlopen at runtime, so QuoxGPU builds on any machine and fails gracefully if libnvidia-ml.so is absent.
2. nvidia-smi (fallback) — Parses --query-gpu CSV output. Used in containers where the NVML shared library is not mounted.

If neither is available, QuoxGPU runs in inference-only mode — it still probes backends and reports models, but has no GPU hardware data.

Fields Reported Per GPU

Inference Backend Detection

QuoxGPU probes configured URLs and auto-detects the backend type by trying health and model endpoints in order: Ollama, vLLM, llama.cpp, NIM. You can also specify the type explicitly in config.

Supported Backends

Installation

Quick Install

curl -sSL https://quox.ai/install-gpu | sudo bash -s -- \
  --collector http://your-quoxcore:9848 \
  --host-id gpu-server-01

Manual Install

# Download binary
curl -L -o /usr/local/bin/quoxgpu \
  https://github.com/quoxai/quoxgpu/releases/latest/download/quoxgpu-linux-amd64
chmod +x /usr/local/bin/quoxgpu

# Create config
mkdir -p /etc/quoxgpu
cat > /etc/quoxgpu/config.yaml <<EOF
host_id: "gpu-server-01"
port: 9846
collector_url: "http://your-quoxcore:9848"
sources:
  - url: "http://localhost:11434"
EOF

# Start
quoxgpu

Configuration

Config file: /etc/quoxgpu/config.yaml

host_id: "gpu-server-01"
port: 9846
collector_url: "http://quoxcore:9848"
heartbeat_interval_sec: 30
gpu_cache_sec: 5
log_level: info

sources:
  - url: "http://localhost:11434"     # Ollama (auto-detect)
  - url: "http://localhost:8000"
    type: vllm                         # Explicit type
  - url: "http://localhost:8080"
    type: llama.cpp

Environment variables override config with the QUOXGPU_ prefix (e.g. QUOXGPU_PORT=9846).

API Reference

All endpoints are on port 9846 (default).

Example: Health Check

$ curl -s localhost:9846/health | jq
{
  "status": "ok",
  "version": "0.1.0",
  "gpus": 2,
  "backends": 1
}

Heartbeat Protocol

QuoxGPU sends heartbeats to the QuoxCORE Collector at POST /api/v1/heartbeat. The payload extends the standard QuoxAgent format with GPU-specific fields:

{
  "version": "0.1.0",
  "host_id": "gpu-server-01",
  "timestamp": "2026-03-10T12:00:00Z",
  "agent": {
    "name": "quoxgpu",
    "version": "0.1.0",
    "type": "quoxgpu",
    "port": 9846
  },
  "gpu_devices": [
    {
      "index": 0,
      "name": "NVIDIA GeForce RTX 4090",
      "vram_total_mb": 24564,
      "vram_used_mb": 8192,
      "temperature_c": 52,
      "utilisation_pct": 45,
      "healthy": true
    }
  ],
  "inference_backends": [
    {
      "type": "ollama",
      "url": "http://localhost:11434",
      "healthy": true,
      "models": [
        { "name": "llama3.1:8b", "loaded": true, "vram_used_bytes": 4500000000 }
      ]
    }
  ],
  "loaded_models": [
    { "name": "llama3.1:8b", "backend": "ollama", "loaded": true }
  ],
  "status": "healthy",
  "capabilities": ["gpu-monitoring", "inference", "backend:ollama"],
  "system": {
    "hostname": "gpu-server-01",
    "os": "linux",
    "arch": "amd64"
  }
}

The Collector stores this data and serves it via GET /api/v1/agents. The dashboard GPUFleetPanel filters agents with gpu_devices or inference_backends fields.

Security

The systemd service includes hardening directives:

• NoNewPrivileges=yes
• ProtectSystem=strict
• ProtectHome=yes
• PrivateTmp=yes
• DeviceAllow restricted to /dev/nvidia*
• Runs as dedicated quox user with video and render group membership

The HTTP API has no built-in authentication. Bind to localhost or use a reverse proxy with auth for production deployments.

Source Code

QuoxGPU is open source: github.com/quoxai/quoxgpu