What Qluster is #
Most operators running more than one QuoxCORE installation, a dev box, a production box, a hosted client box, a demo box, have no shared view of them: no registry, no way to check them all without SSH into each one individually. Qluster is the answer: one operator, many boxes, one pane of glass, built as its own standalone product.
The mental model, set deliberately, is Proxmox for Quox. It maps closely:
| Proxmox | Qluster |
|---|---|
| Datacenter | Your Qluster (the controller) |
| Node (hypervisor host) | A box: one enrolled QuoxCORE installation |
| Left tree: datacenter → nodes | Left tree: Qluster → boxes, grouped by kind |
| Node summary (CPU/mem/disk, uptime, version) | Box summary (status, version, uptime, service matrix, resource gauges) |
pvecm add / join token | Box enrolment: manual add-by-URL, or a scoped token |
Where the analogy deliberately stops matters just as much: no quorum, no shared state, no HA migration, no workload scheduling. Boxes are administratively independent. Qluster is an overlay: if the controller goes down, that is a visibility gap, never an outage on any box. Qluster also manages QuoxCORE applications, not virtual machines; the Proxmox resemblance is a UX paradigm, not a feature set.
Qluster is its own repository (quox-qluster) and its own service: a controller (Node/Express with a local SQLite registry and an SSE event hub, multi-operator session auth with role-based access control) with a separate React glass console, deliberately independent of any single QuoxCORE box's dashboard, auth service or vault.
It runs today on the operator's dev box under systemd, console on port 4200, with a real production box (nw-prod-01) enrolled alongside it.
NoteQluster is not a quox-dashboard module, not CTHULU (a shelved, separate idea for federating across different people's installs), not QCLOUD (which provisions, bills and decommissions hosted boxes; Qluster monitors and operates boxes that already exist, and owns nothing about money), and not Mesh/Matrix federation. Each is a distinct, non-overlapping product.
Boxes, the controller and the fleet registry Stable #
The registry is server truth: it lives in the controller's own SQLite database, refreshed by a server-side poller. The browser never polls a remote box directly, and a fresh browser with an empty cache shows exactly what the controller's database holds.
A jittered 60 second poll per box (5 second per-box timeout, backoff when unreachable) drives a status state machine: online → degraded → unreachable, recorded to the registry and appended to a heartbeat history.
NoteReachable-vs-unreachable semantics were fixed on 2026-07-09 after a real false unreachable reading against the production box. A box that answers at all, even a 502, is up and reads as degraded. Only a genuinely network-dead box (connection refused, timeout, DNS failure) reads asunreachable. A non-critical service being down on an otherwise healthy box does not drag the whole box to degraded.
A box can be muted (alerts_enabled=false): it shows a dimmed badge, is excluded from the fleet-health score, and its uptime is hidden while offline rather than showing a stale number. A box's last-seen timestamp turns amber once it goes stale.
Connecting a box Stable #
The console's connect wizard is capability-tiered, so an operator can add a box for read-only monitoring today and widen what it can do later, without starting over:
| Tier | Adds |
|---|---|
| Monitor | Health, status and version only |
| Full telemetry | Adds a read token for CPU / memory / disk gauges |
| Backups | Adds the backup.trigger and backup.download scopes |
| Let Qluster task it | Adds agent.delegate, the always-HITL governed delegation scope |
Each tier self-provisions what it can and states plainly what still needs a box-side action, typically a read token or a stable LLM key configured for that organisation.
Full enrolment issues a single-use token (15 minute TTL, stored hashed) that exchanges for a long-lived scoped bearer token plus pinned Ed25519 keys in both directions. From then on, every operational request is a signed envelope with a timestamp and nonce for replay protection, and revocation from the box side takes effect within 60 seconds. Scopes are bound to a specific organisation and fail closed.
# scopes: node.read, backup.trigger, backup.download
# no agent.delegate yet, would need a re-enrolment to add it
box nw-prod-01 https://nw-prod-01.quox.ai/collector
checks 4/4 green version 4.2.0NoteA real bug and its fix, for honesty: signed requests were originally signed against the full outbound path, but the box's nginx strips a/collector prefix before its own collector validates the request, so every signed call to a path-prefixed box failed canon verification with a 401. Fixed 2026-07-14 by signing the path the box itself will see; boxes with no path prefix were unaffected and needed no change.
The governed console Stable #
The Datacenter view lists every box as a full-width row with a live status and, where a box reports them, CPU / memory / disk gauges.
The Status Wall gives a NOC-style grid of RAG tiles that animate only on a state change, so a wall display does not flicker.
Selecting a box opens its detail: status, version, uptime, base URL and a per-service health matrix, with an unreachable box's last stored error shown inline rather than hidden behind a click.
Resource metrics (CPU, memory, disk, a 24 hour chart, sparklines, 14 day availability and daily rollups) depend on a small box-side addition, the collector's /api/v1/health/resources endpoint. A box that has not been patched with it shows an honest not reported rather than a fabricated gauge.
Alerting fires on a box leaving unreachable and resolves on recovery, to a webhook or a local beep, with per-box loudness (a production box can be loud, a demo box quiet) and muting honoured. Beta: alert dedup state is held in memory, so it does not persist across a controller restart.
A qluster CLI ships alongside the console, covering the same governed operations from a terminal: 14 command groups (status, boxes, box, delegate, checks, alerts, alert-ack, ask, backup, events, whoami, operators, passwd, qlogs). Live-verified 2026-08-06: qluster status returned the real five-box fleet with live CPU / memory / disk gauges and a health score.
Cross-instance checks and drift Stable #
Read-only fan-out checks run on demand or on a schedule, across every box in the registry:
- Version drift: flags boxes behind the fleet's dominant version, excluding boxes with an unknown version from the vote rather than skewing it.
- Config fingerprint: a sanitised per-box fingerprint, diffed and flagged on drift. Flag only; pushing config to a box is explicitly out of scope.
- Remote WARD chain verify: the controller re-runs the actual hash-chain verification math against a box's own WARD manifest. Nothing but hashes crosses the wire, so this works without any data-sovereignty concern.
- Backup freshness: reads each box's backup list and warns when its newest backup is older than policy.
Live-verified: a remote WARD chain verify returning PASS against the real production chain, which holds 772,000 entries. The check reads a capped 5,000 entries from genesis, a documented limit rather than a silent one; checkpoint-anchored tail verification of the full chain is future work.
Centralised encrypted backups Stable #
A backup pull is a defined sequence, not a raw file copy: trigger the box's own backup → wait, tolerating transient errors → stream the archive down → verify its sha256 against the box's own manifest → encrypt at rest with AES-256-GCM → store. The controller never extracts an archive; restoring stays a deliberate, manual act on trusted ground.
# a real pull from nw-prod-01
size 2.68 GB
sha256 matched # against the box's own manifest
at rest AES-256-GCM encryptednw-prod-01 now backs up on a real schedule: four runs a day at 00:00, 06:00, 12:00 and 18:00 UTC, seven day retention.
Known gapThe box-side backup subsystem itself is fragile, and that is a different stream's problem, not Qluster's. Getting the first clean pull required clearing box-side bugs along the way: a backup-directory ownership mismatch, a cleanup step that tried to remove files it did not own, a five minute timeout against a backup that genuinely takes around ten, and a NULL service-actor id in the audit trail.
Qluster pulls what a box produces, as-is.
Delegating a governed task Stable #
Delegation is always human-in-the-loop, unwaivable. Requesting a task (POST /api/boxes/:id/delegate) only ever creates a pendingrequest; nothing is sent until an operator approves it. On approval the request becomes a signed, scoped envelope, and the target box runs it under its own policy and its own HITL, exactly as if a local operator had asked. Qluster records the outcome; it never overrides a box's own governance to get one.
| Outcome | What it means |
|---|---|
completed | The box ran it and replied |
denied_by_box | A real policy verdict from the box's own governance, sovereignty working as designed, not an error |
box_exec_failed | A box-side runtime or credential fault, for example a stale LLM key for that organisation, not a policy denial |
pending_hitl | The box queued the request in its own inbox for its own admin |
A box can also require its own admin's approval for every delegated task, unconditionally, through its own "require approval for delegated tasks" setting.
Soaked 2026-08-06: 161 governed delegations driven end to end (create pending, approve or deny, send, terminal) against the demo Qluster estate, of which 140 completed, 20 were denied_by_operator at the approval gate, and 1 failed with a box-unreachable reason after the target box's process was killed mid-flight on purpose. Approve-step latency across 71 timed round trips ranged from 8ms to 491ms, median 29ms.
The run also drove failure and edge paths: a delegation to a box not enrolled for delegation was refused before any row was created; approving or denying an already-terminal delegation was correctly rejected with 409, leaving the original outcome untouched; ten delegations approved concurrently all resolved correctly with distinct ids and no interleaving; the killed box resumed normal delegation traffic once restarted.
The controller's WARD evidence chain grew from 93 to 699 entries across the run and still verifies PASS.
Known gapThe demo box used for that soak always answers a delegation with a synchronous completed reply, so the run could not produce a genuine box-side denial (denied_by_box), a box-side runtime failure (box_exec_failed), or a deferred pending_hitl outcome; those remain unproven at volume.
The full path through a real box's own execution still needs that box's collector rebuilt to carry the agent.delegate scope and its/delegate route; until then the scope gate correctly refuses delegation rather than pretending to run it.
nw-prod-01, the enrolled production box, is not currently scoped for delegation. pending_hitl outcomes are also not yet reconciled automatically: if a box queues a delegation in its own inbox, the controller shows pending_hitl but does not yet poll for the eventual verdict.
Colonel Qluster Beta #
Colonel Qluster is the fleet's meta-agent, a chat panel backed by an agent loop that lives on the controller itself, not inside any box. It is read-only: it observes and answers questions across the whole fleet, and it does not mutate anything. The only write path is the governed delegation flow above, which is always HITL.
Five read tools back it today: fleet stats, a single box's status, the recent check runs, recent alerts, and recent backup pulls. A sixth tool, the delegate action itself, creates the pending request described in delegating a governed task rather than acting directly.
Multi-provider and OAuth-first: by default it uses the operator's own Claude subscription, with no API tokens required, and a provider switch to the Anthropic API or OpenAI is available. Live-verified answering real fleet questions over the OAuth path.
Known gapThere is no rate limit on the chat endpoint yet, and a single call can involve several provider round-trips. That needs closing before any public exposure. Internally the agent is still named CaptainQluster in code and a few UI labels; Colonel Qluster is the current product-facing name and a full rename is outstanding, not a functional gap.
QLOGS evidence Stable #
Every controller action and every cross-box communication is captured as an AEE envelope: a spec-conformant record with a ULID id, an ISO timestamp, a typed intent, and acorr/reply_to pair that auto-threads to its causal predecessor on the same correlation, reviving the causality chain automatically. A conformance test asserts every AEE MUST across every call site that emits one.
Each envelope carries an AOCL layer trace (L1 identity, L3 policy, L7 dispatch, L8 outcome), and the honesty split is verified across every delegation outcome:denied_by_box records an L3 deny with an L8 sovereign verdict, whilebox_exec_failed records an L7 failure with no L3 entry, so a runtime fault is never mislabelled as a policy denial.
A per-correlation VOLT evidence bundle is metadata-only: payload keys and hashes, never values. Everything is witnessed onto a controller-local, append-only, content-free WARD hash chain; live chain verification returns PASS, and tamper detection, fail-soft behaviour and survival across a restart are all tested.
The query API filters by intent, action, correlation, box, actor, type, layer, priority and time, with cursor pagination, a cross-box correlation timeline that fails soft rather than ever 500ing, and a bounded NDJSON/CSV export (capped at 50,000 rows) for an auditor.
The QLOGS UI gives a filterable log table, an envelope detail view, a correlation-timeline modal, a layer-stack view with its own honesty notes, and a three-way WARD verdict:VERIFIED, UNVERIFIED (could not check, for example the box was unreachable) or FAILED (tampering). "Could not verify" is deliberately never rendered as "tampered".
The same query and export surface is reachable from the CLI. Live-verified 2026-08-06: qluster qlogs list returned real AEE envelopes carrying AOCL layer results and correlation ids, and qluster qlogs export --format csv produced real rows including the L1 layer decisions.
Coverage is close to total: the delegation lifecycle, enrolment and its failures, backups and refusals, checks, alerts, poll transitions, box add/patch/remove, operator login and logout, configuration changes and Colonel Qluster's own query/reply all emit envelopes.
Known gapQLOGS has no per-organisation scoping today: any authenticated operator session, admin, operator or viewer, sees every box's and every org's evidence. Role-based access now gates which actions an operator can perform (see Stable, beta and planned), but it does not yet gate which boxes' or organisations' evidence a session can see; that remains a hard gate before any multi-tenant use.
Colonel Qluster's individual tool calls are not yet enveloped, only its overall query and reply. A correlation spanning two boxes resolves evidence from only the first-resolved remote box.
After the 90 day AEE prune, a correlation timeline's WARD slice can render empty in that view even though the underlying WARD rows are preserved forever and remain reachable through the WARD query route directly, a timeline-view gap, not a data loss.
Stable, beta and planned #
This section mirrors Qluster's own status and limits documents, the honest maturity map kept with the code and updated at every audit. If the two ever disagree, the documents in the repository win.
Stable Verified and live-verified
- Fleet registry and health poller, server-truth, live-verified against real boxes
- Reachable-vs-unreachable semantics (502 up-but-degraded vs true network-dead)
- Datacenter console, node tree, fleet stats strip and Status Wall
- Muted boxes and stale-data display
- Resource metrics: CPU/mem/disk gauges, 24h chart, sparklines, 14d availability, daily rollups
- Cross-instance checks: version drift, config fingerprint, remote WARD verify, backup freshness
- Secure enrolment: single-use tokens, scoped bearer tokens, Ed25519-signed requests, ≤60s revocation, org-bound fail-closed scopes
- Remote encrypted backup pull: trigger, transient-tolerant wait, stream, sha256-verify, AES-256-GCM at rest
- Agent provider settings: keys encrypted in the controller's own database
- QLOGS: AEE emission, AOCL layer trace, VOLT bundles, the controller WARD chain, the query API and correlation timeline, the QLOGS UI, and near-total action coverage
- A
qlusterCLI: 14 command groups (status, boxes, box, delegate, checks, alerts, alert-ack, ask, backup, events, whoami, operators, passwd, qlogs), live-verified 2026-08-06 against the real fleet - Governed delegation, controller-side: create-pending/approve/deny/send state machine, sovereign scope refusal, unreachable-box handling, concurrent delegations. Soaked 2026-08-06: 161 round trips against the demo estate (140 completed, 20 denied by operator, 1 failed on a deliberately killed box), plus edge and failure paths, zero defects
- Alerting: unreachable/recovery to webhook or beep, per-box loudness, muting honoured; alert lifecycle (fired, escalated, acknowledged, resolved) and the miss-streak counter are persisted in the database, so a controller restart mid-outage no longer fabricates a recovery or loses the dedup state, fixed 2026-07-27
- Multi-operator controller auth: role-based access (admin, operator, viewer), session cookie, rate-limited, LAN-only posture. Live-verified 2026-08-06: a viewer-role session was refused a privileged call server-side ("role "viewer" cannot operators.manage")
- Colonel Qluster: read-only, five fleet tools, multi-provider (Claude-OAuth default, Anthropic API, OpenAI). The OAuth-path deadlock was fixed 2026-07-27 and live-verified with three consecutive real fleet answers
- Cross-box evidence display in the delegation panel:
qlogs show <corr>replays an 11-envelope timeline across controller and box for a real delegation, with the VOLT bundle and WARD entries
Beta Working, live-verified once, still hardening
- Governed delegation, box-side execution: a real box's own policy denial (
denied_by_box) or runtime failure (box_exec_failed) has not been observed live; needs a box-side collector rebuild and a scoped real box
Planned Designed or roadmapped, nothing further shipped
- The delegation green path proven end to end against a real box
- Per-organisation scoping on QLOGS routes
- A hardened, public, multi-tenant
qluster.quox.aideployment - Automatic reconciliation of
pending_hitldelegation outcomes
Known gap Documented limitations, not softened
- WARD chain verify caps at 5,000 entries from genesis on a large real chain; full-chain checkpoint-anchored tail-verify is future work
- The box-side backup subsystem is fragile (a different stream's known issue); Qluster pulls what a box produces as-is
- In-memory replay-nonce state is lost on every controller restart (alert-dedup state was fixed 2026-07-27 and is now persisted)
- Colonel Qluster's chat endpoint has no rate limit yet
pending_hitldelegation outcomes are not reconciled automatically- QLOGS has no per-organisation scoping; any authenticated session sees every box's evidence
- A correlation spanning two boxes resolves evidence from only the first-resolved remote box
- Standalone glass CSS is a copied snapshot of the dashboard's design tokens and will drift; a shared package is future work
- Public deployment is not ready: LAN/VPN-only posture until a dedicated hardening stream lands
Test coverage #
The controller's own suite runs 181 tests covering the fleet registry, poller state machine, checks, enrolment, backup pull and delegation (Phases 0 through 5). QLOGS was closed separately with its own 404 controller tests, including a dedicated AEE-conformance suite, plus UI unit tests for the QLOGS query library. Box-side additions (the enrolment and delegate routes a box must expose) are tested inquox-dashboard's own collector and auth test suites.
QLOGS closed after a three-dimension adversarial audit, protocol conformance, security, and correctness/UI, whose blocking findings were fixed before close: a cross-org delegation gap (a controller scoped to one organisation could target another; now mandatory and membership-checked), a mislabelled 401 error code, and a capped CLI output stream to prevent an out-of-memory condition on a runaway subprocess.
WarningStated plainly, what this corpus does not verify: a public-facing security posture (Qluster has not been through the dedicated hardening pass a public qluster.quox.ai would need, seeSECURITY_AUDIT_HALFTIME.md in the repository for the full checklist), or the real end-to-end delegation green path against a box whose collector has not yet been rebuilt.
Operating notes #
The controller runs under systemd (qluster-controller.service) on the operator's dev box, enabled to survive a reboot and set to restart on failure.
QLUSTER_MASTER_KEY= # required; decrypts stored box read/enroll tokens
QLUSTER_ADMIN_PASSWORD= # bootstraps the admin operator; more via 'qluster operators add'
QLUSTER_PORT=4200
QLUSTER_DB_PATH= # SQLite; persistent across restarts
QLUSTER_POLL_INTERVAL=60 # seconds, jitteredWarningThe server has no dotenv of its own; environment comes from the systemd unit's EnvironmentFile. Starting it by hand with a barenode index.js boots without QLUSTER_MASTER_KEY, so login fails and stored box tokens silently cannot be decrypted, breaking checks and backup pulls without an obvious error at the point of failure. Always manage it throughsystemctl; source the .env file by hand only when deliberately debugging.
The registry database persists across restarts; in-memory state, the replay-nonce store and the alert dedup window, does not, which is acceptable but worth knowing before a restart during an active incident.
Public deployment is not ready. Qluster today is a LAN/VPN-only, multi-operator (roles enforced server-side, not per-org scoped) posture; a dedicated hardening stream (egress policy, secure cookies, a shared session and nonce store, response headers) is required before any internet-facing or multi-tenant deployment.