QuoxDream

A scheduled pass that turns raw Brain2 notes into confidence-scored, deduped memory, the difference between a chunk-search index and a second brain that actually learns.

Brain2 stores raw notes: conversations, transcripts, git history, connector documents, chunked and searchable. That is retrieval, not judgement. QuoxDream is the server-side pass that reads those notes and asks a narrower question of each one: what is actually worth remembering long-term, as a standalone, confidence-scored belief in QuoxMemory?

It was written to close a named gap in the second-brain design: without it, "we are a chunk-FTS RAG with a good name" (the second-brain lens document that scoped this build). QuoxDream is what turns that chunk index into consolidation.

What it does

On each scheduled run, QuoxDream:

1. Reads Brain2 notes written since the last run for a given tenant slice, in strict order, never re-processing and never skipping.
2. Sends each note's text to a strict fact-extraction prompt (via the collector's /chat/complete seam), asking for a JSON array of durable beliefs: facts, preferences, decisions, entities, constraints, or observations, each with a confidence score and an importance tag.
3. For every extracted claim, checks QuoxMemory for a near-duplicate before writing. A duplicate is skipped (noop); a genuinely new claim is written with its confidence, a stability tier, provenance marked model_inference, and a source pointer back to the originating note.
4. Advances a durable cursor so a service restart never reprocesses old notes.

Extraction is best-effort throughout: a malformed completion, a timed-out request, or a bad note is logged and skipped rather than aborting the run or crashing the service.

The cursor and batch model

Two tables carry the state (migration 005-dream.sql, additive-only):

• dream_cursor (one row per (org_id, scope, owner_user_id) tenant slice): holds last_occurred_at and last_note_id. Notes are read in strict (occurred_at, id) order strictly after the cursor, so the pagination is deterministic even when two notes share a timestamp.
• dream_runs (one row per consolidation run): records notes read, beliefs written, beliefs deduped, and any error, for observability. Not yet surfaced anywhere: no dashboard reads this table today.

A single pass works through bounded batches (consolidateSlice), advancing the cursor after each batch rather than only at the end, so a crash mid-run loses at most one batch of progress, not the whole run. consolidateOrg runs this for both an org's own notes and the instance-wide shared-scope notes; per-user-scoped consolidation is deliberately deferred and not built.

Dedup: Jaccard first, semantic second

The default dedup check is word-overlap (Jaccard similarity) between a new claim and the top search results for its content, the same technique the collector's memory-consolidation scheduler already uses, so "near-duplicate" means the same thing across both consolidation paths.

An optional second pass, gated by QUOXDREAM_SEMANTIC_DEDUP, embeds the claim and compares it by cosine similarity against the same candidate set. This only ever adds a second chance to catch a paraphrase Jaccard missed (for example "the org migrated its datastore to Postgres" versus "the team decided to use Postgres"); it never overrides a Jaccard match, and any embedder failure falls back cleanly to the Jaccard-only result. With the flag off, behaviour is byte-identical to the Jaccard-only path.

Env flags

QuoxDream ships dark. Nothing runs until both of the first two are set:

Visibility gap: every flag above is an environment variable only. There is no dashboard toggle, no Settings surface, and no read of dream_runs to show a run history. Documenting the flags here does not close that gap; it names it.

Maturity

Related

• Memory System: the QuoxMemory store QuoxDream writes beliefs into.
• Brain2: the second-brain plugin QuoxDream consolidates notes from.
• Schedulers: the other server-side background passes QuoxDream's scheduler shape is mirrored from. </content>