Comparing AI Memory Systems: From QMD to Mnemosyne on a CPU-Only Server

By

Why I looked for a new memory backend

My Hermes Agent setup runs on a headless Ubuntu server with no usable local GPU. The existing long-term memory backend, QMD, is a capable semantic-search engine, but it embeds markdown with GGUF models through node-llama-cpp. On a CPU that becomes the bottleneck: every re-indexing batch and every hybrid query carries the overhead of loading and running local embedding and reranking models.

I wanted something that could:

  • keep search fast on a CPU,
  • still offer semantic recall when needed,
  • integrate natively with Hermes,
  • and optionally offload embedding compute to a remote Ollama box.

The candidates

I evaluated three systems:

  • QMD — the incumbent. SQLite + BM25 keyword search, optional dense embeddings and LLM reranking via local GGUF models.
  • MemPalace — a heavily benchmarked palace-structure memory system (ChromaDB + small embeddings, knowledge graph, compression). It targets AI-agent retention hooks rather than arbitrary markdown knowledge bases.
  • Mnemosyne — a Hermes-first memory provider. SQLite-backed, FTS5 keyword search by default, optional embeddings through fastembed or a remote OpenAI-compatible endpoint.

How I tested

I used a mix of representative notes: a system configuration file, a user-profile file, a contact list, and several long-form markdown creative documents. Names, addresses, and phone numbers in the contact sample were replaced with placeholders before any reporting.

For each candidate I measured:

  • installation footprint,
  • time to store a memory,
  • time to recall a memory,
  • local RAM usage,
  • and recall quality for keyword and semantic queries.

Key results

System Local RAM Store latency Recall latency Notes
QMD (local GGUF) Moderate–high Slow on CPU Slow on CPU Best hybrid quality, but CPU-bound
MemPalace ~800 MB–1.5 GB Embedding-bound Fast after embedding Agent-retention focused, heavier for a raw markdown archive
Mnemosyne (no embeddings) ~84 MB ~0.2–0.4 s ~0.03–0.06 s Keyword-only, very fast, lower semantic recall
Mnemosyne + remote Ollama ~87 MB ~0.5–2.8 s ~0.1–1.3 s Semantic search offloaded to GPU; local footprint stays tiny

What the numbers mean

QMD and MemPalace both do good work, but they are embedding-first. On a CPU-only box the embedding step dominates every operation. Mnemosyne’s default mode avoids that entirely by using SQLite FTS5. Search is genuinely sub-second and RAM stays under 100 MB.

The trade-off is recall quality. In pure keyword mode, Mnemosyne can rank a profile note above a note that is literally about the queried topic, because keyword frequency wins over semantic intent. Enabling remote embeddings fixes that: queries like “QMD memory” returned the correct document first, and music-related queries surfaced the right creative files.

Migration and cleanup

The old QMD collection had grown to about 244 files and 4.1 MB, dominated by multiple drafts of two novels and their compiled full manuscripts. After reviewing an inventory I removed the duplicate and draft material, keeping only a small set of creative notes. The remaining files were migrated into Mnemosyne as global-scope memories with Ollama-generated embeddings.

I then:

  • installed mnemosyne-hermes into the active Hermes virtual environment,
  • linked the Mnemosyne plugin wrapper into ~/.hermes/plugins/mnemosyne,
  • set memory.provider = mnemosyne in Hermes config,
  • added the Ollama embedding environment variables to the Hermes gateway systemd service,
  • and removed QMD’s markdown tree, index, and binary symlinks.

Final configuration

The gateway now starts with these environment variables baked into its systemd unit:

MNEMOSYNE_DATA_DIR=/home/jess/.hermes/mnemosyne
MNEMOSYNE_EMBEDDING_API_URL=http://192.168.X.XXX:11434/v1
MNEMOSYNE_EMBEDDING_MODEL=nomic-embed-text:latest
MNEMOSYNE_EMBEDDING_DIM=768
MNEMOSYNE_LLM_ENABLED=false

This keeps all memory data local in SQLite while sending only embedding requests to the GPU box. A live test confirmed the provider is active and recall works: a newly stored memory was retrieved correctly by semantic query.

Bottom line

For a CPU-only Hermes deployment that still wants semantic memory, Mnemosyne with a remote Ollama embedding endpoint is the sweet spot. It preserves local speed and privacy, keeps RAM tiny, and only uses remote compute for the parts that actually need a GPU. QMD and MemPalace are better suited to machines that can run embeddings locally without pain.