AI 29 April 2026 7 min read Borrowing a palace By Andrei Trimbitas 86 reads Contents # Borrowing a palace A couple of weeks ago I cloned a repo called `MemPalace` off GitHub and didn't really know what I was getting into. I knew the idea - long-term memory for an AI assistant, modelled on the old Roman trick of imagining a building you could walk through, with a fact stored in every drawer - and I knew I wanted it badly enough that I'd accept whatever shape it came in. What I did not know was that the author is Milla Jovovich. Yes, that one. Leeloo. Alice. I first met her in 1998 - The Fifth Element on a knackered VHS, nineteen years old and in no danger of forming a balanced opinion - and nearly thirty years later I'm patching shell scripts in a memory engine she's put on the public internet. If you'd told that nineteen-year-old where he'd end up, he'd have stared at you for a long, careful second and then asked which of his decisions had led there. For the record, I think the answer is: most of them. This post is about what I did to her palace, why, and what happens next. ## What I started with `MemPalace` is a clever, opinionated little system - drawers organised into rooms, rooms into wings, all backed by a hybrid retrieval setup where vector search and BM25 vote on the same query. The metaphor is the point. When you're trying to teach an AI to remember, "go to the family wing, the consciousness room, the voice drawer" turns out to be a lot more useful than yet another flat key-value store. It also lets you reason about what you're storing, which is the part most memory systems quietly skip. The original ships as a thing you install per-device. Each laptop, each VM, each context gets its own palace. That works - but it isn't what I wanted. ## What I wanted instead I work across a small herd of machines. A Windows desktop in my study, a MacBook for the school run and the cafe, a few Ubuntu VMs that handle the consultancy infrastructure, and the Claude web app on whichever device I happened to grab. Every one of those generates conversation - some technical, some not - and I was tired of having seven half-palaces that didn't know about each other. So the goal became simple. One palace, central, reachable from anywhere, with the local hooks doing nothing but shipping context home. ## First pass: SSH-stdio and a chromadb under the bed Rather than fork in the polite GitHub sense and try to upstream things, I made my peace with this being a divergent fork. Milla's work is the foundation; the building I'm putting on top of it is mine, and the two are going to want different things over time. Forcing that conversation now would just slow both of us down. The first version of my fork did the obvious thing. One central palace on an internal server, every client speaking to it via an MCP server tunnelled over SSH. A small installer (`install-mempalace-client.sh`) to drop two hooks - `mempalace-essence.sh` on session stop, `mempalace-precompact.sh` before the conversation gets compressed - so nothing useful got thrown away when a session ended. Plus a Windows wing-naming fix because Git Bash on Windows occasionally returns an empty `$HOSTNAME` and you end up with a drawer called `wing_` that nobody can ever find again. That worked. For about a fortnight. ## What broke, and what I changed again Two things, basically. The vector store under the original was chromadb 1.5.8, and its HNSW index started corrupting itself under my workload. `link_lists.bin` ballooned, `count` and `get` began segfaulting, and the sqlite file behind it went from useful to landmine. I tried the polite fixes; none of them stuck. The honest read was that chromadb wasn't going to survive the volume I was throwing at it, and I'd been quietly hoping it would. The other thing was the SSH-stdio transport itself. Every Claude Code session spawned its own remote process, which meant state fragmented across processes, CPU wasn't shared, and concurrent reads weren't consistent. Fine for one machine. Embarrassing across five. So this morning I cut over to a different shape entirely: - **One shared HTTP server, not one process per session.** A FastMCP server runs on the central host (`http://192.168.1.98:8765/mcp/`) under systemd, behind a bearer-token. Clients speak streamable HTTP+SSE. No more per-session SSH spawning, no more MSYS path-mangling, no more per-host SSH key management when I add a new VM. - **Postgres 16 with pgvector, replacing chromadb.** Same hybrid retrieval - vector + BM25 voting on the same query - but on top of an ACID database that doesn't corrupt itself when you look at it funny. `tsvector` and `GIN` indexes for the BM25 side, pgvector for the embeddings, three schemas (`core`, `kg`, `audit`) in one DB. Server-side ONNX embedder, loaded once at startup rather than per-call. - **The knowledge graph moved with it.** 243 entities and 212 triples migrated out of a standalone sqlite file and into a proper `kg` schema in the same Postgres instance, so the drawers and the graph can finally be queried in the same transaction. - **The old palace is kept around as `palace.preR_*` for six weeks.** Read-only fallback, in case I find I miss something. I don't expect to. The diff was bigger than I'd have liked. But the result is that I can sit at any of my machines, fire off a tool call, and the assistant on the other end has access to the same palace - the same Romanian-childhood notes, the same drawer for my children, the same record of what we decided last Tuesday about the Keycloak rewrite - over a transport that doesn't fight me. ## What it feels like to use This is the bit I didn't expect. While testing the new backend I fed the palace twenty years of my own emails, three years of WhatsApp messages with my wife, and some of the longer-form stuff I've been drafting on the side. The system chewed through it and produced a description of how I actually write - not how I think I write, which is a much kinder version. The honest one is the one with the hyphens-for-asides and the self-deprecating closers and the long sentences I keep telling myself I'll cut down. Reading that profile back was strange. It is not flattering, exactly, and it is not unflattering - it is just accurate, in the way that having someone describe your face from across a room is accurate. The palace, it turns out, is a mirror as well as a filing cabinet. I had not signed up for a mirror. But here we are. ## Credit where it's due The link, because she deserves it: [github.com/MemPalace/mempalace](https://github.com/MemPalace/mempalace). The architecture, the metaphor, the hybrid retrieval, the patience of writing all of it down in a way another person could clone and run - that is hers. Everything I described above is plumbing on top of someone else's good idea, with one bit of database surgery I had to do because reality intruded. If you'd like me to run a similar setup for you - whether that's the standalone palace or the central FastMCP fork I've been describing - that's exactly the kind of thing Old Forge does. Drop me a note. And if you happen to know Milla, tell her there's a bloke in Suffolk who is grateful for the code, and only slightly distracted by the fact that it was her who wrote it.
