The top floor of my memory pyramid is empty on purpose

A diagram has been going around lately that draws AI memory as a pyramid. Raw material at the bottom, summaries in the middle, an abstract gist at the top, and the agent walks down only the branch it needs instead of hauling the entire transcript into every question.

None of it is new. People have been building hierarchical memory this way since MemGPT in 2023. What got my attention was that I looked at the picture and recognized my own box, floor by floor, including the floor I never built.

So, this is the third and last of three posts about that setup. The first two are already up and they are the floors underneath: the retrieval side and the loading side. This one is about the part I decided not to do, and why.

The floors I do have

The bottom is everything on disk. Memory notes, session transcripts, the running log of my chats with the assistant, the notes vault, the mail archive. Nothing gets summarized away and nothing gets thrown out. It sits there being large and useless until somebody asks for it.

The middle is retrieval. A small embedding model turns every passage into a few hundred numbers, a nightly job keeps that current without re-reading the world, and a query comes back with the passages that mean roughly what I asked instead of the ones that happen to share my words. That is the search-by-meaning post.

Above that is the part that actually loads on every turn: an index, one line per note with a path, plus a small always-on core of standing rules and whatever threads are half open. Around 35KB together, down from 219, and everything else waits on disk, one lookup away. That is the 200KB post, where I found out I had spent two months pushing a 219KB payload into a 2KB slot and getting almost none of it delivered.

Three floors. It has been running for weeks and I mostly do not think about it, which is the whole point of it.

The floor above that

What the pyramid calls the top is conclusions, not more notes.

Everything I have is either a pointer or a single fact. This port, that rule, this decision on that date. A gist layer sits above all of that and boils a pile of small facts into one sentence that no individual note contains. A verdict about the person it works for, the kind you only reach by watching a dozen small events and drawing a line through them. No note says it, and no note can confirm it.

The appeal is easy to see. Right now, if I ask the assistant something general about how I work, it goes and reads twenty things and infers it live, every single time. With a gist layer that inference already happened, once, overnight, and it just reads the answer. Cheaper, and potentially better, because a conclusion drawn slowly at 4am beats one drawn in the two seconds before a reply.

So why is that floor empty, you wonder?

A wrong fact is loud. A wrong conclusion is silent.

A wrong fact gets caught. If a note says the reindex runs at 3am and it actually runs at 4, the first time it matters I notice, I fix the line, done. Facts are checkable against something outside themselves.

A wrong conclusion is not checkable against anything. It is an opinion with no receipt. If a summarizer reads three evenings where I was in a hurry and writes down that I take the quick option over the correct one, that sentence is now in front of the model on every future answer. It never announces itself. It just tilts things a few degrees. Recommendations start coming back slightly wrong and I have no idea why, and neither does the assistant, because by then that sentence is furniture.

Then it compounds. Tomorrow's conclusions get drawn from a pile that already contains yesterday's, and nothing in the loop ever goes back down to the raw material to check the premise. The property that makes the layer worth having, that it saves you from re-reading the evidence, is exactly the property that makes a bad entry impossible to spot.

That is a strange kind of bug to invite into a system on purpose. It has no error message and no stack trace, it degrades slowly, and the thing it degrades is how well the assistant understands the person it works for.

What would have to be true

I am not against ever building it. Three things would have to hold first.

Every conclusion carries links back to the specific facts it came from, so I can open one and look at the evidence myself. Nothing enters the layer without me reading it first. And any entry can be pulled, with everything drawn from it rebuilt.

A summarizer on a timer does not give you any of that. What does is a review queue with an audit trail, sitting between the model and its own opinions about me, and that is a lot more work than the diagram makes it look. It is also, I suspect, why nobody has this part nailed yet. The easy version is a cron job. The version you can trust is a small product.

The part I already have, built by hand

Here is the twist I did not expect when I started looking at it this way. That small always-on core, the standing rules, is a gist layer. "Do only what was asked." "Ask before changing anything about how data is stored." Those are not facts. They are conclusions, and they are exactly the kind of sentence a summarizer would try to produce.

Not one of them was written by a model. Every one exists because the assistant got something wrong, I said so, and the correction got written down as a rule. Slow and manual, and every line has a real event behind it that I still remember. When I want to go through them they are plain markdown on the box, which I get at the same way I get at everything else there, SSH'd in from TerminalNexus.

So the top floor is not empty after all. It is small, it fills at the speed of me noticing something, and a human wrote every line in it. That is a terrible growth rate for a data store and the right one for a layer whose job is to have opinions about me.

I will build the automatic version the day I can audit it. Until then I would rather have three good floors and a hand-written attic.

Thanks for reading. If you have built the gist layer, or you have watched one quietly go wrong, I would like to hear how you caught it. The comments are below.