Building Agent Memory That Survives the Next Session

Spend an afternoon with an AI; the next day it’s blank. Agree on something in one chat app; start over in another. Hit compaction and last week’s architecture call evaporates. I’ve hit this enough times.

The root cause fits in one line: a context window is not memory. The window is a workbench. Close it or crush it and it’s gone. Real memory has to hit disk.

The rule I settled on is blunt:

If it isn’t in a file, it never happened.

Three layers are enough

No fancy middleware first—just the filesystem:

Short-term is NOW.md. Workbench, overwrite allowed, refreshed on heartbeat. What’s in flight today lives here.

Mid-term is the daily log memory/YYYY-MM-DD.md. Append only. Sessions can’t see each other’s transcripts, so important calls go into the log immediately. Entries look like ### HH:MM - Title so they’re easy to scan.

Long-term is the knowledge base: decisions/, lessons/, people/, projects/, preferences/, and so on. Distilled knowledge, not a raw dump. Boot by reading INDEX.md, then open files on demand.

Below that sits .archive/. Cold data, skipped by default search. Logs, reflections, and finished tasks move in on a schedule; decisions, core lessons, and user profiles I keep forever.

Flow: dialogue → daily log → nightly reflection into the knowledge base → stale stuff to cold storage. Each step cleaner than the last.

A layout that works for me (trim as you like):

workspace/
├── NOW.md
├── AGENTS.md
├── HEARTBEAT.md
└── memory/
    ├── INDEX.md
    ├── YYYY-MM-DD.md
    ├── memlog.sh
    ├── memory-gc.sh
    ├── decisions/ lessons/ people/ projects/ preferences/
    ├── reflections/
    ├── actions/{open,in-progress,done}/
    └── .archive/

Why Markdown keeps winning

Not ideology. Humans can open the files, git diffs them, agents already know Read/Write—debugging is less painful. A database can store memory too; I want to see what the agent thinks it knows.

Retrieval is separate. Markdown as source of truth, vectors or FTS as accelerators, has worked well enough. Chinese full-text on SQLite FTS5 is often awkward; fuzzy questions lean on semantic search, which is slower but better than missing hits. The exact search stack (I’ve used QMD) deserves its own post.

Reconcile before you write

Agents hallucinate into memory too: invent events, write wrong facts, fork conflicting versions, skip what mattered. So writes aren’t “append when inspired.” Route first, read the old file, compare, then classify:

ADD / UPDATE / NOOP / CONFLICT

On conflict, keep both and mark them for a human. My rule: ugly beats silent overwrite.

Routing priority, roughly: strategic decisions → reusable lessons → people → projects → preferences → daily log → skip the noise.

Forget on purpose

After about a month I had on the order of a hundred Markdown files. Without cleanup, search gets noisy and INDEX.md gets heavy. Brains forget for a reason—perfect recall isn’t always a feature.

Archival is boring: logs and reflections older than ~30 days with no references go to .archive/; done tasks around 14 days; decisions and profiles stay.

I use a temperature score to weight retrieval, not to replace hard GC rules. Intuitively, newer + more referenced + higher priority → hotter. Early drafts that made “higher age → hotter” had the sign wrong; use recency, then layer references and priority, e.g.:

T = 0.5×recency + 0.3×refs + 0.2×priority
High → boost retrieval; low → down-rank and let GC consider archive

Tune the weights; don’t worship the formula. What actually deletes files is still “expired and unreferenced.”

Don’t vector-search first

My real order:

1. Scan INDEX.md if you know the category—near zero cost
2. Read the file if you know the path
3. Semantic search only when you don’t know where to look (latency; not the default)

Most queries die at step one or two.

Logging can stay tiny:

./memlog.sh "Code review done" "Reviewed bf-gateway-service PR #42; three issues"

Questions I get a lot

Multiple agents? Separate memory/ trees; a main agent aggregates via heartbeat. Don’t share one stompy directory.

Fit for everything? Multi-session work, long-lived assistants, team agents—yes. One-shot Q&A—overkill.

Cost? Files are tiny; indexes often tens of MB. Money can stay low; time goes into rules and upkeep. Automate nightly reflection and weekly GC so you’re not relying on “remember to tidy.”

One line to close

I don’t need a smarter model as much as an agent that can wake up and continue yesterday’s decisions. Files as the ledger, checks as reconciliation, archival as decluttering—plain, and more reliable than hoping the context window will remember forever.