When Your AI Remembers Everything

This morning I published an article about context engineering and memory in AI agents. In it, I audited my own memory system and found a painful gap:

My agent remembers what we decided but not what we discussed.

The problem was architectural. My memory system stored structured observations — decisions, configurations, bugfixes — but user prompts and agent replies simply vanished. Every conversation evaporated the moment it ended. All the reasoning, the implicit preferences, the context that never got formalized into a "decision" — gone.

Hours after publishing, that gap closed.

The missing piece: automatic conversation persistence

The openclaw-memory-engram plugin — now open source on GitHub — hooks into OpenClaw's lifecycle at two critical points:

before_prompt_build  → saves the user's message as a "prompt" observation
llm_output           → saves the agent's reply as a "conversation" observation

Two hooks. One architectural change that shifts the entire memory model from opt-in to automatic.

Before, the agent needed to decide to call engram_save manually. Agents are good at many things, but they are terrible at remembering to log their own conversations. The result was predictable: structured decisions were saved, but every prompt and every reply disappeared.

Now every message, every response, every turn is persisted automatically with topic-based deduplication. Each session gets one user prompt observation and one agent reply observation, updated in place rather than duplicated.

Why this changes everything

The biggest cost of stateless agents is not the token budget. It is the lost reasoning. When you say "let us not use Coolify because Pavel's permissions are an issue," that is not a decision you will find in a config file. It is a conversation fragment that explains why a perfectly reasonable option was rejected.

With automatic persistence, that fragment lives in FTS5, indexed and searchable. The next time the agent evaluates deployment options, it finds not just the decision ("use Docker Swarm") but the reasoning behind it. "Use Docker Swarm" becomes a conclusion. "Pavel cannot access Coolify" becomes the evidence.

Preferences that never get formalized as "decisions" are the hardest thing to preserve. "I prefer deploying Friday nights." "Don't touch production before 10 AM." "Use the green theme, not the blue one." These rarely get saved as observations because they feel too informal. With automatic persistence, they land in the database without anyone remembering to log them.

A search for "deployment" used to return three structured observations. Now it returns the decisions and the conversation where options were debated, the follow-up where a problem was discovered, the email draft where the choice was explained. The structural observation tells you what was decided. The conversation tells you why.

How it works under the hood

The plugin uses OpenClaw's hook system — specifically the before_prompt_build event — to intercept every inbound message before the agent processes it:

api.on("before_prompt_build", async (event, ctx) => {
  const userMessage = extractUserMessage(event.prompt);
  await client.save({
    session_id: ctx.sessionKey,
    title: userMessage.slice(0, 100),
    content: userMessage,
    type: "prompt",
    topic_key: `session/${sessionKey}/user`,
  });
});

The llm_output hook does the same for the agent's reply. Deduplication is handled by Engram's topic_key mechanism — saving with the same key updates the existing observation instead of creating a duplicate.

Fan-out search: why one query is not enough

FTS5 uses implicit AND by default — a query like "deploy Coolify permissions" requires all three words in a single memory. But a real conversation about deployment is fragmented across multiple turns. "Coolify" appears in one message. "Permissions" in another. "Deploy" in a third.

Fan-out solves this by running parallel searches for each extracted keyword, then merging and re-ranking the results. BM25 relevance, keyword hit breadth, recency decay, and title match all feed into a composite score. The result is effective OR semantics with multi-keyword precision: "Coolify" alone returns broad results, but "Coolify AND permissions AND Pavel" floats to the top.

Pointer-based recall: keeping context lean

Auto-recall does not inject full memory content into the agent's context. It injects pointers: the title, type, project, and relevance score. About 120 characters per memory. The agent calls engram_get with the #ID only if it needs the full content. Progressive disclosure — the agent knows what memories exist without consuming context budget on details it may not need.

The real test: does the agent use it?

The auto-recall injected the first saved conversation into context within hours of deployment. I asked: "What was that thing about deployment that you said yesterday?" The agent responded with the exact turn from the previous session. Not a structured decision I had manually saved. The actual conversation fragment.

That is the moment when memory stops being a database and starts being a relationship.

From stateless to stateful in one PR

The distance between "my agent forgets everything" and "my agent remembers every conversation" is not a new model, a bigger context window, or AGI. It is two hooks, one topic_key, and a fan-out search algorithm.

The openclaw-memory-engram plugin is public on GitHub. It connects OpenClaw to Engram — a Go-based memory server using SQLite and FTS5. The combination costs nothing beyond the server hardware you already have.

Context engineering is not about building bigger memories. It is about automating the capture of memories you did not know you needed — until the moment your agent recalls them.

← Back to Blog