AI Agent Memory Architecture: What I'd Build Before a Graph

AI agent memory architecture is the system around a stateless model that decides what gets stored, retrieved, updated and forgotten. In its architecture guide, Cognee puts the write path, temporal validity and deletion alongside retrieval rather than treating memory as another vector index. My takeaway: the useful change is not a model that remembers, but making memory an explicit application responsibility.

For the AI-assisted editorial workflows I work on, I'd start with that distinction. I would not want an assistant to treat an earlier editorial decision as current simply because the old conversation remains easy to retrieve.

This is an architecture guide, not evidence that a particular memory engine will improve my application. I'd use it to identify missing responsibilities before choosing another database.

The memory architecture I'd draw before choosing storage

The Cognee guide separates working memory from three persistent memory types. Working memory is the current prompt: instructions, recent turns, tool results and injected memories. The surrounding system decides which persistent information earns space there.

• Episodic memory: messages, tool traces and session summaries.
• Semantic memory: facts, entities and relationships, including when those facts are valid.
• Procedural memory: instructions, skills, tools and retrieval weights.

I find this split more useful than starting with a vendor diagram. It forces me to ask whether I need a past conversation, a current fact or an instruction. Those are different retrieval jobs, even if I initially store them in the same system.

The guide also draws a boundary between read-only retrieval over a corpus and read-write agent memory. Its point is that retrieval alone leaves extraction, conflict resolution and forgetting unspecified. I would keep that distinction in the design document without getting stuck arguing over whether every implementation deserves the label RAG.

Building an AI agent memory architecture incrementally

Start with the questions memory must answer

I'd write a small set of representative questions before adding storage. Can the assistant recover a previous decision? Identify the current owner of something? Explain what was believed at an earlier point? Follow a reusable instruction?

For a newsroom CMS, I would test the difference between recovering an earlier editorial discussion and retrieving the currently approved instruction. I would want those answers to carry different evidence, not just differently ranked chunks.

I'd also keep a no-memory baseline. The guide recommends evaluating task success with and without memory, retrieval precision and p95 latency. My adoption decision would depend on those results, not on whether a demo remembers my name.

Make writes deliberate before making them automatic

The source describes several write triggers: every turn, the end of a session, an explicit remember action or a background job. It then separates extraction, entity resolution, reconciliation and persistence.

I'd begin with explicit writes and end-of-session extraction for a narrowly defined workflow. That is my preference for keeping the first implementation inspectable, not a claim that these triggers are universally better. Before enabling broader extraction, I'd want to inspect what the system considers durable.

Each candidate fact would need its source and timestamp. I'd define which identifiers establish entity identity, and what happens when two mentions might refer to the same entity. I would not let fuzzy matching silently become the authority on identity.

Preserve changes instead of silently overwriting them

The guide's reconciliation policy is to supersede facts with validity windows. An earlier fact receives a valid_to value; its replacement receives valid_from. Both remain available for historical queries.

I'd choose that policy for facts where history matters. I would still distinguish superseding a fact from deleting it: the former preserves history, while the latter must remove the relevant information and derived artifacts across stores.

Here is a small TypeScript example of the eligibility check I'd test. It is my application code, not a Cognee API example. It assumes already validated timestamps and uses an inclusive start and exclusive end.

type MemoryFact = {
  id: string;
  tenantId: string;
  sourceId: string;
  text: string;
  validFrom: number;
  validTo: number | null;
};

function eligibleFacts(
  facts: readonly MemoryFact[],
  tenantId: string,
  asOf: number
): MemoryFact[] {
  return facts.filter((fact) =>
    fact.tenantId === tenantId &&
    fact.validFrom <= asOf &&
    (fact.validTo === null || asOf < fact.validTo)
  );
}

I would test the exact replacement boundary, historical queries and cross-tenant records. This helper is deliberately not an authorization system. The source calls for storage-layer scope enforcement; I would apply that before retrieval and retain application checks as a separate safeguard.

Route reads, then enforce a context budget

The guide's read path rewrites and classifies a query, retrieves candidates, ranks them and trims them before prompt injection. It combines vector search, keyword search and graph traversal rather than sending every question through similarity search.

I'd introduce those routes according to the questions established earlier. Exact identifiers would get a lexical path. Historical questions would carry an explicit time constraint. Relationship questions would be my reason to evaluate graph traversal, not an excuse to add a graph beforehand.

Every injected memory would carry provenance. I'd reserve a fixed context budget and define what gets dropped first. I would also inspect the final injected context during evaluation; looking only at retrieved candidates would not tell me what the model actually received.

Which memory store would I choose first?

My default would be the smallest storage arrangement that can pass those tests. For a prototype concerned only with recalling similar past text, I'd consider a vector-only implementation. I would not mistake that starting point for a complete design.

The source presents Postgres with pgvector, relational graph tables and an episodic log as a practical middle path. I'd evaluate that before operating separate graph, vector and relational systems, particularly if the team wants one transactional boundary.

That choice has a trade-off. The guide notes that a combined engine is constrained by its weakest storage capability, while separate systems allow independent scaling but require cross-store consistency. I'd move to separate systems only after identifying a concrete limitation, not because the reference architecture contains three boxes.

Before either choice, I'd define how to detect incomplete writes. The guide highlights the problem of a graph node without its embedding, or an embedding without its graph node. I'd want an explicit recovery path rather than assuming every write always completes.

How I'd compare Cognee, Mem0, Graphiti and Letta

The source describes Mem0 as write-path focused, with extraction followed by decisions to add, update, delete or ignore candidates. It describes Graphiti as a temporal knowledge-graph engine, and Letta as emphasizing editable working-memory blocks and model-controlled memory tools.

Cognee describes its own approach as a combined extraction pipeline, graph and vector persistence, temporal validity and feedback-based reweighting. Its guide also lists a single-Postgres deployment option.

I would treat these as architectural starting points, not a performance ranking. This comparison comes from Cognee's own article. It gives me questions to investigate, but I would not use it alone to declare a winner.

I'd test each candidate with the same corrected fact, historical query, tenant boundary and deletion request. My preference would go to the implementation whose behavior I can explain and inspect when a result is wrong.

What AI agent memory architecture does not solve

I would not treat persistence as verification. The guide includes extraction, reconciliation and source attribution, but I would still require evidence before promoting an extracted claim into trusted application state. A remembered statement and an approved instruction would remain different things in my design.

Nor would I treat feedback as a substitute for evaluation. The source describes reinforcing useful memories and down-weighting or correcting harmful ones. I'd first define what counts as a successful outcome and inspect proposed updates before allowing them to influence procedural instructions.

Deletion would be a release gate for me. The guide explicitly warns that forgotten content can resurface through orphaned embeddings. I'd test forgetting across the log, graph and vector index, then repeat the original query within the same tenant scope.

I'd also advise teams building a read-only document assistant to ignore much of this until they have a genuine cross-session write requirement. If the task only needs current source material and the current conversation, I would not add a durable memory pipeline just to match an agent diagram.

For my own adoption decision, I'd start with one workflow and require it to handle a corrected fact, explain its source and forget it completely. If it cannot do those things, I would keep the feature narrow rather than give the assistant more to remember.