How AI Agent Memory Actually Works: RAG, Semantic and Episodic Memory, Vector Database

If you've used ChatGPT or Claude, you've probably noticed something: you don't have to keep re-explaining who you are. It knows you're working on that side project. It knows you prefer TypeScript over JavaScript. It remembers you complained about a bug three days ago and brings it up again without you asking.

That's not magic — it's a memory system. And once you see how it's built, you start noticing it everywhere: in your coding assistant, in customer support bots, in every "AI agent" product that claims to "remember you."

This article breaks down what's actually happening under the hood.

The starting point: an AI with no memory at all

Strip everything away, and a chatbot is just this:

Three things go into every single reply:

• User Prompt — whatever you just typed
• Chat History — everything said earlier in this conversation
• System Prompt — the instructions defining how the AI should behave (its personality, rules, role)

This is enough for a basic back-and-forth. Ask a question, get an answer, ask a follow-up, it stays coherent. But close the tab, open a new chat, and none of it exists anymore. The AI has no idea who you are. This works fine for a one-off question like "explain recursion to me," but it falls apart the moment you expect any continuity.

The problem: sessions are goldfish

Say you're building a support bot for an online store. A customer asks: "Where's my order from last week?"

The bot has no way to know. It only has this conversation's chat history — nothing about past purchases, nothing about who this customer is, nothing about a complaint they filed two weeks ago. The moment the chat ends, all of it evaporates.

This is the exact same reason a new ChatGPT conversation used to know nothing about you, until these tools added actual memory on top. The conversation itself was never the problem — the problem was that nothing survived between conversations.

To fix this, you need something outside the conversation. That's where three types of memory come in.

The three pillars of agent memory

1. Procedural memory — "how should I behave?"

This is the AI's habits and rules. Think of it as onboarding notes for a new employee: "If the customer sounds angry, apologize first before troubleshooting."

If you've used Claude Code or Cursor and set up a Skill.md or CLAUDE.md file with your coding conventions, that's procedural memory.

It's just a file — no database, no vectors, just plain instructions the AI reads every time.

2. Semantic memory

This is durable factual information that doesn't change frequently overtime, like: who you are, what your company does, your preferences.

Real example: you tell Claude once that you're building a fintech app for freelancers. Weeks later, in a completely new conversation, it still knows that context.

That fact got pulled out of the conversation and saved separately, in what's called a vector database — a store that lets the AI search by meaning, not just keywords, so it can pull back only the relevant facts instead of your entire history.

This retrieval process, searching a large store for just the relevant bits, is called RAG (Retrieval-Augmented Generation).

It matters because you can't just dump your entire company's history into every prompt; even at a 1M-token context window, that's slow, expensive, and honestly unnecessary.

RAG fetches only the top handful of relevant facts (commonly called "top-K" search) and feeds just those into working memory.

3. Episodic memory.

Episodic memory is the raw timeline: "Customer ordered on June 3rd. Delivery delayed on June 7th. Filed a complaint on June 9th."

It's also stored in a vector database, but unlike semantic memory, it's time-stamped and event-based — basically a permanent log of everything the agent has seen.

When you scroll back through ChatGPT's conversation history and it recalls something you said a month ago in an unrelated chat, that's episodic memory being retrieved.

I know you're probably confused that if episodic memory already records everything, why do we need semantic memory too? Isn't that just storing the same thing twice?

Let's clear that up.

What's the difference between Episodic and Semantic Memories?

As you keep chatting, Episodic memory just keeps piling up.

Every message, every event, forever. If the AI had to search through all of that raw history every single time you asked a question, it'd be slow and expensive.

So periodically — say, every 20 conversations, or after some threshold of activity. a summarizer agent kicks in. It looks at the recent raw episodic logs and boils them down into a handful of durable facts, which get written into semantic memory. This is the exact same reason you can edit or delete specific "memories" in ChatGPT or Claude settings — those are the compressed, semantic facts, not the raw chat logs.

That's the loop: raw events pile into episodic memory → get periodically compressed by a summarizer → distilled facts land in semantic memory → semantic memory gets pulled into working memory on future requests, cheaply.

Putting it all together

Once you see it laid out, the whole system is just four moving parts working together:

• Working memory — what's actually sent to the LLM right now
• Procedural memory — fixed behavioral rules, stored as files
• Semantic memory — compressed, durable facts, fetched via RAG
• Episodic memory — the raw, timestamped history everything gets built from

None of this requires exotic infrastructure — a vector database, a scheduled summarization job, and some plain files will get you most of the way there. The reason it feels like "the AI remembers me" isn't one big trick. It's just these four pieces quietly working together every time you hit send.

That's it for this article, I hope you got to learn something.

If you're interested more about AI, Agents, and how these fancy terms work, follow my checkout more of my blogs

See you in the next one. Have a great one!