Memory in Conversational AI: Why Context Persistence Matters

I’ve spoken elsewhere about what capabilities started to make these models extremely useful. Intelligence was obviously one of them—going from GPT‑2 to GPT‑3 was a pretty big leap. I’ve also mentioned context: when we started to see models like GPT‑4 and its successors become extremely capable, largely because you could throw a lot of text—or code—into the context and get the model to do interesting things. It went beyond just trying to write a better sentence or fix a paragraph, and started doing the same kind of work many information workers have to do every day.

But one of the simplest things that made these models feel a little more magical—aside from just being good at natural language—was memory.

And I don’t mean the kind of “memory” people think of as a long ChatGPT conversation. I mean something even more basic: being able to ask a chatbot a question and come back ten minutes later, and have it remember what you asked before. It seems trivial now, but it wasn’t.

Even after GPT‑3 came out, Meta released what they described as a highly capable chatbot, with a demo you could interact with. But it was stateless. You could ask it a question and it would give you a fairly good answer (for what the models could do at the time), and the question and answer would show up in a message history. But if you asked it something that depended on what you’d said two minutes earlier, it had no recollection at all.

What they’d effectively done was: generate responses, display them in a history, but give the model no access to that history. They were trying to solve everything “in model” and not add a retrieval system to pull previous answers or re-inject them into context. And the context window was tiny, so even including a reasonable conversation history was basically out of the question. The result was that, when it came out, it didn’t feel like a very serious effort—not because the language was incoherent, but because it missed that single little detail: awareness of what was said before.

The researchers had made a model that could produce coherent natural-language responses given a prompt, but they’d decided not to give it any sense of shared history about something you talked about a minute ago. And I think the researchers were focused on, “Look, it really gave a good answer to this question,” which was interesting. But for people actually trying to talk to it, it didn’t feel like conversation.

This stood out to me because when I was a kid writing my first chatbots—back in the 1980s—on my Commodore 64, one of the things I did was ridiculously simple. I’d have it ask you something like, “What’s your favorite color?” Then I’d save the answer as a variable so it could come up later. Now, having it tell you, “Your favorite color is red,” because you told it red isn’t exactly a technological marvel. But at age 12 it felt important, because it seemed to me the purpose of a conversation wasn’t just whatever question you asked in the moment. It was the awareness of what happened before.

In other words: a shared history. Not treating every interaction as completely new.

And I think that’s part of the essence of personality—the idea that there is history and there is memory. It’s probably one of the reasons stories about people who lose the ability to form long-term memories are so chilling: it messes with identity. If you lose that history, it’s hard to know who you are. You’re just static. You don’t change.

So the point is: sometimes it’s a little detail, like the ability to remember something earlier in a conversation. And I think that’s one of the things that stood out with ChatGPT—integrating a much more robust memory system so you could have a genuinely sophisticated conversation.

For me, it’s still delightful when I open a new thread, talk to ChatGPT about a project I’m working on, and it says something like, “Oh yeah, that would be good for this thing you’re trying to do.” That’s magical. It’s a scaled-up version of what I was trying to do at 12—and it’s still magical.