Eternal Sunshine of the Spotless Model
On agentic memory, and the tricky business of forgetting.
There are some things we just know.
They’re ingrained in our memories, and whether we like it or not, we just can’t scoop them out of our brains. For example, if anyone’s seen the movie Hereditary, there are multiple scenes (I won’t spoil which if you haven’t seen it) that will stay with me until I die, no matter how much I would very much like to forget them.
This also holds true for LLMs, which can be problematic when you’re working in regulated industries, when you need to fulfil someone’s right to be forgotten, or if it’s somehow learned some deeply embarrassing personal secrets about you.
Don’t You (Forget About Me)
An AI agent broadly retrieves and recalls data in the following ways…
The first is via external memory; a database, blob storage, a text file, anything that an AI agent has to query and pull information from. For us human-folk, an example of this type of memory or storage would be an address book, or a list of contacts on our phone. It’s information that we can look up whenever we need it, but it’s not necessarily something we can just easily recall. An agent doesn’t actually know any of this information, it looks it up. In the same way I don’t know my partner’s phone number, or our anniversary, or… actually, let me stop there. This type of memory is where an agent keeps anything personal or likely to change.
Then there is cache memory. It’s essentially our short-term memory, like when we need to memorise a phone number for a short while before we find a pen and paper to write it down. This memory normally clears after a certain period of time, or when we end a chat session for example. After a while, the memory/data just disappears, like how I’ll forget someone’s name approximately 5 seconds after they tell me for the first time.
Finally there is in-weight memory, this refers to all of the data baked into the model itself during training. Nothing gets looked up here, it’s all been learned. This data has been scattered across billions of tiny numbers and layers within the model until it’s part of how the model itself “thinks”. For example, if I asked you to complete the phrase “Practice makes …” your brain might automatically complete the phrase “Practice makes perfect” without you having to do any real deep thinking.
This is essentially your own “in-weight memory”. It’s what you know that you don’t need to look up, and it’s what’s very very very hard to forget.
Let’s play this out; let’s consider Logan. Logan is extremely privacy-savvy, and he’s asked us to delete everything we hold on him from our agentic system. From the interactive demo below, try deleting his data from our external store. Then, we’ll fine-tune a model on his details and try the exact same thing again.
Looks like this interactive demo didn’t load, please try enabling JavaScript. It would have walked you through this:
We're asked to delete Logan's data from a database, and the deletion just works. The record is gone, and asking our model for information about Logan turns up nothing.
However, when we try to delete the same data once it's been trained into the model's weights, nothing happens: there's no record to remove, and the model will still answer questions about them. From here, there are two options: cheap machine unlearning, which is fast but leaves the weights leaking traces of what it just erased, or a full retrain from scratch, which works but costs a huge amount of time and resources.
Delete, Done, Dusted
Luckily, that first delete just worked. Logan’s data was a single row you could point at, you removed it, and now the agent draws a blank when you ask. It’s like ripping a page out of a notebook and shredding it; once it’s gone, it’s gone. This means that deleting Logan’s personal data in this case is straightforward.
Can’t Touch This
But the second you fine-tuned the model on Logan, that same delete isn’t really possible. There’s no row left to remove. His details aren’t in a specific location that you can just get rid of, they’re scattered across millions of parameters, tangled up with everything else the model has ever learned.
“Deleting” this data now means inherently changing the model itself.
Just like we can’t consciously “delete” memories that are ingrained in our brains, we can’t force an LLM to outright forget what it knows. The information is part of it’s very fabric.
What we can do, is retrain the model from scratch, or try some machine unlearning. However, retraining is slow and expensive, and unlearning hasn’t yet been shown to be reliably effective.
Only full retraining actually guarantees that the data is gone; the cheaper methods estimate a data point’s influence and subtract it, which is an approximation that can leave traces behind. So if you ask where Logan lives in a slightly different way, or particularly nicely, the model might just tell you. For example, researchers recovered over half of supposedly forgotten content from a model by simply by padding the prompt with filler text (Jang et al.). Much of what passes for unlearning teaches the model to decline the question, not necessarily to forget the answer.
Not only that, but the unlearning process itself can ultimatlely provide an attack surface. Given a model’s parameters before and after machine unlearning, researchers were able to recover the features and labels of the very data that was supposed to have been “forgotten” (Hu et al.).
Total recall
When building an agentic system, we have to be deliberate and considerate about how data enters our model(s), because that choice determines whether or not the data can ever really leave again. When we question “should we fine-tune a model?”, we should also ask “what kind of data would we be training it on, and do we need it to be forgotten at some point?”.
Making the wrong choice here can lead to permanent regret, which is ultimately why I’ve still never gotten a tattoo, despite threatening to for years. It’ll be any day now. Maybe a dragon, or a tiger, or my dog, we’ll see.
Suggestions welcome.