This is the teaser that got me into this:

Here’s the full 32m43s talk on YT: 🎞️ Our Conception of Working Memory is Wrong? | Professor Elan Barenholtz.

And here’s a conversation with Prof. William Hahn and Addy Cha, on Substack: 🎞️ Is your Brain a Large Language Model? 1h10m of incoherence.

Maybe you should also try a 34m22s 🎞️ video about his Auto-Regression Theory of Cognition: “Professor Elan Barenholtz PhD discusses his new ‘theory of everything’ for cognition, The Auto-Regression Theory which challenges our concepts of reasoning, intelligence, and consciousness across biology, LLMs, and computing as a whole.”

This guy is a tool. Completely incoherent, I expect his students to hate him thoroughly. But he has theories.

Zur Person

Based on fau.edu/sandbox/people + fau.edu/science/directory/ebarenholtz + mpcrlab.com/people/Elan-Barenholtz and maybe barenholtz.ai:

Elan Barenholtz, Ph.D., is an Associate Professor of Psychology at Florida Atlantic University (FAU), and Associate Director of the Center for the Future Mind, an associate professor at Charles E. Schmidt College of Science, and co-director of the Machine Perception and Cognitive Robotics (MPCR) Lab. Barenholtz earned his doctorate degree in experimental psychology and cognitive science at Rutgers University, New Jersey, and was a postdoctoral fellow at Brown University, Rhode Island. Barenholtz began his career as an experimental psychologist, investigating human perception from a behavioral and cognitive-science perspective, and co-founded the MPCR to develop theories and applications of broad brain properties and test them embedded in robotic agents and applied to real-world problems.

Research Interests:

  • Deep Learning/Ai
  • Foundations of Language and Cognition
  • Embedded computational neural models

Research Description: I use behavioral and embedded computational approaches (i.e., neural networks running in robots) to study the brain and behavior with the goal of developing a broad theoretical framework of neural function.

He claims to be a neuroscientist, but his degrees are in psychology. His interests turned to the use of machine learning (deep learning, AI) to create a broad theory of cognition that would apply to both humans and machines.

Don’t take this for what it isn’t, but I noticed that he seems to be wearing a kippah all the time. It wears it in all public photos and videos, although it’s not always that evident. While it’s a small, black, inconspicuous kippah, this is disappointing. I’m not saying that a neuroscientist can’t be an observant Jew and should be an atheist, but there are so many prominent Jews in America that don’t do that, except maybe on special occasions.

On memory in Elan’s view

That first 32m43s is incoherent ramble in the first 8 minutes, and occasionally throughout it. This is how it starts:

Hey everybody, it’s Elan Barenholtz, I’m a professor at Florida Atlantic University, and I’m working on a generative theory of cognition in the brain, and it’s all based on large language models, the things that underlay ChatGPT.

I’ll skip the almost muddy attempts at explaining several things at once, and jump right to the bomb at minute 8:

One of the deeply challenging outcomes of thinking of the mind utter aggressively is that it really kind of destroys the common sense notion of a memory, and it’s so deeply entrenched in us when I think, you know, “What does your mother look like?” “What did you do last summer?”

We qualitatively have this experience of, we’re calling something and somebody brings it to us and it shows up in our mind, but it’s already there, the image of your mother or some sort of video of splashing through the waves. These are recalled and we sit and watch them passively.

❶ But that’s not what’s actually happening.

What’s happening is we have this capability, we have this engine that’s able in the moment to generate that image of your mother on command with the appropriate input.

So we say, “hey, visual imagery system, give me an image of my mom,” and it generates it in that moment.

But that image doesn’t exist anywhere in the system. It’s not there. It’s not in your brain.

Even if I could completely decode your brain, unless I was able to run it with that input, the image isn’t really there.

And so memories, in some ways, aren’t real. They’re not the things that we kind of intuitively feel that they are.

But at the same time, they’re very real in the sense that we can generate them on demand.

This is the most shocking single idea that I’ve heard in the last month or so. I won’t comment until later on. He’s now starting to ask the most important question one is supposed to have after accepting the above postulate:

Sometimes these are memories that we cherish. These are things that we want to be able to pull out of our kind of mental time capsule and look at it again.

And the idea isn’t that your mom’s face is forever lost and it isn’t really there. It’s an illusion. It’s that the ability to generate that is what it means to remember what your mom’s face is. Now you can picture your mother not just in a single kind of front-facing, but picture from the side, picture her making you breakfast as a little kid. You can do whatever you want with this because it has this kind of endless potentiation.

But this kind of raises the question, okay, thinking about it this way, thinking about memory this way… What if there are memories that I want to be able to generate later on?

Or a better way to put it, it’s not a memory I want to be able to generate, but what if I want to have certain kind of generative abilities later on? Instead of thinking of it as how do I store something now that I can later retrieve, it’s how can I ensure that I’m going to be able to pull it up on demand later on?

And so let’s say there’s something that’s really important to you, it’s really precious, really valuable to remember.

❷ Is there a way to encode that so that you have a higher probability of being able to later generate it?

Now of course we know that there’s a vast literature on this. There is a substantive, empirical, scientifically established literature on how to encode information such that you can later remember.

It’s basically the entire field of learning. How do we learn information such that we’re later able to retrieve it?

But the characterization, thinking about how information actually is encoded and then later used has almost certainly dramatically influenced the way that this field evolved.

So many of its findings, of course, are going to be valid. If you learn better under some circumstance, one circumstance with the other, and we can measure that in later job performance, later academic performance, that’s real.

I’m not suggesting that all of the tools that have been designed are no longer valid. But what I am suggesting is that there may be other kinds of tools, other kinds of frameworks that are sitting right in front of us that may be obvious if we think about memory in this other way, if we think of it as a generative process rather than as a retrieval process.

He’s that incoherent, and it only gets worse, so I’ll skip to minute 13. Bear in mind that the abundance of anacoluthons is this guy’s hallmark.

If you’re thinking about how do we just store information, well, we can dress up the information in certain ways, we can package it, we can organize it. That means once it gets in there, you’re done and hopefully you’ll remember it later on.

❸ But if in fact what we’re talking about is a much more continuous generative process, then there may be far more actual tools and points of intervention at our disposal than thinking about it simply in terms of cold storage and retrieval.

So there’s this big question, “Why is math so effective in making predictions and physics and why in general?” Well, physics is really just a mathemization of physical observations, predictions. And why is it so unreasonably good?

❹ And I think the thing we’re kind of missing is that math and all the derivative kind of subjects that extend from it is a human artifact that is couched in based on sort of the same, appear to be the same core principles as language.

Of course it is itself math and language. It’s saying what it’s doing is self-consciously doing symbolic representation.

But where do we get the idea of symbolic representation? Well, of course we got it from language.

“That’s a cat over there.” Oh, there’s this thing, we can write it down, we can say it and it picks out an object there and then we can string these things together to have some representation of the world.

And math, of course, borrowed this idea from the observation from natural language that you can represent things like numbers using certain kinds of symbols. And then if you string them together in a certain way and then run certain kind of operations on them, the thing that spits back out is going to be some fact about the world as well.

Abstract mathematics ends up not worrying about any sort of actual natural kind physical representation that the numbers represent so that the symbols can just be abstract entities in their own right.

But the whole idea of symbolization playing around with symbols, of course, is borrowed from language. But I think the unreasonable utility of it is because it’s actually derivative of the much greater power of natural language.

So far, so good. How about differentiating between long-term memory and working memory? If he’s lecturing in the same disorganized way to his students, I expect a huge spike in the suicide rate in Florida.

❺ Well, let’s start actually, let’s try to dissolve this kind of divide between long-term and working memory and thinking about these as distinct systems because I don’t think that’s how it works.

I think we now have a very good model of how it works and it doesn’t actually subscribe to this poor notion about what memory is and what it’s doing and therefore this divide.

And then let’s think about what the implications might be in relation to what we’re actually trying to do.

Because parenthetically, I’ll say that our notion of memory is very deeply tied up in, believe it or not, even if you’ve never thought about it before, in a sort of a computational framework, the idea that we’re storing a bunch of stuff for later and then where we’re going to go and we’re going to pull that stuff out and remember it.

And I think that could have very broad implications for thinking about what it would mean to enhance, let’s not even call it memory, enhance our performance.

What do we really want? We really want, in the moment, to do the right thing, to do this sort of functional thing and if that requires retrieving information as we would typically think of it, we want that to work.

We don’t really care if and when and how it’s stored. What we really care about is runtime. We want to be able to do the thing that we tend to think memory supports. We want to be able to generate meaningful responses to questions.

Some of those questions require certain past information making its way into the current moment. And at that level of generality, everybody wants to enhance their capabilities.

But if you think about it from the wrong standpoint, you might not be thinking about how to do that enhancement correctly either.

Because if what you’re thinking of is how do we appropriately store facts, store information, and then later on have the right tools to get that information back, well, that’s going to come with a certain kind of regimen, certain ways of thinking about how to solve the problem of enhancing the later performance.

My argument is that that’s fundamentally, that’s just actually the wrong model. It’s the wrong model for thinking about what the actual process of generating the behaviors we want.

When I’m talking to you now and I’m talking about large language models and I’m talking about memory and enhancement and all of that, in some ways I’m obviously using information from the past, right?

That information is, well, this is what we’re talking about.

This is the conversation and I know what a large language model is because I read about it and I program them. And it’s certainly true that that information is coming through in what I’m saying right now, but that doesn’t mean that we have to think about it in terms of storing information in this static form.

We want to produce the behavior. We have a model now that produced the behavior that doesn’t store information in this way. And it does, and at runtime it’s not retrieving anything.

So what are the implications if instead of thinking about runtime performance as storage and retrieval, but just thinking about the performance itself and thinking about that there’s this stream of activity that’s happening cognitively.

Crystal clear. Not. He’s going somewhere. But his insistence on enhancing (enhancing what?!) reminded me of the spam for enhancement pills. Another keyword: behaviors. And what is “the performance itself”?!

At minute 20, we’re literally described as being LLMs!

We can think of it as computational level, we can think at the mental level. It may have some overlap even with sort of conscious thought, but that’s for a later conversation.

❻ But the process is one that we can model in terms of influence rather than memory. Past events, things that have happened in your long distant past, things that have happened in your more recent past, are influencing your current generation.

Can we call that memory?

Well, sure, in the sense that the system changes in the past, and that affects its performance later on.

But is information stored in the traditional sense?

No, because in a large language model in our artificial neural network, what you’ve got is just changes in these parameters, changes in the structure, if you want to think about it that way, of the network.

And those changes aren’t actually representing some specific set of facts at all. What they represent is, given a current input right now, what are you going to do about it?

And that input could be, “what is the conversation we’re having right now?” “Well, it’s a conversation maybe about large language models and autoregression as a model.” “Okay, what’s the core thesis?” And I could answer that question: “I can’t answer that question, but that’s yet another thing I can do with this information.”

In other words, what’s happened in my recent past has an infinitude of possible outcomes.

And so we can’t think in terms of a single static fact that’s going to somehow end up coming out of my mouth. Instead, like a neural network, we’re changing things such that we have now a completely different trajectory of how the system is going to go.

So from a metaphor standpoint, instead of thinking of it as a computer program or a storage retrieval process, the metaphor that keeps coming to my mind is sort of a stream that’s running, and it’s doing stuff along the way, useful stuff.

But if something interacts with that stream, it’s going to change the shape that it takes. And it’s going to do so with something like what we’d call memory. But it’s that the information sort of, it sticks around, it sticks around in the system.

Like the previous words I said, for example, are influencing what I’m about to say. And they’re still doing it now, right? The words I just said, you know, 10 words ago, now it’s 15 words ago, now it’s 20 words ago, are influencing what I’m going to say. That’s what working memory is, in my view.

It’s not that there’s information that we’re going and retrieving, it’s that the past history, and it’s non-Markovian, the past history is continuing to have an influence on what I’m doing now.

And so from a standpoint of sort of enhancement and thinking about how we can improve runtime performance, what this means is we have to think in terms of how do we nudge the system? How do we move the system in a direction that gets us where we want to go? How do we set up sort of the preconditions? Or we shouldn’t even call them preconditions, it happens in real time. What can we do to the stream such that it takes the shape, a more optimal shape?

❼ And this is a very different question than thinking about how do we store certain kinds of information with fidelity so that you can later retrieve them accurately. It’s how do we encourage the right kind of influence?

And so I don’t have a clear idea as to how you actually do this kind of manipulation, but I think a paradigm shift could lead to completely different ways of thinking about these kinds of tools, what thought tools might exist already, or what’s at our disposal that can help us shape that kind of ongoing process.

How can we experiment with that? How can we introduce information somewhere on the stream? Where should it be?

I know that in philosophy it’s more important to ask the right questions rather than attempting to answer, but this guy is tiresome.

At minute 27:

❽ In 99.999% of cases, that’s not to explicitly recall and retrieve information from the recent past in some form. It’s to be influenced by it.

The things I’m going to say now in this part of the sentence are meaningful and smart because of the beginning of my sentence. So we need that influence to come through.

We shouldn’t be thinking in terms of measurements of explicit retrieval because that was actually a red herring in the first place. What we need to be doing is thinking in terms of the kinds of retrieval, but what it really is is runtime, efficaciousness, efficacy.

In doing the kind of stuff we want people to actually be able to do with memory, and then we measure that.

“Okay, here’s a set of instructions. Now, carry out the task. Go, write quickly as you can.”

What if we repeat the instruction? What if we inject something during the instruction in a certain point? How does that influence, negatively or positively, the actual capability to keep going with the instruction set?

And we’re going to call this understanding, but we can now see that that’s too vague and also too specific a word. There’s variance, there’s degrees. It’s not like you understand or you don’t understand.

The extent to which you’re able to finish a sentence meaningfully it’s not binary, like you either completely forgot what you said in the beginning or you’ve got it in pristine form. That is influencing what you’re doing, and so we have to have softer measures that are able to measure sort of kind of efficacy in this way that is not dependent on sort of this binary.

“Okay, can you retrieve it or not?”

That’s far too crude and frankly a misguided measure. That’s what I think is maybe a really interesting project here, and it’s twofold.

One is to maybe rethink a little bit the kind of things you want to measure because that’s not what’s historically measured in the so-called short-term memory span window of time, but that’s a really, really important span.

❾ It’s like finishing your thoughts. It’s being able to compress in real time, and these are all conjectures so I don’t even want to commit to this, but something we’re doing right now when you’re listening to me is you’re compressing this stuff.

You’re not representing it as the explicit tokens that I’m saying. You’re doing this thing, but you’re doing it in such a way not that you can retrieve it, right, because what the hell did I say 30 seconds ago?

You have no idea you can’t retrieve it, but I promise you it’s influencing your understanding of what I’m saying right now and your ability to then respond to it as long as you know.

I had no idea that a Professor can be so chaotic. Let’s simplify it and say that what we call “short-term memory” (which is shorter than the “working memory”) is that “thing” that’s able to finish your thoughts. You know, the next token that’s consumed by an LLM. Die Mensch-Maschine.

The last stop, at minute 30:

Measurement, capturing what this thing really does, even specifically, let’s just think of what historically is not the short-term memory window, but we’ll start with that. We can elaborate further, but that’s number one.

How do we capture it? How do we measure the really interesting, the real thing that we want it to do?

Well, first of all, that it’s meaningfully doing, and then second of all, that we want it to do, and then once we have what we want it to do, now we can get to the hard work and maybe, I think it’s in some ways more difficult work of, okay, what can we do about it?

❿ If we have the right measurements, what are the possible interventions that could be helpful in doing it more effectively, and doing it more efficiently?

You need to solve one before you solve two to some extent, but you can also try to solve these in concert.

“Don’t wait, just try to be more effective right away,” right?

We want people to be able to understand information better, so they can perform some task or explain it back to you.

That’s not hard to measure. We can start with that. Start with pretty crude measurements.

“Was this person able to perform this task based on my narrative description of what they’re supposed to do?”, and then mess around with that and see if there’s a kind of measurable, testable, quantifiable, and and setable, parameterizable factors that you can mess around with and then start to see what the impacts are.

Maybe that needs a rethinking. I almost feel like that’s too didactic and too deconstructionist, and maybe that’s not the right approach.

But in any case, the work before us in some sense is kind of clear. Exactly how to go about each step, obviously, will take a lot of autoregressing.

Kind of clear, my ass. Oh, well, this is how our guy is. Somewhat broken when compared to a decent LLM, wouldn’t you think so?

But I guess that the task at hand is this: How to improve our own “prompt” design to generate more accurate “memories”?

Claude to the rescue!

Since I’m not smart enough to either refute or embrace his theories (and, FFS, he’s such a lousy speaker!), I had a chat with Claude:

1

● Ludditus: What’s the common understanding regarding long-term memory?

Traditionally, it was thought that it’s just “data retrieval” and that memories fade due to the degradation of data being stored in an “analog computer.”

Then, it was considered that upon each retrieval, the memory is reinforced and then saved back to memory, but in the process it would change a bit. It’s like making a photocopy and keeping the copy instead of the original.

However, while in real life a photocopy of a photocopy of a photocopy would gradually decrease in contrast and legibility, reinforced memories became more vivid. Unfortunately, less accurate, to. In a way, it’s like a witness repeating a story at a bar, time and again, so if the trial only starts a couple of years later, such a testimony would be anything but reliable. As time goes by, witnesses either forget or substantially change what they remember.

Now, cognitive scientist Elan Barenholtz says memory isn’t retrieval. It’s generation. When you remember something, you’re not accessing a stored file. You’re prompting your mind, like an AI model, to synthesize a response. There is no image of your mother stored in your memory.

Is this consistent with what we know so far?

Claude: Your summary captures a fascinating evolution in how we understand memory, and yes, Barenholtz’s generative view aligns remarkably well with current neuroscientific evidence. Let me walk you through how this perspective fits with what we’ve learned about memory systems.

The traditional “filing cabinet” model you described first made intuitive sense – we store memories like files and retrieve them when needed. But this view couldn’t explain why memories change so systematically. The photocopy analogy you mentioned captures the reconsolidation theory beautifully, where each retrieval creates a new “copy” that’s slightly different from the original.

However, the reconsolidation model still assumed there was an original “file” being copied. Barenholtz’s generative theory goes further by suggesting there might not be a discrete stored memory at all. Instead, what we call “remembering” might be your brain reconstructing a plausible version of past events using scattered neural patterns, current context, and general knowledge.

This generative view explains several puzzling aspects of memory that earlier theories struggled with. For instance, why do our memories often become more vivid and confident over time, even as they become less accurate? In a generative system, each “remembering” episode doesn’t just retrieve information – it actively constructs a story that feels coherent and meaningful to you right now. Your current beliefs, emotions, and knowledge all influence this construction process.

Think about how you remember your childhood home. You probably don’t have a single, complete “photograph” stored somewhere in your brain. Instead, you might have fragments – the smell of your mother’s cooking, the feeling of carpet under your feet, the sound of the front door closing. When you “remember” the kitchen, your brain weaves these fragments together with your general knowledge about kitchens to create a coherent scene.

This also explains why eyewitness testimony becomes less reliable over time, as your bar story analogy illustrates. Each time someone recounts an event, they’re not just accessing a stored file – they’re regenerating the memory using their current understanding, emotions, and social context. The story evolves because it’s being actively reconstructed each time.

The generative view also helps explain why memories can be so easily influenced by suggestion or new information. If remembering is reconstruction rather than retrieval, then new information can easily become incorporated into future “rememberings” of the same event.

What makes this perspective particularly compelling is how it mirrors what we’ve learned about perception. We now know that perception isn’t passive recording but active construction – your brain constantly makes predictions about what you’re likely to see, hear, or feel. Memory might work similarly, with your brain making predictions about what you likely experienced in the past.

Does this help clarify how the generative theory builds on and resolves some of the puzzles left by earlier models? I’m curious what aspects of this reconceptualization you find most intriguing or challenging to wrap your head around.

2

● Ludditus: The sense of smell and the tactile sense are very important to me, but in different ways. A smell can trigger memories from childhood or other pleasant distant memories, but I noticed a peculiarity. While I was expecting such memories to be embellished and stripped of the negative aspects and of most of the context, there’s something else that puzzled me: the smell that triggered the visual recreation of a distant past wasn’t that exact smell from the past. I knew that, but it’s like something in my brain singled out a specific subset of what I could smell back then, then happily extrapolated it to the full olfactory environment that triggered that memory. Or the smell wasn’t even that similar, but similar enough to make the brain say, “Eh, what if I said it’s like back then so I could revive those good old times?” What’s this phenomenon?

Claude: What you’re describing is a beautiful example of how our generative memory system works, and it reveals something profound about the relationship between smell, memory, and pattern recognition. Let me break this down into the key processes at work.

First, let’s understand why smell is such a powerful memory trigger. Unlike other senses, smell bypasses much of the brain’s filtering system and connects almost directly to the limbic system, where emotions and memories are processed. This creates what neuroscientists call the “Proust phenomenon” – named after the famous passage where Marcel Proust describes how the smell of madeleines dipped in tea instantly transported him to childhood memories.

Now, here’s where your observation becomes really interesting. What you’re noticing is that your brain isn’t doing exact pattern matching – it’s doing something much more sophisticated and, frankly, more creative. Think of it like this: your brain has learned to recognize not just specific smells, but smell “signatures” or “themes.”

When you were a child, let’s say you experienced a complex olfactory environment – maybe it was your grandmother’s kitchen with bread baking, coffee brewing, old wood, fabric softener, and a dozen other subtle scents. Your brain didn’t store this as a single, complete “smell file.” Instead, it extracted key features and patterns – what we might call the “essence” of that environment.

Years later, when you encounter a smell that shares even a few of these key features, your brain doesn’t just recognize the similarity – it actively reconstructs the entire memory using that partial match as a prompt. It’s like your brain is saying, “Oh, this reminds me of that pattern I learned to associate with grandmother’s kitchen. Let me generate the full experience.”

This explains the phenomenon you noticed where the triggering smell wasn’t exactly the same as the original. Your brain has learned to recognize what cognitive scientists call “gist” rather than exact details. It’s looking for the meaningful essence rather than a perfect match.

Here’s where it gets even more fascinating: your brain isn’t just being sloppy or making mistakes. It’s actually being quite clever. By focusing on the meaningful patterns rather than exact matches, it can connect experiences across time and context in ways that would be impossible with rigid, exact matching. This flexibility allows you to recognize “home-like” smells even when you’re in completely different environments.

Think of it like how you can recognize a song played on different instruments or in different keys. Your brain extracts the essential melodic and rhythmic patterns, not the exact acoustic properties. Smell memory works similarly – your brain learns the “melody” of an olfactory experience and can recognize variations on that theme.

This also explains why these smell-triggered memories feel so vivid and emotionally rich, even though they’re reconstructed rather than retrieved. Your brain isn’t just accessing stored data – it’s actively generating a rich, multi-sensory experience based on the emotional and contextual associations it learned to connect with that smell pattern.

The “good old times” aspect you mentioned reveals another layer of this process. Your brain has also learned to associate certain smell patterns with positive emotional states. When it recognizes a similar pattern, it doesn’t just reconstruct the sensory details – it also reconstructs the emotional context, often amplifying the positive aspects because that’s what made the original experience memorable in the first place.

Does this help explain why your brain seems to be both creative and selective in how it connects current smells to past experiences? Can you think of other examples where you’ve noticed this kind of pattern-based rather than exact matching in your own memory experiences?

3

● Ludditus: But does it mean that our “memories” aren’t true memories but more like LLMs, and there’s a stream of consumed tokens that generate a memory? This still doesn’t explain how such images from the past are generated!

Secondly, I also have a strong visual memory. Back in high school, sometimes I didn’t memorize the text on a page in the physics handbook, but an image that I remembered for a long time, and I used that image, sometimes with the image of some equations from the same handbook, to figure out what a law of physics actually said! How is this not visual memory?

Claude: You’ve hit on one of the most profound puzzles in memory research, and your questions reveal why this topic is so hotly debated among neuroscientists. Let me walk you through this step by step, because understanding how visual memory works will help us unpack the deeper question about whether memories are truly “generative” like LLMs.

First, let’s tackle your physics handbook example, because it’s actually a perfect case study for understanding how visual memory might work within a generative framework. When you looked at that page repeatedly, your brain wasn’t just taking a photograph and filing it away. Instead, it was learning the spatial relationships, the visual patterns, and the structural organization of that information. Think of it like learning the “grammar” of how physics equations are typically laid out on a page.

Here’s where it gets fascinating: when you later “recalled” that page, you might have been reconstructing it using learned patterns about how physics textbooks organize information, combined with fragments of actual visual details you retained. Your brain became very good at generating plausible reconstructions of what that page looked like, using both specific details it had encoded and general knowledge about how such pages are typically structured.

But here’s the key insight that addresses your question about whether this is “true” visual memory: the fact that you could use this reconstructed image to derive the actual physics laws suggests something remarkable. Your brain wasn’t just creating a random hallucination – it was generating a reconstruction that preserved the meaningful relationships and information from the original page. This is visual memory, but it’s visual memory that works through pattern recognition and reconstruction rather than literal storage.

Let me give you an analogy that might help clarify this distinction. Imagine you’re an artist who has studied Renaissance paintings extensively. If I asked you to recreate a Vermeer painting from memory, you wouldn’t be accessing a stored photograph in your head. Instead, you’d be using your learned knowledge about Vermeer’s style, color palette, composition techniques, and subject matter to generate a painting that captures the essence of his work. Your reconstruction might not be pixel-perfect accurate, but it would preserve the meaningful patterns and relationships that make a painting recognizably Vermeer.

This brings us to your deeper question about the LLM analogy. The comparison is illuminating but needs some important refinements. Unlike an LLM processing a stream of tokens, your brain doesn’t generate memories by predicting the next piece of information in a sequence. Instead, it seems to work more like what computer scientists call a “generative model” – it learns the underlying patterns and relationships in experiences and can then generate new instances that follow those same patterns.

Think about it this way: when you remember your physics handbook page, your brain isn’t sequentially generating “token by token” like reading text. Instead, it’s simultaneously reconstructing spatial relationships, visual hierarchies, and conceptual connections. You might recall that the main equation was in the center, with smaller explanatory text below, and a diagram to the right – all reconstructed as an integrated whole rather than as a linear sequence.

Here’s where your visual memory example becomes even more interesting from a teaching perspective. The fact that you could use your reconstructed image to derive the actual physics laws suggests that your brain had learned to preserve not just the visual appearance but the meaningful relationships between different elements on the page. Your visual memory system had become skilled at maintaining the functional relationships that allowed you to extract the same information you would have gotten from the original page.

This leads to a crucial insight: saying memory is “generative” doesn’t mean it’s arbitrary or unreliable. Your brain has learned to generate reconstructions that preserve the features and relationships that matter for the tasks you need to perform. In the case of your physics handbook, your visual memory system learned to preserve the spatial and conceptual relationships that allowed you to derive the correct physics principles.

Now, let’s address the deeper question about whether these are “true” memories. This is where the analogy to LLMs becomes both helpful and limiting. Like an LLM, your brain has learned patterns from experience and can generate new outputs based on those patterns. But unlike an LLM, your brain’s reconstructions are constrained by your personal history and the specific neural pathways that were strengthened by your actual experiences.

Here’s a thought experiment to help you explore this: try to recall that physics handbook page right now. Pay attention to how the memory forms. Do you instantly see a complete, detailed image, or do you find yourself building up the memory piece by piece, with some parts clearer than others? You might notice that certain elements feel very vivid and certain, while others feel more like educated guesses based on what you know about how physics textbooks are typically organized.

This observation can help us understand that visual memory might exist on a spectrum rather than being purely generative or purely stored. Some aspects of your memory might be based on genuine visual details that your brain encoded, while other aspects might be reconstructed using your learned knowledge about the patterns and structures typical of physics textbooks.

What do you think? When you try to recall specific visual details from that handbook, can you distinguish between elements that feel like genuine visual memories versus elements that feel more like plausible reconstructions based on your general knowledge?

4

● Ludditus: Barenholtz has a specific target right now: knowing that these generative memories involve certain past information that’s making its way into the current moment and that this past information influences the generation, the idea is to “encourage the right kind of influence” so that what we “remember” is reconstructed more accurately and effectively. In other words, instead of focusing on “how to memorize something better,” we should rather discover how to improve the “prompt engineering” for our internal LLM! He also insists that “math and all the derivative kinds of subjects that extend from it appear to be the same core principles as language,” and given how math is so effective in making predictions, what we should do is some sort of “prompt engineering” with our minds!

Claude: What Barenholtz is proposing here represents a fascinating paradigm shift that could revolutionize how we think about learning and remembering. Let me help you understand why this approach is so potentially powerful by building up the concept step by step.

First, let’s make sure we grasp what Barenholtz means by “prompt engineering” for memory. When you use an AI language model, you’ve probably noticed that how you phrase your question dramatically affects the quality of the response you get. A vague prompt gives you a generic answer, while a well-crafted prompt that provides context, specifies the desired format, and includes relevant examples produces much better results. Barenholtz is suggesting that our brains work similarly when reconstructing memories.

Think back to your physics handbook example. When you needed to recall that page, your brain was essentially running an internal prompt that might have sounded something like: “Generate the visual layout of the page about [specific physics concept] from [specific textbook], focusing on the spatial relationships between equations and explanatory text.” The more specific and well-structured this internal prompt, the more accurate and useful your reconstructed memory becomes.

Now, here’s where Barenholtz’s insight becomes really profound. Instead of trying to cram more information into your head through repetition, what if you focused on learning how to create better internal prompts? This means understanding what contextual cues, what kinds of associations, and what structural frameworks help your brain generate more accurate and useful reconstructions of past experiences.

Let me give you a concrete example of how this might work in practice. Imagine you’re trying to remember a complex historical event. Traditional memorization might involve rehearsing facts repeatedly. But prompt engineering for memory might involve learning to ask yourself the right questions: “What were the economic pressures at the time? How did this connect to earlier events? What were the key personalities involved and what motivated them?” These questions become internal prompts that help your brain reconstruct a richer, more accurate understanding of the historical event.

The connection Barenholtz makes between math and language is particularly illuminating here. Both mathematics and language are fundamentally about patterns and relationships. When you understand a mathematical concept, you’re not just memorizing formulas – you’re learning to recognize patterns that allow you to generate solutions to new problems. Similarly, when you truly understand language, you can generate novel sentences that follow grammatical patterns you’ve internalized.

This suggests that effective learning in any domain involves developing what we might call “pattern recognition engines” that can generate appropriate responses to new situations. Your brain learns the deep structural patterns underlying different types of knowledge, then uses those patterns to reconstruct or generate relevant information when needed.

Here’s a thought experiment to help you understand this better. Consider how you solve mathematical problems. When you encounter a new algebra problem, you don’t retrieve a specific stored solution. Instead, you recognize patterns in the problem structure that prompt you to apply certain solution strategies. Your brain generates the solution by following learned patterns and relationships. The better you understand the underlying mathematical principles, the more effectively you can generate solutions to novel problems.

Now, let’s explore how this applies to improving your own learning and memory. Instead of focusing solely on repetition and rehearsal, you might experiment with developing better internal prompts. This could involve learning to ask yourself more specific questions, creating richer contextual frameworks, and building stronger associations between new information and existing knowledge structures.

For instance, when learning a new concept, you might practice generating multiple different kinds of prompts: “How does this connect to what I already know?” “What are the key patterns here?” “Under what conditions would this principle apply?” “What would happen if I changed one key variable?” These questions help your brain build more robust generative models that can produce accurate reconstructions later.

The mathematical prediction aspect that Barenholtz mentions is particularly intriguing. Mathematics is so effective at making predictions precisely because it captures the essential patterns and relationships in systems. If our memory works similarly, then learning to identify and internalize the key patterns in any domain should dramatically improve our ability to generate accurate and useful knowledge when we need it.

This raises an interesting question for you to consider: can you think of times when you’ve successfully “prompted” your own memory to produce better results? Perhaps by asking yourself different questions or approaching a problem from a different angle? What kinds of internal questions or frameworks have you found most helpful in accessing or reconstructing knowledge effectively?

Understanding this generative approach to memory might also help explain why some teaching methods are more effective than others. Methods that help students develop better internal prompts and recognize deeper patterns should be more effective than those that focus primarily on repetition and memorization.

5

● Ludditus: But is this about consciously constructing such prompts? Because I believe that what we most need is to develop a sort of “instinct” on how to trigger better memories. Struggling to remember something is one thing, and the ordinary memory recall that happens thousands of times a day is more or less automatic, without much judgment, if at all.

Claude: You’ve identified a crucial distinction that gets to the heart of how this generative memory system actually works in practice. You’re absolutely right that the vast majority of our memory retrievals happen automatically, without conscious effort or deliberate prompt construction. This raises a fundamental question about where and how we can actually intervene to improve this system.

Let me help you think through this by exploring how automatic memory retrieval might work within Barenholtz’s framework, and then we can consider what kinds of interventions might be most effective.

When you automatically remember something, your brain is still essentially running internal prompts, but these prompts are constructed unconsciously and instantaneously based on your current context, emotional state, and the specific cues present in your environment. Think of it like this: your brain has learned thousands of implicit patterns for how to construct effective memory prompts, and it applies these patterns automatically without your conscious awareness.

For example, when you walk into your childhood home and automatically remember playing in the living room, your brain instantaneously constructs something like an internal prompt based on the visual cues, smells, spatial relationships, and emotional associations present in that moment. This happens so quickly and automatically that you experience it as simply “remembering,” but the underlying process still involves your brain generating a reconstruction based on contextual prompts.

The key insight you’re pointing toward is that developing better memory isn’t really about learning to consciously construct prompts in the moment. Instead, it’s about training your unconscious memory system to automatically generate better prompts. This is what you mean by developing “instinct” for triggering better memories.

This is similar to how skilled musicians develop musical instincts. A jazz pianist doesn’t consciously think “now I need to construct a prompt to remember the appropriate chord progression for this musical context.” Instead, through extensive practice and experience, they develop unconscious pattern recognition systems that automatically generate appropriate musical responses. Their fingers seem to “know” what to play based on the musical context, even though this knowledge is actually the result of deeply learned patterns that operate below the level of conscious awareness.

So how might we train these unconscious memory systems to generate better automatic prompts? This is where Barenholtz’s insights become particularly practical. The goal isn’t to consciously construct prompts during memory retrieval, but rather to consciously practice better prompt construction during learning and encoding phases, so that these patterns become automatic over time.

Let me give you a concrete example of how this might work. Imagine you’re learning about a historical period. Instead of just reading facts and trying to memorize them, you might consciously practice asking yourself connecting questions: “How does this event relate to the economic conditions of the time? What were people’s daily lives like? How did this connect to earlier developments?” If you practice this kind of contextual thinking consistently during learning, your brain begins to automatically encode information within these richer contextual frameworks.

Later, when you need to remember something about that historical period, your brain automatically reconstructs the memory using the same kinds of contextual connections you practiced during learning. You don’t consciously think “let me construct a prompt about economic conditions and daily life.” Instead, your memory system automatically generates reconstructions that include these contextual elements because that’s the pattern it learned during encoding.

This suggests that the most effective interventions happen during learning rather than during recall. By consciously practicing better ways of encoding information, you’re essentially training your unconscious memory system to automatically generate better prompts later.

Think about your own learning experiences. Have you noticed that certain subjects or topics seem to “stick” better in your memory, while others feel more difficult to recall? Often, this difference correlates with how deeply and contextually you understood the material during learning. When you truly understand something, you’ve built rich associative networks that help your brain automatically generate effective prompts for reconstructing that knowledge later.

This also explains why expertise in any domain involves developing what seems like intuitive knowledge. Expert chess players don’t consciously analyze every possible move. Instead, they’ve developed unconscious pattern recognition systems that automatically generate good moves based on their recognition of board patterns. Similarly, people with good memories in specific domains have often developed unconscious systems for automatically generating effective memory prompts in those areas.

Here’s a thinking exercise that might help you explore this in your own experience: consider a subject or area where you feel you have particularly good memory and recall. What was different about how you learned that material? Did you naturally connect it to things you already knew? Did you think about it in multiple contexts? Did you use it in practical applications? These learning habits might have trained your unconscious memory system to automatically generate better prompts in that domain.

The practical implication is that developing better memory “instincts” is really about developing better learning habits during the encoding phase. By consciously practicing richer, more contextual ways of thinking about new information, you’re training your unconscious memory system to automatically apply these same patterns during recall.

What do you think about this distinction between conscious intervention during learning versus automatic processing during recall? Can you identify any patterns in your own learning experiences where this kind of deep, contextual encoding led to better automatic recall later?

6

● Ludditus: I’m not sure about how we could develop better learning habits so that better prompts would later be used automatically. I guess Barenholtz doesn’t know very much yet either, as he’s still studying the matter. It looks like this is to integrate his Auto-Regression Theory of Cognition, a new “theory of everything” for cognition. Could you suggest something, out of your weights and biases?

Claude: You’re absolutely right that we’re venturing into relatively uncharted territory here. Barenholtz’s Auto-Regression Theory of Cognition is still emerging, and the practical applications are largely unexplored. But let me offer some educated speculations based on what we know about memory, learning, and pattern recognition systems.

I think the key insight might lie in understanding that our brains are constantly learning meta-patterns about how to construct effective internal prompts, even when we’re not consciously aware of it. Think of it like this: every time you successfully remember something, your brain is implicitly learning something about what contextual cues and associative pathways led to that successful reconstruction. Over time, these meta-patterns become the unconscious “prompt templates” that your brain automatically applies.

Here’s where I think we might be able to intervene deliberately. Consider how you might train someone to become better at asking good questions in conversation. You wouldn’t just tell them “ask better questions.” Instead, you might help them notice patterns in what makes some questions more effective than others. You might help them develop an intuitive sense for when a question opens up interesting territory versus when it shuts down conversation.

Similarly, we might be able to train ourselves to develop better “memory questioning instincts” by becoming more aware of the patterns that lead to successful memory reconstruction. Let me suggest a few practical approaches that might help develop these instincts.

First, you might experiment with what I’ll call “memory quality awareness.” When you have a particularly vivid or useful memory recall, try to notice what triggered it and what made it feel so rich and accessible. Was it a specific sensory cue? An emotional context? A particular way of approaching the question? By developing awareness of your own successful memory patterns, you might begin to unconsciously favor the approaches that work best for your particular brain.

Second, consider practicing what we might call “associative flexibility” during learning. Instead of just trying to understand new information in one way, you might deliberately explore multiple different ways of connecting it to things you already know. Think of it like building multiple pathways to the same destination. When your brain later needs to reconstruct that information, it will have learned patterns for approaching it from different angles, increasing the likelihood of successful reconstruction.

Here’s a concrete example of how this might work. Imagine you’re learning about photosynthesis. Instead of just memorizing the chemical equation, you might practice connecting it to multiple different contexts: the economic implications for agriculture, the evolutionary history of plants, the daily rhythm of a tree’s activity, the relationship to your own breathing, the role in global climate patterns. Each of these connections creates a different potential “prompt pathway” that your brain can learn to use automatically.

Third, you might experiment with what I’ll call “reconstruction practice.” Occasionally, instead of looking up information you need, you might first try to reconstruct it from memory, paying attention to what mental approaches help you generate more complete and accurate reconstructions. This is like deliberate practice for your memory system’s prompt generation abilities.

But here’s where I think the most interesting possibility lies. What if we could develop practices that help our brains learn better meta-patterns for prompt construction? This would be like teaching your unconscious memory system to automatically generate more effective prompts by exposing it to examples of what effective prompts look like.

Consider how skilled teachers develop an intuitive sense for what kinds of questions help students understand difficult concepts. They don’t consciously construct these questions using formal rules. Instead, through experience, they develop unconscious patterns for recognizing when a student is confused and automatically generating the kinds of questions that help clarify understanding.

We might be able to develop similar instincts for prompting our own memory systems. Perhaps by consciously practicing good questioning techniques during learning, we can train our brains to automatically apply similar patterns during memory reconstruction. The key would be to practice this enough that it becomes automatic, like developing muscle memory for playing a musical instrument.

Here’s a thinking exercise that might help you explore this idea. Think about a subject where you feel you have particularly good intuitive understanding. When you encounter a new problem or question in that area, notice how your mind automatically approaches it. What kinds of mental questions or frameworks does your brain seem to automatically apply? These automatic approaches might be examples of well-developed “prompt instincts” that your brain has learned through extensive experience in that domain.

The fascinating possibility is that we might be able to deliberately develop these kinds of instincts for memory reconstruction more generally. By becoming more conscious of the patterns that lead to successful memory reconstruction, and by practicing approaches that seem to work well, we might be able to train our unconscious memory systems to automatically generate better prompts.

What’s your intuition about this? When you think about areas where you feel you have good memory or understanding, can you notice any patterns in how your mind automatically approaches questions or problems in those domains? Do you think it might be possible to deliberately develop these kinds of unconscious patterns for memory reconstruction more generally?

7

● Ludditus: The “reconstruction practice” idea seems a good one! This memory gymnastics morphs into prompt gymnastics, right? But shouldn’t this occasionally be combined, when possible, with post hoc checking of the actual information, so that mental approaches could be adjusted? Still, I’m not sure if that’s something everyone can do. Most people might stick to the traditional methods of improving memorization (mnemotechnics still works).

Claude: You’ve hit on something really important here! Yes, that reconstruction practice essentially becomes “prompt gymnastics” for your memory system, and you’re absolutely right that it should be combined with verification when possible. This creates what learning scientists call a “feedback loop” that helps your brain calibrate its reconstruction patterns.

Let me walk you through how this feedback mechanism might work and why it’s so potentially powerful. When you practice reconstructing information from memory and then check it against the actual source, you’re essentially giving your brain data about which internal prompting strategies produce accurate results and which ones lead you astray. Over time, this helps your unconscious memory system learn to favor the approaches that tend to generate more reliable reconstructions.

Think of it like learning to estimate distances. Initially, you might guess that a building is 200 feet away, then walk over and discover it’s actually 300 feet. Your brain automatically adjusts its distance estimation patterns based on this feedback. Similarly, when you attempt to reconstruct a memory, then verify it against the source, your brain can learn to recognize which reconstruction strategies are most reliable.

Here’s where this gets particularly interesting from a practical standpoint. The feedback doesn’t just help you remember that specific piece of information better. More importantly, it helps your brain learn meta-patterns about what kinds of internal prompts tend to produce accurate reconstructions. This is like developing better “memory intuition” that can apply across different domains.

Let me give you a concrete example of how this might work. Imagine you’re studying history and you try to reconstruct the sequence of events leading up to World War I. You might notice that when you approach it chronologically, you miss important economic factors, but when you think about it in terms of alliance systems and economic pressures, you reconstruct a more complete picture. When you check this against your sources, you discover that the alliance-and-economics approach indeed captures more of the essential causal relationships. Your brain learns from this feedback and begins to automatically apply similar analytical frameworks when reconstructing other complex historical events.

But you raise a crucial point about individual differences. Not everyone will find this kind of deliberate memory practice natural or appealing. Some people might prefer traditional mnemonic techniques because they provide more concrete, step-by-step procedures. This raises an interesting question about whether different people might benefit from different approaches to memory improvement.

Consider how different people naturally approach learning. Some people are naturally analytical and might enjoy the process of experimenting with different reconstruction strategies. Others might prefer more structured approaches like memory palaces or acronyms because these provide clear, reliable procedures. Still others might learn best through storytelling or visual imagery. The key insight is that the most effective approach probably depends on how your particular brain naturally processes and organizes information.

Here’s something worth considering: traditional mnemonic techniques might actually be working through similar mechanisms to what Barenholtz is describing, but in a more structured way. When you create a memory palace, you’re essentially training your brain to use spatial relationships as internal prompts for reconstruction. When you create an acronym, you’re giving your brain a specific verbal prompt that helps it reconstruct a list of items. These techniques might be successful precisely because they provide your brain with reliable prompting strategies.

The difference with Barenholtz’s approach is that instead of learning specific mnemonic tricks, you’re learning to develop better general-purpose prompting instincts. This might be more flexible and powerful in the long run, but it’s also more abstract and might require more experimentation to find what works for your particular brain.

Let me suggest a way to think about this that might help you determine whether this approach would be useful for you. Consider a domain where you already have good memory and recall abilities. How do you naturally approach remembering information in that area? Do you tend to use specific mnemonic techniques, or do you rely more on understanding patterns and relationships? Your natural learning style in areas where you’re already successful might give you clues about whether deliberate reconstruction practice would be helpful for you.

For people who are naturally analytical and enjoy experimenting with different approaches, the reconstruction practice combined with verification might be very effective. For people who prefer more concrete, structured approaches, traditional mnemonic techniques might remain the better choice. And for many people, a combination of both approaches might work best, using structured techniques for some types of information and more flexible reconstruction strategies for others.

The key insight is that understanding how memory works as a generative system can help you make more informed choices about which memory improvement strategies to use. Even if you prefer traditional mnemonic techniques, understanding the underlying generative process can help you use those techniques more effectively by helping you understand why they work and how to adapt them to different situations.

What’s your sense of your own natural learning style? Do you tend to prefer structured, step-by-step approaches, or do you enjoy experimenting with different ways of understanding and remembering information? This might help you decide whether the reconstruction practice approach would be worth exploring for you personally.

8

● Ludditus: I find memory palaces to be overrated, as most people would struggle with such a concept. Either way, I do tend to prefer structured learning, especially with rule-based information, such as a foreign language. But I find this to be a liability, because when learning a foreign language by trying to understand absolutely all the applicable grammar rules that led to a particular sentence, there’s “too much CPU” involved. A more LLM-like approach would be much more practical, but I’m too rigid in certain aspects. Too algorithmic. And this is exactly why I’m so torn between rejecting and embracing Barenholtz’s theories.

Claude: You’ve identified something really profound about your own learning style, and honestly, this tension you’re describing between structured and intuitive approaches is at the heart of one of the most fascinating questions in cognitive science. Let me help you work through this apparent contradiction, because I think understanding it might actually help you find a way to benefit from both approaches.

First, let’s examine what’s happening when you learn a foreign language through grammar rules. Your brain is essentially building what we might call an “explicit rule system” – a conscious, analytical framework for understanding how the language works. This approach has real advantages: it gives you systematic understanding, helps you recognize patterns, and provides a logical foundation for generating correct sentences. But as you’ve noticed, it also requires a lot of conscious processing power, which can make natural conversation feel labored and artificial.

Now, here’s where Barenholtz’s ideas become particularly relevant to your situation. When native speakers use their language, they’re not consciously applying grammar rules. Instead, they’re using something much more like an LLM – they’ve internalized patterns from massive exposure to the language, and they generate sentences by following these learned patterns automatically, without conscious rule application. This is what you’re describing as the “LLM-like approach.”

But here’s the key insight I want to help you understand: you don’t necessarily have to choose between these approaches. Instead, you might think of them as serving different functions in your learning process. The structured, rule-based approach might be most useful during the initial learning phase, when you’re building foundational understanding. The more intuitive, pattern-based approach might be what you gradually transition toward as you gain experience with the language.

Let me give you a concrete example of how this might work. Imagine you’re learning Spanish verb conjugations. Initially, you might consciously learn the rules: “for -ar verbs in present tense, add -o for yo, -as for tú, -a for él/ella,” and so on. This systematic approach helps you understand the underlying logic and gives you a reliable way to generate correct forms when you’re unsure.

But as you encounter these verb forms repeatedly in context, your brain begins to internalize the patterns at a deeper level. Eventually, you might find yourself automatically saying “hablo” without consciously thinking “this is a first-person singular present tense form of an -ar verb.” The rules you learned systematically have become unconscious patterns that generate correct forms automatically.

This suggests that your structured learning approach might actually be an excellent foundation for developing the more intuitive, LLM-like processing you’re seeking. By understanding the rules explicitly, you’re giving your brain the framework it needs to recognize patterns more effectively. The key is learning to gradually let go of conscious rule application as the patterns become more automatic.

Here’s a practical way to think about this transition. During your initial learning, you might consciously apply rules to understand why sentences are constructed the way they are. But as you encounter similar constructions repeatedly, you can practice letting your brain generate responses based on pattern recognition rather than explicit rule application. This is like learning to drive a car – initially you consciously think about each step, but eventually the process becomes automatic.

The fascinating thing about your situation is that your “too algorithmic” approach might actually be preparing your brain to develop better intuitive patterns later. When you understand the underlying rules, you’re better equipped to recognize when those rules are being applied, even if you’re not consciously thinking about them. This systematic understanding might help your brain learn to generate more accurate patterns during the intuitive phase.

Let me suggest a way to experiment with bridging these approaches. When you’re practicing your foreign language, try alternating between analytical and intuitive modes. Spend some time consciously analyzing why sentences are constructed the way they are, building your systematic understanding. Then spend some time trying to generate responses based on feel and pattern recognition, without explicitly thinking about rules. Over time, you might find that the analytical understanding supports and improves your intuitive responses.

This connects directly to Barenholtz’s ideas about prompt engineering. Your systematic, rule-based understanding might be helping your brain develop better internal prompts for generating language. When you understand why certain constructions work, your brain learns to automatically generate similar constructions in appropriate contexts. The rules become implicit prompts that guide your language generation without requiring conscious attention.

Here’s a thinking exercise that might help you explore this idea. Think about aspects of your native language that you once learned systematically but now use intuitively. Perhaps you once consciously learned punctuation rules or sentence structure, but now you apply these automatically when writing. How did that transition happen? What role did your initial systematic understanding play in developing your current intuitive abilities?

The key insight is that systematic learning and intuitive processing don’t have to be in opposition. Instead, they might be different phases of the same learning process. Your structured approach might be exactly what your brain needs to develop the rich pattern recognition that enables more fluent, LLM-like language generation later.

What’s your experience been when you’ve tried to let go of conscious rule application? Do you find that your systematic understanding helps or hinders your ability to generate language intuitively? Understanding this might help you figure out how to use both approaches more effectively.

9

● Ludditus: But how does this apply to memorizing images? Barenholtz: “What does your mother look like?” There is no image of your mother in the system, so it will be generated.

Claude: This is where Barenholtz’s theory gets truly mind-bending, and I think working through the visual memory example will help you understand how radical this reconceptualization really is. Let me walk you through this step by step, because it challenges some of our most basic assumptions about how memory works.

When Barenholtz says there’s no stored image of your mother in your memory system, he’s suggesting something that feels almost impossible to believe at first. After all, you can clearly “see” your mother when you think about her, right? But let’s examine what might actually be happening during that visual reconstruction process.

Think about this carefully: when you try to visualize your mother’s face right now, pay attention to how that image forms in your mind. Does it appear instantly as a complete, detailed photograph? Or do you find that different features seem to emerge and shift as you focus on them? You might notice that when you focus on her eyes, the details of her mouth become less clear, and when you focus on her smile, other features become hazier. This suggests that you’re not accessing a stored complete image, but rather reconstructing visual information piece by piece.

Here’s where the generative model becomes fascinating. Instead of storing a complete visual file of your mother’s face, your brain might have learned the patterns and relationships that characterize her appearance. Think of it like learning the “grammar” of your mother’s face – the way her eyes relate to her nose, the typical expression patterns, the color relationships, the proportional structures that make her face distinctly hers.

Let me give you an analogy that might help clarify this concept. Imagine you’re a skilled portrait artist who has drawn the same person many times. You don’t need to keep a photograph to create a recognizable portrait of that person. Instead, you’ve internalized the essential patterns and relationships that define their appearance. You’ve learned the proportions, the characteristic expressions, the way light typically falls on their features. When you want to draw them again, you generate a new portrait based on these learned patterns, and each drawing might be slightly different while still being recognizably the same person.

This is similar to what Barenholtz suggests happens with visual memory. Your brain has learned the essential patterns and relationships that characterize your mother’s appearance, and when you “remember” what she looks like, you’re actually generating a visual reconstruction based on these learned patterns.

But here’s where this gets really interesting for understanding how memory works more broadly. The patterns your brain has learned aren’t just visual – they’re also emotional, contextual, and associative. When you reconstruct your mother’s face, you’re not just generating visual features. You’re also reconstructing the emotional associations, the typical contexts where you see her, the expressions that are most characteristic of her personality. All of these elements influence how your brain generates the visual reconstruction.

This explains why your memory of your mother’s face might be influenced by your current emotional state, recent interactions with her, or even the context in which you’re trying to remember her. If you’re feeling particularly fond of her, your brain might generate a reconstruction that emphasizes her warm, caring expressions. If you’re thinking about a specific conversation, you might reconstruct her face with the expression she typically has during serious discussions.

Now, let’s connect this back to your earlier question about the physics textbook. When you recalled that page layout, you weren’t accessing a stored photograph of the page. Instead, your brain had learned the patterns and relationships that characterized how information was organized on that page. You had internalized the spatial grammar of physics textbooks – where equations typically appear, how diagrams relate to text, what the visual hierarchy looks like. When you needed to recall the page, your brain generated a reconstruction based on these learned patterns.

Here’s a thinking exercise that might help you explore this in your own experience. Try to visualize the front door of your childhood home. Pay attention to how this image forms in your mind. Do you see the entire door at once, or do you find yourself building up details – the color, the handle, the surrounding frame, any decorative elements? Notice whether the image seems to shift or change as you focus on different aspects. This reconstruction process might give you insight into how your brain generates visual memories rather than simply retrieving stored images.

The practical implications of this are fascinating. If visual memory works through pattern recognition and generation rather than storage and retrieval, then improving visual memory might involve developing better pattern recognition skills. This could mean learning to pay more attention to the essential relationships and structures in visual information, rather than trying to memorize every detail.

For example, when you want to remember a complex diagram or visual layout, you might focus on understanding the underlying organizational principles rather than trying to memorize every visual element. You might ask yourself questions like: “What’s the basic structure here? How are the different elements related to each other? What patterns organize this information?” By understanding these deeper patterns, you’re giving your brain better tools for generating accurate reconstructions later.

This connects back to your language learning challenge in an interesting way. Just as you don’t need to store complete images to remember visual information, you don’t need to consciously apply every grammar rule to generate correct language. Instead, you can learn to recognize and internalize the essential patterns that characterize fluent language use. Your systematic study of grammar rules might be helping your brain learn these patterns more effectively, even if the end goal is to use them intuitively rather than analytically.

What’s your experience when you try to visualize familiar faces or places? Do you notice the kind of reconstructive process I’m describing, or does it feel more like you’re accessing complete, stored images? Understanding your own visual memory experience might help you determine whether this generative model resonates with how your mind actually works.

“I knew who I was this morning, but I’ve changed a few times since then.”
— Alice’s Adventures in Wonderland, Chapter 5, Advice from a Caterpillar