The big million-dollar question that I  have, that I've been trying to get the  

answer to through all these interviews with  AI researchers: How does the brain do it? 

We're throwing way more data at  these LLMs and they still have a  

small fraction of the total capabilities  that a human does. So what's going on? 

This might be the quadrillion-dollar  question or something like that. 

You can make an argument that this is  the most important question in science. 

I don't claim to know the answer. I also don't think that the answer  

will necessarily come even from a lot of smart  people thinking about it as much as they are. 

My overall meta-level take is that we have  to empower the field of neuroscience to  

just make neuroscience a more powerful  field technologically and otherwise,  

to actually be able to crack a question like this. Maybe the way that we would think about this now  

with modern AI, neural nets, deep learning,  is that there are certain key components of  

that. There's the architecture. There's  maybe hyperparameters of how many layers  

you have or properties of that architecture. There is the learning algorithm itself. How  

do you train it? Backprop, gradient descent,  is it something else? How is it initialized?  

If we take the learning part of the system, it  still may have some initialization of the weights. 

And then there are also cost functions. What is it being trained to do? What's the  

reward signal? What are the loss  functions, supervision signals? 

My personal hunch within that framework  is that the field has neglected  

the role of these very specific loss  functions, very specific cost functions. 

Machine learning tends to like  mathematically simple loss functions. 

Predict the next token, cross-entropy, these  simple computer scientist loss functions. 

I think evolution may have built a lot of  complexity into the loss functions actually,  

many different loss functions  for different areas turned on at  

different stages of development. A lot of Python code, basically,  

generating a specific curriculum for what  different parts of the brain need to learn. 

Because evolution has seen many times what was  successful and unsuccessful, and evolution could  

encode the knowledge of the learning curriculum. In the machine learning framework, maybe we can  

come back and we can talk about where do  the loss functions of the brain come from? 

Can different loss functions lead  to different efficiency of learning? 

People say the cortex has got the universal  human learning algorithm, the special sauce  

that humans have. What's up with that? This is a huge question and we don't know. 

I've seen models where the cortex… The cortex  typically has this six-layered structure,  

layers in a slightly different  sense than layers of a neural net. 

Any one location in the cortex has six physical  layers of tissue as you go in layers of the sheet. 

And those areas then connect to each other  and that's more like the layers of a network. 

I've seen versions of that where what  you're trying to explain is just,  

"How does it approximate backprop?" And what is the cost function for that? 

What is the network being asked to do, if you  are trying to say it's something like backprop? 

Is it doing backprop on next token prediction or  is it doing backprop on classifying images or what  

is it doing? And no one knows. But one thought  about it, one possibility about it, is that it's  

just this incredibly general prediction engine. So any one area of the cortex is just trying to  

predict… Basically can it learn to predict  any subset of all the variables it sees  

from any other subset? Omnidirectional  inference, or omnidirectional prediction.  

Whereas an LLM is just seeing everything in  the context window and then it computes a  

very particular conditional probability which  is, "Given all the last thousands of things,  

what are the probabilities for the next token." But it would be weird for a large language  

model to say "the quick brown fox blank blank  the lazy dog" and fill in the middle versus  

doing the next token, if it's doing just forward. It can learn how to do that stuff at this emergent  

level of the context window and everything, but  natively it's just predicting the next token. 

What if the cortex is natively  made so that any area of cortex  

can predict any pattern in any subset of  its inputs given any other missing subset? 

That is a little bit more like "probabilistic AI". A lot of the things I'm saying, by the way, are  

extremely similar to what Yann LeCun would say. He's really interested in these energy-based  

models and something like that is like, the  joint distribution of all the variables. 

What is the likelihood or unlikelihood  of just any combination of variables? 

If I clamp some of them and I say that  definitely these variables are in these states,  

then I can compute, with probabilistic sampling  for example—conditioned on these being set in  

this state, and these could be any arbitrary  subset of variables in the model—can I predict  

what any other subset is going to do and sample  from any other subset given clamping this subset? 

And I could choose a totally different subset and  sample from that subset. So it's omnidirectional  

inference. And so there could be some parts of  the cortex, there might be association areas  

of cortex that predict vision from audition. There might be areas that predict things that  

the more innate part of the brain is going to do. Because remember, this whole thing is riding on  

top of a lizard brain and  lizard body, if you will. 

And that thing is a thing  that's worth predicting too. 

You're not just predicting do  I see this or do I see that. 

Is this muscle about to tense? Am I about to have a reflex where I laugh? 

Is my heart rate about to go up? Am I about to activate this instinctive behavior? 

Based on my higher-level understanding… Like I can  match somebody has told me there's a spider on my  

back to this lizard part that would activate if  I was literally seeing a spider in front of me. 

You learn to associate the two so that  even just from somebody hearing you say  

"There's a spider on your back" Well, let's come back to this. 

This is partly having to do with Steve Byrnes’  theories, which I'm recently obsessed about. 

But on your podcast with Ilya, he said, "Look,  I'm not aware of any good theory of how evolution  

encodes high-level desires or intentions." I think this is very connected to all of  

these questions about the loss functions and  the cost functions that the brain would use. 

And it's a really profound question, right? Let's say that I am embarrassed for saying  

the wrong thing on your podcast  because I'm imagining that Yann LeCun  

is listening and he says, "That's not my theory. You described energy-based models really badly." 

That's going to activate in me  innate embarrassment and shame,  

and I'm going to want to go hide and whatever. That's going to activate these innate reflexes. 

That's important because I might otherwise get  killed by Yann LeCun's marauding army of other… 

The French AI researchers  are coming for you, Adam. 

So it's important that I have  that instinctual response. 

But of course, evolution has never seen Yann  LeCun or known about energy-based models or  

known what an important scientist or a podcast is. Somehow the brain has to encode this desire to  

not piss off really important people in the tribe  or something like this in a very robust way,  

without knowing in advance all the things  that the Learning Subsystem of the brain,  

the part that is learning cortex and other parts…  The cortex is going to learn this world model. 

It's going to include things  like Yann LeCun and podcasts. 

And evolution has to make sure that those neurons,  whatever the Yann-LeCun-being-upset-with-me  

neurons, get properly wired up to the shame  response or this part of the reward function.  

And this is important, right? Because  if we're going to be able to seek status  

in the tribe or learn from knowledgeable  people, as you said, or things like that,  

exchange knowledge and skills with friends but  not with enemies… We have to learn all this stuff. 

It has to be able to robustly wire these learned  features of the world, learned parts of the world  

model, up to these innate reward functions,  and then actually use that to then learn more. 

Because next time I'm not going to try to piss off  Yann LeCun if he emails me that I got this wrong. 

We're going to do further learning based on that. In constructing the reward function, it has to  

use learned information. But how can evolution,  

which didn't know about Yann LeCun, do that? The basic idea that Steve Byrnes is proposing  

is that part of the cortex, or other areas like  the amygdala that learn, what they're doing is  

they're modeling the Steering Subsystem. The Steering Subsystem is the part with  

these more innately programmed responses  and the innate programming of these series  

of reward functions, cost functions,  bootstrapping functions that exist. 

There are parts of the amygdala, for example,  

that are able to monitor what those  parts do and predict what those parts do. 

How do you find the neurons that  are important for social status? 

Well, you have some innate heuristics of social  status, for example, or you have some innate  

heuristics of friendliness that  the Steering Subsystem can use. 

And the Steering Subsystem actually has  its own sensory system, which is crazy. 

We think of vision as being  something that the cortex does. 

But there's also a Steering Subsystem,  subcortical visual system called the  

superior colliculus with innate ability  to detect faces, for example, or threats. 

So there's a visual system that  has innate heuristics and the  

Steering Subsystem has its own responses. There'll be part of the amygdala or part  

of the cortex that is learning  to predict those responses. 

What are the neurons that matter in the  cortex for social status or for friendship? 

They're the ones that predict those  innate heuristics for friendship. 

You train a predictor in the cortex and you  say, "Which neurons are part of the predictor?" 

Those are the ones that, now you've  actually managed to wire it up. 

This is fascinating. I feel like I still don't  understand… I understand how the cortex could  

learn how this primitive part of the brain would  respond to… Obviously it has these labels on,  

"here's literally a picture of a spider,  and this is bad, be scared of this." 

The cortex learns that this is bad  because the innate part tells it that. 

But then it has to generalize to,  "Okay, the spider's on my back. 

And somebody's telling me the spider's  on your back. That's also bad." 

Yes. But it never  

got supervision on that. So how does it…? Well, it's because the Learning Subsystem  

is a powerful learning algorithm that does have  generalization, that is capable of generalization. 

The Steering Subsystem, these  are the innate responses. 

You're going to have some built into your Steering  Subsystem, these lower brain areas: hypothalamus,  

brainstem, et cetera. Again, they have their  

own primitive sensory systems. So there may be an innate response. 

If I see something that's moving fast toward my  body that I didn't previously see was there and  

is small and dark and high contrast, that might be  an insect skittering onto my body. I am going to  

flinch. There are these innate responses. There's  going to be some group of neurons, let's say,  

in the hypothalamus, that is the I-am-flinching  or I-just-flinched neurons in the hypothalamus. 

When you flinch, first of all, it’s a  negative contribution to the reward function. 

You didn't want that to happen, perhaps. 

But that's a reward function that  doesn't have any generalization in it. 

I'm going to avoid that exact situation  of the thing skittering toward me. 

Maybe I'm going to avoid some actions  that lead to the thing skittering. 

That's a generalization you can get, what  Steve calls downstream of the reward function. 

I'm going to avoid the situation where  the spider was skittering toward me,  

but you're also going to do something else. There's going to be a part of your amygdala,  

say, that is saying, "Okay, a few milliseconds,  hundreds of milliseconds or seconds earlier,  

could I have predicted that flinching response?" It's going to be a group of neurons that  

is essentially a classifier  of, "Am I about to flinch?" 

And I'm going to have classifiers for that  for every important Steering Subsystem  

variable that evolution needs to take care  of. Am I about to flinch? Am I talking to a  

friend? Should I laugh now? Is the friend high  status? Whatever variables the hypothalamus,  

brainstem, contains… Am I about to taste salt? It's going to have all these variables and for  

each one it's going to have a predictor. It's going to train that predictor. 

Now the predictor that it trains,  that can have some generalization. 

The reason it can have some generalization is  because it just has a totally different input. 

Its input data might be things like the word  "spider", but the word "spider" can activate  

in all sorts of situations that lead to the  word "spider" activating in your world model. 

If you have a complex world model  with really complex features that  

inherently gives you some generalization. It's not just the thing skittering toward me,  

it's even the word "spider" or the concept of  "spider" is going to cause that to trigger.  

This predictor can learn that. Whatever spider  neurons are in my world model, which could even  

be a book about spiders or somewhere, a room  where there are spiders or whatever that is… 

The amount of heebie-jeebies that this  conversation is eliciting in the audience… 

Now I'm activating your Steering Subsystem,  your Steering Subsystem spider hypothalamus  

subgroup of neurons of skittering insects  are activating based on these very abstract  

concepts in the conversation. If you keep going, I'm going  

to put in a trigger warning. That's because you learned this.  

The cortex inherently has the ability to  generalize because it's just predicting  

based on these very abstract variables and  all these integrated information that it has. 

Whereas the Steering Subsystem  only can use whatever the superior  

colliculus and a few other sensors can spit out. By the way, it's remarkable that the person who's  

made this connection between different pieces of  neuroscience, Steve Byrnes, is a former physicist. 

For the last few years, he’s  been trying to synthesize— 

He's an AI safety researcher.  He's just synthesizing. This  

comes back to the academic incentives thing. I think that this is a little bit hard to say. 

What is the exact next experiment? How am I going to publish a paper on this? 

How am I going to train my grad student to  do this? It’s very speculative. But there's  

a lot in the neuroscience literature and  Steve has been able to pull this together. 

And I think that Steve has an answer to Ilya's  question essentially, which is, how does the brain  

ultimately code for these higher-level desires  and link them up to the more primitive rewards? 

Very naive question, but why can't we achieve  this omnidirectional inference by just training  

the model to not just map from a token to next  token, but remove the masks in the training so  

it maps every token to every token, or come up  with more labels between video and audio and  

text so that it's forced to map one to each one? I mean, that may be the way. It's not clear to  

me. Some people think that there's a different  way that it does probabilistic inference or a  

different learning algorithm that isn't backprop. There might be other ways of learning,  

energy-based models or other things like that,  that you can imagine that is involved in being  

able to do this and that the brain has that. But I think there's a version of it where what  

the brain does is crappy versions of backprop to  learn to predict through a few layers and that  

it’s kind of like a multimodal foundation model. LLMs are maybe just predicting the next token. 

But vision models maybe are trained in learning  to fill in the blanks or reconstruct different  

pieces or combinations. But I think that  

it does it in an extremely flexible way. If you train a model to just fill in this  

blank at the center, okay, that's great. But what if you didn't train it to fill  

in this other blank over to the left? Then it doesn't know how to do that. 

It's not part of its repertoire of predictions  that are amortized into the network. 

Whereas with a really powerful inference  system, you could choose at test time,  

what is the subset of variables it needs  to infer and which ones are clamped? 

Okay, two sub-questions. One, it makes you wonder  whether the thing that is lacking in artificial  

neural networks is less about the reward function  and more about the encoder or the embedding… Maybe  

the issue is that you're not representing video  and audio and text in the right latent abstraction  

such that they could intermingle and conflict. Maybe this is also related to why LLMs seem bad  

at drawing connections between different ideas. Are the ideas represented at a level of generality  

at which you could notice different connections? Well, the problem is these questions  

are all commingled. If we don't know if it's  

doing a backprop-like learning, and we don't  know if it's doing energy-based models, and we  

don't know how these areas are even connected  in the first place, it's very hard to really  

get to the ground truth of this. But yeah, it's  possible. I think that people have done some work. 

My friend Joel Dapello actually did something  some years ago where he put a model—I think it  

was a model of V1, specifically how the  early visual cortex represents images—as  

an input into a convnet and that improves  some things. It could be differences. The  

retina is also doing motion detection and  certain things are getting filtered out. 

There may be some preprocessing  of the sensory data. 

There may be some clever combinations of which  modalities are predicting which or so on,  

that lead to better representation. There may be much more clever things than that. 

Some people certainly do think that there's  inductive biases built in the architecture  

that will shape the representations differently  or that there are clever things that you can do. 

Astera, which is the same organization that  employs Steve Byrnes, just launched this  

neuroscience project based on Doris Tsao's work. She has some ideas about how you can build vision  

systems that basically require less training. They build into the assumptions of the design  

of the architecture things like objects are  bounded by surfaces and surfaces have certain  

types of shapes and relationships of how  they occlude each other and stuff like that. 

It may be possible to build more  assumptions into the network. 

Evolution may have also put  some changes of architecture. 

It's just I think that also the cost functions  and so on may be a key thing that it does. 

I want to talk about this idea that you just  glanced off of which was amortized inference. 

Maybe I should try to explain what I think  it means, because I think it's probably wrong  

and this will help you correct me. It's been a few years for me too. 

Right now, the way the models work is that  you have an input, it maps it to an output,  

and this is amortizing a process, the real  process, which we think is what intelligence is. 

It’s that you have some prior over how  the world could be, what are the causes  

that make the world the way that it is. And then when you see some observation,  

you should be like, "Okay, here's  all the ways the world could be. 

This cause explains what's happening best." Now, doing this calculation over every possible  

cause is computationally intractable. So then you just have to sample like,  

"Oh, here's a potential cause. Does this  explain this observation? No, forget it. Let's  

keep sampling." And then eventually you get the  cause, then the cause explains the observation,  

and then this becomes your posterior. That’s actually pretty good. Bayesian inference  

in general is of this very intractable thing. The algorithms that we have for doing that tend  

to require taking a lot of samples, Monte  Carlo methods, taking a lot of samples.  

And taking samples takes time. This is like  the original Boltzmann machines and stuff. 

They're using techniques like this, and still  it's used with probabilistic programming,  

other types of methods often. The Bayesian inference problem,  

which is basically the problem of perception,  given some model of the world and given some data,  

how should I update my… What are the  missing variables in my internal model? 

And I guess the idea is that neural networks  are hopefully… Obviously, mechanistically,  

the neural network is not starting  with, "Here is my model of the world,  

and I'm going to try to explain this data." But the hope is that instead of starting with,  

"Hey, does this cause explain this observation?  No. Did this cause explain this observation? Yes."  

What you do is just like observation… What's the cause that the  

neural net thinks is the best one? Observation to cause. So the feedforward  

goes observation to cause to then the output that… You don't have to evaluate all these energy values  

or whatever and sample around  to make them higher and lower. 

You just say, approximately that  process would result in this being  

the top one or something like that. Exactly. One way to think about it  

might be that test-time compute, inference-time  compute is actually doing this sampling again. 

You literally read its chain of thought. It's actually doing this toy example we're  

talking about where it's like, "Oh,  can I solve this problem by doing X? 

Nah, I need a different approach." This raises the  question. I mean, over time it is the case that  

the capabilities which required inference-time  compute to elicit, get distilled into the model. 

So you're amortizing the thing which  previously you needed to do these rollouts,  

these Monte Carlo rollouts, to figure out. In general, maybe there's this principle that  

digital minds which can be copied, have  different tradeoffs which are relevant,  

from biological minds which cannot. So in general, it should make sense  

to amortize more things because you can  literally copy the amortization, or copy  

the things that you have sort of built in. This is a tangential question where it  

might be interesting to speculate about. In the future, as these things become more  

intelligent and the way we train them becomes more  economically rational, what will make sense to  

amortize into these minds, which evolution did not  think was worth amortizing into biological minds? 

You have to retrain every time. First of all, I think the probabilistic  

AI people would be like, of course you need  test-time compute, because this inference  

problem is really hard and the only ways we know  how to do it involve lots of test-time compute. 

Otherwise it's just this crappy approximation  that's never going to… You have to do infinite  

data or something to make this. I think some of the probabilistic  

people will be like, "No, it's  inherently probabilistic and amortizing  

it in this way just doesn't make sense." They might then also point to the brain and say,  

"Okay, well the brain, the neurons are stochastic  and they're sampling and they're doing things. 

So maybe the brain actually is doing more like  the non-amortized inference, the real inference." 

But it's also strange how perception can  work in just milliseconds or whatever. 

It doesn't seem like it uses that much sampling. So it's also clearly doing some baking things  

into approximate forward passes  or something like that to do this. 

In the future, I don't know. Is it already a trend to some  

degree that things that people were having to  use test-time compute for, are getting used  

to train back the base model? Now it can do it in one pass. 

Maybe evolution did or didn't do that. I think evolution still has to pass  

everything through the genome to build  the network and the environment in which  

humans are living is very dynamic. So maybe, if we believe this is true,  

there's a Learning Subsystem per Steve  Byrnes, and a Steering Subsystem,  

that the Learning Subsystem doesn't have a  lot of pre-initialization or pretraining. 

It has a certain architecture, but  then within lifetime it learns. 

Then evolution didn't actually  amortize that much into that network. 

It amortized it instead into a set of innate  behaviors in a set of these bootstrapping  

cost functions, or ways of building  up very particular reward signals. 

This framework helps explain this mystery that  people have pointed out and I've asked a few  

guests about, which is that if you want  to analogize evolution to pretraining,  

well how do you explain the fact that so little  information is conveyed through the genome? 

So 3 gigabytes is the size  of the total human genome. 

Obviously a small fraction of that is  actually relevant to coding the brain. 

Previously people made this analogy, that actually  evolution has found the hyperparameters of the  

model, the numbers which tell you how many  layers there should be, the architecture,  

basically, how things should be wired together. But if a big part of the story is that increased  

sample efficiency aids learning, generally makes  systems more performant, is the reward function,  

is the loss function—and if evolution found  those loss functions that aid learning—then  

it actually makes sense how you can build  an intelligence with so little information. 

Because the reward function, in Python  the reward function is literally a line. 

So you just have a thousand lines like this,  and that doesn't take up that much space. 

Yes. It also gets to do this generalization  thing with the thing I was describing where  

we were talking about the spider, where it  learns just the word "spider" which triggers  

the spider reflex or whatever. It gets to exploit that too. 

It gets to build a reward function that  actually has a bunch of generalization  

in it just by specifying these innate  spider stuff and the Thought Assessors,  

as Steve calls them, that do the learning. That's potentially a really compact solution  

to building up these more complex  reward functions too, that you need. 

It doesn't have to anticipate everything  about the future of the reward function. 

It just has to anticipate what variables  are relevant and what are heuristics  

for finding what those variables are. And then it has to have a very compact  

specification for the learning algorithm and  basic architecture of the Learning Subsystem. 

And then it has to specify all this Python code  of all the stuff about the spiders and all the  

stuff about friends, and all the stuff about  your mother, and all the stuff about mating  

and social groups and joint eye contact. It has to specify all that stuff.  

So is this really true? I think  that there is some evidence for it. 

Fei Chen and Evan Macosko and  various other researchers have  

been doing these single-cell atlases. One of the things that scaling up  

neuroscience technology—again, this is one of my  obsessions—has done through the BRAIN Initiative,  

a big neuroscience funding program, is  they've basically gone through different  

areas, especially of the mouse brain, and  mapped where the different cell types are? 

How many different types of cells are  there in different areas of cortex? 

Are they the same across different areas? Then you look at these subcortical regions,  

which are more like the Steering Subsystem  or reward-function-generating regions. 

How many different types of cells do they have? And which neuron types do they have? 

We don't know how they're all connected  and exactly what they do or what the  

circuits are or what they mean, but you can just  quantify how many different kinds of cells there  

are with sequencing the RNA. And there are a lot more weird  

and diverse and bespoke cell types  in the Steering Subsystem, basically,  

than there are in the Learning Subsystem. Like the cortical cell types, it seems like  

there's enough to build a learning algorithm  up there and specify some hyperparameters. 

And in this Steering Subsystem, there's like  a gazillion, thousands of really weird cells,  

which might be like the one for the spider flinch  reflex and the one for I'm-about-to-taste-salt. 

Why would each reward function  need a different cell type? 

Well, this is where you get  innately wired circuits. 

In the learning algorithm part, in the Learning  Subsystem, you specify the initial architecture,  

you specify a learning algorithm. All the juice is happening through  

plasticity of the synapses, changes of  the synapses within that big network. 

But it's a relatively repeating  architecture, how it's initialized. 

It's just like how the amount of Python  code needed to make an eight-layer  

transformer is not that different from one  that makes a three-layer transformer. You're  

just replicating. Whereas all this Python code for  the reward function, if superior colliculus sees  

something that's skittering and you're feeling  goosebumps on your skin or whatever, then trigger  

spider reflex, that's just a bunch of bespoke,  species-specific, situation-specific crap. 

The cortex doesn't know about  spiders, it just knows about layers. 

But you're saying that the only way to write this  reward function is to have a special cell type? 

Yeah, well, I think so. I think you either  have to have special cell types or you have to  

somehow otherwise get special wiring rules  that evolution can say this neuron needs  

to wire to this neuron, without any learning. And the way that that is most likely to happen,  

I think, is that those cells express  different receptors and proteins that say,  

"Okay, when this one comes in contact  with this one, let's form a synapse." 

So it's genetic wiring, and  those need cell types to do it. 

I'm sure this would make a lot more sense  if I knew 101 neuroscience, but it seems  

like there's still a lot of complexity, or  generality rather, in the Steering Subsystem. 

So if the Steering Subsystem has its own visual  system that's separate from the visual cortex,  

different features still need  to plug into that vision system. 

So the spider thing needs to plug into it and  also the love thing needs to plug into it,  

et cetera, et cetera. So it seems complicated. It's still complicated. That's all the more  

reason why a lot of the genomic real estate  on the genome, and in terms of these different  

cell types and so on, would go into wiring  up the Steering Subsystem, pre-wiring it. 

Can we tell how much of the  genome is clearly working? 

So I guess you could tell how many  are relevant to producing the RNA  

that manifest or the epigenetics that manifest  in different cell types in the brain. Right? 

Yeah. This is what the cell types help you get at. I don't think it's exactly like, "Oh, this percent  

of the genome is doing this", but you could say,  "Okay, in all these Steering Subsystem subtypes,  

how many different genes are involved in  specifying which is which and how they wire? 

And how much genomic real estate do those  genes take up versus the ones that specify  

visual cortex versus auditory cortex? You're just reusing the same genes to  

do the same thing twice. Whereas the spider  

reflex hooking up… Yes, you're right. They have to build a vision system and they  

have to build some auditory systems and  touch systems and navigation-type systems. 

Even feeding into the hippocampus and stuff  like that, there's head direction cells. 

Even the fly brain has innate  circuits that figure out its  

orientation and help it navigate in the world. It uses vision, figures out its optical flow of  

how it's flying and how its flight  is related to the wind direction. 

It has all these innate stuff that I think  in the mammal brain we would all lump that  

into the Steering Subsystem. There's a lot  of work. So all the genes that basically go  

into specifying all the things a fly has to  do, we're going to have stuff like that too,  

just all in the Steering Subsystem. But do we have some estimate of like,  

"Here's how many nucleotides, here  are many megabases it takes to—" 

I don't know. I mean, I think you might  be able to talk to biologists about this. 

I mean, we have a lot in common  with yeast from a genes perspective. 

Yeast is still used as a model for some amount of  drug development and stuff like that in biology. 

And so much of the genome is just going towards  you having a cell at all, it can recycle waste,  

it can get energy, it can replicate. And then what do we have in common with a mouse? 

So we do know at some level that the differences  between us and a chimpanzee or something—and that  

includes the social instincts and the more  advanced differences in cortex and so on—it's  

a tiny number of genes that go into this  additional amount of making the eight-layer  

transformer instead of the six-layer  transformer or tweaking that reward function. 

This would help explain why the  hominid brain exploded in size so fast. 

Presumably, tell me if this is  correct, but under this story,  

social learning or some other thing increased  the ability to learn from the environment. It  

increased our sample efficiency. Instead of  having to go and kill the boar yourself and  

figure out how to do that, you can just be like,  "The elder told me this is how you make a spear." 

Now it increases the incentive to have a  bigger cortex, which can learn these things. 

Yes and that can be done with a relatively few  genes, because it's really replicating what  

the mouse already has, making more of it. It's maybe not exactly the same and there  

may be tweaks, from a genome perspective,  you don't have to reinvent all this stuff. 

So then how far back in the history of the  evolution of the brain does the cortex go back? 

Is the idea that the cortex has always figured  out this omnidirectional inference thing,  

that's been a solved problem for a long time? Then the big unlock with primates is that we  

got the reward function, which increased the  returns to having omnidirectional inference? 

It’s a good question. Or is the omnidirectional inference  

also something that took a while to unlock? I'm not sure that there's agreement about that. 

I think there might be specific  questions about language. 

Are there tweaks, whether that's  through auditory and memory,  

some combination auditory memory regions? There may also be macro-wiring where you need  

to wire auditory regions into memory regions or  something like that, and into some of these social  

instincts to get language, for example, to happen. But that might also be a small number of gene  

changes to be able to say, "Oh, I just  need from my temporal lobe over here,  

going over to the auditory cortex, something." There is some evidence for the Broca's area,  

Wernicke's area. They're connected with the  

hippocampus and so on and prefrontal cortex. So there's like some small number of genes  

maybe for enabling humans to  really properly do language. 

That could be a big one. But is it that something changed about the  

cortex and it became possible to do these things? Or is that that potential was already there,  

but there wasn't the incentive to expand  that capability and then use it, wire it  

to these social instincts and use it more? I would lean somewhat toward the latter. 

I think a mouse has a lot of similarity  in terms of cortex as a human. 

Although there's Suzana Herculano-Houzel's  work on how the number of neurons scales  

better with weight with primate brains  than it does with rodent brains. 

So does that suggest that there actually was some  improvement in the scalability of the cortex? 

Maybe, maybe. I'm not super deep on this. There may have been changes in architecture,  

changes in the folding, changes  in neuron properties and stuff  

that somehow slightly tweak this. But there's still a scaling. either way. 

That's right. So I'm not saying  

there isn't something special about humans in the  architecture of the Learning Subsystem at all. 

But yeah I think it's pretty widely  thought that this is expanded. 

But then the question is, "Okay, well, how  does that fit in also with the Steering  

Subsystem changes and the instincts  that make use of this and allow you  

to bootstrap using this effectively?" But just to say a few other things,  

even the fly brain has some amount, even very far  back… I mean, I think you've read this great book,  

A Brief History of Intelligence, right? I think this is a really good book. 

Lots of AI researchers think this  is a really good book it seems. 

You have some amount of learning going back  all the way to anything that has a brain. 

Basically you have something like  primitive reinforcement learning,  

going back at least to vertebrates. Imagine a  zebrafish. Then you have these other branches. 

Birds may have reinvented something cortex-like. It doesn't have the six layers,  

but they have something a little bit cortex-like. So some of those things after reptiles, in some  

sense birds and mammals both made a somewhat  cortex-like, but differently organized thing. 

But even a fly brain has associative learning  centers that actually do things that maybe look  

a little bit like this Thought Assessor concept  from Byrnes, where there's a specific dopamine  

signal to train specific subgroups of neurons  in the fly mushroom body to associate different  

sensory information with, "Am I going to get  food now?" or "Am I going to get hurt now?" 

Brief tangent. I remember reading in one  blog post that Beren Millidge wrote that  

the parts of the cortex which are associated with  audio and vision have scaled disproportionately  

between other primates and humans, whereas  the parts associated, say, with odor have not. 

And I remember him saying something like  that this is explained by that kind of  

data having worse scaling law properties. Maybe he meant this, but I think another  

interpretation of actually what's happening  there is that these social reward functions  

that are built into the Steering Subsystem  needed to make use more of being able to  

see your elders and see what the visual  cues are and hear what they're saying. 

And in order to make sense of these cues  which guide learning, you needed to activate  

the vision and audio more than odor. I mean, there's all this stuff. 

I feel like it's come up in  your shows before, actually. 

But like even the design of the human eye where  you have the pupil and the white and everything,  

we are designed to be able to establish  relationships based on joint eye contact. 

Maybe this came up in the Sutton episode. I  can't remember. But yeah, we have to bootstrap  

to the point where we can detect eye contact  and where we can communicate by language. 

That's like what the first couple  years of life are trying to do. 

Okay, I want to ask you about RL. So currently, the way these LLMs are trained,  

if they solve the unit test or solve  a math problem, that whole trajectory,  

every token in that trajectory is upweighted.  What's going on with humans? Are there different  

types of model-based versus model-free that  are happening in different parts of the brain? 

Yeah, I mean, this is another one of these things. Again, all my answers to these questions,  

any specific thing I say, it’s all just saying  that directionally we can explore around this. 

I find this interesting, maybe I feel  like the literature points in these  

directions in some very broad way. What I actually want to do is go and  

map the entire mouse brain and figure this  out comprehensively and make neuroscience  

a ground-truth science. So I don't know,  basically. But first of all, I think with  

Ilya on the podcast, he was like, "It's weird  that you don't use value functions, right?" 

You use the dumbest form of RL basically. Of course these people are incredibly smart  

and they're optimizing for how to do it on GPUs  and it's really incredible what they're achieving. 

But conceptually it's a really dumb form of RL,  even compared to what was being done 10 years ago. 

Even the Atari game-playing stuff was  using Q-learning, which is basically  

a kind of temporal difference learning. The temporal difference learning basically  

means you have some kind of a value function of  what action I choose now doesn't just tell me  

literally what happens immediately after this. It tells me what is the long-run consequence  

of that for my expected total  reward or something like that. 

So you would have value functions  like… The fact that we don't have  

value functions at all in the LLMs is crazy. I think because Ilya said it, I can say it. 

I know one one-hundredth of what he does about  AI, but it's kind of crazy that this is working. 

But in terms of the brain, I think there are  some parts of the brain that are thought to do  

something that's very much like model-free RL,  that's parts of the striatum and basal ganglia. 

It is thought that they have a certain  finite relatively small action space. 

The types of actions they could take, first  of all, might be like, "Tell the brainstem  

and spinal cord to do this motor action?  Yes or no." Or it might be more complicated  

cognitive-type actions like, "Tell the thalamus  to allow this part of the cortex to talk to this  

other part," or "Release the memory that's in the  hippocampus and start a new one or something." 

But there's some finite set of actions  that come out of the basal ganglia,  

and that it's just a very simple RL. So there are probably parts of other  

brains and our brain that are just doing  very simple naive-type RL algorithms. 

Layering one thing on top of that is that  some of the major work in neuroscience,  

like Peter Dayan's work, and a bunch of work that  is part of why I think DeepMind did the temporal  

difference learning stuff in the first place. They were very interested in neuroscience. 

There's a lot of neuroscience evidence that  the dopamine is giving this reward prediction  

error signal, rather than just reward, "yes  or no, a gazillion time steps in the future." 

It's a prediction error and that's consistent  with learning these value functions. 

So there's that and then there's  maybe higher-order stuff. 

We have the cortex making this world model. Well, one of the things the cortex world  

model can contain is a model of  when you do and don't get rewards. 

Again, it's predicting what  the Steering Subsystem will do. 

It could be predicting what  the basal ganglia will do. 

You have a model in your cortex that has  more generalization and more concepts and  

all this stuff that says, "Okay, these types of  plans, these types of actions will lead in these  

types of circumstances to reward." So I have a model of my reward. 

Some people also think that  you can go the other way. 

So this is part of the inference picture. There's this idea of RL as inference. 

You could say, "Well, conditional  on my having a high reward,  

sample a plan that I would have had to get there." 

That's inference of the plan  part from the reward part. 

I'm clamping the reward as high and inferring the  plan, sampling from plans that could lead to that. 

So if you have this very general  cortical thing, it can just do. 

If you have this very general model-based  system and the model, among other things,  

includes plans and rewards, then  you just get it for free, basically. 

So in neural network parlance, there's a  value head associated to the omnidirectional  

inference that's happening in the— Yes, or there's a value input. 

Oh, okay. Interesting. Yeah and it can predict.  

One of the almost sensory variables it can  predict is what rewards it's going to get. 

By the way, speaking about amortizing things,  

obviously value is like amortized  rollouts of looking up reward. 

Yeah, something like that. It's like a  statistical average or prediction of it. 

Tangential thought. Joe Henrich and others have  this idea for the way human societies have learned  

to do things like, how do you figure out that this  kind of bean, which actually just almost always  

poisons you, is edible if you do this ten-step  incredibly complicated process, any one of which  

if you fail, at the bean will be poisonous? How do you figure out how to hunt this seal in  

this particular way, with this particular weapon,  at this particular time of the year, et cetera? 

There's no way but just like  trying shit over generations. 

And it strikes me this is actually  very much like model-free RL happening  

at a civilizational level. No, not exactly. Evolution is the simplest algorithm in some sense. 

If we believe that all of  this can come from evolution,  

the outer loop can be extremely not foresighted. Right, that's interesting. Just hierarchies  

of… Evolution: model-free… So what does that tell you? 

Maybe the simple algorithms can just  get you anything if you do it enough. 

Right. Yeah, I don't know. 

So, evolution: model-free. Basal ganglia:  model-free. Cortex: model-based. Culture:  

model-free potentially. I mean you pay  attention to your elders or whatever. 

Maybe there's like group selection or whatever  of these things is like more model-free. 

But now I think culture, well,  it stores some of the model. 

Stepping back, is it a disadvantage or an  advantage for humans that we get to use  

biological hardware, in comparison  to computers as they exist now? 

What I mean by this question is, if there's  "the algorithm", would the algorithm just  

qualitatively perform much worse or much  better if inscribed in the hardware of today? 

The reason to think it might…. Here's  what I mean. Obviously the brain has  

had to make a bunch of tradeoffs which  are not relevant to computing hardware. 

It has to be much more energetically efficient. Maybe as a result it has to run on slower speeds  

so that there can be a smaller voltage gap. So the brain runs at 200 hertz,  

it has to run on 20 watts. On the other hand, with robotics  

we've clearly experienced that fingers are way  more nimble than we can make motors so far. 

So maybe there's something in the brain that  is the equivalent of cognitive dexterity,  

which is maybe due to the fact that  we can do unstructured sparsity. 

We can co-locate the memory and the compute. Yes. 

Where does this all net out? Are you like, "Fuck, we would  

be so much smarter if we didn't have to  deal with these brains." Or are you like— 

I think in the end we will get  the best of both worlds somehow. 

I think an obvious downside of  the brain is it cannot be copied. 

You don't have external read-write access  to every neuron and synapse, whereas you do. 

I can just edit something in the weight matrix in  Python or whatever and load that up and copy that.  

In principle. So the fact that it can't be  copied and random-accessed is very annoying. 

But otherwise maybe it has a lot of advantages. It also tells you that you want to somehow do  

the co-design of the algorithm. It maybe even doesn't change it  

that much from all of what we discussed,  but you want to somehow do this co-design. 

So yeah, how do you do it with  really slow low-voltage switches? 

That's going to be really important for energy  consumption. Co-locating memory and compute. I  

think that hardware companies will probably  just try to co-locate memory and compute. 

They will try to use lower voltages,  allow some stochastic stuff. 

There are some people that think that all  this probabilistic stuff that we were talking  

about—"Oh, it's actually energy-based models,  so on"—it is doing lots of sampling. It's not  

just amortizing everything. The neurons  are also very natural for that because  

they're naturally stochastic. So you don't have to do a random  

number generator in a bunch of Python  code basically to generate a sample. 

The neuron just generates samples  and it can tune what the different  

probabilities are and learn those tunings. So it could be that it's very co-designed with  

some kind of inference method or something. It'd be hilarious…. I mean the  

message I'm taking from this interview is  that like all these people that folks make  

fun of on Twitter, Yann LeCun and Beff Jezos and  whatever, I don’t know maybe they got it right. 

That is actually one read of it. Granted, I haven't really worked on AI at all  

since LLMs took off, so I'm just out of the loop. But I'm surprised and I think it's amazing how  

the scaling is working and everything. But yeah, I think Yann LeCun and Beff  

Jezos are kind of onto something about the  probabilistic models or at least possibly. 

In fact that's what all the neuroscientists and  all the AI people thought until 2021 or something. 

Right. So there's a bunch of cellular stuff  happening in the brain that is not just about  

neuron-to-neuron synaptic connections. How much of that is functionally doing  

more work than the synapses themselves are doing  versus it's just a bunch of kludge that you have  

to do in order to make the synaptic thing work. So with a digital mind, you can nudge the synapse,  

sorry the parameter, extremely easily. But with a cell to modulate a synapse  

according to the gradient signal, it  just takes all this crazy machinery. 

So is it actually doing more than it  takes extremely little code to do? 

I don't know, but I'm not a believer in  the radical, "Oh, actually memory is not  

synapses mostly, or learning is mostly  genetic changes" or something like that. 

I think it would just make a lot of sense, I  think you put it really well for it to be more  

like the second thing you said. Let's say you want to do weight  

normalization across all the weights coming  out of your neuron or into your neuron. 

Well, you probably have to somehow tell  the nucleus of the cell about this and  

then have that send everything back  out to the synapses or something. 

So there's going to be a lot of cellular changes. 

Or let's say that you just had a lot of  plasticity and you're part of this memory. 

Now that's got consolidated  into the cortex or whatever. 

Now we want to reuse you as a  new one that can learn again. 

There's going to be a ton of cellular  changes, so there's going to be tons  

of stuff happening in the cell. But algorithmically, it's not really  

adding something beyond these algorithms. It's just implementing something that in a  

digital computer is very easy for us to go  and just find the weights and change them. 

In a cell, it just literally has to do  all this with molecular machines itself  

without any central controller. It's kind of  incredible. There are some things that cells do,  

I think, that seem more convincing. One of the things the cerebellum has  

to do is predict over time. What is the time  delay? Let's say that I see a flash and then  

some number of milliseconds later, I'm going  to get a puff of air in my eyelid or something. 

The cerebellum can be very good at  predicting what's the timing between  

the flash and the air puff, so that now  your eye will just close automatically. 

The cerebellum is involved in that  type of reflex, learned reflex. 

There are some cells in the cerebellum where  it seems like the cell body is playing a role  

in storing that time constant, changing that time  constant of delay, versus that all being somehow  

done with like, "I'm going to make a longer  ring of synapses to make that delay longer." 

No, the cell body will just  store that time delay for you. 

So there are some examples, but I'm not a  believer out of the box in essentially this  

theory that what's happening is changes in  connections between neurons and that that's  

the main algorithmic thing that's going on. I think there's very good reason to still  

believe that it's that rather  than some crazy cellular stuff. 

Going back to this whole perspective of how our  intelligence is not just this omnidirectional  

inference thing that builds a world model, but  really this system that teaches us what to pay  

attention to what the important salient  factors are to learn from, et cetera. 

I want to see if there's some intuition we  can drive from this about what different  

kinds of intelligences might be like. So it seems like AGI or superhuman  

intelligence should still have this ability  to learn a world model that's quite general,  

but then it might be incentivized to pay attention  to different things that are relevant for the  

modern post-singularity environment. How different should we expect  

different intelligences to be? I think one way to think about this  

question is, is it actually possible to  make the paperclip maximizer or whatever? 

If you try to make the paperclip maximizer,  does it end up just not being smart or  

something like that because the only reward  function it had was to make paperclips? 

I'd say, can you do that? I don't know. If I  channel Steve Byrnes more, I think he's very  

concerned that the minimum viable things in the  Steering Subsystem that you need to get something  

smart is way less than the minimum viable set of  things you need for it to have human-like social  

instincts and ethics and stuff like that. So a lot of what you want to know about  

the Steering Subsystem is actually the  specifics of how you do alignment essentially,  

or what human behavior and social instincts  is versus just what you need for capabilities. 

We talked about it in a slightly different  way because we were sort of saying, "Well,  

in order for humans to learn socially, they  need to make eye contact and learn from others." 

But we already know from LLMs that  depending on your starting point,  

you can learn language without that stuff. So I think that it probably is possible to  

make super powerful model-based RL optimizing  systems and stuff like that that don't have  

most of what we have in the human brain reward  functions and as a consequence might want to  

maximize paperclips. And that's a concern. But you're pointing out that in order to  

make a competent paperclip maximizer, the  kind of thing that can build spaceships and  

learn physics and whatever, it needs to  have some drives which elicit learning,  

including say curiosity and exploration. Yeah, curiosity, interest in others,  

interest in social interactions. But that's pretty minimal I think. 

And that's true for humans, but it might  be less true for something that's already  

pretrained as an LLM or something. So most of why we want to know  

the Steering Subsystem, I think if I'm  channeling Steve, is alignment reasons. 

How confident are we that we even have the  right algorithmic conceptual vocabulary  

to think about what the brain is doing? What I mean by this is that there was one  

big contribution to AI from neuroscience  which was this idea of the neuron in the  

1950s, just this original contribution. But then it seems like a lot of what we've  

learned afterwards about what the high-level  algorithm the brain is implementing, from  

the backprop to if there's something analogous to  backprop happening in the brain to "Oh is V1 doing  

something like CNNs" to TD learning and Bellman  equations, actor-critic, whatever… It seems  

inspired by this dynamic where we come  up with some idea, maybe we can make AI  

neural networks work this way, and then we notice  that something in the brain also works that way. 

So why not think there's more things like this. There may be. I think the reason that I think  

that we might be onto something is  that the AIs we're making based on  

these ideas are working surprisingly well. There's also a bunch of just empirical stuff. 

Convolutional neural nets and  variants of convolutional neural nets. 

I'm not sure what the absolute latest  is, but compared to other models in  

computational neuroscience of what the visual  system is doing, they are just more predictive. 

You can just score, even pretrained  on cat pictures and stuff, CNNs,  

what is the representational similarity that they  have on some arbitrary other image compared to the  

brain activations measured in different ways? Jim DiCarlo's lab has this brain score and  

the AI model is actually… There  seems to be some relevance there. 

Neuroscience doesn't necessarily  have something better than that. 

So yes, that's just recapitulating what  you're saying, that the best computational  

neuroscience theories we have seem to  have been invented largely as a result  

of AI models and finding things that work. So find backprop works and then saying, "Can  

we approximate backprop with cortical circuits?"  or something. There's been things like that. Now,  

some people totally disagree with this. György Buzsáki is a neuroscientist who has  

a book called The Brain from the Inside Out where  he basically says all our psychology concepts,  

AI concepts, all this stuff is just made-up stuff. What we actually have to do is figure out what  

is the actual set of primitives  that the brain actually uses. 

And our vocabulary is not  going to be adequate to that. 

We have to start with the brain and  make new vocabulary rather than saying  

backprop and then try to apply that  to the brain or something like that. 

He studies a lot of oscillations and stuff  in the brain as opposed to individual neurons  

and what they do. I don't know. I think  that there's a case to be made for that. 

And from a research program design perspective,  one thing we should be trying to do is just  

simulate a tiny worm or a tiny zebrafish, almost  as biophysical or as bottom-up as possible. 

Like get connectome, molecules,  activity and just study it as a physical  

dynamical system and look at what it does. But I don't know, it just feels like AI is  

really good fodder for computational neuroscience. Those might actually be pretty good models. We  

should look at that. I both think that there  should be a part of the research portfolio  

that is totally bottom-up and not trying to apply  our vocabulary that we learn from AI onto these  

systems, and that there should be another big part  of this that's trying to reverse engineer it using  

that vocabulary or variant of that vocabulary. We should just be pursuing both. 

My guess is that the reverse engineering one  is actually going to work-ish or something. 

Like we do see things like TD learning,  which Sutton also invented separately. 

That must be a crazy feeling to just like— Yeah, that's crazy. 

This equation I wrote down is like in the brain. It seems like the dopamine is  

doing some of that, yeah. So let me ask you about this. 

You guys are funding different groups that are  trying to figure out what's up in the brain. 

If we had a perfect representation,  however you define it, of the brain,  

why think it would actually let us  figure out the answer to these questions? 

We have neural networks which are way  more interpretable, not just because we  

understand what's in the weight matrices,  but because there are weight matrices. 

There are these boxes with numbers in them. Even then we can tell very basic things. 

We can kind of see circuits for very basic pattern  matching of following one token with another. 

I feel like we don't really have an  explanation of why LLMs are intelligent  

just because they’re interpretable. I think I would somewhat dispute it. 

We have some description of what  the LLM is fundamentally doing. 

What that's doing is that I have an  architecture and I have a learning  

rule and I have hyperparameters and I have  initialization and I have training data. 

But those are things we learned because we  built them, not because we interpreted them  

from seeing the weights. The analogous thing to  

connectome is like seeing the weights. What I think we should do is we should  

describe the brain more in that language of  things like architectures, learning rules,  

initializations, rather than trying to find the  Golden Gate Bridge circuit and saying exactly  

how this neuron actually… That's going to be  some incredibly complicated learned pattern. 

Konrad Kording and Tim Lillicrap  have this paper from a while ago,  

maybe five years ago, called "What does  it mean to understand a neural network?" 

What they say is basically that you could  imagine you train a neural network to compute  

the digits of pi or something. It's like  some crazy pattern. You also train that  

thing to predict the most complicated  thing you find, predict stock prices,  

basically predict really complex systems,  computationally complete systems. 

I could train a neural network to do  cellular automata or whatever crazy thing. 

It's like, we're never going to be able to fully  capture that with interpretability, I think. 

It's just going to just be doing really  complicated computations internally. 

But we can still say that the way it got that  way is that it had an architecture and we gave it  

this training data and it had this loss function. So I want to describe the brain in the same way. 

And I think that this framework that  I've been kind of laying out is that  

we need to understand the cortex and  how it embodies a learning algorithm. 

I don't need to understand how  it computes "Golden Gate Bridge." 

But if you can see all the neurons, if you have  the connectome, why does that teach you what  

the learning algorithm is? Well, I guess there are a  

couple different views of it. So it depends on these different  

parts of this portfolio. On the totally bottom-up,  

we-have-to-simulate-everything  portfolio, it kind of just doesn't. 

You have to make a simulation of the zebrafish  brain or something and then you see what are  

the emergent dynamics in this and you come up  with new names and new concepts and all that. 

That's the most extreme  bottom-up neuroscience view. 

But even there the connectome  is really important for doing  

that biophysical or bottom-up simulation. But on the other hand you can say, "Well,  

what if we can actually apply some ideas from AI?" We basically need to figure out,  

is it an energy-based model or is  it an amortized VAE-type model? 

Is it doing backprop or is  it doing something else? 

Are the learning rules local or global? If we have some repertoire of possible  

ideas about this, just think of the connectome  as a huge number of additional constraints that  

will help to refine, to ultimately  have a consistent picture of that. 

I think about this for the Steering Subsystem  stuff too, just very basic things about it. 

How many different types of dopamine signal  or of Steering Subsystem signal or thought  

assessor or so on… How many different  types of what broad categories are there? 

Like even this very basic information  that there's more cell types in the  

hypothalamus than there are in the cortex,  that's new information about how much  

structure is built there versus somewhere else. How many different dopamine neurons are there? 

Is the wiring between prefrontal and auditory the  same as the wiring between prefrontal and visual? 

The most basic things, we don't know. The problem is learning even the  

most basic things by a series of bespoke  experiments takes an incredibly long time. 

Whereas just learning all that at once by  getting a connectome is just way more efficient. 

What is the timeline on this? Presumably the idea of this is,  

first, to inform the development of AI. You want to be able to figure out how  

we get AIs to want to care about what other  people think of its internal thought pattern. 

But interp researchers are making progress on  this question just by inspecting normal neural  

networks. There must be some feature… You can do interp on LLMs that exist. 

You can't do interp on a hypothetical model-based  reinforcement algorithm like the brain that we  

will eventually converge to when we do AGI. Fair. But what timelines on AI do you need  

for this research to be practical and relevant? I think it's fair to say it's not super practical  

and relevant if you're in an AI 2027 scenario. And so what science I'm doing now is not going  

to affect the science of ten years from now. Because what's going to affect the science  

of 10 years from now is the  outcome of this AI 2027 scenario. 

It kind of doesn't matter that much  probably if I have the connectome,  

maybe it slightly tweaks certain things. But I think there's a lot of reason to think  

maybe that we will get a lot out of this paradigm. But then the real thing, the thing that is the  

single event that is transformative for the  entire future or something type event is still  

more than five years away or something. Is that because we haven't captured  

omnidirectional inference, we haven't figured  out the right ways to get a mind to pay attention  

to things in a way that makes sense? I mean, I would take the entirety of  

your collective podcast with everyone  as showing the distribution of these  

things. I don't know. What was Karpathy's  timeline, right? What's Demis's timeline? So  

not everybody has a three-year timeline. But there are different reasons and I'm  

curious which ones are yours. What are mine?  

I don't know, I'm just watching your podcast. I'm trying to understand the distribution. 

I don't have a super strong  claim that LLMs can't do it. 

But is the crux the data efficiency or…? I think part of it is just that it is  

weirdly different from all this brain stuff. So intuitively it's just weirdly different  

than all this brain stuff and I'm  kind of waiting for the thing that  

starts to look more like brain stuff. I think if AlphaZero, and model-based RL  

and all these other things that were being  worked on 10 years ago, had been giving us  

the GPT-5 type capabilities, then I would  be like, "Oh wow, we're both in the right  

paradigm and seeing the results a priori. So my prior and my data are agreeing." 

Now it's like, "I don't know  what exactly my data is. 

Looks pretty good, but my prior is sort of weird  so I don't have a super strong opinion on it." 

So I think there's a possibility that  essentially all other scientific research  

that is being done is somehow obviated. But I don't put a huge amount of  

probability on that. I think my timelines  

might be more in the 10-year-ish range. If that's the case, I think there is  

probably a difference between a world where we  have connectomes on hard drives and we have an  

understanding of Steering Subsystem architecture. We've compared even the most basic properties  

of what are the reward functions, cost function,  architecture, et cetera, of a mouse versus a shrew  

versus a small primate, et cetera. Is this practical in 10 years? 

I think it has to be a really big push. How much funding,  

how does it compare to where we are now? It's like low billions-dollar scale funding  

in a very concerted way I would say. And how much is on it now? 

So if I just talk about some of the specific  things we have going on with connectomics…  

E11 Bio is our main thing on connectomics. They are trying to make the technology of  

connectomic brain mapping several  orders of magnitude cheaper. 

The Wellcome Trust put out a report a year  or two ago that said to get one mouse brain,  

the first mouse brain connectome would  be a several billion dollars project. 

E11 technology, and the suite of efforts  in the field, is trying to get a single  

mouse connectome down to low tens of  millions of dollars. That's a mammal  

brain. A human brain is about 1,000 times bigger. If with a mouse brain you can get to $10 million  

or $20 million, $30 million, with technology,  if you just naively scale that, a human brain  

is now still billions of dollars, to just do one  human brain. Can you go beyond that? Can you get  

a human brain for less than a billion? But I'm not sure you need every neuron  

in the human brain. We want to, for example,  

do an entire mouse brain and a human Steering  Subsystem and the entire brains of several  

different mammals with different social instincts. So with a bunch of technology push and a bunch of  

concerted effort, real significant progress  if it's focused effort can be done in the  

hundreds of millions to low billions scale. What is the definition of a connectome? 

Presumably it's not a bottom-up biophysics model. So is it just that it can estimate  

the input-output of a brain? What is the level of abstraction? 

You can give different definitions and one of the  things that's cool… So the standard approach to  

connectomics uses the electron microscope and  very, very thin slices of brain tissue. It's  

basically labeling. The cell membranes are going  to show up, scatter electrons a lot and everything  

else is going to scatter electrons less. But you don't see a lot of details of the  

molecules, which types of synapses,  different synapses of different  

molecular combinations and properties. E11 and some other research in the field  

has switched to an optical microscope paradigm. With optical, the photons don't damage the tissue,  

so you can wash it and look  at fragile gentle molecules. 

So with E11's approach, you can get  a "molecularly annotated connectome." 

That's not just who is connected to who by  some synapse, but what are the molecules that  

are present at the synapse? What type of cell is that? 

A molecularly annotated connectome, that's not  exactly the same as having the synaptic weights. 

That's not exactly the same as being  able to simulate the neurons and say  

what's the functional consequence of  having these molecules and connections. 

But you can also do some amount  of activity mapping and try to  

correlate structure to function. Train an ML model basically to predict  

the activity from the connectome. What are the lessons to be taken  

away from the Human Genome Project? One way you could look at it is that  

it was a mistake and you shouldn't have spent  billions of dollars getting one genome mapped. 

Rather you should have just invested in  technologies which have now allowed us to  

map genomes for hundreds of dollars. Well, George Church was my PhD advisor and he's  

pointed out that it was $3 billion or something,  roughly $1 per base pair for the first genome. 

Then the National Human Genome Research Institute  basically structured the funding process right. 

They got a bunch of companies  competing to lower the cost. 

And then the cost dropped like a million-fold in  10 years because they changed the paradigm from  

macroscopic chemical techniques to these  individual DNA molecules which would make a  

little cluster of DNA molecules on the microscope  and you would see just a few DNA molecules at a  

time on each pixel of the camera. It would give you a different,  

in parallel, look at different fragments of DNA. 

So you parallelize the thing by millions-fold. That's what reduced the cost by millions-fold. 

With switching from electron microscopy  to optical connectomics, potentially even  

future types of connectomics technology,  we think there should be similar patterns. 

That's why E11, the Focus Research Organization,  started with technology development rather than  

starting with saying we're going to do a human  brain or something and let's just brute force it. 

We said let's get the cost  down with new technology. 

But then it's still a big thing. Even with new next-generation technology,  

you still need to spend hundreds  of millions on data collection. 

Is this going to be funded with  philanthropy, by governments, by investors? 

This is very TBD and very much  evolving in some sense as we speak. 

I'm hearing some rumors going  around of connectomics-related  

companies potentially forming. So far E11 has been philanthropy. 

The National Science Foundation just  put out this call for Tech Labs,  

which is somewhat FRO-inspired or related. You could have a tech lab for actually going and  

mapping the mouse brain with us and that would be  philanthropy plus government still in a nonprofit,  

open-source framework. But can companies  accelerate that? Can you credibly link  

connectomics to AI in the context of a company  and get investment for that? It's possible. 

I mean the cost of training  these AIs is increasing so much. 

If you could tell some story of not only are  we going to figure out some safety thing,  

but in fact once we do that, we'll also be  able to tell you how AI works… You should go  

to these AI labs and just be like, "Give me one  one-hundredth of your projected budget in 2030." 

I sort of tried a little bit seven or eight  years ago and there was not a lot of interest.  

Maybe now there would be. But all the things that  we've been talking about, it's really fun to talk  

about, but it's ultimately speculation. What is the actual reason for the energy  

efficiency of the brain, for example? Is it doing real inference or  

amortized inference or something else? This is all answerable by neuroscience. 

It's going to be hard, but  it's actually answerable. 

So if you can only do that for low billions of  dollars or something to really comprehensively  

solve that, it seems to me, in the grand scheme  of trillions of dollars of GPUs and stuff,  

it actually makes sense to do that investment. Also, there's been many labs that have been  

launched in the last year where they're raising  on the valuation of billions for things which  

are quite credible but are not like, "Our  ARR next quarter is going to be whatever." 

It's like we're going to discover materials and— Yes, moonshot startups or billionaire-backed  

startups. Moonshot  

startups I see as on a continuum with FROs. FROs are a way of channeling philanthropic  

support and ensuring that it's open  source public benefit, various other  

things that may be properties of a given FRO. But yes, billionaire-backed startups, if they can  

target the right science, the exact right science. I think there's a lot of ways to do moonshot  

neuroscience companies that would  never get you the connectome. 

It's like, "Oh, we're going to upload the  brain" or something, but never actually get  

the mouse connectome or something. These fundamental things that you  

need to get to ground truth the science. There are lots of ways to have a moonshot  

company go wrong and not do the actual science. But there also may be ways to have companies or  

big corporate labs get involved  and actually do it correctly. 

This brings to mind an idea that you had  in a lecture you gave five years ago about. 

Do you want to explain behavior cloning? Actually this is funny because the first  

time I saw this idea, I think it might  have been in a blog post by Gwern. 

There's always a Gwern blog post. There are now academic research  

efforts and some amount of emerging  company-type efforts to try to do this. 

Normally, let's say I'm training  an image classifier or something. 

I show it pictures of cats and dogs or whatever  and they have the label "cat" or "dog". 

And I have a neural network that's supposed to  predict the label "cat" or "dog" or something. 

That is a limited amount of information per label  that you're putting in. It's just "cat" or "dog".  

What if I also had, "Predict what is my  neural activity pattern when I see a cat  

or when I see a dog and all the other things?" If you add that as an auxiliary loss function or  

an auxiliary prediction task, does that sculpt the  network to know the information that humans know  

about cats and dogs and to represent it in a way  that's consistent with how the brain represents  

it and the kind of representational dimensions  or geometry of how the brain represents things,  

as opposed to just having these labels? Does that let it generalize better? 

Does that let it have richer labeling? Of course that sounds really challenging. 

It's very easy to generate lots  and lots of labeled cat pictures. 

Scale AI or whatever can do this. It is harder to generate lots and  

lots of brain activity patterns that correspond  to things that you want to train the AI to do. 

But again, this is just a technological  limitation of neuroscience. 

If every iPhone was also a brain scanner,  you would not have this problem and we would  

be training AI with the brain signals. It's just the order in which technology  

has developed is that we got GPUs  before we got portable brain scanners. 

What is the ML analog, what you'd be doing here? Because when you distill models,  

you're still looking at the final  layer of the log probs across all— 

If you distill one model into  another, that is a certain thing. 

You are just trying to copy  one model into another. 

I think that we don't really have a  perfect proposal to distill the brain. 

To distill the brain you need a  much more complex brain interface. 

Maybe you could also do that. You could  make surrogate models. Andreas Tolias and  

people like that are doing some amount of neural  network surrogate models of brain activity data. 

Instead of having your visual cortex do the  computation, just have the surrogate model. 

So you're distilling your visual  cortex into a neural network to  

some degree. That's a kind of distillation.  This is doing something a little different. 

This is basically just saying I'm adding an  auxiliary… I think of it as regularization  

or I think of it as adding an auxiliary loss  function that's smoothing out the prediction  

task to also always be consistent  with how the brain represents it. 

It might help you with things like  adversarial examples, for example. 

But what exactly are you predicting? You're predicting the internal state of the brain? 

Yes. So in addition to predicting the label,  a vector of labels like yes cat, not dog, yes,  

not boat, one-hot vector or whatever of yes, it's  cat, instead of these gazillion other categories,  

let's say in this simple example. You're also predicting a vector which  

is all these brain signal measurements. So Gwern, anyway, had this long-ago blog  

post of like, "Oh, this is an intermediate thing. We talk about whole brain emulation, we talk about  

AGI, we talk about brain-computer interface. We should also be talking about this  

brain-data-augmented thing that's trained  on all your behavior, but is also trained on  

predicting some of your neural patterns." And you're saying the Learning System is  

already doing this through the Steering System? Yeah, and our brain, our learning system also has  

to predict the Steering Subsystem as an auxiliary  task. That helps the Steering Subsystem. Now,  

the Steering Subsystem can access that predictor  and build a cool reward function using it. 

Separately, you're on the board of Lean,  which is this formal math language that  

mathematicians use to prove theorems and so forth. Obviously there's a bunch of conversation right  

now about AI automating math. What's your take? Well, I think that there are parts of math that it  

seems like it's pretty well on track to automate. First of all, Lean was developed for a number of  

years at Microsoft and other places. It has become one of the Convergent  

Focused Research Organizations to kind of  drive more engineering and focus onto it. 

So Lean is this programming language where instead  of expressing your math proof on pen and paper,  

you express it in this programming language Lean. And then at the end, if you do that that way,  

it is a verifiable language so that you can  click "verify" and Lean will tell you whether  

the conclusions of your proof actually follow  perfectly from your assumptions of your proof. 

So it checks whether the proof  is correct automatically. 

By itself, this is useful for mathematicians  collaborating and stuff like that. 

If I'm some amateur mathematician  and I want to add to a proof,  

Terry Tao is not going to just believe my result. But if Lean says it's correct, it's just correct. 

So it makes it easy for collaboration to happen,  but it also makes it easy for correctness of  

proofs to be an RL signal in very much RLVR. Formalized math proofing—so formal means  

it's expressed in something like Lean and  verifiable—is now mechanically verifiable. 

That becomes a perfect RLVR task. I think that that is going to just  

keep working, it seems like there is at least one  billion-dollar valuation company, Harmonic, based  

on this. AlphaProof is based on this. A couple  other emerging really interesting companies. 

I think that this problem of RLVRing the  crap out of math proving is going to work  

and we will be able to have things that search  for proofs and find them in the same way that  

we have AlphaGo or what have you that can  search for ways of playing the game of Go. 

With that verifiable signal, it works. So does  this solve math? There is still the part that has  

to do with conjecturing new interesting ideas. There's still the conceptual organization of  

math of what is interesting. How do you come up with new  

theorem statements in the first place? Or even the very high-level breakdown of  

what strategies you use to do proofs. I think this will shift the burden of  

that so that humans don't have to do  a lot of the mechanical parts of math. 

Validating lemmas and proofs and checking if  the statement of this in this paper is exactly  

the same as that paper and stuff like that. That  will just work. If you really think we're going  

to get all these things we've been talking about,  real AGI would also be able to make conjectures. 

Bengio has a paper, more like a theoretical paper. There are probably a bunch of other  

papers emerging about this. Is there a loss function for  

good explanations or good conjectures? That's a  pretty profound question. A really interesting  

math proof or statement might be one that  compresses lots of information and has lots  

of implications for lots of other theorems. Otherwise you would have to prove those  

theorems using long complex passive inference. Here, if you have this theorem, this theorem is  

correct, and you have short passive  inference to all the other ones. 

And it's a short compact statement. So it's like a powerful explanation  

that explains all the rest of math. And part of what math is doing is making these  

compact things that explain the other things. It's like the Kolmogorov complexity  

of this statement or something. Yeah, of generating all the other statements,  

given that you know this one or stuff like that. Or if you add this, how does it affect the  

complexity of the rest of the network of proofs? So can you make a loss function that adds,  

"Oh, I want this proof to be a  really highly powerful proof"? 

I think some people are trying to work on that. So maybe you can automate the creativity part. 

If you had true AGI, it would  do everything a human can do. 

So it would also do the things that  the creative mathematicians do. 

But barring that, I think just RLVRing the crap  out of proofs, I think that's going to be just  

a really useful tool for mathematicians. It's going to accelerate math a lot and  

change it a lot, but not necessarily  immediately change everything about it. 

Will we get mechanical proof of the Riemann  hypothesis or something like that, or things like  

that? Maybe, I don't know. I don't know enough  details of how hard these things are to search  

for, and I'm not sure anyone can fully predict  that, just as we couldn't exactly predict when Go  

would be solved or something like that. I think it's going to have lots  

of really cool applied applications. So one of the things you want to do is you want to  

have provably stable, secure, unhackable software. So you can write math proofs about software and  

say, "This code, not only does it pass these unit  tests, but I can mathematically prove that there's  

no way to hack it in these ways, or no way to mess  with the memory", or these types of things that  

hackers use, or it has these properties. You can use the same Lean and same proof  

to do formally verified software. I think that's going to be a really  

powerful piece of cybersecurity that's relevant  for all sorts of other AI hacking the world stuff. 

And if you can prove the Riemann hypothesis,  you're also going to be able to prove insanely  

complex things about very complex software. And then you'll be able to ask the LLM,  

"Synthesize me a software  that I can prove is correct." 

Why hasn't provable programming  language taken off as a result of LLMs? 

I think it's starting to. One challenge—we are  actually incubating a potential Focused Research  

Organization on this—is the specification problem. So mathematicians know what interesting theorems  

they want to formalize. Let's say I have some code  

that is involved in running the power grid or  something and it has some security properties,  

well what is the formal spec of those properties? The power grid engineers just made this thing,  

but they don't necessarily know how  to lift the formal spec from that. 

And it's not necessarily easy to come up with the  spec that is the spec that you want for your code. 

People aren't used to coming up with formal  specs and there are not a lot of tools for it. 

So you also have this user interface  plus AI problem of what security  

specs should I be specifying? Is this the spec that I wanted? 

So there's a spec problem and it's  just been really complex and hard. 

But it's only just in the  last very short time that  

the LLMs are able to generate verifiable proofs  of things that are useful to mathematicians,  

starting to be able to do some amount of that  for software verification, hardware verification. 

But I think if you project the  trends over the next couple years,  

it's possible that it just flips the tide. Formal methods, this whole field of formal  

methods or formal verification, provable software. It’s this weird almost backwater of the more  

theoretical part of programming languages  and stuff, very academically flavored often. 

Although there was this DARPA program  that made a provably secure quadcopter  

helicopter and stuff like that. Secure against… What is the  

property that is exactly proved? Not for that particular project,  

but just in general. Because obviously things  

malfunction for all kinds of reasons. You could say that what's going on in this  

part of the memory over here, which is supposed  to be the part the user can access, can't in any  

way affect what's going on in the memory over  here or something like that. Things like that. 

So there's two questions.  One is how useful is this? 

Two is, how satisfying, as a  mathematician, would it be? 

The fact that there's this application towards  proving that software has certain properties  

or hardware has certain properties, if that  works, that would obviously be very useful. 

But from a pure… Are we going  to figure out mathematics? 

Is your sense that there's something about finding  that one construction cross-maps to another  

construction in a different domain, or finding  that, "Oh, this lemma, if you redefine this term,  

it still satisfies what I meant by this term. But a counterexample that previously knocked  

it down no longer applies." That kind of dialectical thing  

that happens in mathematics. Will the software replace that? 

Yeah. How much of the value of this sort of pure  mathematics just comes from actually just coming  

up with entirely new ways of thinking  about a problem, mapping it to a totally  

different representation? Do we have examples? I don't know. I think of it maybe a little  

bit like when everybody had to write  assembly code or something like that. 

The amount of fun cool startups that got  created was just a lot less or something. 

Fewer people could do it, progress was more  grinding and slow and lonely and so on. 

You had more false failures because you  didn't get something about the assembly  

code, rather than the essential  thing of was your concept right. 

Harder to collaborate and stuff like that. And so I think it will be really good. 

There is some worry that by not learning to  do the mechanical parts of the proofs that  

you fail to generate the intuitions that inform  the more conceptual parts, the creative part. 

It’s the same with assembly. Right. So at what point is that applying? 

With vibe coding, are people not learning  computer science or actually are they vibe  

coding and they're also simultaneously  looking at the LLM that's explaining  

these abstract computer science concepts to  them and it's all just all happening faster? 

Their feedback loop is faster and they're learning  way more abstract computer science and algorithm  

stuff because they're vibe coding. I don't know, it's not obvious. 

That might be something about the user interface  and the human infrastructure around it. 

But I guess there's some worry that people don't  learn the mechanics and therefore don't build  

the grounded intuitions or something. But my hunch is it's super positive. 

Exactly, on net, how useful that will be  or how much overall math breakthroughs,  

or math breakthroughs even that we care about,  will happen? I don't know. One other thing that  

I think is cool is the accessibility question. Okay, that sounds a little bit corny. 

Okay, yeah, more people  can do math, but who cares? 

But I think there's lots of people  that could have interesting ideas. 

Like maybe the quantum theory  of gravity or something. 

Yeah, one of us will come up  with the quantum theory of  

gravity instead of a card-carrying physicist. In the same way that Steve Byrnes is reading  

the neuroscience literature and he hasn't  been in the neuroscience lab that much. 

But he's able to synthesize across the  neuroscience literature and be like, "Oh,  

Learning Subsystem, Steering Subsystem.  Does this all make sense?" He's an  

outsider neuroscientist in some ways. Can you have outsider string theorists  

or something, because the math is  just done for them by the computer? 

And does that lead to more innovation  in string theory? Maybe yes. 

Interesting. Okay, so if this approach works and  you're right that LLMs are not the final paradigm,  

and suppose it takes at least 10 years  to get the final paradigm in that world. 

There's this fun sci-fi premise where you  have… Terence Tao today had a tweet where  

he's like, "These models are like automated  cleverness but not automated intelligence." 

And you can quibble with the definitions there. But if you have automated cleverness and you have  

some way of filtering—which if you can formalize  and prove things that the LLMs are saying you  

could do—then you could have this situation  where quantity has a quality all of its own. 

So what are the domains of the world which could  be put in this provable symbolic representation? 

So in the world where AGI is super far away,  maybe it makes sense to literally turn everything  

the LLMs ever do, or almost everything  they do, into super provable statements. 

So LLMs can actually build on top of each other  because everything they do is super provable. 

Maybe this is just necessary because you have  billions of intelligences running around. 

Even if they are super intelligent, the only way  the future AGI civilization can collaborate with  

each other is if they can prove each step. They're just brute force churning out…  

This is what the Jupiter brains are doing. It's a universal language, it's provable. 

It's also provable from the perspective of,  "Are you trying to exploit me or are you  

sending me some message that's trying  to hack into my brain effectively?" 

Are you trying to socially influence me? Are you actually just sending me just the  

information that I need and no more for this? So davidad, who's this program director at ARIA  

now in the UK, he has this whole  design of an ARPA-style program,  

a sort of safeguarded AI that very heavily  leverages provable safety properties. 

Can you apply proofs to…  Can you have a world model? 

But that world model is actually not  specified just in neuron activations,  

but it's specified in equations. Those might be very complex equations,  

but if you can just get insanely good at just  auto-proving these things with cleverness,  

auto-cleverness… Can you have explicitly  interpretable world models as opposed to  

neural net world models and move back basically  to symbolic methods just because you can just  

have insane amount of ability to prove things? Yeah, I mean that's an interesting vision. 

I don't know in the next 10 years whether that  will be the vision that plays out, but I think  

it's really interesting to think about. Even for math, I mean, Terence Tao is  

doing some amount of stuff where it's not about  whether you can prove the individual theorems. 

It's like let's prove all the theorems en  masse and then let's study the properties  

of the aggregate set of proved theorems. Which are the ones that got proved  

and which are the ones that didn't? Okay, well that's the landscape of all the  

theorems instead of one theorem at a time. Speaking of symbolic representations,  

one question I was meaning to ask you is,  how does the brain represent the world model? 

Obviously nets out in neurons, but  I don't mean extremely functionally. 

I mean conceptually, is it in something  that's analogous to the hidden state of  

a neural network or is it something  that's closer to a symbolic language? 

We don't know. There's some  amount of study of this. 

There's these things like face patch neurons  that represent certain parts of the face that  

geometrically combine in interesting ways.  That's with geometry and vision. Is that  

true for other more abstract things? There's this idea of cognitive maps. 

A lot of the stuff that a rodent  hippocampus has to learn is place cells and,  

where is the rodent going to go next and  is it going to get a reward there? It's  

very geometric. And do we organize concepts  with an abstract version of a spatial map? 

There's some questions of can  we do true symbolic operations? 

Can I have a register in my brain that copies a  variable to another register regardless of what  

the content of that variable is? That's this  variable binding problem. Basically I don’t  

know if we have that machinery or is it more  like cost functions and architectures that  

make some of that approximately emerge, but  maybe it would also emerge in a neural net? 

There's a bunch of interesting neuroscience  research trying to study this, what the  

representations look like. But what’s your hunch? 

Yeah, my hunch is that it's going to be a huge  mess and we should look at the architecture,  

the loss functions, and the learning rules. I don't expect it to be pretty in there. 

Which is that it is not a  symbolic language type thing? 

Yeah, probably it's not that symbolic. But other people think very differently. 

Another random question speaking of binding, what  is up with feeling like there's an experience? 

All the parts of your brain which are modeling  very different things, have different drives,  

and at least presumably feel like there's  an experience happening right now. 

Also that across time you feel like… Yeah, I'm pretty much at a loss on this one.  

I don't know. Max Hodak has been  giving talks about this recently. 

He's another really hardcore neuroscience  person, neurotechnology person. 

The thing I mentioned with Doris maybe also sounds  like it might have some touching on this question. 

But yeah, I don't think anybody has any idea. It might even involve new physics. 

Here’s another question which  might not have an answer yet. 

Continual learning, is that the product  of something extremely fundamental at the  

level of even the learning algorithm? You could say, "Look, at least the way  

we do backprop in neural networks is that  you freeze the weight, there's a training  

period and you freeze the weights. So you just need this active inference  

or some other learning rule in  order to do continual learning." 

Or do you think it's more a matter of architecture  and how memory is exactly stored and what kind of  

associative memory you have basically? So continual learning… I don't know. 

At the architectural level, there's probably  some interesting stuff that the hippocampus  

is doing. People have long thought this.  What kinds of sequences is it storing? 

How is it organizing, representing that? How is it replaying it back? What is it  

replaying back? How exactly does  that memory consolidation work? 

Is it training the cortex using  replays or memories from the  

hippocampus or something like that? There's probably some of that stuff. 

There might be multiple timescales of  plasticity or clever learning rules  

that can simultaneously be storing short-term  information and also doing backprop with it. 

Neurons may be doing a couple things:  some fast weight plasticity and some  

slower plasticity at the same time, or  synapses that have many states. I mean,  

I don't know. From a neuroscience perspective,  I'm not sure that I've seen something that's super  

clear on what causes continual learning except  maybe to say that this systems consolidation  

idea of hippocampus consolidating cortex. Some people think it is a big piece of this  

and we still don't fully understand the details. Speaking of fast weights, is there something  

in the brain which is the equivalent of  this distinction between parameters and  

activations that we see in neural networks? Specifically in transformers we have this  

idea that some of the activations are the  key and value vectors of previous tokens  

that you build up over time. There's the so-called fast  

weights that whenever you have a new token,  you query them against these activations,  

but you also obviously can't query them against  all the other parameters in the network which  

are part of the actual built-in weights. Is there some such distinction that's analogous? 

I don't know. I mean we definitely  have weights and activations. 

Whether you can use the activations in these  clever ways, different forms of actual attention,  

like attention in the brain… Is that  based on, "I'm trying to pay attention"... 

I think there's probably several different  kinds of actual attention in the brain. 

I want to pay attention to  this area of visual cortex. 

I want to pay attention to the  content in other areas that is  

triggered by the content in this area. Attention that's just based on reflexes  

and stuff like that. So I don't know. There's  not just the cortex, there's also the thalamus. 

The thalamus is also involved in somehow  relaying or gating information. There's  

cortico-cortical connections. There's  also some amount of connection between  

cortical areas that goes through the thalamus. Is it possible that this is doing some sort of  

matching or constraint satisfaction or matching  across keys over here and values over there? 

Is it possible that it can do stuff like that?  Maybe. I don't know. This is all part of the  

architecture of this corticothalamic system. I don't know how transformer-like it is or if  

there's anything analogous to that attention. It’d be interesting to find out. 

We’ve got to give you a billion dollars  so you can come on the podcast again and  

tell me how exactly the brain works. Mostly I just do data collection. 

It's really unbiased data collection so all the  other people can figure out these questions. 

Maybe the final question to go  off on is, what was the most  

interesting thing you learned from the Gap Map? Maybe you want to explain what the Gap Map is. 

In the process of incubating and coming up  with these Focused Research Organizations,  

these nonprofit startup-like moonshots  that we've been getting philanthropists  

and now government agencies to fund,  we talked to a lot of scientists. 

Some of the scientists were just like, "Here's  the next thing my graduate student will do.  

Here's what I find interesting. Exploring  these really interesting hypothesis spaces,  

all the types of things we've been talking about." Some of them were like, "Here's this gap. 

I need this piece of infrastructure. There's no combination of grad students in my lab  

or me loosely collaborating with other labs with  traditional grants that could ever get me that. 

I need to have an organized engineering  team that builds the miniature  

equivalent of the Hubble Space Telescope. If I can build that Hubble Space Telescope,  

then I will unblock all the other researchers in  my field or some path of technological progress in  

the way that the Hubble Space Telescope lifted the  boats and improved the life of every astronomer." 

But it wasn't really an  astronomy discovery in itself. 

It was just that you had to put this giant mirror  in space with a CCD camera and organize all the  

people and engineering and stuff to do that. So some of the things we talked to  

scientists about looked like that. The Gap Map is just a list of a lot of  

those things and we call it a Gap Map. I think it's actually more like a  

fundamental capabilities map. What are all these things,  

like mini Hubble space telescopes? And then we organized that into gaps for  

helping people understand that or search that. What was the most surprising thing you found? 

I think I've talked about this before,  but one thing is just the overall size  

or shape of it or something like that. It's a few hundred fundamental capabilities. 

So if each of these were a deep tech  startup-size project, that's only  

a few billion dollars or something. If each one of those were a Series A,  

that's only… It's not like a trillion dollars to  solve these gaps. It's lower than that. So that's  

one thing. Maybe we assumed that, and that's  what we got. It's not really comprehensive.  

It's really just a way of summarizing a lot  of conversations we've had with scientists. 

I do think that in the aggregate process, things  like Lean are actually surprising because I did  

start from neuroscience and biology and it  was very obvious that there's these -omics. 

We need genomics, but we also need connectomics. We can engineer E. coli, but we also need to  

engineer the other cells. There's somewhat obvious  

parts of biological infrastructure. I did not realize that math proving  

infrastructure was a thing and that  was emergent from trying to do this. 

So I'm looking forward to seeing other  things where it's not actually this hard  

intellectual problem to solve it. It's maybe slightly the equivalent  

of AI researchers just needing GPUs  or something like that and focus and  

really good PyTorch code to start doing this. Which are the fields that do or don't need that? 

So fields that have had gazillions of dollars  of investment, do they still need some of those? 

Do they still have some of those gaps  or is it only more neglected fields? 

We're even finding some interesting  ones in actual astronomy, actual  

telescopes that have not been explored. Maybe because if you're getting above a  

critical mass-size project, then you have to  have a really big project and that's a more  

bureaucratic process with the federal agencies. I guess you just need scale in every single  

domain of science these days. Yeah, I think you need scale in  

many of the domains of science. That does not mean that the  

low-scale work is not important. It does not mean that creativity,  

serendipity, etc., and each student pursuing  a totally different direction or thesis that  

you see in universities is not also really key. But I think some amount of scalable infrastructure  

is missing in essentially every area  of science, even math, which is crazy. 

Because mathematicians I thought just needed  whiteboards, but they actually need Lean. 

They actually need verifiable programming  languages and stuff. I didn't know that. 

Cool. Adam, this is super  fun. Thanks for coming on. 

Thank you so much. My pleasure. Where can people find your stuff? 

Pleasure. The easiest way  now… My adammarblestone.org  

website is currently down, I guess. But convergentresearch.org can link to  

a lot of the stuff we've been doing. And then you have a great blog,  

Longitudinal Science. Longitudinal Science, yes, on WordPress. 

Cool. Thank you so much. Pleasure.