I want to begin with an observation.

The observation is that in this venue, there's probably in the order of

a thousand, maybe a couple thousand computers.

There's some in your pockets.

You brought some with you. There's computers running various systems

in the

buildings. There are a few thousand computers.

But if you think about instances of notations of symbols, graphics, signage,

maps, things that people have written down,

notes in your notebook.

There are probably at least 100 times, if not a thousand times more instances

of notations of ways that you write things down to communicate,

to remember

things, to work on them in this venue.

And my observation

is that even though we may not be conscious of it at all times,

the impact of notations,

how we write things down to communicate and represent things, is potentially

greater than the impact of machines that we build to augment our

intelligence. They're both ways to make us smarter,

make us more effective at

what we want to do, but notations possibly has a greater

impact, even though we

may not be conscious of it. And I call this idea notational

intelligence.

And for the rest of our time together today, I want to talk through first what

makes good notations, why do some ways of writing things down

feel more

satisfying and more useful,

and dive into some attributes of good notations.

And then I want to think through how we might be able to use some of their

modern tools of deep learning to think about new kinds of notations we can

create. A great example of what I call notational

intelligence is when you do

long addition. If I ask you to add three numbers

or four-digit numbers, let's

say 217 and 868.

Maybe some of you can do it in your head.

I can't really do it that fast in your head.

But then you do this.

You can move some numbers around. I hope I'm not making a mistake.

That's embarrassing.

It's not this. It's this. Okay. And in this calculation,

I didn't really do the calculation purely in my head.

It's not just that I became more intelligent in my brain.

It's also not that the chalkboard is a machine or a calculator.

There's nothing behind this. There's something in the way

that we represent

information that allows us, the combined system of my brain

and the things that

I'm writing down, the shapes that I'm putting on the board,

to be smarter.

And so that's what I call notational intelligence.

And there are different kinds of notation.

I want to just walk through a few desirable properties that I

think make certain kinds of notation useful and satisfying.

And the first is abstraction.

Abstraction is the idea that you can write down one symbol

or one kind of

shape, and it can stand for a whole set of things

that have similar properties.

So a good example of this is in algebra, you might say, y = mx +

b,

and b is not just a single object. b stands for a whole set of things.

Maybe it's a set of integers, maybe it's a set of reals.

All of these variables are individual symbols

that we can manipulate, so I can

move this term to the left, and really I'm manipulating a whole family

of

relationships between numbers or family of ideas.

And that is really powerful because you can start to manipulate larger objects,

potentially entire families of equations or relationships by just moving

symbols around on the board. The second and third properties

that make great

notation

are suggestiveness

and what our favorite mathematician, Terry Tao, calls natural transformations.

And they're related kind of abstract ideas.

So I'll try to first state it, and then I'll give you an example.

Suggestiveness is the idea that if two things when you write them down

look the

same, then they probably share some property

that is implied by the similarity

of the shapes. And natural transformation

is a related idea that when you could

apply some operation to the shapes in the space of writing graphical space,

that they have some corresponding meaning in what you're doing with the

underlying ideas. So what does that mean?

A good example of this comes from two different ways that early inventors of

calculus wrote down the idea of derivatives,

Leibniz and Newton.

Leibniz was coming from a more mathematical,

pure mathematical background.

He wrote down derivatives as such.

You would do like a dy over dx, and this is a ratio of differentials.

It's the ratio of a tiny movement in x to a tiny movement in y, right?

Remember this from high school.

And you can manipulate this in a few different ways.

You can maybe multiply it by other ratio of differentials.

You can take squares of things and although there

are rigorous ways you can

manipulate this, you can also just kind of eyeball certain

things, and you can

operate mechanically on the symbols in a way that means something in the space

of ideas. Whereas Newton, coming from a physics background,

would say something

like, "Oh, I only really care about derivatives over

time, about how fast

things are moving and accelerating." So he'd say something

like, "There's

position, and maybe the first derivative

is like a dot over the value, and the

second derivative of acceleration is like two dots over the value."

And this is

from an abstraction perspective, it's really nice.

It's a lot simpler, but you can't really operate on these

things in a way that

makes any sense. There's not that much mathematical implied by the

shape of the

notations. And so in practice, we find, of course, that this is the more common

way to work with derivatives in mathematic literature because that Leibniz

notation is both more suggestive and more amenable to natural

transformations.

And then finally, the fourth family of properties

that we really care about for

good notations is that they take advantage that they

are graphical.

To be graphical means to take advantage of the implicit biases that we have in

our visual perception system to make sure that whenever we operate on things in

these two dimensions, that they are kind of naturally intuitive because of

the

way that our brain works and our perception system works.

One way that this is the case is that notations that we work with here

and in

the venue and in the signages are all flat.

They could be three-dimensional.

You could imagine ways of writing things down involve CAD models and videos and

other kinds of dimensions, but flat notations

are useful because we can move

them around. They're mobile. You can scale them up

and down really easily.

So you can take notations that describe really complex,

really large objects

and really small objects like bacteria, and you can then bring to bear the same

tools of geometry, like arrows and shapes and measurement and ratios,

onto all

of these notations, different kinds of maps and graphs

and plots, because they

ultimately boil down to 2D shapes that are flat, that you can move around and

you can copy and you can print out.

A really great example of graphical notation is the coordinate plane.

So we could take something like Y=MX+B, and we could just say, okay, we'll draw

a Y-axis here, X-axis here, and maybe this is the line that we want to

represent with this equation. And this is useful for manipulating

numbers and algebraic terms, but this is really what gets us to understand the

relationships between numbers, right?

If I move the value of B up and down, maybe it means that the line moves up and

down because that's moving the Y-intercept, or maybe the slope is M,

and so it

represents something physical.

And so the coordinate plane is a really, really useful notation for us because

it allows us to take advantage of the implicit kind of visual understanding

that we have as humans, kind of just moving around our space.

Another

example of really great visual notation that is one of my favorite examples

because of how surprising it is, is the arrow. We have arrows here.

I think pretty much every speaker that we've had today has drawn an arrow at

some point, pointing to something or gesturing at a direction of movement.

So it's really kind of we're a fish in the water.

We're used to the idea of arrows.

How old do you think this arrow symbol is in recorded history?

What do you think is the earliest example of the arrow symbol we can find?

It turns out that the earliest example of arrows that we can find is about 300

years old. It's from 1737.

That's pretty wild, right? Before that, if you wanted to draw something and you

want to point at something, you had to draw a hand pointing at the

thing,

which is pretty crazy. And so even something as basic as a

coordinate plane or

even the arrow symbol, somebody had to think of and invent,

and that invention

allows us to work with these other kinds of notations much, much more easily.

Another great kind of notation that we are all used to as programmers and

practitioners in computer science is programming languages.

Programming languages describe a totally new family of ideas.

The idea of programs didn't really exist when a lot of this work was done.

And yet, programming languages are a kind of notation, a thing

that you write

down on paper, on a computer screen, to talk about a family of concepts that

are programs, that are these kind of complex dynamic objects

that allow

computers to be turned into useful machines.

And programming languages have all these elements. They use abstraction.

They allow for abstraction in that you have variables and functions that

abstract over important concepts.

They are suggestive and immutable to natural transformation,

in that if you're

refactoring programs, you'll often take a term, take a function,

replace the

term with a function. You're moving values around,

and you're performing

actions in the space of written programs, in the space of symbols that have

corresponding meaning in the space of actual programs, this abstract object

that exists inside a computer, right?

And then finally, they're maybe more graphical than you

might initially think,

because when you indent code, when you syntax highlight code, you

are using the

implicit biases that we have as humans of how we perceive color and how we

perceive space to denote useful ideas in the space of programs like variable

scope. So these are how we get good notations.

So now

we're in 2026,

we don't have to just invent arrows anymore.

We can bring the more powerful tools that we have at our disposal, like deep

learning,

to try to think about how we might kind of

in more intentional event, new kinds of abstractions for new kinds

of ideas. When we build these models,

as some of the folks talked about previously,

we have models learning these new kinds of weird ideas.

When you look inside a model, maybe they learn features that

are not really

interpretable to us.

Maybe if you look inside a model and apply interpretability techniques,

you get

things like LoRA layers and strange new features

that we don't really have

words for. But if we could somehow figure out how to

mechanize the process of

getting good notation, maybe we can mass produce new kinds of

ways to write

down ideas that we don't yet have words for or yet have notations for.

And so how do we invent new notations with deep learning?

There's actually some prior art in this space.

In the field of cognitive science, a lot of people use deep learning models,

things like ConvNets and ResNet, as kind of laboratory models for human

perception to try to imagine what the evolution of written language may

have looked like. So they use neural networks as a proxy for

how humans see the

world, and then try to use that to imagine how we may have evolved

from things

like pictograms to modern language.

That's not exactly what we're going to do here.

We're going to instead try to use the same tools of deep learning to try to

imagine what totally new notations, unconstrained from natural

notations that we've learned about so far, might look like, and try to actually

produce a toy model of inventing a new kind of notation.

So when you want to invent a new notation, you need two things.

First thing you need

is you need to decide on an input domain.

What do you want to represent with the notation?

And the second thing you need

is the constraints that are inherent to the medium

that you're working in.

So the constraints of the medium.

For example, so if I'm trying to write things on a

chalkboard, the constraints

of the medium are not just that it's flat, but also that maybe you need high

contrast. Maybe I can't do shading super well,

so it needs to be just pure

black and white.

Maybe it needs to be made of strokes instead of pixels because I'm writing with

something that generates lines.

But if you're using a model that's generating raster images,

maybe your medium

actually can allow for pixel images and raster images, and we'll see exactly

that. And so just to produce a toy model of

inventing a new notation with deep

learning, let's say we're going to pick a really

simple input domain, which is

just a vector of size N, where only one of the values can be

one and the rest has to be zero. So imagine maybe it's zero

1000,

and this is a vector of size five where exactly one value is one, right?

So we call that a one-hot vector.

And let's say that's vector N, and we have N values.

Another way to think about this object is it's like an alphabet.

The English alphabet has 26 letters. Each letter can only be one letter.

A letter can't be A and B. And so we're going to try to train a model

to learn

a new alphabet that is N letters big.

So we have this input that's just a vector.

We put this first into an image generator model G, and this is going to

generate some kind of an image. In our case, this is a ResNet-style

convolutional model in my little toy model, and so maybe it generates a 32 by

32 image.

And then we're going to have a different model try to read what this model

wrote down to represent this idea. So this is a generator.

It's writing something down, and then we have a decoder that's trying

to decode

this back into what the decoder thinks the generator tried to write down,

which we call V prime N. And it's the same size, right?

And so we optimize this whole thing end to end with gradient descent to try to

make sure, there was another arrow, try to make sure that the generator and

decoder learn to talk to each other.

In other words, when we take some vector, put it through the generator to

produce an image, put it through a decoder to

try to figure out what it meant. Hopefully, the values V prime N and VN are

similar. We try to minimize the difference.

And at this point, we have a handout

that we'll start handing out. And if we don't

have enough handouts for everybody, if you don't get a handout,

you can also go

to the URL linus.zone/compile,

and you'll see the same thing. The handout shows you two interesting

things.

The first

page is in the process of training this thing, where we have in

the case of the experiment that you're looking at there,

it's a vector of size

1024. So the model is learning an alphabet that is 1024

characters or letters big. It's learning 1024 different symbols.

It has to learn to distinguish 1,000 different concepts.

It's doing that in a 32 by 32 grayscale pixel space.

And you'll see that the first page is the outcome.

So it's what do these letters look like that these models have made up in their

own space, unconstrained from human perception.

In one of the pages, it's that.

In the other page, it's a sequence of learning.

Well, you see that in the beginning, you're starting with noise, and the model

learns basic ideas like black and white, and then over time, it learns more and

more complex shapes, which is pretty interesting to see.

The last trick that I forgot to mention in building this thing is that if you

just train this naively, the model's going to learn some really

basic hacks.

Maybe the first letter is the first pixel being on,

second letter is the second

pixel being on, and so on and so forth.

We don't really want that because that doesn't really mean anything to

humans.

So we have to go back to our graphical implicit biases of the human visual

perception and say, what kinds of invariance do we want to

impose in the

medium? For example, we want symbols to mean the same thing,

whether they're

big or small. And so we impose a scale invariant.

We make sure that when we get a generator image out of the

generator, maybe we

make it smaller and make sure that a smaller symbol decodes to the same

thing.

We make it rotational invariance.

If you take a letter that this thing has learned

and turn it around, it still

means the same idea. And so we have rotational invariance,

scale invariance.

We also vary the color a little bit, so a brighter or a darker version of the

same symbol roughly means the same idea.

And so if you train this system, this kind of autoencoder with an image in

the

middle, with these invariants, the model, it turns out, learns to encode a

family of ideas or domain of ideas in its own kind of visual space.

In essence, it's learned its own graphical

representation of ideas.

The reason I've divided this board here in two like this is that on the left

side is our world. It's the things that we've, by culture, by communities of

practice, invented to support us, and they've evolved really organically,

right? And in the right is this totally alien world,

where we've taken a model

not just as a way to learn what's already out there, but as a simulator that

can simulate anything and learn anything.

They don't have to be learning just the laws of physics that exist in our

world. They don't have to just learn the

languages that we already know.

We can instead use these to try to imagine new kinds of languages, new kinds of

ways of writing things down, new laws of physics.

And I think this idea of thinking of a computer and a model as a simulator for

anything, not just as a simulator for our specific

world and experience and way

of being, is really powerful.

And while we get a lot of value out of simulating things that exist in our

world, I think it's maybe much more interesting

to try to use models as a

laboratory for imagining other kinds of being.

Other ways of imagining things, other concepts,

other ways of writing things

down.

And this was really what I thought for a long time about, and I would push

everyone here with all of your new tools and your keyboards to think about it.

How do we use models that we know and love to not just model the current

world,

but to speak of unspeakable ideas and to see what we really can't imagine

yet

that might come in the future? Thank you so much, guys,

and have a wonderful

conference.