So today I'm going to be talking about

intelligence efficiency, right?

And how we can maximize it.

I'm going to start off with three graphs.

And I want to talk about where we are today as an industry

and where I think we

should go.

So where are we today, right?

ChatGPT came out, LLM boom occurred,

intelligence skyrockets over time. We've seen this, right?

We've all seen the benchmarks. Intelligence is only increasing.

It's increasing at an exponential rate.

Right?

Additionally, with that,

we've seen tokens skyrocket over time.

We've seen token maxing occur in the industry, and we've seen industry as a

whole really adopt LLMs into their production processes,

causing this to occur.

Right?

But additionally, what I think we've seen

is that

these models are getting smarter, but they're not getting more efficiently

smart, right? The more money you're paying,

we see the state of the art over

here.

This is the fable. This is maybe last year's models.

The more money you're paying for intelligence is actually having

diminishing

returns, right? And this is not necessarily what the world

was sold, right?

We were sold with LLMs that you could buy intelligence at a unit

economic

price. But what we're actually paying for

is tokens, right?

And this is not true, right?

We all know this as technical folk.

We know that intelligence does not equal tokens,

yet people are trying to buy this but are actually purchasing this, right?

And there's a fundamental difference there that exists, and I think will

actually come to bite

the model providers in the ass a little bit, right?

We see this logarithmic curve to some extent, while folks want a more linear

curve, right?

Some data to back that up. I work at Ramp, like Dwarkesh said,

so we see a lot of this in the data as well.

Just some stats is that we've seen a 13X monthly token spend

since January

2025, and we've also seen some political things

happen in the

technological space, right? We've seen Uber,

we've seen Meta clamping down on

this token masking and really

making sure that this exponential doesn't necessarily continue as it affects

the bottom line.

We also recently did a version of SWE-bench of Ramp, and we saw this

more logarithmic curve, where intelligence

is actually getting less efficient

over time, not more efficient. And some of the research I've been doing

personally is how can we make sure that this graph is more linear rather than

more logarithmic, right? And what I found is

that context seems to really be

the key in order to provide for the models, we provide more context, they tend

to become more efficient and need less work to complete the same

processes.

I want to give some mathematical foundations of this.

So the way I think about LLMs,

and AI as a whole,

and agentic processes, is that they are really entropy

reduction machines, right? So entropy is a measure of chaos in a state.

And if you think about LLMs, what they do is they take very high

entropy

states, they take PDFs, they take information from the web,

they take

information from very scattered fields and a prompt, and they organize it

into what the user is asking for, right?

They fundamentally reduce the entropy of state.

I think out of this

field of thought, we can actually derive equations

that show how introducing

context into the field makes LLMs and AI as a whole more efficient.

So there's two fundamental views of entropy.

There's an information theoretic view of entropy, which is a Shannon entropy

definition, which says that the entropy of a state after an

observation is

fundamentally reduced. I think this is pretty obvious, right?

You have some chaotic system, you measure it,

you understand a little bit more,

and so the chaos of the system, at least our local representation,

is reduced,

right? Additionally, the thermodynamic view of entropy says

that the amount you can reduce entropy in a system is

upper bounded by the amount of work you put in.

And combining these two equations, we can see that there's an upper bound on

the change of entropy in a system,

both from the work you put in and from the information we already have

about

the system.

The way I explain this to people is let's say you're cleaning your room,

right?

You have to know where the dirty clothes are to then put in the work and put

them in the right place, right? So to reduce a chaotic system to a

less chaotic system, you have to have both.

And I think modern LLMs really focus on how much work we can put in, right?

Chain of thought, reasoning. We see that the more work you put in, you can

reduce the entropy system. We've seen it.

We know it's not linear, but it's true.

But they're not necessarily focusing on the information piece of it all, right?

And I'm going to give three examples,

to try to illustrate how providing context, memory information, whatever you

want to call it, into the system,

particularly in the latent space, can make LLMs more energy

or more intelligent, efficient. Okay?

And to give some context,

the current methods in the field to provide context are like RAG or memory,

but

we all know that these have problems, right? They don't scale that well.

They're typically used for needle in a haystack tasks or semantic search tasks,

and really struggle with multi-hop reasoning queries, right?

Where multi-hop reasoning, if you don't know,

is taking a bunch of facts,

putting them together, and constructing a logical answer from

that.

They also both happen in token space, which is fundamentally constrained for

the models,

rather than a latent space, right?

And I'm going to talk about three examples in the latent space that I believe

are efficient ways to convey context to the underlying models and increase the

intelligence efficiency.

So this is going to be in-

Multi-agent systems.

My chalk is getting small. Okay.

In the context window,

and as inputs, right? So

these are the three fields of research I've been looking at over the last few

months, and I want to explain them. So let's start with multi-agent systems.

So I'm going to give a model of multi-agent systems

and then explain how it can

be improved and the results we've seen from that improvement.

So multi-agent systems, we can say, pretty abstractly, there's a supervisor

agent, right? And then can spawn

worker agents.

Right? I'm pretty sure everyone

knows about this kind of stuff, right?

And they all complete a task, but let's say this is time down here.

The problem with this, right, is it's extremely inefficient.

The amount of tokens that this uses is really astronomical.

And in the current economic state where token maxing is largely

unsustainable, we need to fix this somehow, right?

Additionally, the supervisor agent, as it's doing work, is generating context.

It's exploring the system. We also see that the worker agents are all

generating their own context itself,

and this is not shared in the global state, right?

A context a worker discovers here might have been discovered here, and now

we're putting in extra work and becoming very inefficient, right?

So, part of the research I did is,

let's say we share context. We maintain a global state of context,

which I

did as a

global KV cache that was shared among the agents.

And we can, when we spawn a new agent,

take the task that it's spawned with, filter out the information we need from

the KV cache. We did this with a

compression algorithm, essentially.

And then spawn the worker with this initialized KV cache that essentially

contains the context of the system that's important to the worker, right?

When we

judge this system versus the system without context sharing on a multitude of

tasks, we saw, in total, there was a 42%

to 57%

reduction in worker agent tokens,

and a total reduction of 21% to 31%

total tokens with the same accuracy, right?

So they become highly more efficient just by producing shared context,

introducing information into the global system, right?

Additionally, some work I also did is about in-context learning,

right? We've all seen DeepSeeks release the

sparse attention model, right?

Or hopefully, we've all seen that.

And we've seen that sparse attention has really taken the industry by storm.

It makes sense, right? We need a more efficient way than dense

attention.

But the way I want to investigate sparse attention is modeling it as a

reranker model, right? Fundamentally, if you think of sparse attention, it's

looking at one token saying what information of the past tokens

are necessary

in order for me to generate the next token, right?

And this is very similar to how reranker models work in RAG.

And so I was wanting to say, okay, if this could model a reranker system, it

would be evidence that more efficient model was partly,

at least,

because of accurate context retrieval over its own token space,

or latent space, I guess. And so what we did

is we took the queries and the

keys generated by the Lightning indexer from the DeepSeek model, and we

computed document scores over a question-answer dataset that was

multi-hop to try to see if it would accurately get the right

documents for a

question. I'm just going to write the equations of

how we did that.

Feel free to come up to me after and ask about any of these because I'm

going

pretty quick.

But essentially, we took the ReLU of the cost of the normed

query over the queries. Basically, we have a dataset with

a bunch of documents and multiple-choice questions over those

documents.

The queries are from the questions in the multiple-choice dataset, and we

did this with the

normed keys from the docs itself.

And then we could compute a score

for each doc that was just a sum of these cost scores

times each head, and then we just took the mean over all

tokens in the doc,

right? And what we found is that using this pretty simple scoring

method

from the sparse attention mechanism, we actually match or

beat state-of-the-art

current reranker models, showing how this essentially mirrors a

reranker and is

essentially just context retrieval over the

context window. So here, two minutes left, but I just want to reiterate that

these are both examples of fundamental information introductions to the system,

context retrieval, that make the full system much more

efficient, right?

Sparse attention is way more efficient than dense attention.

Multi-agent systems, we showed that there was

a huge reduction in average tokens used in order to get the same result.

And the last thing I'm going to talk about is some work I'm doing with the SNAP

Lab at Stanford to reduce the number of tokens we actually need in the input

rather than the

output,

while maintaining state-of-the-art results.

So we have this idea of

when doing

LLM systems, why do we have to put the entire doc in

the context window when

doing RAG? Instead, what if we just treated memory or context

injection as a modality as we do text, as we do image, as we do video, where

it's a full modality onto itself, and how would that work, right?

And so what we did ... is we built a memory module,

is what we called it, where

we took docs in as well as a query.

This is over a multi-hop reasoning dataset.

We built an architecture that

took these in, produced 16 latent representations,

and then injected them directly into the frozen LLM. Here, we used Qwen 8B.

And our theory was, okay, if we can inject 16 latency

representations that

tend to take the information in the docs from the query that is necessary, how

will it perform? Directly into the frozen LLM.

What we found is using the trivia QA dataset,

we actually beat

RAG 50, where we took the 50 most relevant docs to

the query and injected them

directly in the context window in the token space by a fair margin.

So

using this approach, we could

inject the 16 latent space representations and get a 63% exact match

over the, like I said, the trivia QA dataset, while RAG 50 got a

55% exact match. And fundamentally, this actually used

372 times less tokens representations in the

input. So it's doing better. It's more intelligent, efficient,

while using

372 less tokens. So it's incredibly more efficient just by

introducing

the correct context through this method in the latent space.

Okay, so I hope that I've convinced you all

that latent space representations

of context and context retrieval is incredibly important in the LLM sense.

I want to leave you with what I think would be the holy grail of a

model that could introduce context accurately.

I think it should be low latency, so it can be used at inference time.

I think it should scale over large corpuses, and I think that there should be

essentially zero switching costs to the base LLM, so that when the next model

came out, you could introduce the most

state-of-the-art reasoning model while still maintaining this context

introduction into the global system.

These are research topics that I'm exploring so

that we can introduce them in

the large AI tooling systems that we're using at Ramp, such as Inspect,

where

we're actually only generating more context than we ever have before.

Thank you.