[music]

[music]

I would like to start with a little bit

of like a change. So I'm sure many of

you spend your days prompting agents,

right? In fact, some of you are doing

this at this very moment, right? Yeah, I

know what you're doing. So instead, I'm

I'm going to turn the ship around. I'm

going to prompt you for this talk. So

you are the audience of a talk about

harness hacks. The speaker is a tiny bit

nervous, so he might speak quickly.

Listen carefully. Laugh at the jokes if

they're funny. Hold your questions until

the end. Find me. And if something is

unclear, it is his fault, not yours. All

right, take a deep breath.

Let's begin. So, I'm Ivan. I work at

victor.com and I do a lot of open source

work related to local inference agents

and agentic systems. And if you want to

find out more, my GitHub handle is

fairly easy to type. So, feel free to.

And first about the product where I

work, victor.com. Well, AI employee, I

think many people here heard these words

one too many times already. So, let me

tell you what makes him special. So,

first of all, I checked this morning and

in our production workspace we have 173

integrations connected right now

totaling 5,400 tools at once. Then we

have 1,300

skills in the same workspace also at

once. And it's a single agent. It's not

a multi- agentic system. no sub aents

nothing like that and it works we just

use it like it automates half of our

company and the growth that company is

experiencing wouldn't be there without

that second thing is that it's

continuous agent so you can start your

work in slack you can continue in open

code and then you can finish on the call

while you are I don't know going

somewhere then uh it's a newer

generation of harnesses so it's double

loop by design meaning it's async native

so You can steer it, you can join uh the

trajectories mid run and it happens

asynchronously.

Then lastly, it works wherever you work.

It's a true work scale AI.

So Slack, Teams, all of the APIs you can

think of, OpenAI compatible, entropic

compatible, ACP, of course, MCP so that

you can use with your traditional tools.

So harness hacks, but let's talk a

little bit about the harness itself.

harness engineering, right? What what is

the harness itself? Uh for the purpose

of this talk, it will be very simple. So

harness is everything that sits between

the world and the model. And some people

might uh say, okay, is the inference

engine a part of the harness? Could be

why not

let's start in the beginning. So May

2020 we have GPT3 coming out and there

is a paper called language models are

few short learners by open AI and the

thing is GPT3 is not capable of

following instructions it completes next

token right so how do we make it work

how do we make it do it what we want it

to do well the answer is we give it

examples and we make the thing that we

want to do the next most probable output

for the

And the paper starts with a translation

example. It's very simple. So you have a

word translated version and then you end

with a

placeholder for the model to fill.

Then we are moving to a slightly

different area. So what examples are

doing they are just priming model latent

states. So the thing that you want is

the next most probable output. Right?

But it's not the only way to get there.

we can get there in a slightly different

way. What if we just type the

instruction? Well, okay, it can actually

put the model into the same exact state

and an awesome orc Luther AI uh released

this paper in February 2021 called

prompt programming for large language

models which describes exactly that. And

since then it was about searching how to

prime that internal state of the model

to do more of the things we want and

less of the things that we don't want

to.

Then we move to October 2021 and AI

chains. I I like many of you know uh

where this is going but that actually

happened a year later. So this was

actually a research paper from Google

research um in October 2021 and uh they

found out that well okay uh prompt is

one task but if we want to achieve more

delegated work then we chain them and

then it gives us the results we need.

And then a year later, longchain put it

into a very viable framework.

Uh May 2022,

we are on a trajectory where we see that

model becomes slightly um more

responsible for what it outputs and

learns a little bit of the next probable

outputs step by step. And they found out

that u if you just append this tiny

thing in front of your task, let's think

step by step, the model outputs exactly

that and ends up in a state where it's

more likely to give an expected [snorts]

answer. Such a simple thing and all ever

since we were hunting for for approaches

like that, right?

Next, we are moving to October 2022. So

text generation has no side effects.

and how model acts in the world. Well,

let's let's build a small framework

around it, right? Let the model reason

about what it sees. Then let it output

some action. Then we execute that action

and then the world replies. And that I

think many of you already seen this loop

where you implemented it countless times

here, right? React classic paper on on

how harnesses are built. So um and

lchain which we talked about shipped a

react chain within 3 weeks after it was

released and so agentic frameworks uh

began

many others too. Um

then we are moving further. So we seen

examples few shot we seen framing zero

shot when we're just trying to uh put

the model into specific state. We seen

chains of prompts. Then with react, we

seen loops. And what's next? Well, just

add more dimensions to it, right? Let's

make it a three. So three of thoughts.

Let model consider multiple

possibilities explicitly and then

estimate which one is more probable to

happen or better for our outcome. Choose

that and continue based on that option.

So and repeat that in a loop or any part

of other framework. And on game of 24,

uh, GPT4 with this approach went from 4%

success rate to 74,

but of course at a high far higher

inference costs.

Uh, there are always trade-offs, aren't

they? So then we are moving to July

2023. Uh, I want to ask how many people

here have promised a reward for a model

if if it outputs syntactically correct

JSON.

Okay. Okay. So, that's not great. And if

we want more predictable outputs, of

course, we have to build a framework

around it. And that's another thing

where we had to make our inference

engine a tiny bit of of our harness

actually to make sure that they produce

what they want. So what if we take the

center which is already a part of our

inference pipeline anyway and let's use

it as a parser. Why not good hug and a

grammar could decide which tokens are

even allowed to be generated and we

don't have to care about invalid outputs

or retries again later. It just works by

default. Right?

Then uh in September 2023

uh we moving on to the next thing. So we

gave model prompts, we gave examples,

but then the model what if we we give it

a task to optimize its own prompt where

it will lead. Well, turns out it's a

pretty good idea and the models can

research language space quite well and

uh OPRO organization found out that just

giving model a way to optimize its own

prompt improves it outputs significantly

on a specific task. And the most

interesting part is that they launched

optimization hill climb on let's think

step by step. They uh t the model to

find a better version of that and models

are trained on on our outputs on human

outputs. Right? So the best performing

version they found was let's take a deep

breath and think step by step. So I find

it fascinating that they did that.

uh then July 2024

we have

uh we are entering the era when we want

to see better quality outputs of our

models and what else can we hack well we

have a sampler again we hacked it to

produce grammarss we want hm okay we're

also using it to produce more varied

outputs well let's let's actually try to

repeat generation 250 times pick best

candidates and then see if if it helps

on equality with with our given task. Oh

wow. And it did help quite a lot

actually. And imagine this, we are doing

this and then we're picking best

trajectories. We are training models on

them. Then we repeat and boom, we have

something that can improve itself over

time.

Then uh in June 2025

uh we observed another phenomenon

happening. So context windows always

short right and models can never keep up

with with the requirements that we have

for them and it's unlikely to happen. So

we had to find another hack to actually

to make them work in in such conditions.

So every task was outgrowing context

windows and we stopped trying to fit the

task. Let's split the work across

several windows, context windows. Give

each a fresh context and let let the

model be in in in its native training

context window so that it produces

higher quality output and yeah it helped

a lot.

Then November 2025

a very interesting hug. So uh harness

engineering is hard. So we are not

creating a lot of harnesses, right?

Because it's hard to optimize, hard to

adjust. But the thing is by that moment

in time models already learned quite a

bit from training data about the models

themselves, the harnesses, the

behaviors, what's needed, what's not,

how training data distribution looks

like, how the task distribution looks

like. And it's only reasonable if we

give model back some of that control so

that we can extract that knowledge from

the model, right? Well, okay, let's

start with a very simple harness. two

tools, one calls bash and another reads

files and tell the model well figure out

the rest. Write yourself tools, use them

and complete the task. And they observed

that the results on Sweetbench verified

went to 77.4%

with this approach. Such an amazing

thing. So models can learn about

previous optimizations from their

training data and just apply them in the

best possible format that is appropriate

for the model. And of course it can be

attached to all of the other things that

we've seen before to hill climb to

optimizations and it can happen

iteratively and we'll talk a little bit

more about that. So rethinking the

evaluation of harness evolution was the

name of the paper and it's a great one.

Great. So almost now August 2026 right

we are entering an era where AI is a

commodity. it's available for consumers

and we just we we want to optimize it

more because nobody will wait for an

assistant that takes 40 seconds for a

first reply, right? And the thing is uh

our tokens are very slow to arrive but

Python functions h they are faster

than tokens and why not just start

executing code as soon as model

generates it. So speculative code uh

execution model generates a function

call and we run it as soon as it is

syntactically uh valid. So that when the

model finish generating we already have

another payload to send to our API and

there is no waiting and the end results

um are arriving to the user faster. The

paper is called speculative programmatic

tool calling uh by Alex Jang a very cool

thing and it observes about 1.2x 2x uh

speed up in the uh small error

experiments.

Great. So all of the previous slides

they were actually on one of the three

trajectories. I think some of you

noticed the small pictograms in there.

So this is the decoding of that and u

this will help us to actually establish

what's next because now we are moving

into the area of like what will happen.

Let's try to fantasize a little bit

about it based on what we've seen and

the first trajectory is about shape of

work. So what we sent to the model and

how it's structured and of course it

started with just a simple prompt. So we

just sent a piece of text. A piece of

text flies out. That's it. Then we

started chaining them and then we

started looping them. And then there is

a tree. Well, what's next? Well, the

answer in the previous instances was

just adding more and more dimensions.

And how do we add more dimensions to a

tree? Well, we make it an a cyclic or

even cyclic graph. And I I think we'll

see more designs like that going

forward. The second trajectory is also a

very interesting one and I think many of

you observed it is that um the model

slowly eats the harness and possibly it

will start eating the world later in

mysterious ways. We'll talk about it. So

how it manifests? It manifests uh in in

changes like for example when cloud code

when migrates into fable 5 they removed

80% of their system prompt because it's

no longer needed. It doesn't add

anything to the model behavior. The

model behaves in the way they want

already and it doesn't add anything. So

we can just throw it out and model

already knows about it. Great. So in

there we started with think step by

step. We explore different formats of

how that can look like. We let the model

to try to find better versions of how to

ask itself questions. Then we let the

model to modify the harness itself. Uh

now we are at the stage where model can

also define execution shapes and

trajectories. Uh I think many of you

used ultra code from cloud code, right?

So I think that's just the beginning of

where it starts. And the next two things

we'll talk about will be about hill

climb because I believe eventually most

of the tasks will turn out into a hill

climb exercise of sorts. And then about

model eating the world. How scary it

sounds. We'll see. And the last one is

about the scale. So we started with a

single interaction with the model.

Single prompt, single output, then many

tries, best of end, then sub agents each

working on its own uh portion of task

and then of course more of that.

Let's go. So the first one uh we will

turn most tasks into a hill climb. What

does that mean? It means that you

probably observe that many companies are

starting to collect trajectories as much

as they can and this is a good thing

because those trajectories are very

valuable sources of of gradient and the

gradient is the scars artifact in this

new age. What we want is a clear and

useful signal on what to improve and why

and then the models can can already do

that autonomously. So hunting for that

signal will will become the next thing

that many companies will do and already

doing actually. So we set an objective,

we define experiments in that objective.

Uh then we have a clear metric to

optimize then we decide to keep some of

the experiments and build on top of that

and the whole thing loops as many times

as as we needed to.

The next thing is work becomes a graph.

So loops chains yeah great but graph can

have all of that. It can be a chain, it

can be a loop, it can have all kinds of

interesting topologies. So it cannot

parallel work, fan outs, funins, joins.

It cannot retries roads that are

conditional or things that are only

available after an intermediate result

and ultimately it will be just an easier

paradigm to orchestrate and the models

already know all of it like you don't

have to explain it. Uh this is a very

interesting exercise. I encourage all of

you to ask your coding agent next time,

please draw me a mirror diagram of how

you can complete this task with sub

agents and it will show you like amazing

amount of of understanding of how to

actually split work in such pieces and

and and use them.

Great. So the world retreats into the

model. What does it mean? Well, it means

that initially we have to explain the

world to the model and as they got

better, we we had to do less and less of

that explaining up to a point where now

most models they they understand that

okay uh

we'll just say it like that. Of course,

for the purpose of the talk, they

understand quote unquote that they are

large language models acting in the

world through some sort of harness and

they can own it and modify it. And it

means that eventually uh this trajectory

will lead to models having better and

better understanding of the world, its

state and how reactions of the model

modify that state up to a point where

eventually the model will not need to

actually observe the world. It will have

such a good internal model that it will

just need to check that okay my

expectation aligns with real reality.

And that's very similar to how another

kind of intelligence operates, right?

And then uh we have a swarm a very

interesting thing uh scale grows up

prompt model many tries many sub aents

we have a graph where we are uh this

told phrase like a city of geniuses

about modern data centers and the scale

will only grow larger up to a point

where you will say that okay I have a

civilization of experts working on my

task could be so how will we prompt AGI

based on what we discussed

I think about three things. So probably

through an agent that makes the case for

you because that capability you won't be

able to access it directly. There will

be an intermediate that translates,

explains, guards. The prompt might be

short because the model will have such a

good understanding of the world that

eventually your prompt might look like a

shrug in the morning and then the work

is done in the evening. Then the work

behind your prompt might be civilization

sized.

That's it. Thank you.

[applause]