Harnesses. I think this is a very

important topic. A lot of you are

thinking through building your own

harnesses right now. Um, I'm very

excited to introduce Harrison. I first

noticed Harrison on Twitter in 2022,

back in the GPT-3 era. And Harrison was

one of the first people thinking about,

"Okay, we have these models. How can we

build an entire harness around them so

that they're not just um auto-complete

uh tasks, but that they start acting as

virtual collaborators or agents?" Um and

Harrison, like the ecosystem has grown

so much since 2022, and I've seen you

you also grow a lot in terms of how you

think about building agents, building

harnesses, how to eval them, etc. Um so,

I'm very excited to have you talk today.

I think the talk's going to be both

about harnesses and evals. And then

format again will be 15 minutes or so of

presentation content, uh 15 minutes of

Q&A. Thanks for joining us, Harrison.

>> Cool. Um my name's Harrison, co-founder

CEO of LangChain. I want to talk about

evals and harnesses in the context of

kind of owning your own intelligence.

So, when we talk about intelligence,

we're normally talking about agents.

What exactly makes up an agent? At

LangChain, we think there's kind of like

three main parts. There's a harness that

orchestrates a model and some context.

And if you're talking about owning your

intelligence in general, you probably

want to own all three parts of these.

And so, owning the model, I'm not going

to talk too much about. Lin was here

from Fireworks and talking about open

weight models and owning that. Uh big

part of this is also the ability to

switch models. Uh

there used to be this concept of kind of

being like cloud agnostic and being able

to switch clouds uh back in the day.

Same thing exists, but for models. You

want to be able to switch to avoid

lock-in, but also to just use the best

model when it's available. Context, you

want to own all the the context that

your agent uses. Whether that is memory,

uh whether that is semantic knowledge,

um whether that is previous

conversations. You can these can help

personalize and guide the agent as it

goes along. And then the last bit is the

harness, and that's what I really want

to focus on.

So, how how do you how do you really own

your harness? What does that even mean?

What's the main job of a harness? The

main job of a harness is to bring

context to the model at the right point

in time.

And so, it does all the orchestration

around the the fixed context, the the

dynamic context. It brings it into the

context window of the model, shows it

something, gets some response, and then

does something with that.

And so, agents need to do all these

different things in order to accomplish

their jobs. There's a ton of

domain-specific stuff that they need to

do as well, but they need to interact

with external systems. These external

systems, when you interact with them,

they emit more context that can get fed

back into the agent into the loop. And

so, the harness is the thing that really

orchestrates all of this together.

Agents at [snorts] their their kind of

like simplest, when everyone talks about

agents, what they really talk about is

just an LLM running in a loop calling

tools.

Um and uh this this is a really simple,

but really general architecture. Some

request comes in, the the LLM makes some

uh generation. That generation may

include a tool to call. If it does, you

invoke those tools, and you pass that

observation back to the LLM. And and

this is this is the core architecture

behind pretty much every agent out there

today.

But they're all different in like

slightly different ways. And so, on the

left here, this is kind of like uh the

the the base core kind of like loop. But

there's a bunch of different things that

you can do in your particular harness at

different stages.

And so, this is uh over here, this is

So, so we built LangChain, which is a

really really base minimal harness, and

that's LangChain over here. And then

this is Deep Agents. Deep Agents is kind

of like our model-agnostic and and more

general-purpose version of of Quad Code.

And so, it does more things. It connects

to file systems. Uh it has skills. It

has sub-agents.

It's built on top of this really simple

harness, but we customize it by using

these uh these levers over here. So, you

can run particular code snippets before

the agent's invoked, before each model

call. You can kind of like wrap these

model calls, you can wrap the tool

calls, and you can and you can customize

this core uh simple loop in a lot of

really powerful ways just by just by

using kind of like small what we call

kind of like middleware constructs.

There's other ways to customize the

harness as well, but this is kind of

emerged as uh there's a concept of hooks

and plugins in a lot of the coding

agents as well, and that's essentially

what they're doing. They're taking this

base loop that's running, and they're

adding little hooks or plugins at

various points to let you customize it.

And so, a lot of the stuff that you can

do when you can customize, this is all

done by that concept of middleware, by

just modifying that core loop. So, the

agent's still running in a loop, it's

still doing that same simple

architecture, but through that you can

give it access to a sandbox, you can

give it access to a file system, you can

give it access to sub-agents, you can

give it access to to memory, you can

have summarizations. So, summarization,

if if if we go back to this thing,

summarization would come in before the

model. Before the model's invoked, you

check if the context is too long, and

then you summarize it. And so, that that

you can add into this core loop through

this concept of middleware, same with

context offloading,

um which which is a way of basically

taking large tool calls and dumping

them. That kind of wraps the tool call.

And so, the point is there's there's

this really simple kind of like general

architecture of an agent. All of these

more advanced agent harnesses are

basically doing that loop, but adding in

a bunch of stuff while still running the

core loop. And so, as you think about

kind of like building or customizing

your own harness,

these are the different places that you

can insert things into. You can add your

own summarization step, you can add your

own handling of particular tool calls,

and that's one way that you can

customize kind of like the agent to your

particular domain and the harness to

your particular domain.

The other way that you can customize the

harnesses that the agent runs in is by

having a more explicit kind of like

cognitive

So, this used to be the way that a lot

of people would build agents in kind of

2023, 2024 because the models weren't

good enough to run in a loop. And so, in

order to get it to do particular things,

you would have these very bespoke

cognitive architectures. And so, this

one over here is

for

a deep research example

where it would generate some some sub

questions, fan them out, and then go and

execute them. And then this one over

here is for a code review bot. And you

can see that there's these very kind of

like the bespoke steps.

A lot of this has gone into the harness

now. And by the harness, I mean it's

still this core loop. These might be

added as particular kind of like

modifications to that core loop. But,

for a lot of really particular kind of

like flows, we do see people still using

cognitive architectures like these to

really guide it in particular ways. One

thing that we recommend to people is to

start with a general harness.

That's the easiest to get started. It's

going to be quickest to time to value.

And then as you kind of narrow in on the

use case that you want to be excellent

at, you can start to add more of these

kind of like gates and checks around it

to to to guide it into particular ways.

One question that we get a lot is when

to kind of think about building your own

harness versus using an off-the-shelf

harness. Um a lot of the off-the-shelf

harnesses are uh work with particular

models. So, the off-the-shelf harnesses

include things like Claude Code or

Claude Agent SDK, which works with

Anthropic models, Codex, which works

with OpenAI models. I think this is a

big open question in the industry. My

answer generally is the more in

distribution you are of what the models

are trained on, then the better the

off-the-shelf harness will be. As soon

as you start to move further and further

out of distribution, then the then

you'll probably want to tune your

harness in some way.

There's different ways to tune the

harness as well. So, the models may be

in distribution on particular things

that you are doing on an out of

distribution task. So, what I mean by

that is if you think about something

like like legal AI, which Gabe just

talked about, and I think he mentioned

how they have their own harness,

there are things that are in legal AI

that are still in distribution of the

the the main models. So, for example,

editing files is something that the main

models have all been RL'd on. And

they've actually all been RL'd in very

particular ways. So, OpenAI and and

Claude models edit files in in different

ways in their harnesses, and as a

result, their models are actually best

at editing files in different ways.

Now, the the models themselves are out

of distribution on this larger task of

legal AI, but they're in distribution on

this task of editing files. So, if you

think about building a harness that

works there, you'll probably want a

custom harness, but you'll want it to

use the edit file tool that is in

distribution for

the the the model that you're using. So,

one of the things we do in Deep Agents,

for example, so Deep Agents is our

customizable harness, we actually have

this concept of model profiles, where

for things that are in distribution of

models like editing files, we basically

switch between different edit file

implementations depending on which

model's being used. And so, I think

that's an example of customizing the

overall harness when it's out of

distribution for a task, but keeping

smaller in distribution parts

as close to the model layer as as

possible.

The second big part of what I want to

talk about is evals and observability.

And so, I think as you're experimenting

with all parts of an agent, whether it's

the model or the harness or the context,

you're going to want to know what's

going on inside of this system, and

you're going to want to be able to

evaluate it. And so, these are useful

tools that you can use, again, not just

for custom harnesses, but also for

custom models.

So, there was a great Twitter article

that Satya wrote uh

2 weeks ago,

um where he talked about a lot of these

concepts. And there's three quotes in

particular that kind of stood out for

me. One, create your private evals

because eval defines what good looks

like inside the organization. Two,

retain ownership of your organization's

memory, traces, feedback, though that

bold is mine, decisions, and

institutional context. And then three,

you create your own continuous learning

loop hill climbing machine that will

allow your AI investments to compound

the value of your firm. And so I think

these speak to the importance of evals

and observability and the learning loop

that they power in in really owning your

intelligence and compounding it. So how

exactly do they do that?

So evals. Gabe was here talking about

how they built benchmarks for the legal

domain. I think every company what when

they're building a mission-critical

agent, they will build benchmarks for

that agent.

Um you can use it to define and catch

regressions or you can hill climb on

that benchmark. Again, either by

adjusting the harness or adjusting the

model.

Things that we see becoming the industry

standard for defining these benchmarks

is Harbor. Harbor is an open-source eval

runner. It's uh created by the makers of

Terminal Bench 2, which is one of the

industry standard benchmarks for

benchmarking coding agents, and it's

become pretty popular for a variety of

domains.

What it lets you do, so this is so this

is Frontier Bench, which is another uh

coding benchmark. You get this nice

benchmark and you can compare different

agent harnesses, different models,

different reasoning efforts, and you can

get this nice benchmark and you can see

how all these different harnesses and

all these different models do on your

task. And so having a benchmark for your

task will become really really important

when you're trying to define it.

What exactly is Harbor? It's pretty

simple. At a high level, it consists of

agent that you you run an agent against

the data set. A data set has a bunch of

different tasks. Generally, they're run

in sandboxes because they are a lot of

these different tasks and you might want

to parallelize them. And as we talk as I

talk about in a little bit, each task

has its own kind of like environment. So

this is what a Harbor task looks like.

So, on the right, you can see that it

has an environment. This is where you

define the environment that the agent

runs in. A lot of these longer running,

more stateful agents need to interact

with their environment. And so, you

basically spin up a sandbox, give it its

own environment that's defined in a

Dockerfile, and run it there.

There's then uh a solution, which is

basically this this is uh a kind of like

golden solution that you use to sanity

check it, so it's not that interesting.

Test is more interesting. This is

basically the verifier for the the agent

run.

The test scripts can do anything. They

can run code, they can run unit tests,

they can run another LLM as a judge,

they can run an agent as a judge. You

basically define how the agent is scored

in this test. And then instruction.md is

the prompt that the agent is given. And

that's kind of like the core of Harbor.

You define these tasks, which are

bundled up things that can be run in a

sandbox, and then you run a bunch of

them against agents. And agents again

consist of models and harnesses, and you

score how how they do.

When you do all of that, what do you

get? You get some nice results that you

can compare. So, this is LangSmith, the

the platform that we build for evals and

observability. And so, you can see here

a bunch of different experiments. Uh we

have a great integration with Harbor.

You can see the the feedback scores. In

this case, it's a single reward

function. You can also track latency and

tokens. So, when you're benchmarking

agents, you probably don't just care

about accuracy. You also probably care

about latency and and and cost. And so,

you'll want to track all of those.

And then for a particular experiment

that you run, these are these would be

the different tasks that are in a in a

in a Harbor data set.

Talking about a little bit about

observability. Uh observability sounds

basic, but I think it's really important

and really underrated for agents,

actually. So, when agents mess up, they

mess up because an LLM call goes wrong.

Why might it go wrong? It might go wrong

for one of two reasons. One, the model

is not good enough. Two, the context

that the LLM received isn't good enough.

And so, I actually think it's the the

second one that more often than not

causes issues. And so, having really

good observability into what is going

into the context window of the the model

and then how that context is

accumulated, what steps were run, what

tools were run, how does that context

get there? All of that is really

important for debugging your agent when

it when it goes wrong.

So this is one view of observability

that we have. This is intended to be a

more kind of like user-friendly view

where we actually represent it. This is

similar to what you might see in kind of

like Claude code. We we kind of like

hide some of the tool calls so you can

see seven tool calls up there.

And so we try to make it really easy to

kind of like skim through this. Most

agent paths these days come in the form

of trajectories. Trajectories are

basically you can think of them as the

list of messages that you see kind of

like Claude code running. So when you

run Claude code or another agent, you

type in a human message, it then makes a

bunch of tool calls. Those are all

messages under the hood and then it

responds and then you type in another

human message. That's kind of like this

message trajectory that is becoming more

and more of a central part of of these

agents. But that's not enough to fully

debug it and so we also have this full

kind of like trace and you can click

into particular things and see exactly

what goes on inside the model. And this

type of observability is pretty

important for knowing what's going on.

Evals and observability really let you

set up this data flywheel and compound

the intelligence as you as you start to

use the agent, as your users start to

use the agent and you start to get

feedback.

So this is this is a slide that one of

our team members presented at Swyx's AI

engineering fair actually around a

recipe for continuously improving

agents.

At a high-level it's really simple. You

build an agent, you start running it,

you collect lots of traces, you then

curate the trace data, and then you run

experiments on on that data that you

create. And so it's really simple,

but of course there's a lot of

complexity under the hood. So so one

thing that's really important for this

is is feedback. Getting feedback either

from the environment or from synthetic

source. So from the environment, one

thing that I think is really

underestimated in agent design is

actually UX design of how you present

the agent to your users. If you present

it in a really intelligent way, you can

actually end up getting a lot of

feedback from them. They may not click

thumbs up or thumbs down explicitly. No

one really doing that. But if you if you

design the UX in a clever way, you can

get some of that feedback. The other

thing you can do is you can start to get

synthetic feedback. So you can run what

we call kind of like online evaluators

over these traces to judge things. So

Gabe was talking about an experiment

that we did with Harvey where we we

significantly reduced the cost of some

of these LLM as a judge type thing. So

if you imagine running Opus over every

single trace that comes into your

system, that's going to rack up a big

bill. And so you want a really cheap and

fast way of doing this. So we've

fine-tuned some SLMs for actually doing

this, but you can of course use

off-the-shelf models with custom

prompting to do it. Or you can just use

code if some of the things that you want

to test are are simple enough.

Um so this is the the the full a part of

it. Um the curating the trace data,

feedback is a big part there. And then

the other thing is is when you use that

data to update what happens, you can

update any part of the agent with this

with this kind of like system. Um so you

can update the harness by doing harness

engineering. You can update the model by

do a by doing fine-tuning on that. You

can update the context by doing memory.

Um and and so the part that we are that

we think most about at LangChain is the

harness engineering part of that. And so

I want to show a really quick demo of of

one of the things that we added to help

with that. But I I think Trajectory is

talking next on some fine-tuning that

can be done. Uh and and so it's a very

similar process where you run the agent,

get some traces, use that data in some

way to improve the system. What is the

system? It's these three pieces. Any of

these can be updated in some way.

Um and so yeah, this is the full

end-to-end um

uh uh flow that you might want to do.

One of the things that we think about is

how can you automate this as much as

possible because this is tricky and

takes a lot of time.

Um and so that's one of the things that

we've been thinking about for the past

few months. I want to do a quick demo of

what we call LangSmith Engine, uh which

is basically an agent that sits on top

of your traces and does all this work.

So if we look at what that work was, you

know, you've got these traces, it's it

the work from there is curating the

traces and running some experiments,

suggesting fixes to one of the three

things. And as I mentioned, we mostly

focus on the harness engineering bit. Um

so uh in the demo I want to show uh what

this looks like and how it represents

that. So hopefully this will work. If

not, it's not that big of a deal.

Perfect. Okay. So this is LangSmith.

This is a bunch of traces we have coming

in. Um we have this tab called engine

over here. Um this is an agent. It runs

in the background. It creates what we

call kind of like issue boards. So this

is the part of curating data. It will

look for It will It will It will

basically under the hood is a coding

agent that has access to our LangSmith

CLI. The LangSmith CLI, you can filter

traces for feedback and other things

like that. So we give it a nice big

prompt and some sub-agents that help it

basically go out and explore this data

and identify issues and see what common

things are. And then it will create

these issues right here. And so here um

it's created an issue. It gives a

description of it. It has It links to

the traces so I can go see some

supporting evidence. And then down here,

I guess this is very simple changes to

the prompts. Um but here it's updating

part of uh of the context in this case.

Um here it's also updating some

instructions. Um and we can see here

that it's adding uh some code to go into

the harness as well. And so this is uh

something we launched in the past few uh

months and I think speaks to this data

flywheel, which again is a very simple

thing. Run agent, get traces,

see patterns, fix. Um and and this is

our attempt at automating it. Um, that's

all I've got. Happy to take any

questions on harnesses or evals.

>> Um, this is great, by the way. Really

appreciate the whole whole presentation.

Uh, Engine itself is an agent, right?

That's given a prompt and go search over

things. Uh, have you run Engine on

Engine?

>> We have it running, yeah. So, we got

slight So, Engine also hooks up to Slack

and sends it kind of like reports about

itself. Um, and yeah, that's how we

that's how we dogfood it. Yeah.

We also uh we also created uh

uh what we call kind of like issue bench

for Engine, which again is like a harbor

a harbor formatted uh

uh

benchmark, basically, that we're

constantly benchmarking different models

and different harnesses on.

Um, and so, it's

I think Gabe talked about this a little

bit, but one of the benefits of having a

benchmark is you can you can benchmark

it on a bunch of different harnesses and

see what they're good and bad at. So, we

we uh I think a few weeks ago we ran our

our own kind of like deep agents and

then Codex and then Claude Code on this.

And we saw that Codex was doing a really

interesting thing, where it would write

itself a bunch of small scripts to run

against these traces, and it was doing

that really aggressively and actually

allowing it to perform really well. So,

we we did a sprint to do what we call

the kind of like codis- codexification

of Engine and basically take that

learning and and bring it into kind of

like the the core Engine harness. And

so, I think that's another uh a benefit

of having a benchmark is you can just

run a bunch of different things on it

and see how they actually perform and

then bring those things back into your

core kind of like agent harness.

>> Super cool talk. Um, to which extent do

you think that like harnesses will

converge into one thing and users will

be educated to do that and the models

will be best for that versus

diversifying here, every company has

their own

uh way of doing things uh optimized for

them?

>> Yeah. Yeah, really good question, one

that we think a lot about. And um, I I

chatted with Eno from Factory, who also

kind of like thinks a lot about this.

Um, and and I think there's some stuff

we talked about this.

Um,

I I think

uh

I think there's

I I think the model I I don't know is

the is the honest answer. I think uh

some things that I've seen is that the

general purpose harnesses have gotten

good enough to work for a lot of basic

tasks at least when you're getting

started. So, I would recommend getting

started with like an off-the-shelf

harness, whether it's Deep Agents or

Codex or Cloud Code or something like

that. Because I think the models are now

good enough and the things that we've

learned about what these makes these

models good, access to file systems,

sub-agents, things like that, those are

those those are kind of like good

enough. Um, I think we often see that

the more out of distribution you get,

the more you're going to want to

customize the harness. And it's a scale,

right? So, like it uh it it it it at the

extreme end of a scale, you might want

to build a complete kind of like

cognitive architecture that that is

really focused on things. A A reason

Another reason you might want to do

that, by the way, is kind of like for

predictability and control. And so, we

have a lot of customers in financial

services where they want where they need

kind of like predictability. And so,

they we show them something like Deep

Agents and they're like, "Woah, woah,

woah. That's way too like scary an agent

for us. We want like more of this kind

of like custom cognitive architecture

where we can really control things." Um,

but but then on the other end, uh you

know, you could just use an

off-the-shelf harness and there's things

in the middle like hooks or middleware

that you can use to kind of So, so it's

a spectrum as well. The more out of

distribution you get, the more custom

harness you're going to want to have.

Um, and then there's other like weird

things where like again, like um, I I I

think uh both OpenAI and Anthropic are

getting really good at coding, but

they've landed on different ways to kind

of like edit files um that are like, you

know,

that are like actually pretty different.

Um, you know, I I think they have some

benchmark and I think he was uh he he

thought that one way was just better

than the other way just like strictly

superior

and and so that's like like

so I think the model labs will kind of

converge and that they all seem to be

kind of like really good at coding the

harnesses will converge to to kind of

like being really good at coding if they

keep on going down that path but they're

at the same time there are these like

really small differences and I don't

really know how to explain those either

and and

right now I think those show up most

concretely in small things but you could

imagine what what if one lab really goes

down kind of like bio and those

harnesses become really good at kind of

like bio agent things then then then the

harnesses themselves start to diverge

and so

I I I don't know is the answer to the

fast moving space that's why evals and

observability are important and I think

and and I think we

to measure all of that.

Cool. Awesome. Thank you guys.

>> [applause]