All

right. Thank you all for for joining for

automating massive refactors with uh

with parallel agents. Um super excited

to talk to you all today about uh you

know what we're doing with open hands to

really automate large scale chunks of

software engineering work. Lots of uh

lots of toil related to tech debt, code

maintenance, code modernization. Uh

these are tasks that are super

automatable. Uh you can throw agents at

them, but they tend to be way too big

for like you know a single just one

shot. So it involves a lot of what we

call agent orchestration. Uh we're going

to talk a little bit about how we do

that uh with Open Hands and also just

more generically.

Uh a little bit about me. Um my name is

Robert Brennan. I'm the co-founder and

CEO at Open Hands. Uh my background is

in dev tooling. I've been working in

open source dev tools for over a decade

now. I've also been working in natural

language processing for about the same

amount of time. Um uh I've been really

excited over the last few years to see

those two fields suddenly converge as

LMS are really good at writing code. Um

and I'm super excited to be be working

in the space. Uh then open hands is an

MIT licensed coding agent. Open hands

started at open dev about a year and a

half ago when Devon first launched their

uh demo video of a fully autonomous

software engineering agent. Uh my

co-founders and I saw that got super

excited about you know what was possible

what the future of software engineering

might look like. uh but realized that

that shouldn't happen in a black box,

right? If our adopts are going to

change, we want that change to be driven

by the software development community.

We want to have a say in that change. Um

and so we started opens uh then open dev

as a way to give the community a way to

help drive what the future of software

engineering might look like in an AI

powered world.

Uh so hopefully not uh controversial for

me to say that software development is

changing. Um, I know my workflow has

changed a great deal, uh, in the last

year and a half. Um, uh, I would say

now, like, you know, pretty much every

line of code that I write goes through

an agent. Uh, rather than me opening up

my IDE and typing out lines of code, I'm

now asking an agent to do the work for

me. I'm still, you know, doing a lot of

critical thinking. You know, a lot of

the the mentality of the job hasn't

changed, but what the actual work looks

like has changed quite a bit. Uh, but

what I want to convince you all of is

that it's still changing. We're still

just in the first innings of this

change. We still haven't realized all

the um all the impact that large

language models are have already brought

to the job and are going to continue to

bring to the job as they improve. Uh I

would say even if you froze large

language models today and they didn't

get any better, you would still see the

job of software engineering changing

very drastically over the next two to

three years as we figure out ways to

operationalize the technology. Uh I

think there's still a lot of uh sort of

psychological and organizational hurdles

to adopting uh large language models

within software engineering. Um and

we're seeing a lot of those hurdles

disappear as time goes on.

A brief history of kind of how we got

here. U everything started I would say

with what I call contextware code

snippets. um some of the first large

language models it turned out were very

good at writing chunks of code

especially things that they'd seen over

and over again. So you could ask it to

write bubble sort. Uh you could ask it

for you know small algorithms you know

how to how to access a SQL database

things like that. Uh it was able to

generate little bits of code. It was

able to to you know it seemed to

understand the logic a bit. But this was

totally context unaware right it was

just dropping code into a chat window

that you had asked for. It had no idea

what project you were working on what

the context was.

Shortly thereafter we got these

contextaware code generation. Uh so like

GitHub copilot as autocomplete um was

probably like the the best example here

right uh so you actually was in your IDE

it could see you know where you're

typing you know what the what the code

you're working on in uh and it could

generate code that was specific to your

codebase that reference you know local

variable names that reference you know

local table names in your database uh

huge huge improvement for um uh you know

our productivity so instead of copy

pasting back and forth between the chat

GBT window and your IDE now All of a

sudden, you can see the little robot get

its eyes. It can see inside your

codebase and it can actually generate

relevant code for your for your your

codebase.

And then I think the the giant leap

happened in early 2024 um with the

launch of Devon and then uh the next day

the launch of open devon now open hands.

Uh this is where we first started to see

autonomous coding agents. So this is

when AI started not just writing code

but could run the code that it wrote and

it could Google an error message that

came out, find a stack overflow article,

apply that to the code, add some debug

statements into the code and run it and

see what happens. Basically automating

the entire inner loop of development. Um

this was this was a huge uh step

function forward. Um you can see the

little the little robot gets arms in

this picture. Um this was a this was a

huge jump at least at least in my own

productivity. um being able to like just

write a couple sentences of English,

give it to an agent and let it churn

through the task until it's got

something that's actually working,

running, tests are passing.

And then now what we're seeing is uh

parallel agents, what we're calling

agent orchestration. Uh folks are

figuring out how to get multiple agents

working uh in parallel, sometimes

talking to each other, sometimes

spinning up new agents under the hood.

Um you know, agents creating agents. Um

this is uh I would say kind of bleeding

edge of what's possible. Um people are

just starting to experiment with this

are just starting to see success with

this at scale but there are some uh some

really good tasks that are um uh very

amenable to this sort of workflow. Uh

and it has the potential to really uh

automate away a huge mountain of tech

that sits under you know every

contemporary software company.

a little bit about kind of like the the

market landscape here. Um, again, you

can kind of see that same evolution from

left to right where we really started

with, you know, plugins like GitHub

copilot inside of our existing IDEs and

we got these like AI AI empowered IDEs,

ids with like AI tacked onto them. Um, I

would say your your median developer is

kind of adopting local agents now. They

may be running cloud code locally for uh

one or two things. Um, maybe some ad hoc

tasks. Uh your early adopters though are

starting to look at cloud-based agents,

agents that get their own sandbox

running in the cloud. This allows uh

those early adopters to run as many

agents as they want in parallel. U it

allows them to run those agents much

more autonomously than if they were

running on their local laptop, right? If

it's running on your local laptop,

there's nothing stopping the agent from

doing rmrf slash trying to delete

everything in your home directory,

whatever it might do, installing some

weird software. Whereas if it's got its

own like containerized environment

somewhere in the cloud, you can run a

little bit more safely knowing that you

know the worst it can do is ruin its own

environment uh and um uh you don't have

to like sit there babysitting it and

hitting the Y key every time it wants to

run a command. Uh so those cloud-based

environments much more scalable uh a bit

more secure. Um and then uh I would say

at the far right here what we're really

just seeing the top like 1% of early

adopters uh start to experiment with is

orchestration. this idea that you not

only have these agents running in the

cloud, but you have them talking to each

other. Uh you're coordinating those

agents, you know, on a larger task. Uh

maybe those agents are spinning out sub

aents within the cloud that have their

own sandbox environments. Uh some really

cool stuff happening there. Uh I would

say, you know, with open hands, we we

generally started with cloud agents. Uh

we've leaned back a little bit and built

local CLI similar to cloud code in order

to meet developers where they are today.

you know these these types of

experiences are much more comfortable

for developers. Uh you know we've been

using autocomplete for decades just got

million times better with GitHub go-

pilot. Um I would say these experiences

on the right side are very foreign to

developers. They feel very strange to

like give off a pass to an agent or a

fleet of agents uh and let them do the

work for you. It feels kind of like uh

for me at least uh the jump that I made

when I went from being an IC to being a

manager um is is what it feels like

going from writing code myself to giving

that code to agents. Uh so very very

different way of working. I think one of

the developers have been very slow to

adopt. Uh but again the top 1% or so of

engineers that we've seen adopt the

stuff on the right side of this uh

landscape. Uh they've been able to get

you know massive massive lifts in

productivity and tackle huge backlogs of

tech that other teams just weren't

getting to.

Uh some examples of where you would want

to use orchestration rather than a

single agent. Uh typically these are

tasks that are going to be very

repeatable and very automatable.

Uh so some examples are things like the

basic code maintenance tasks, right?

Every codebase has to uh you know

there's there's a certain amount of work

to do to just keep the lights on, right?

To keep dependencies up to date to uh

make sure that any vulnerabilities get

solved. Uh we have one client for

instance that is using open hands to uh

remediate CDEs throughout their entire

codebase. They have tens of thousands of

developers, thousands of thousands of

repositories. Um and basically every

time a new vulnerability gets announced

in an open source project, they have to

go through their entire codebase, figure

out which of their repos are vulnerable,

uh submit a poll request to that

codebase to uh actually uh you know

resolve the CVD, update whatever

dependency, fix breaking API changes. Uh

and they have seen a 30x improvement on

time resolutions for these CVDs by doing

uh orchestration at scale. uh they

basically have a setup now where every

time an ACV gets announced, new

vulnerability comes in. Uh they kick off

an open hand session to scan a repo for

that vulnerability. Uh make any code

changes that are necessary and open up a

pull request and all the downstream team

has to do is click merge, validate the

changes.

Um you can also do this for like

automating documentation and release

notes. Um there's a bunch of

modernization challenges that uh

companies face. Um, for instance, uh,

you might want to add type annotations

to your Python codebase if you're

working with Python 3. Um, you might

want to split your Java, you know, like

a monolith into microservices. Um, these

are the sorts of tasks that are still

going to take a lot of um, thought for

an engineer. You know, you can't just

like one shot it with code and say like

uh, you know, refactor my model if it's

microservices, but it is still very real

work, right? You're still just kind of

like copying and pasting a lot of code

around. So if you thoughtully or trade

agents together, they can do this. Um a

lot of migration stuff. So migrating

from like old versions of Java to new

versions of Java. We're working with one

client to migrate a bunch of Spark 2

jobs to Spark 3. Um we've uh used Open

to migrate our entire front end from

React uh from Redux to Zustand. U so you

can do these very large migrations.

Again, lots of very growth work. still

takes a lot of um thinking from a human

about how they're going to orchestrate

these agents. Um and there's a lot of

tech that uh detecting unused code

getting rid of that um you know we we

have one client who's using our SDK to

basically scan their data.logs every

time there's a new error pattern go into

the codebase and uh add error handling

fix whatever problem is uh is cropping

up. Um, so lots of things that you know

are a little too big for a single agent

to just one shot. Um, but are super

automatable are good tasks to handle

with an agent as long as you're

thoughtful about orchestrating them.

A bit about why these aren't onestopable

tasks. Uh, some of them are

technological problems, some of them are

more like human psychological problems.

On the technology side, you have a

limited amount of context uh that you

can give to the agent. So extremely long

running tasks are tasks that span like a

very large code base. Usually you don't

really have enough there. You're going

to have to uh compact that context

window to the point the agent might get

lost. Uh we've all seen the laziness

problem. Uh I've tried to launch out

some of these types of tasks. And the

agent will say, "Okay, I migrated three

of your 100 services. I need to hire a

team of six people to do the rest." Um

uh the agents often lack domain

knowledge within your codebase, right?

They don't have the same intuition that

you do for the problem.

Uh and errors compound when you go on

these really long trajectories with an

agent. Uh a tiny error in the beginning

is going to uh you know compound over

time. The agent is going to basically

repeat that error over and over and over

again for every single step that it

takes in its task. Uh and then on the

human side uh you know we do have this

intuition for the problem we can't

convey. You know say you want to break

your model into microservices. You

probably have a mental model of how

that's going to work. Uh if you just

tell the agent break the model with

microservices it's just going to take a

shot in the dark. based on patterns seen

in the past without any real

understanding of your codebase.

Uh we have some difficulty decomposing

apps for agents and understanding like

what agent can actually get done uh in

one shot. Um uh we also like you you uh

do need this intermediate review

intermediate checkin from the human as

the agent's doing its work. We'll talk a

little bit about what that loop looks

like later. Uh but it's again not

something you can just like tell an

agent to do and expect the final result

to come in. have to kind of approve

things as the agent goes along. Uh and

then not having a true definition of

them. I think uh if you don't really

know what finish looks like for this

project, it's hard to tell the agent.

Uh on these types of orchestration

paths, want to make it super clear that

we don't expect every developer to be

doing agent orchestration. Um, we think

most developers are going to use a

single agent locally uh for you know

sort of ad hoc tasks that are common for

engineers building new features uh

fixing a bug things like that. I think

running quad code locally uh in a

familiar environment alongside an IDE is

probably going to be a common workflow

at least for the next couple years. Uh

what we're seeing is that a small

percentage of engineers who are early

adopters of agents who are really

excited about agents are finding ways to

orchestrate agents to t tackle like huge

mountains of tech debt at scale and get

a much bigger lift in productivity for

that smaller select set of tasks. Right?

You're not going to see 3,000% lifted

productivity for all software

engineering. Probably going to get more

of that, you know, 20% lift that

everybody's been reporting. uh but for

some select tasks like CDE remediation

or codebased modernization you can get a

massive massive lift you can do you know

ending your years of work in a in a

couple weeks

I want to talk a little bit about what

these workflows look like in practice so

this loop probably looks pretty familiar

if you're used to working with local

agents um this is very typical loop that

looks a lot like the inner loop of

development for you know nonI coding as

well but basically you know you give the

agents a prompt

uh it does some work in the background.

Maybe you babysit it and watch, you

know, everything it's doing and hit the

Y key every time it wants to run a

command. Uh then the agent finishes, you

look at the output. Uh you see the tests

are passing. You see if this actually

satisfies uh what you asked for and then

maybe you prompt the agent again to get

it to get a little closer to the answer.

Or maybe you're satisfied with the

result. You uh you know, you commit the

results and and push.

For bigger orchestrated tasks, this

becomes a little bit more complicated.

Uh basically what you need to do is uh

you or maybe handinhand with cloud you

want to decompose your task into a

series of tasks that can be executed

individually by agents. Uh then you'll

send off an agent for each one of those

individual tasks and you'll do one of

those one of those agents for each of

the individual tasks. And then finally

at the end uh you maybe with the help of

an agent are going to need to pull in

all the output together from all those

individual agents into a single change

uh and merge that into your codebase.

Very importantly there's still a lot of

human in the loop here. Um you need to

review not just the final output of the

collated result but uh the intermediate

outputs for each agent. Um I like to

tell folks the goal is not to automate

this process 100%. It's something like

90% automation. Uh that's still, you

know, an order of magnitude productivity

lift. Um I think this is this is really

tricky to get right. This is where a lot

of like thought comes into the process

of like how am I going to break the tax

down so that I can verify individual

step uh and so that uh I can actually uh

automate this whole process without just

ending up with a high coded mess.

Uh this is a typical git workflow that I

like to use for tasks like this. Uh

typically we'll start a new branch on

our repository. Uh we might add some

high level context to that branch using

like an agent or an open hand the

concept of a micro agent. Uh but I just

a markdown explaining you know here's

what we're doing here. Uh just so the

agent knows okay we're migrating from

Redux is us andor we're going to migrate

these Spark 2 jobs to Spark 3. uh you

might want to put some kind of

scaffolding in place. Uh I'll talk a

little bit more about examples of of uh

scaffolding later. Uh you're going to

create a bunch of agents based on that

on that first branch. Uh the idea is

that they're going to be submitting

their work into that branch and it's

basically going to accumulate our work

as we go along and then eventually once

we get to the end we can rip out our

scaffolding and merge that branch into

main. Uh now for uh if you're you're

kind of getting started with this I

would suggest limiting yourself to about

three to five concurrent agents. Uh I

find more than that your brain starts to

break. Uh but for folks that have really

adopted orchestration at scale uh we see

them running hundreds even thousands of

agents concurrently. Usually a human is

not uh in the loop for you know one

human is not on the hook to review every

single one but maybe those agents are

sending out pull requests to individual

teams things like that. Um, so you can

scale up very aggressively once you

start to get a feel for how all this

works and you feel like you have a very

good way of getting that human input

into the loop.

I'm going to kick it off to uh my

coworker Calvin here. He's going to talk

about uh a very very large scale

migration uh basically u eliminating

code smells from the open hands database

that he did using our refactor SDK up

here.

Open

hands excels at solving open tasks. Give

it a focused problem something like fix

my failing CI add and debug this end

point and it delivers. But like all

agents it can stumble when the scope

grows too large. Let's say I want to

refactor an entire code base. Maybe

enforce certifiing update with your

dependency or even migrate from one

framework to another.

These are not tasks. They're sprawling

interconnected changes that can touch

hundreds of files.

To battle problems at this scale, we're

using the open hands agent SDK to build

tools designed to specifically

orchestrate collaboration between humans

and multiple agents.

As an example, let's work to eliminate

code from the open answer.

Here's the repository structure. Just

the core agent definition has about 380

files uh spanning 60,000 lines of code.

Says a lot about the volume of the code

but not much about the structure. So

let's use our new tools to visualize the

dependency graph of this chunk of the

repository.

Here each node represents a file. The

edges show dependencies who imports who.

And as we keep zooming out it becomes

clear this tangled web is why

refactoring at scale is hard. To make

this manageable, we need to break the

scrap up into humanized chunks. Think PR

size batches that an agent can handle a

human can understand.

There are many ways to bash based on

what's important to you. Graph theoretic

algorithms give strong guarantees about

the structure of edges in between

induced batches, but for our purposes,

we can simply use the existing directory

structure to make sure that semantically

related files appear inside the same

batch. Navigating back to the dependency

graph, we can see that the codes of the

nodes are no longer randomly

distributed. Instead, they correspond to

the batch that each of those associated

files exist. Zooming out and zooming

back in, we easily find a cluster of

adjacent notes that are all the same

color, which indicates that an agent is

going to access all of those files

simultaneously.

Of course, this graph is still large and

incredibly tangled. To construct a

simpler view, we'll build a new graph

where nodes are batches and the edges

between those nodes are dependencies

that are inherited from the files within

each of those patches. This view is much

simpler. We can see the entire structure

on our screen at the same time.

But this is something we have with using

a graph. We can identify batches that

have no redies and expect the files that

go. Dispatch, for example, add 16. Looks

like it's in the file. It's probably

empty. Let's check.

Now, this is a tool intended for human

AI collaboration. So, once we know that

this file is empty, we might determine

that it's better to move it elsewhere.

Or maybe we're okay keeping it inside

this batch. And all that we want to do

is add a note to ourselves or reach so

we know the contents.

Of course, when refactoring code, it's

important to consider the complexity of

what it is you're moving. This batch is

trivial. Let's find one that's a little

bit more complex. Here's a batch that

has four files. They all do and the

complexity measures reflect this. These

are useful to indicate to a human that

we should be more careful when this for

example the first examples.

You need to identify what's wrong in the

first place. Enter the verifier.

There are several different ways of

defining the verifier based on what you

care about. You consider it to be

programmatic. So it calls a match

command. This is useful if your

verification is checking unit tests or

running a lender or a text.

Instead though, because I'm interested

in code smells, I'm going to be using a

language model that's going to be

looking at the code and trying to

identify any problematic patterns based

on a set of rules that I provided.

Now, let's go back to our first batch

and actually put this verifier to use.

Remember, this batch is trivial and

fortunately the verifier recognizes it

as such. It comes back with a nice

little report indicating which person

identified and didn't. And status of

this batch is turned to completed green.

Good.

And this change in status is also

reflected in the batch graph. Navigating

back and toggling the color display, we

can see that we have exactly one node

out of many completed and the rest are

still yet to be handled. But this

already gives us a really good sense of

the work that we've done and how it fits

into the bigger picture.

So now our strategy for ensuring that

there are no code smells in the highly

of our repository is straightforward. We

just have to ensure that every single

node on this batch graph turns green. So

let's go back to our batches and

continue verifying till we run across a

failure.

We'll keep going in dependency, making

sure that we pick nodes that don't have

any dependencies on other batches that

we have yet to analyze. This next batch

is about as simple as the first, but

because the init file is a little bit

more complex. The report that gets

generated is a little bit more verbose.

Continuing down the list, we come across

the bash we identified earlier with some

chunky files of relatively high code

complexity. And this batch happens to

give us our first tree later. Notice

that the status turns red instead of

green. Now this batch has more files

than what we've seen in the past. So the

verification report is proportionally

longer. Looking through see that it is

listing file by file. The code that is

identified in which

I see one file is particularly egregious

with its violations. We'll have to come

back to that.

And if we zoom all the way back out to

the bash graph and look at the status

indicators, we'll see the two green

nodes representing the batches we've

already successfully verified. We'll

also see the red representing the batch

that we just saw that verification. Now,

our student goal is to turn this entire

graph green. This red node presents a

little bit of an issue. To convert this

red node into a green node, we need to

address the problems that the verifier

found using the next step of the

pipeline, the fixer.

Just like the verifier, the fixer can be

defined in a number of different ways.

The programmatic fixer can run a batch

command or you can feed the entire batch

into a language model and hope it

addresses the issues in a single step.

But by far the most powerful fixer that

we have uses the open agent SDK to make

clean copy of the code instead of an

agent that has access to all sorts of

tools to run tests, examine the code,

look at documentation on the do whatever

it needs to to address these issues. So

let's go back to the scaling dash and

run the fixer and see what happens.

Now this part of the demo is sped up

considerably, but because we're

exploring these patches in dependency

order, while we're waiting, we can

continue to go down the list, running

our verifiers, and spinning up new

instances of the open agent using the

SDK until we come across a node that's

blocked because one of its extreme

dependencies is still complete.

When the fixer is done, the status of

the batch is set. We'll need to rerun

verification in the future to make sure

the associated returns again.

Looking at the report that the fixer is

returned, there's not much information,

just the title of the DR. We've set this

up so that every fixer produces a nice

tidy for request ready for human

approval. Just because the refactor is

automated doesn't mean it needs to be

viewed.

And here's the generated. and the agent

does an excellent job of summarizing the

code smells that identified the changes

made to address those as well as any

changes that they have to make. It's

also less helpful for the reviewer and

some notes for anybody working on this

part of the code in future.

And when we look at the content of this,

we see it's very risky. All the changes

are tightly focused on addressing the

code snails that we provided earlier.

And we've only modified a couple hundred

lines of code, the bulk of which is

simply refactoring messed block into its

own function call.

Not all the scope to be this small, but

our batching strategy and narrow

instructions ensure that the scope of

the changes are well considered. This

helps to improve performance, but it

also will easily

from here. The full process for removing

code smells from the entirety of code

becomes clear. Use the verify to

identify problems. Use the fixer to spin

up the address those problems. Review

and merge those PRs. Unblock new fixes

and repeat until that entire screen.

We've already used this tool to make

some pretty significant changes to the

code including typing and improving

test. And we could not have done it

without the open HSDK power everything

under the hood. All

right. So, that's the uh open hands

refactor SDK powered by our open hands

agent SDK. Uh we're going to walk

through a little bit later on the

workshop how to build something a little

simpler but very similar where we get

parallel agents working together to fix

tasks that were discovered by initial

agent.

Uh I want to talk a little bit about

strategy for both decomposing tasks and

sharing context between these agents.

These are both really big important

parts of agent orchestration. Uh so

effective task decomposition

uh you're really looking to uh break

down your very big problem into tasks

that a single agent can solve, a single

agent can one shot. Um something that

can fit in a single commit, single pull

request. Um super super important

because you don't want to be, you know,

constantly iterating with each of the

sub agents. You want each one, you want

a pretty good guarantee that each one is

just going to one-shot the thing. you'll

be able to rubber stamp it and get

merged into your ongoing branch.

Uh you want to look for things that can

be parallelized. This is going to be a

huge way to increase the uh the speed of

the task. Um you know, if you're just

executing a bunch of different agents

serially, you might as well just have a

single agent moving through the task

serially. U the more you can

parallelize, the more you get many

agents working at once, the faster

you're going to able to move through the

task uh and iterate. Um, you want things

that you can verify as correct very

easily and quickly. Ideally, you'll have

something where you can just like look

at the CI/CD status and have good

confidence that if everything's green,

you're good. Uh, maybe you'll need to

click through the application itself,

something like that, run a command

yourself to verify that things look good

to you. Uh, but you want to be able to

very quickly understand whether an agent

has done the work you asked it to or

not. U, and you want to have clear

dependencies and order in between tasks.

Uh you notice these these uh criteria

are pretty similar to how you might

break down work for an engineering team,

right? You need to make sure that you

have tasks that are maybe separable,

tasks that like different people on your

team can execute in parallel and then

colle the results together. You want to

know uh once I get task A done, then

that unlocks tasks B, C, and D and then

once those are done, we can do E. Um so

very similar to breaking down work for a

team of engineers.

Uh there are a few different strategies

for breaking down a very large refactor

like the one we saw challenges do. Uh

the simplest like most one is to just go

piece by piece. You know you might

iterate through every file in the

repository, every directory, maybe every

function or class. Um you know this this

uh is a fairly straightforward way to do

things. It works well uh if those um

dependencies are can be kind of executed

um you know without depending on one

another too much. Um so good examples

might be like adding type annotations

throughout your pipeline codebase. Um

uh and then you know at the very end

once you've migrated every single file

say you can collect all those results

into a single PR.

A slightly more sophisticated thing

would be to create a dependency tree. Um

and the idea here is to add some

ordering to that piece by piece approach

where you know you start as we saw

Calvin do you start with like the leaf

nodes in your dependency graph right you

start with maybe your utility files get

those migrated over um and then anything

that depends on those you know it's

going to have those those initial fixes

in place and the dependencies can uh can

start working through um you know their

their set of the process. You can

basically back your way up to whatever

the entry point of the application is.

Uh this is often a a better way to

proceed. Um it's more kind of a

principal approach for how you're going

to order through these tasks.

Another example is to create some kind

of scaffolding that allows you to live

in both the like pre-migrated and post

migrated worlds. Um we did this uh for

example when migrating our React state

management system. Uh we basically had

an agent set up uh some scaffolding that

would allow us to to work with both

Redux Redux and Zustan at the same time.

Um pretty ugly, not something you would

actually really want to do. Um but it

allowed us to test the application as

each individual component got migrated

from the old state management system to

the new state management system. Uh and

then we sent off parallel agents for

each of the components. uh I got each

component done and then at the very end

once everything was using zestand we

were able to rip out all of the u all

the scaffolding so there was no more

mention of redux and everything was

working but having that scaffolding in

place allowed us to validate you know as

each agent finished its work for just

that one component we could validate the

application was still working that

component still works uh we didn't have

to do everything all at once we got some

kind of human uh feedback from the

agents

uh next I want to talk a bit about

context sharing uh as you go through a

big large scale project like this uh

you're going to learn things right

you're going to figure out okay what I

my original mental model wasn't actually

complete I didn't actually uh you know

understand the problem correctly um your

agents might uh run into that you know

you might have a fleet of agents you got

10 agents running they're all hitting

the exact same problem you kind of want

to share the solution of that problem so

they're not all getting stuck right

there's a bunch of different strategies

for doing this context sharing between

agents

Uh, one strategy that I think the most

naive thing you can do is share

everything. Basically, every agent sees

every other agent's context. Uh, this

is, uh, not great. Uh, it's basically

the same thing as just having a single

agent working iteratively through the

task. Uh, you're going to leave your

context window really quickly if you do

something like this. Uh, so this is this

is not going to help.

Uh, a a better value approach would be

to have the human being just sort of

manually enter information into the

agents. Uh if you have a chat message, a

chat window with each agent, you can

just paste in like hey use library 1.2.3

instead of 1.2.2.

Um the human can also modify like an

agent MD or micro agent to pass messages

to these agents. Uh but this does

involve manual human effort. Um it

involves a lot more like babysitting of

the agents. So it's it's not super

scalable.

Uh you can also have the agents

basically share context with each other

through a file like agent MD. Uh you can

allow the agents to actually modify this

file themselves. Uh maybe they send a

pull request into the file as they learn

new things. Uh downside here is that

sometimes agents will try and learn

unimportant things. Uh they can get kind

of aggressive about pushing information

to this file. Uh so doing some kind of

human review seems to help.

And then last uh this is probably the

most like leading edge idea here. Um,

but you can basically give each change

in a tool that allows it to send

messages to other agents. Uh, it could

be like a broadcast message that goes

out to all the other agents. Uh, or it

could be, uh, you know, pointto-point

conversation. Uh, this is super, uh, fun

to experiment with. We're doing a lot

uh, to experiment with this now, uh,

with our SDK. Um, but it's, uh, it's

tricky to get right. It's, uh, you you

once you get agents talking to each

other, you're like increasing the, uh,

level of non-determinism in the system.

Uh, things can get a little bland. Uh I

have an example here on the right of uh

this is from a doctor's report where

they had two agents just talk to each

other. They just entered into a loop of

wishing each other zen perfection.

Um

cool. Uh now I want to work through an

exercise. Uh I would love it if you all

want to follow along. Um you can access

this presentation for uh copy pasting

purposes at uh dev.shophands-workshop.

Um, we'll work through some coding

exercises with the open hands SDK

specifically to uh do CD remediation at

scale. Um, we're going to write a script

that will take in a GitHub repository,

scan it for open source vulnerabilities

for CDEs. Um, uh, and then set up a

parallel agent for every single

vulnerability we find to solve that and

open up a poll request.

So, dub.shophandworkshop.

uh let me know anybody can access it.

>> It's gonna be the slideshow.

>> So, so it should be the slideshow if you

want to. There will be um

uh copy pasteable prompts and uh links

and stuff like that around slide 29.

>> Got it.

>> We'll get there.

Uh so in terms of how this process is

going to work,

uh basically we're going to start with

one agent that runs a CVE scan on this

repository. It's going to stand for

vulnerabilities. Uh what's nice about

using an agent for this is it can look

at the um uh the repository and decide

how am I going to scan for

vulnerabilities, right? Am I going to

use trivia to scan a Docker image? Uh am

I going to run npm audit on a

package.json?

uh so it can it can basically detect the

programming language to figure out how

am I going to stand for CDES here. Uh

then once we have our list of

vulnerabilities, we're going to run a

separate agent for each individual

vulnerability. Uh each of these agents

is going to research whether or not it's

solvable. Uh it's going to update the

relevant dependency, fix any breaking

API changes throughout the codebase, and

then open up a poll request. Uh what's

nice about this is that we can merge

those individual PRs once they're ready.

You

>> show the link again. Yeah.

Uh what's nice about running the solving

in parallel is that you know we get we

get a bunch of different PRs. Uh so we

can merge them as they're ready. If one

agent gets stuck, one of the

vulnerabilities isn't solvable. All the

other ones are still going to work. Uh

maybe we get to 90% or 95% solved. Uh we

don't have to get to 100% in order to

have any value here. Uh just some quick

pseudo code of what this is going to

look like.

Uh so this is an example using the

openhance SDK of how to create an agent.

You can see we create a large language

model. Um we then pass that large

language model to an agent object along

with some tools. Uh a terminal, a file

owner, a pass tracker for planning. Uh

we give it a workspace and then we just

tell it we want to do run. Uh this is a

pretty like naive hello world example.

We'll see how it gets a little bit more

complicated as we progress through this

particular task.

Uh but then once that first agent is

done, we're going to iterate through all

of the vulnerabilities to get back out.

Um and then for each one, we'll send off

a new agent uh asking it to solve that

particular CDE.

All right. So, uh to get started here,

uh it would say create a new GitHub

repository. Uh we start save our work

there. Uh you're also going to need both

a GitHub token and an LLM token.

Uh, I would, uh, if you sign up for for

OpenHands app.allands.dev, you can get a

$10 free credit u LLM credits there. Um,

if you're already an existing user, let

me know and I can I can bump up your

your existing credits for the purpose of

this exercise.

Um, then we're going to start uh an

agent server. Uh, this is a um uh

basically like a Docker container that's

going to house all the work that our

agents are doing. Uh this is a great way

again to run agents securely and more

scalably. So instead of running the

agents on our local machine to solve all

these CVEes uh we're going to run them

inside of a container. Hypothetically if

we were doing thousands of CVEs we could

run this in like a Kubernetes cluster so

that you know we have as many

workstations as we want for our agents

but for the purposes of this exercise

we'll just run one one Docker container

as a home for our agents. Um then we can

create uh an agent of be or an open

enhance micro agent to uh you know start

working through this task. I'm going to

be using the openhand CLI as we go here.

Um you're welcome to check out the open

hand cli. You can also use cursor or pod

code or whatever you're used to using uh

as we uh kind of bode our way through a

CD remediation process with open hands.

Uh I'm going to give it a couple

minutes. I'm going to walk through

creating my GitHub repo, getting my

GitHub token, etc. Um

uh if you all have any trouble feel free

to raise your hand and come around and

uh help you know getting it all out etc.

You said app.allhands.dev

app.

>> Yeah.

So, I've got my new GitHub repo here.

Uh, so I'm gonna add a quick open hands

micro agent here.

Perfect.

I'm just going to tell

a

uh process for remediating

with agents.

relevant talks

for the open hand SDK are at

open hands SDK.

So some data opens a little bit of

context

similar to agent. Um we now have officed

uh to get a token. I'm not actually

going to do it here so that was my token

but you can go to GitHub settings your

profile

then developer settings

personal access tokens.

I like to do classic tokens.

Uh classic token. Give it a name and

then uh the repo scope is really what

you'll need. Uh that way we can open up

pull requests uh to solve to the CS

involved.

>> We did a classic token not the new

thing.

I I haven't gotten a link

used to you're welcome to do. I guess

you could create a new repository.

>> I haven't got to them either. So,

>> I'm not with you.

>> Back in the old days.

>> So, what permissions do we need to

>> uh just the repo permission?

Also, it's going to show you sign up for

app.alland.dev.

Um,

you go to

piece under your profile here, you can

get your open API key, your L key here.

I won't show it, but

this will allow you to use our proxy

step.

Last, I'm gonna

start up some agent server here. You'll

probably want to copy paste this out of

the presentation.

Got my repo close

dinner.

Maybe

that's back here. If you do want to work

with the open hand cli

tool install open hands

I'm going to start up the open hands

CLI.

Again, you can use cloud code, cursor,

whatever else if you want. Uh you folks

need a little more time with the setup.

key get token set up.

Sorry, check.

Uh so I'm gonna start with

this first prompt. Uh basically what

we're going to do is we're going to

point our agent uh at the open hands SDK

point it at the documentation

uh and just ask it to basically check

that our LLM API key is working that it

can actually do an LLM deletion. This

will be like a very basic hello world.

just kind of get started here. Um, I'm

going to tell it uh I'm using I'm using

the open hands uh key that I generated

at app.allands.dub.

Um, so I'm telling it to use this open

handbon 4 model. Uh, you can replace

this with enthropic. If you want to use

just like a regular anthropic API key.

Uh, you may need to set this model a bit

differently depending on if you're using

open AI using light.

You can look at the light all docs to

figure out if you have an open API key

or an open AI key. Uh you can look at

the light all docs to figure out which

model plug for the string. But I'm just

going to copy paste this as is.

Sorry, what's the step for uh agents.md

or the one for open hands?

>> So I would say just create a u a file

either the agents.md if you're working

with a a tool that's compatible with

that or

uh for open hands we have it's called a

micro aent I can get to it. Uh so

openhands.openhandsmicroagent

by convention repo.mmd is the

description of the repository you're in.

Um and I just gave it a couple links to

the SDK documentation uh and the

repository for the SDK so it has access

to you know basically the the API docs

there.

This is kind of an optional step. Make

things a little easier though.

is doing. All right, it thinks it's got

something good. So, let's see what's

going on.

Python CV solver

need

environment variables.

I'm using the to set my brightness here.

Make sure I don't check those in.

One more time.

Got a small error. Looks like the agent

didn't quite get the API doc right.

Let's uh paste the error back. See what

happens.

Let's try again.

Of

course, never never go.

Not there.

She's working.

version.

>> Let's use club.

UV

tool install that breaks.

>> Yeah.

>> You know what version of UV you're on?

>> I'm on 096.9.6.

>> What error are you getting?

>> I don't know why.

No executables are provided by package

open hands. Removing tool error failed

to install entry points.

>> I'm newish to the Python world. So I

assumed I was doing silly.

>> You could try updating on 111 which is

what I'm on. But okay. Yeah, I'll try.

>> Another question.

>> Yeah.

>> Um, so I was able to I see you running

through the CLI. I was able to run this

on the like all all.dev.

>> Yeah. Cool. and it submitted a PR and

created it. Looks good.

>> Awesome.

>> Why are you doing it through the CLI?

>> Uh really just for

um

normally I actually prefer to work

through the web UI here. Um

I think uh being able to like run and

show that script is working locally. Uh

it's like a little bit better of a hand

out. I actually like to work through the

web UI normally and then have the agent

push and I pull locally if I really want

to work locally, but figured that was

just extra extra steps for presenting

purposes.

Yeah, feel free to use the the web or

the tool.

Looks like I

API key here. Come

Jesus.

200.

>> What's that?

>> Should we get 200?

>> Uh yeah, you should get something like

this. Uh like I just got finally uh

where the other one says hello.

Just

section.

Anybody managed to get connection

working?

>> I think so. I've created the file.

>> Nice.

Uh just a quick view of what this looks

like

in the first

basically you can see we create an tell

what model we want to use what I key we

want we want to use

and then just send a quick message to to

the to make sure it's actually working.

Uh all right for the second time I'm

going to move towards prompt two. Uh so

here we're going to actually start to do

some work for the uh so we're going to

tell um you know the agent we're working

with uh we want to use the SDK to create

a new agent uh that's going to take in a

GitHub repository. Uh it's going to

connect to a remote workspace uh running

at localhost 8000. Again that's the the

docker start command from before. If you

haven't already run that now's a good

time to get Docker running. uh Docker

run this agent server. Uh

it's going to uh clone our repository

into that Docker container. Uh we're

going to create an agent that's going to

work inside that Docker container and

we're going to tell that agent to scan

this repository for any

with the open ends CLA. Is there a way

to interrupt and get it to stop?

>> Uh, hit control P or pause. Yeah.

>> And then can I insert my corrections?

>> Yeah. Then you can type me a message or

just type continue.

Yeah,

>> I got the CLI to install, but I had to

add - AI.

>> Seems on PI that there's a D AI version,

but then it says in the docs.

>> I don't I think the AI one is

deprecated, but it is it is a usable

CLI. You want to use that

service that one off our team.

Did you get the dash AI1 to work?

Because as soon as I tried to run it, it

crashed. Oh,

>> oops. It installed. I was so happy.

>> Yeah, it installed and then it it didn't

work.

>> There's a deprecation warning when I go

to version. So, yeah,

>> there is a if you want to download an

executable binary on our release page.

>> Okay,

>> that might be straightforward. You can

also run it in a docker container. Um,

if you CLI docs,

I think there's a UV run as well.

Try UV run.

the version.

The version

that's for the not open AI regular.

>> Okay.

Thank you.

Okay, supposedly have an agent working

here. Let's see. Going to run it with

repo. It should have a few CVs in it.

Let's see if we find any vulnerable

by default. Open hands. We'll uh

we'll visualize the output here. So, we

can see the agent working uh even with

the SDK. pretty similar to how we saw

the uh uh CLI.

Uh you can see it's task list.

It's uh

the repository.

It's uh doesn't have trivia itself. So

it's like trivia. It's basically doing

what we would expect an agent to do. Uh,

we've been a task. We can't get to it.

So, we're running Trivia now.

show a bit about what this what this

generated code looks like. Uh you can

see so we we instantiated our LLM in the

first step.

Now we're actually passing this LLM to

an agent. We're also giving it terminal

tool and file editor tool. Uh we're

creating this remote workspace that's

connecting to our Docker container so

that a can start working in its own

environment. Uh we create what's called

a conversation which is basically one

chunk of context that the is going to

manage as it goes about it its work. Uh

we pass it a task with some clear

instructions for what it's supposed to

do and then send that send that task.

Looks like that initial scanner agent is

almost done.

Looks like that agent ran just fine. Got

these results.

I'll keep uh keep plugging along here.

We've got an agent that's uh scanning

for vulnerabilities.

Uh so the next thing I'm going to ask

this to do is basically we're going to

reach into the environment and get the

vulnerability list out from it. Uh the

idea is we're going to have it save as

the vulnerabilities to a JSON file. Uh

then we can on that workspace object

inside of the docker container we can

run execute command in order to get

those vulnerabilities back out. We also

have some some options for like

manipulating files uh within the

workspace. Uh then for now we're just

going to iterate over the

vulnerabilities.json file, print it down

just so we can see we were able to reach

into this workspace and get some

information back out.

All

right. Supposedly

good to go. See what happens.

Sheep.

Got some vulnerability results.

Agent's finished. Let's see if our

script can get

results back.

person. Jace

Um,

One more time.

>> What is the observation event? So for

every

states uh there's a there's an action

and then an observation. So it might be

run this command and then an observation

comes back with the output of that.

>> Uh it's more than a it's it's the

basically the entire trajectory the

agent takes of events and then there's

two kinds of events actions and

observations. So fans whenever we get

calls with the LM it comes back with an

action to take or basically a tool call

uh and then the observation is like a

tool call.

If anyone stuck on anything, happy to

come around to free to raise a hand.

Number

three.

>> Nice. Yeah, it looks like it's printing

the CV list. Yeah, that looks good.

create like

a specific sub agent for each script we

are running. Why you overating

the same file again and again?

>> So the the process we're going through

here with the five the five prompts this

is really

uh to demonstrate what it would feel

like to actually like build with our

SDK, right? uh this is not the way that

I would this is the way I I would maybe

like work if I was actively working on a

problem you know I could have just given

you this this whole fully packaged code

base pre-built right yeah

>> that had all this built but uh

is that what you're asking like why are

we why are we pasting these prompts in

one by one

>> eventually we get a very large script

right we should break it several

separate files or sections

>> yeah yeah yeah no I think there's

there's definitely better ways to

organize this code than to have one

single script just uh easier for demo

purposes. Yes, I do have a I do have a

demo repo um I think it's openhand CVE

demo that uses special classes. There's

a single, you know, CVE agent subassm

that's a little bit more than just this

one script.

We're still pressing JSON.

Seems

Yes.

Focus.

Enough of us.

That's beautiful question.

Our

SC

the open source models.

We're actually

I don't know what I'll be doing.

All right.

The thing is

I mean

>> Yeah.

Heat.