HomeVideos

Build the thing that builds the thing | BRK245

Now Playing

Build the thing that builds the thing | BRK245

Transcript

1127 segments

0:06

d

0:06

afternoon. All right, let's see

0:15

if we get this working here.

0:17

Yeah. So don't expect a regular

0:21

presentation here. My talk is

0:24

about build the thing that

0:26

builds the thing. I want to

0:27

inspire you a little bit. How

0:31

to approach building software

0:33

in the future. You know, it's

0:35

been not even a year since we

0:39

have coding agents. And it's

0:41

kind of like half a year since

0:44

they got good. And there's no

0:48

stopping done. So a lot of the

0:51

ways we build software in the

0:53

past, I feel are obsolete. And

0:59

we we haven't really figured it

1:00

out yet. I'm curious here, is

1:05

anyone here who maintains open

1:07

source? Yes. How are you

1:12

handling the pain? So. I kind

1:18

of stumbled into into open claw

1:22

people. I saw this all these

1:26

articles where people say, oh,

1:28

Peter made 43 projects that

1:31

failed. And then finally, like

1:33

one worked. And I had to laugh

1:38

at this because none of these

1:41

projects failed. All of these

1:43

projects are things I built for

1:45

myself to work faster to like.

1:53

Mostly actually to like help my

1:55

agents to work faster. Like

1:57

anytime you build software now

2:00

your job is no longer to, to

2:01

figure out how you build

2:03

software faster. Really your

2:05

job should be, what can I do to

2:07

help my agent to build software

2:10

faster and even more? So it's

2:13

all about closing the loop. You

2:16

know? Last year I built, I

2:21

spent a few months building a

2:22

SaaS. And the agent would write

2:28

code and then it would like,

2:30

tell me it's done. And then I

2:32

would look at the browser, I

2:33

would at the agent. It would

2:35

like go in a loop, it would fix

2:37

it. And then eventually we

2:38

would be there. And that was

2:41

until I figured out like, hey,

2:43

no, the agent just can't make a

2:45

screenshot, you know, just like

2:46

humans would like anything that

2:50

where you feel. This is

2:54

something I do repeatedly, you

2:56

should start thinking, how can

2:58

I automate this? How can I, how

3:00

can I help my agent to close

3:01

the loop? So when I guide you a

3:04

little bit through all my

3:05

failed projects. And give you a

3:09

little bit of the, the, the

3:11

thought process, I put on this

3:14

website a little bit. It's pure

3:16

slop, but I think it's kind of

3:18

nice for that. All the little

3:20

tools that we built to build

3:24

Opencore, and I don't think

3:26

it's even complete. So my, my

3:32

first problem was. That open

3:38

cloud client exploded. And then

3:40

the issues and PRS started

3:42

increasing. And I didn't know

3:43

that this number actually, once

3:46

it reaches five K, it stays at

3:47

five K and kind of masks it. At

3:49

some point we had like more

3:51

than 10,000 issues because

3:52

people don't they don't look,

3:54

they just like, oh, I mean,

3:57

people, the agents don't look,

3:59

you know, it's like, let's not

4:00

pretend any of that is written

4:02

by humans anymore. And I know a

4:04

lot of open source people get

4:05

really mad at all the slop

4:07

they're getting, but I feel.

4:12

You can't fight the world. You

4:13

have to figure out a way how to

4:15

deal with that. So I see

4:17

differently. Yes, there's a lot

4:18

of slop, but the real problem

4:23

is that it's still called pull

4:24

requests. I call it prompt

4:27

request because what you really

4:29

want is anytime someone opens

4:30

an issue, a PR, it's just a

4:33

signal that somebody is not

4:34

happy with what you built and

4:37

is asking for a change.

4:38

Actually, like these two things

4:40

are like interchangeable. Like

4:42

mostly doesn't matter if it's

4:45

an issue or PR. Sometimes an

4:46

issue is actually nicer because

4:47

I can work fresh if it's a pull

4:49

request. And like somebody used

4:51

Claude and like I, it's

4:53

probably takes me more time to

4:54

like make my agent read the

4:56

code and then like at the code

4:57

and then rewrite the code. But

5:01

the first thing, the first

5:02

thing I had to had to solve was

5:04

how do I get this number down?

5:06

And I tried, obviously you try

5:08

tools, right? I tried code

5:11

Revit, I tried, I tried a bunch

5:13

of tools and they were nice,

5:15

but they ultimately would just

5:17

give me a review. I wanted more

5:21

I reviews are nice, but I kind

5:22

of want something that

5:23

automatically groups and closes

5:26

the issues and I couldn't find

5:27

anything good. And in this day

5:29

and age you don't actually need

5:31

to wait anymore to like find

5:32

stuff. You can just build your

5:35

own stuff. So I built Cloud

5:37

Sweeper again, totally sloppy

5:39

website because the website

5:40

doesn't matter. But it's

5:47

actually really simple to build.

5:48

Like, I don't know what all

5:50

these tools were their motives

5:52

because ultimately, anytime

5:54

someone opens an issue or a

5:56

pull request, I just start a

5:59

codex in the cloud. We are a

6:01

GitHub action and have a nice

6:06

prompt that explains the codex.

6:08

Like what is, what's the goal

6:11

of the project? What's our

6:13

vision? I do this now for any

6:15

of my open source projects. I

6:17

have a vision.md, and part of

6:20

my standard prompt is that my

6:21

agent reads that file. And in

6:22

that file is kind of like what

6:24

stuff that yes, we totally want

6:27

what stuff that definitely we

6:28

don't want. And based on those

6:30

rules. Klaus Reaper will run.

6:35

And we'll just either comment

6:37

on the issue or close the issue.

6:40

And through that, I was able to

6:41

like close around 15,000 issues.

6:44

I mean, yes, I also operate

6:46

under the assumption that have

6:47

infinite tokens. You could

6:49

definitely do this a little bit

6:50

more efficient, but we didn't

6:52

stop there. Once we had a

6:55

system that could close issues.

6:58

Of course, we also wanted to

6:59

review issues. And it's not a

7:06

static process right at our

7:08

scale. Any time a fixed lens on

7:15

main, there's a good chance

7:17

that I solved an issue that

7:18

I've never read because there's

7:19

like many thousand issues open.

7:21

So what we do is actually we

7:23

rerun Klaus Reaper on every

7:26

issue and pull request at least

7:28

once a week, depending on how

7:29

many tokens you want to spend.

7:31

You could do it every day. So

7:34

what regularly happens is, is

7:35

that I, I have an integration

7:39

on discord. And then and then

7:41

the report will tell me, hey,

7:42

close this issue from three

7:43

months ago that I've never seen

7:46

just because, because Codex

7:47

would run on that issue. And

7:49

then a person that wrote an

7:50

issue will get a, will get a

7:52

response. Hey, we fixed this on

7:54

main. And again, it's something

7:55

I don't have to deal with it. I

7:57

automated it, it solved the

7:59

problem. I also built a nice

8:03

dashboard. So see, right now

8:06

there's only 1818 Codex working

8:11

right now. And this is how it

8:15

looks. It's just one long

8:18

comment that explains what it

8:19

does. And it gives people array

8:23

of issues that that they need

8:25

to solve with typical stuff.

8:29

The next thing I, I had to

8:32

solve now, now the project

8:33

gained momentum. I added more

8:36

intainers and some people

8:39

turn out to be really good.

8:40

Some people, they kind of

8:42

showed up for a few days and

8:45

then disappeared. And I was

8:47

very liberal in the beginning

8:48

of adding people and I lost a

8:51

little bit overview. So I

8:54

wanted a dashboard. So I just

8:55

built a dashboard. So now we

8:58

have reports of open cloud.

8:59

This is not public for everyone

9:00

because it because you want a

9:03

little bit more data. Actually,

9:04

I'm not just looking at GitHub

9:06

issues. I also look at all our

9:08

discord stuff. So to me, being

9:12

a maintainer doesn't just mean

9:13

you push code. Domain also

9:15

means that you participate in

9:17

discussions, which we use

9:18

discord for that. So so this

9:20

system integrates all data from

9:22

discord. Now that brought me to

9:26

the next problem. How do I get

9:27

the data from discord? Well,

9:32

let's just build this crawl.

9:35

Like in this day and age, any

9:37

problem is just a prompt away.

9:39

Like building this thing is

9:40

never the hard thing.

9:41

Maintaining this thing is

9:42

actually the hard thing. But

9:45

again, even even on even now

9:48

when I, when I maintain stuff,

9:51

I don't think about the project

9:54

level anymore. I. It looks like

9:59

this. I basically, I go one

10:02

level beyond. I go into my

10:03

projects folder and ask Codex,

10:05

hey, let's triage. What are the

10:07

things that I should work on?

10:08

And it just finds a list of all

10:10

the, all my projects and open

10:13

issues. And then I basically

10:14

say, yes, do all of them

10:16

because I already have. My

10:20

slope is better than your slope.

10:23

No, I don't do all of them

10:25

because there's still sometimes

10:28

people report weird stuff and I

10:30

don't fully trust my agents. So

10:32

I want to I want to read that.

10:35

Right? So again, I'm a lazy

10:38

person. I could click on each

10:39

of those URLs, but it's kind of

10:42

like slow. So I built something

10:44

where you just select a text

10:47

and I built myself an issue

10:49

browser because again, it was

10:51

like, like there was like just

10:53

a few clicks too much. It

10:54

annoyed me a little bit too

10:56

much. And the critical, the

10:57

critical thing that you are is

10:59

like, your time is limited. So

11:00

you need to build the tools to

11:02

build the tools. So again, I

11:03

built this little slot, right

11:05

issue browser where I can copy

11:09

by now any format because it

11:11

turns out there's Clankers

11:13

given an almost unlimited

11:15

amount of versions, how to

11:16

format things. So I anytime

11:19

something didn't work, I just

11:21

fixed it to make it work. And

11:22

now I basically parses all the

11:24

formats that it ever seen. And

11:25

I can very quickly click

11:27

through this list. In many

11:31

cases, the agents are already

11:33

so good because I have this

11:34

vision in D file that the

11:35

selection is mostly right. And

11:37

I usually say, do those eight

11:39

don't do those two because I

11:40

know this needs actual time.

11:43

You know, anytime you work with

11:46

agents, you want to optimize

11:49

what they can do autonomously.

11:50

And it doesn't matter how long

11:52

it takes. You want to, you want

11:54

to really manage the focus time

11:57

where you want to be in the

11:59

loop for the things that are

12:01

important and all the things.

12:03

What is that even gmail draft

12:05

create? Yeah, that's probably a

12:07

nice feature. Oh yeah. Let's

12:08

look at this. This is a nice

12:10

feature on Gorg. Another thing

12:13

that I built because I got

12:15

annoyed that Google doesn't

12:16

have a CLI, and I want my

12:18

agents to access all my Google.

12:20

Yes, like Codex has gmail

12:22

integration and now. Calendar

12:26

integration, but only for one

12:28

account against an annoyed me.

12:30

So I just built a CLI that can

12:32

do everything anytime

12:33

something's missing because I,

12:35

I control the whole stack. My

12:37

agent controls the whole stack.

12:38

So it can just fix itself like

12:41

so like you own the whole

12:42

universe. Anything is fixable.

12:44

And this is all these issues

12:46

are super nice because this

12:47

would eventually add a feature.

12:50

Codex can not only build it.

12:55

But also end to end verify it

12:57

because I of course have like a

12:58

demo account on GOG so it can

13:00

end to end test the whole

13:02

feature. And I basically have

13:03

to do nothing other than saying

13:04

yes, and I can ship it and I

13:06

can ship it with confidence

13:08

because it's end to end tested.

13:15

So then I built this role for

13:16

the maintainer reports. Turns

13:18

out this is like this solves a

13:21

much bigger problem because

13:22

when you do open source. And

13:27

you have multiple thousands of

13:28

issues, it's very hard to know

13:29

what's actually the thing that

13:31

I should work on. So now I, I

13:35

hooked up this crawl to Codex

13:39

and it will, it will go through

13:41

all my channels and I can

13:43

basically ask it, what are

13:44

people screaming the most

13:47

correlated to open issues and

13:48

PRS and find me the top five

13:50

things I should work on. It

13:53

should like match it with my

13:54

vision document, pick the stuff

13:56

that you can work autonomously,

13:58

and then basically it can run

14:00

off and work for doesn't matter

14:02

how many hours you can, you can

14:04

parallelize it. And then what

14:11

you always should know when you

14:12

build, even when you highly

14:14

automate with agents, is you

14:15

should have this. The you

14:17

should have the overview of

14:18

your system so you will know

14:19

which issues are like. Probably

14:21

fine, probably fine. Oh, this

14:22

is hairy. I need to look at

14:23

that more specifically. So I

14:27

built this crawl and then of

14:29

course all my maintenance

14:31

wanted this crawl. But again,

14:33

it there's no, there's no

14:34

public way to get all this data.

14:36

So I used a bot multi that was

14:38

in discord to like use him to

14:41

access all the data. But then

14:43

how do you distribute it to, to

14:45

the other maintainers. Well.

14:49

What's the, what's the simplest

14:51

backup system there is GitHub.

14:55

So, so all the, all the data

14:58

from our discord is not just in

15:02

some discord store, just in

15:06

some, some format that

15:08

serializes the database. And

15:11

anyone that uses discord can

15:12

just use this as an alternative

15:14

data server if they don't have

15:16

a bot token, because I didn't

15:17

want to give my bot token to

15:19

all the people, this is good

15:21

enough. Again. Problem solved.

15:24

Now, if you're like me and

15:27

you're maintaining a lot of

15:29

open source projects, what I

15:32

always wanted was a way to have

15:36

this little dashboard of like,

15:38

what actually needs my

15:39

attention? Where are the most

15:41

pull requests? Should I do

15:43

another release? But there

15:44

wasn't really anything good. So

15:46

I just built this bar and now I

15:49

can filter on this.

15:52

Specifically how many days

15:53

since the last release. So

15:55

whenever I do like my little

15:56

fixes and software, if you

15:58

don't constantly garden your

16:02

software, it's dead because

16:04

dependencies, everything needs

16:06

updating. Systems change. So

16:07

now I have a nice dashboard to

16:09

know when should I do another

16:11

release? And I can basically

16:12

filter on stuff that need

16:14

attention. And again, the

16:18

beauty of all these little, all

16:19

these little tools that you

16:21

build to help you build the

16:23

thing, they're not system

16:26

critical. So all these tools

16:29

are. Yes. I actually didn't

16:31

look at any of that code

16:32

because it doesn't matter, like

16:34

for, for stuff that a lot of

16:37

people run open claw, I read

16:40

more code than people would

16:41

think. But for all these little

16:43

helper tools. Vibe anyway, and

16:48

if maybe you use them a lot,

16:49

then you can iterate on them.

16:51

Maybe they don't work out and

16:53

no cost. Yeah, we talked about

16:58

a little browser here. There's

17:00

another thing now that I. I use

17:04

more and more to work on things

17:06

in parallel. I hit another

17:09

problem. There's only so many

17:11

GitHub tokens that you can use.

17:14

And because we have so many

17:15

issues, and I instructed my, my

17:17

agent to look at find related

17:20

issues, I always ran out of

17:23

tokens. And I know that I, I

17:25

don't know if the GitHub people

17:27

are here like they are good

17:28

friends and I have someone

17:29

literally clicking a button

17:31

once an hour to reset Peter's

17:32

rate limits. But the problem is

17:35

these people need sleep. It's

17:36

not clankers. So so when I was

17:39

working weird hours, like I

17:41

wasn't around, I couldn't ping

17:42

her to reset my tokens. Again.

17:48

I find being annoyed is the

17:51

very best way to solve problems.

17:54

So I built octopus and you're

17:57

like, my feelings were like

17:58

transported into this this

18:00

things feelings. But the

18:05

principle is the principle is

18:06

very, very easy. Octopus. Let's

18:13

see if this actually works. But

18:18

again, the principle is very

18:19

easy. You as a as an individual

18:23

have 5000 API tokens per hour.

18:27

But if you install a GitHub app,

18:29

it gives you 15,000 tokens per

18:31

hour. So I just built this

18:33

thing that shims GG and then

18:37

calls this thing on Cloudflare.

18:40

If it's a read only if it's if

18:42

it's a non mutating read only

18:43

action from a public repository,

18:45

it can use the GitHub app and

18:48

then use the token from the

18:49

GitHub app and not my token

18:51

only if it's something that's

18:52

mutating or it doesn't fit that

18:54

that match it will use my token.

18:55

Bam! Again. Problem solved. And.

19:03

All the tools are so good now,

19:06

especially especially

19:07

Cloudflare, I didn't have, I

19:10

just explained the agent what I

19:12

want. And then with the

19:17

combination of really smart

19:19

models, browser use, you don't

19:22

have to do anything anymore.

19:23

Like literally, you could

19:24

configure Cloudflare itself.

19:26

You probably still want to have

19:27

a look at it because you want

19:30

to review if it actually

19:31

doesn't mess up your security

19:32

settings. But the idea is any

19:38

time, any time you feel you

19:40

feel pain, think of go, go, go,

19:43

go out and think about what,

19:45

what can I do to like get rid

19:47

of that pain? And then usually

19:50

it will not only help you, it

19:52

will help your whole team. And

19:53

it's, it's a and it compounds

19:56

all the little tools are built.

19:57

Help me to build faster.

19:59

Another, another problem I had.

20:03

How are we on time? Actually, I,

20:05

I think we're good. Who here

20:12

had this problem? Who's crazy

20:15

enough to run like more than,

20:16

let's say ten coding sessions

20:18

except for two? Yeah. You guys

20:21

need to level up. So. The

20:29

project grew. It kind of

20:31

exploded. The code and the

20:34

features grew. People reported

20:36

bugs, our test case grew and

20:39

the tests were becoming more

20:42

and more pain. You know, that's

20:44

that's the stuff kind of creeps

20:45

in. It takes like, it took like

20:47

10s and it took 30s and took a

20:49

minute. Then it took two

20:51

minutes, then it took five

20:53

minutes. Then I optimized it

20:54

for my Mac studio to work in

20:56

one minute. And everybody

20:57

complained that that like,

20:59

their computers were melting

21:01

because they didn't have like

21:02

40 cores. So we had this shared

21:06

painting again. And around that

21:09

time, I saw that. Oops, I saw

21:15

that. What's it called?

21:18

Blacksmiths. They, they started

21:20

something. Test box concept. So

21:26

the idea is. Your computer

21:32

power is limited. So you want

21:34

to give the agent a little box

21:36

where it can run the tests or

21:38

do whatever it needs to be CPU

21:41

intensive. And what's beautiful

21:44

for a while. But the problem

21:45

was their stuff was very new

21:47

and it was like. It was down

21:50

for at least two hours a day.

21:52

And by then, our team was so

21:54

dependent on it that like,

21:55

nothing would get done anymore

21:57

in those two hours because CI

21:58

would break down and then the

21:59

test would break down. So I was

22:01

like, this is fine. I'll just

22:03

build something that wraps it

22:04

with alternatives. And that's

22:06

how Crab Box was born, because

22:08

you have to keep it with the

22:09

weird crustacean names. Again,

22:13

very simple, a very simple

22:14

concept. You spin up a little

22:16

infernal machine in the cloud,

22:19

and then it syncs your changes

22:22

in there. Your reports are

22:23

already checked out. It runs

22:24

the tests in there, and then

22:26

you pay the all the CPU stuff

22:29

that you would need to run the

22:30

test on your machine. You can

22:32

use really beefy hardware. You

22:34

can massively parallelize.

22:36

Everyone's happy. And then this,

22:40

this tool quickly became like

22:43

universally loved because it

22:46

would it would use our main

22:50

provider. And if that runs down,

22:51

it would just use AWS or Azure.

22:54

And then people started making

22:55

pull requests. And now it

22:56

supports like 20 providers. But

23:01

it didn't stop there because I

23:03

recognized this is actually way

23:05

more useful because oftentimes

23:06

we have bugs where, oh yeah, on,

23:10

on windows there is a Eprom

23:12

issue, even though we use node

23:14

and it should be cross-platform,

23:15

there's like always little

23:16

weird details where

23:19

cross-platform is a lie. And

23:21

then in the past, I would just

23:23

sometimes just like wipe it

23:25

because I'm too lazy to spin up

23:27

my windows computer, I could

23:28

spin up a VM, but then how do I

23:31

get it into the VM? And then

23:32

kind of like have to do get

23:34

cortex into the VM because I

23:36

can't, I can do git anymore

23:37

without an agent because it's 3

23:39

a.m. and I'm tired. You know,

23:41

you have to fight friction like,

23:43

like ultimately we are being

23:46

lazy is good. So I noticed that

23:50

I'm, I'm pushing slop because

23:53

I'm too lazy to come up with a

23:55

good testing system. So I added

24:00

cross-platform support to grab

24:01

box. And now anytime there's

24:03

something that's hairy, I just

24:05

tell my agent, hey, test it on

24:07

all the races. And it will not

24:09

just create one box, it will

24:10

create three boxes. Or maybe,

24:12

maybe there's a weird issue

24:13

that's only on Red hat. Sorry,

24:14

Sally, I love Red hat and it

24:16

will spin up four boxes and you

24:18

can like, you can scale it

24:20

infinitum and like increase

24:21

your confidence that your code

24:23

actually works. And it can do

24:24

that in parallel, which is

24:26

really amazing. And then once I

24:29

built that got me thinking like.

24:33

I have this bug, but it's like

24:37

actually only on windows on

24:38

this browser. So I kind of need

24:41

UI for that. People already

24:43

solved that. Just add VNC into

24:45

it. So now I have a crap box

24:47

that like can recreate exact

24:50

scenario, and then my agent

24:51

opens a browser with VNC and in

24:54

there there's already a browser

24:55

waiting in a terminal and maybe

24:57

telegram or whatever I need so

24:59

I can recreate the issue end to

25:01

end and test it. That was

25:03

amazing. But still, like I had

25:06

to click myself. Oh, you know,

25:11

like laziness is good. So that

25:15

got us the idea. Wouldn't it be

25:16

cool if the Clanker could do it?

25:18

Of course they can do it. They

25:20

have computer vision models

25:21

could really good. So so now

25:25

Crap box also has a little

25:26

little tools for screenshot and

25:28

click and type. And now I can

25:30

actually, my agent can have its

25:32

own little box with UI that it

25:34

can use computer automation in

25:36

to control the UI. I was still

25:41

not good enough. Like I don't

25:45

want to watch the clanker. I

25:46

just want to see video. So what

25:48

we built next was mantis, where

25:50

now we have an agent that I can

25:53

ping on a, on a pull request

25:56

that'll spin up boxes, make a

25:59

video of the bug, fix it, make

26:01

a video of the fix and actually

26:03

look at the video and verify it.

26:05

And then all I get, I can

26:07

basically see an issue. I can

26:09

ping Klaus to fix it, and it'll

26:12

create the PR mantis will do

26:13

the visual verification, and

26:15

all I have to do is like, look

26:17

at the video and press merge.

26:21

And that's all built based on

26:23

me being annoyed. The next

26:29

thing I noticed. Like you would,

26:34

you would use Codex, you would

26:36

like do the prompt and then you

26:39

would do slash review because

26:40

you're like, you're kind of

26:42

unsure. Maybe maybe I should

26:43

review it. And then Codex would

26:44

find three issues and they're

26:46

like, damn, they're like, all

26:48

of them sound real. SI fix it

26:51

and run off for ten, 20 minutes,

26:53

run all the tests again, and

26:55

you're like, oh, is it really

26:57

all fixed? Let's just let's

26:58

review again and will run for

27:01

ten minutes. It would find two

27:02

new issues because it can't

27:05

show me all the issues at once.

27:07

It's just like, I can't do it.

27:09

You type fix again, you wait 20

27:12

minutes, you do something else,

27:13

go back and like, yeah, but is

27:15

it really fixed this? Let's

27:17

review again. It would run for

27:20

20 minutes, 30 minutes. It

27:21

would find another issue. And I

27:23

don't know, have you done this?

27:25

Like I did this. And then there

27:27

was one time I did this ten

27:29

times. It was the whole day.

27:32

Like I added support for line.

27:34

I don't even use line, but like

27:37

somebody was very passionate

27:38

about it. I wanted to get it in.

27:40

And then all day I did this

27:42

review and I was like, is it

27:44

enough? Is enough? Like, like,

27:47

I feel like I'm the clanker. So

27:51

I built auto review and in the

27:54

beginning I thought, oh, this

27:56

is kind of annoying. I probably

27:57

need hooks or like, maybe I for

27:59

Codex because Codex is open

28:00

source. I could totally do that.

28:02

But that turns out it's

28:03

actually way easier than that

28:04

because agents are really good

28:06

at following instructions for

28:08

last at least 4 or 5 months. So

28:10

now there is one line in my

28:13

agents.md that says, before you

28:17

commit or land a PR, if you

28:20

haven't done auto review it,

28:22

run auto review. And again, you

28:25

know, review. Some people say,

28:27

look at the code with fresh

28:28

eyes. Yeah, good luck with that.

28:30

You know, like context doesn't

28:32

really work that way. But. But

28:43

you can just call. I don't know

28:47

if you find the but you get the

28:50

idea. You can just call the

28:52

agent that you use as CLI with

28:55

the commands you should do. So

28:57

you get a fresh context, but

28:58

just calling like I Codex calls

29:02

Codex to like run auto review.

29:05

There was a bug sometimes.

29:06

Sometimes Codex was calling

29:07

Codex, calling codex and

29:08

calling colleagues. And it took

29:09

very long. I fixed that, but

29:12

now I have a very reliable

29:14

system that automatically runs

29:17

many rounds of review and. I

29:25

also tuned the file to explain

29:28

the agent. Not every review

29:32

thing is valid. You know,

29:35

sometimes. They would find very

29:39

weird edge cases. So it's a

29:41

little bit up to you to explain

29:43

the agent, the, the, the, the

29:47

conditions, what you think is

29:49

valid running, for example, in

29:50

our case, we don't expect that

29:53

you randomly edit plugin files

29:55

while the system is running.

29:57

The agent doesn't know that. So

29:58

like by default it would write

30:00

very defensive code to like

30:02

deal with all of those things.

30:04

So, so it's always a good idea

30:05

to write all these invariants

30:07

into your agent file. That's a

30:11

good idea in general because

30:13

you know that, but the clanker

30:15

doesn't. And every time you

30:17

start a new session, Clanker

30:19

memory is pretty much gone. So

30:22

you the more you tell it about

30:25

your project, the better the

30:27

time you'll have. I went all in

30:30

from very long agent files to

30:31

like very short ones, and now

30:33

mine are kind of long again,

30:35

but with meaningful stuff.

30:39

Every time you do that, it's

30:40

also don't don't write the text

30:44

yourself. You know, none of

30:45

this is written yourself. You

30:46

don't know how to write to the

30:48

client in a way that Clanker

30:50

writes to a clanker. So like

30:51

you're going to explain what

30:52

you want from the agent, and

30:54

the agent will write it in its

30:56

own words. And then every once

30:57

in a while, go into meta mode

30:59

and just ask the agent, hey, is

31:01

there anything that's confusing

31:02

in your agent file? And then

31:03

they will usually tell you,

31:04

yeah, well, this is confusing

31:06

because here it says this and

31:07

here it says this and then

31:09

cleaned it up. Clean up any

31:10

confusion because that's just

31:11

token burner. And we'll build a,

31:13

we'll give you a worse results.

31:16

But I think if, if you only

31:20

take about two ideas from this

31:22

talk, try auto review and try

31:23

grab box. It totally

31:25

revolutionized how we build

31:26

software like my, my trust in

31:29

in. Prompting and shipping is

31:34

much, much higher with that

31:36

stuff. There's another thing

31:46

when we get PRS. I would love

31:52

to know how people came to that

31:55

PR like. The good and the bad

32:00

thing with agents is like a one

32:02

line prompt can create ten

32:03

lines, ten, ten, ten pages of

32:06

content. And you don't always

32:10

know how much, how much work

32:11

was put in. So very early on in

32:14

our contributor guidelines, I

32:15

asked people to please send me

32:17

the prompts and nobody would do

32:20

it because there was just no

32:21

good system. You know, I was

32:23

fighting against people's

32:25

laziness. Of course they will

32:26

not do it. That one took me a

32:29

long time to figure out, but

32:30

now I built a skill. If you use

32:32

open claw and you contribute,

32:34

you send a pull request to open

32:35

claw. Your agent will ask you,

32:37

hey, do you want to send a a

32:41

privacy filtered version of the

32:42

prompt? If you do that, we will

32:45

look at your PR faster and then

32:47

the user can type yes. And then

32:50

the agent will upload, will,

32:51

will, will look on the disk,

32:53

will find its own transcript,

32:56

will cut the transcript to the

32:58

interesting parts, and then

33:01

we'll add those transcript to

33:02

the pull request. And suddenly

33:05

I have a much more interesting

33:07

data point about how much

33:09

effort was put in, because

33:10

that's ultimately what you're

33:11

after. Like I can, somebody can

33:13

say, fix issue number and send

33:17

PR and it's like four words,

33:19

five tokens. And or somebody

33:21

could actually spend a few

33:23

hours and like have an actual

33:24

discussion and you don't always

33:25

know. So this has been

33:27

extremely helpful in, in, in

33:29

figuring out what's total slop

33:32

and what's kind of good slop.

33:35

And again, like, I could have

33:37

done this half a year ago, just,

33:39

I just didn't come to mind that

33:41

actually it's that easy.

33:43

Oftentimes we're not limited

33:46

anymore by. By intelligence.

33:50

The models are good enough now.

33:53

Everything that we are limited

33:54

of is our own imagination. How

33:56

what you do with it and how to

33:58

build things. Now, I built auto

34:01

review and I was like, oh, this

34:03

is amazing. And it made me

34:05

really bad. I felt really bad

34:07

for all the code that I

34:08

committed without review. Now

34:10

that I, I saw how much stuff

34:13

the agent finds each time and I

34:16

saw GPT five is good, but and

34:18

yes, it is really good, but

34:21

still, there's so many boo boos

34:24

like the. Especially on large.

34:26

There's just mistakes and like

34:28

difficult things. The agent

34:29

will only find. After doing

34:31

multiple rounds of reviews. So

34:34

it got me worried like, oh, I

34:36

wish I had done this earlier,

34:38

but you can't really. There's

34:40

no slash review for a project

34:42

that has a million lines of

34:43

code. There's too much like

34:46

what should the agent focus on?

34:48

There's too much for, for. The

34:53

context. It annoyed me. You

34:59

know what, what, what happens

35:00

when I'm annoyed? So I built

35:02

cloud Patch. And the principle

35:06

is very simple. Like, oh, you

35:07

have a big project. Let's take

35:09

it apart into like 50 little

35:11

sub things, subsections by

35:13

feature or whatever, whatever

35:15

else you can, like, you can

35:16

separate it and then have the

35:18

agent review each of those

35:20

things separately. And then you

35:24

can, you can download this, run

35:25

this on, on your project. And I

35:27

guarantee you you'll find a lot

35:29

of interesting things. It's not,

35:33

it's not quite as perfect as

35:35

paired change because you would

35:37

have to probably scale it up to

35:39

thousands of code sessions,

35:43

even with unlimited token.

35:44

That's a problem. But it's

35:47

really good at like cleaning up

35:48

code bases. This is something

35:57

that's not fully finished yet.

36:02

But, you know, I got I was so

36:04

into crap box. The more I

36:11

worked with the team, the more

36:14

did you had this where you ask

36:16

your teammate and your teammate

36:17

will pass stuff from your from

36:19

the agent into chat. And then

36:21

you would you would comment on

36:23

it and then they would pass

36:24

stuff into into the coding

36:25

agent and they would pass stuff

36:27

back into the chat. I'm like, I

36:29

don't like, like, like I,

36:33

you're just intermediary. I

36:34

want to talk to your clanker.

36:36

So, so that's the last thing I

36:40

built this, this crap fleet,

36:41

which is basically we already

36:43

have crap box where the tests

36:45

run. Couldn't Codex run in

36:46

there as well. Turns out, yes,

36:49

it can. And then you have

36:52

multiplayer codecs. So so so

36:56

now if I work on something and

36:58

I want a second opinion, I can

37:00

just send a link to one of my,

37:02

my maintainers and like, he can

37:03

look at the session, talk to

37:05

the session, take over the

37:06

session. It's like multiplayer

37:09

clanking. There's more stuff.

37:18

We kind of rebuild slack. Yeah,

37:21

I don't know what happened

37:23

there, but we ran into this

37:26

issue where we had all these

37:31

model message channels, and you

37:35

often want to test it end to

37:37

end, but there was sometimes

37:40

features are very difficult to

37:43

understand with all the complex

37:45

API. You kind of want to see

37:46

see it in the UI. And I built

37:48

this automation, but man,

37:50

anytime I logged in, there was

37:52

like a Cloudflare capture or

37:54

like, like all these, all these

37:56

companies really don't like it

37:57

if you spin up something in a

37:59

box, especially not if it's

38:01

Linux based and there's nobody

38:02

logged in and it just feels

38:03

like it feels like someone's

38:05

trying to hack the system.

38:06

We're just trying to like make

38:07

it work. But I can see why

38:09

people don't like it or why

38:10

their systems would freak out.

38:12

So we just built our own

38:13

messaging platform to like test

38:16

to like, have real end to end

38:18

tests with something that we

38:19

have under control. And it got

38:22

a little bit out of hand. And

38:23

now we kind of have a backup

38:25

chat system that works.

38:27

Whenever our team is down, it's

38:29

called click clock. There's so

38:43

much more. Where were we? I

38:49

think I showed you, I showed

38:50

you a good part. Yes, there's

38:52

of course there's way more

38:53

crawlers. But my, my big, my

38:58

big takeaway for this is listen

39:01

to your listen to your body.

39:02

Listen when you're annoyed.

39:04

Annoyance is an amazing signal

39:06

for. We can automate this. And

39:09

then anytime you automate

39:10

something, it compounds all

39:12

these little tools. They're

39:12

working together, right? I, I

39:14

can only do the reports because

39:16

I use disk crawl and I could

39:18

only use crab fleet because I

39:20

have crab box. Even though all

39:22

of these are weird names, they

39:23

are incredibly helpful in

39:26

helping us ship a massive

39:28

codebase with a very small team.

39:30

And with that, I do think we

39:32

have a few minutes of Q&A.

39:34

There is a microphone. There's

39:38

a microphone here, so should.

39:51

As you're building. So. It's

40:00

making some noise. Is it

40:01

working? Yes. There we go.

40:03

Sorry. As you're building this

40:05

ecosystem and becoming more

40:07

like enterprise, quote unquote,

40:09

like, do you feel that there

40:11

are times where you want to

40:14

build something, but the

40:16

dependency on the other things

40:19

get in the way of you improving?

40:25

Hasn't happened so far yet.

40:28

It's less dependencies. It's

40:30

more like all those little

40:31

tools compound and make

40:32

building new tools faster in

40:35

many ways. Sometimes I use one

40:38

tool in another tool many other

40:40

times, and that's a general

40:42

trick. When you work with

40:43

agents, if you've sold

40:44

something once, you can just

40:45

tell the agent, look there, I

40:47

already solved this there. They

40:48

are really good at pattern

40:49

matching. So I would actually

40:50

say it really compounds.

40:51

Sometimes something doesn't

40:52

work out, but then you learn

40:54

from it like, like the the

40:56

token thing. Where was it? Too

41:02

many things. The token thing,

41:06

the first version I built was

41:08

just on my machine. That was

41:14

really awkward with the GitHub

41:16

app. So I iterated a bunch on

41:18

it, and eventually it was like,

41:20

I ripped it all out and I just

41:22

told my agent, remove this

41:23

feature here, rebuild this with

41:25

this new idea, but take that

41:27

part from that tool and like

41:29

push it into your tool. And

41:31

then like, basically you save

41:33

the prompting, right? Because

41:34

you already, you can just tell

41:35

the agent, look around this,

41:36

look at the git history.

41:37

They're very smart at figuring

41:39

out the context. Even better,

41:40

if you just press the, the, the

41:43

button and you just like, you

41:45

just talk about it sometimes.

41:47

Sometimes we just talk in the

41:48

team about it and I just press

41:50

the button and Codex will just

41:51

listen in on the conversation

41:53

and just start building while

41:54

we've been discussing, even

41:56

though there's like side

41:56

chatter and blah blah, but like

41:58

it's clever enough to

41:59

understand like, oh, this

42:01

comment was a joke and this is

42:03

important. So you just like.

42:06

You can just do stuff. Thank

42:08

you. All right. Hi, Peter, this

42:14

is Divya. My biggest question

42:17

is this is awesome. But imagine

42:19

you have a billion things on

42:21

your plate as a solutions

42:24

engineer. And I love the

42:26

concept of building my own

42:27

agents, but I simply don't have

42:28

the time to actually go ahead

42:31

and build tools to help me be

42:34

more efficient and save time.

42:36

Do you notice, do you notice

42:38

meme and like Stone age where

42:39

they had this car with

42:42

rectangular. And and then the

42:46

guy comes along with like the

42:47

round one and they're like, oh,

42:49

we don't have time. We're so

42:50

busy. So, so the beauty is all

42:54

these tools, they can be

42:56

massive slop. It's like you can

42:59

literally, you can really tell

43:00

your cortex, hey, look at this

43:01

session and see the pain that I

43:03

had to do. Build me a tool that

43:05

reduces the pain. This is the

43:06

rough idea. You don't even have

43:08

to give it a programing

43:10

language. Nothing matters for

43:10

like those little tools like I

43:12

use go. I don't even like go.

43:13

Like I know, I know that my

43:15

agents are fast and go, so who

43:17

cares? And then just like, do

43:18

it on the side, you know, it's

43:20

like there's always space

43:21

somewhere in the corner for a

43:23

little window for like

43:24

experiments. And it also takes

43:25

a fun a little bit back. And

43:27

then it, it really compounds

Interactive Summary

The video explores the philosophy of building software in the era of coding agents by creating 'tools to build the tools.' The speaker demonstrates how to automate repetitive maintainer tasks, such as triaging GitHub issues, code reviews, cross-platform testing, and managing API rate limits. The core message is that developers should view frustration as a signal to build an automated solution, leveraging agents to handle the heavy lifting and compound efficiency, even if the tools themselves are built quickly and 'sloppily'.

Suggested questions

3 ready-made prompts