So what is context engineering?

>> It's kind of like deabstracting the

abstractions that have been layered on

top of rag memory, agentic history. At

the end of the day, they're all

different ways to pass tokens into a

model.

>> What is a smart zone and what is a dumb

zone?

>> The less context window you use, the

better outcomes you'll get always.

>> A new paradm that is spreading up is

loop engineering. What do you think is

bad about it?

>> Problem with loops is like at a certain

point, you're going to generate so much

code that you can't read it anymore. We

built a lights off software factory in

July of 2025 and by November we had shut

it down.

>> Can we talk about what you mean by token

harder and token smarter?

>> I'm in a group chat called

hyperengineering and it's all like

people trying to max out their cloud

subs. That's my idea of token harder and

the goal is

what happens when you let AI agent ship

code for [music] months and no developer

reads a single line. Today's guest tried

exactly that. He built a lights off

software factory and four months later

he had no choice but to shut it down as

things just [music] stopped working.

Dexory is the founder of human layer and

the person who coined the term context

engineering days before Andre Carpathy

and Tubiluska made it famous. He spent

the last two years talking to hundreds

of AI engineers about what actually

works [music] when you build with LMS

and is testing the most extreme ideas

with his own team. In today's

conversation we discuss [music] context

engineering, what it is and the physics

of context windows, including what the

dump zone is. loop [music] engineering

from the Ralph Wim technique to the slow

loops that Dex's team runs every night

to wake up to code cleanup PRs [music]

the rise of software factories from a

NATO conference in 1968 through DevOps

[music] to today's agentic factories

specdriven development and why specs

always drift from the code itself and

many more. If you want to understand

increasingly important concepts like

concept engineering and harness

engineering or want to know how far you

can push the let agents build everything

idea from someone who pushed it further

than almost anyone then this episode is

for you. This episode is presented by

antithesis. If you work with agents your

job is no longer just writing [music]

code. It's specifying and testing it and

antithesis is the most effective method

of verifying agenda code today. Today's

episode is brought to you by Buildkite,

the CIOS platform trusted by OpenAI,

Entropic, Cursor, Nvidia, Uber, Canva,

and more. Today, we're talking about

pushing the right context into models so

that they write better code. Right after

that starts working, your agents will

write more code, a lot more. Trusting

that code avalanche is where many teams

face a challenge today. Every change

that an agent makes still has to be

built, tested, and proven safe before it

ships. Worked on my machine is not

enough. So, you obviously need CI. But

when agents are pushing five, 10, or 50

times the commit volume into your

pipelines, faster CI runners won't save

you. Shaving 30 seconds off a single

build is meaningless when the queue is

100 plus jobs deep. What you really want

is a CI system that gets faster as the

volume grows and CI that offers instant

parallelization to give you unlimited

concurrency and to intelligently route

changes at runtime. This is what Bill

Kite does and why global software

leaders continue to rely on it. The same

architecture that observed the scale of

Shopify and Uber a decade ago now runs

about 1.4 billion job minutes a week

across cursor meta and snowflake. While

the rest of the CI world are cracking

under the weight of rearchitecting your

platform, build kite continues to

reliably grow. Agents running on your

infrastructure or build kites. Any

cloud, any chip, your secrets, your

scale, every artifact and log is

captured. So when something fails,

either you or your agents have immediate

insight for why. As you're ensuring the

context you'll give to your agents,

think about how you'll verify what they

hand back. If your system is buckling

under the increased volume, head to

buildkite.com/pragmatic.

30-day all access trial, no credit card,

and an actual human engineer on standby.

His name's Ola, and he's very helpful.

So, Dex, welcome to the podcast.

>> Super stoked to be here, dude. Before we

get into some of the context engineering

and some of some of the the more spicy

stuff as well, how did you get into

tech? How did you fall in love with

computers?

>> Oh, man. So, I uh I was doing undergrad

as a as a physics major. Um, and I

realized that uh I didn't like academia.

And there's like basically like two or

three paths out of physics is basically

you go get a PhD or you go into finance

or you go do programming. At that time,

this was, you know, 2012, 2011 when it

was like in the middle of undergrad and

deciding what to do. And I had done an

internship when I was in high school. I

was working with NASA researchers to a

jet propulsion lab in California. They

had just gotten this really highfidelity

like the most uh you know fine grain

data set of altitudes like the heights

of very at very like top topographical

map of the south pole of the moon.

>> And the south pole of the moon is really

interesting because some of the craters

there are so deep because of the angle

it has. It got hit by meteor storms like

no other part of the moon. So there's

very deep craters that have never seen

sunlight.

>> And so there's frozen liquid water in

there from the formation of the moon.

And so scientists were really interested

in getting down there and exploring. And

uh so we had this really fine grain map

and it's like okay cool. Let's build

software so that I I have point A to

point B. I know the limitations of my

rover can you know max incline up is

this, max incline down is that find a

path from point A to point B that

doesn't like break those rules of the

incline. So I was you know 17. I had

never cracked a CS textbook. So I wrote

I basically like wrote a really naive

bad version of Dystra's algorithm for

pathf finding. Uh so I was in college. I

was like, I don't know if I want to do

the academics thing, but I really

enjoyed programming back in the day. And

so, uh, so I decided to go I got like

half of a CS minor and then started

working on a API platform team at a

software company in Chicago and

>> Sprout Social, right?

>> Yes. And uh, basically never went back.

>> Yeah. And then then where did you go

from there? Where did you pick up like

the parts of the trade? Because very

early on, your first job that's not

really common. you were doing platform

engineering back in you know more than a

decade ago. From that point, it took me

about two or three months to notice that

like the most valuable work that was

being done in the company was being done

by like of course it's obvious like the

first couple engineers who know

everything and understand where

everything was and like you spend a day

on a support ticket from a customer and

they solve it in 5 minutes but like you

have to solve it so you learn and

whatever. And I realized like the most

valuable people in the company were the

people that were building the developer

platform CI/CD sandbox environments

preview stuff. And so I kind of like

that was my first step into the journey

and I've basically been obsessed with

software factories since that like three

or six months into my first job.

>> We talk about software factories now but

you're you're talking about software

factories back then. So like you were

you're you were starting to already

think that this is how we can produce

better software inside this is pre AI

world right?

>> Well and I'm always surprised like

there's a huge class of developers that

say I don't want to work on CI/CD. I

hate CI/CD. I'm like really because

building the thing that builds the thing

and building the thing that builds the

thing that builds the thing is like as

software engineers we're lazy. We want

to do the most high lever thing that

makes our job easier. So how do we if we

can build a thing that helps us build a

thing that helps us move faster then

that's the best use of my time as a as a

lazy engineer. And then you went to

another startup uh as aspiration.

>> Aspiration. Yeah.

>> Aspiration also platform engineering.

>> Yeah. I was brought in and then like

three 3 months into the job, the VP of

engineering who hired me quit or got

fired. I don't know. There was some

drama about it. I probably shouldn't

talk about it. And then I was there for

about a year uh and was kind of like

acting CTO for a while like hired a

couple people, helped hire the new VP of

engineering, but I was out of there. I

don't think I'll ever do consumer again.

I think I'm actually a B2B guy.

>> Good to know. And then you went to

replicator where you spent like a good

like like solid like four years and went

from engineer for deployed engineer to

product manager. Yeah, I did core

engineering for like two years. We were

building a container orchestrator like

before Kubernetes, before Docker Swarm

was really a thing. We built our own

orchestrator. The founders had this

vision that like, oh, Docker is going to

make it much easier to ship on-prem

software. And when I say onrem, I don't

mean literally like a a rack in a colo.

It's more like, hey, look, bring the app

to where the data is rather than sending

the data up to some cloud vendor.

>> And Docker makes it much more much

easier to package up apps and and and

and move them around. And so they had

this thesis that like basically you

could build a platform that the

experience that you get when you use

GitHub enterprise which is like you

install it and it has this admin panel

but then you just get GitHub running in

your data center and your code never has

to leave your your data center. Suddenly

you could build a generic SAS where

everybody could have that. So I did two

years as an engineer there and then our

head of sales. We parted ways with our

head of sales and uh honestly I was

having a lot of arguments about the

software factory with our CTO and it's

kind of like almost like a too many

cooks in the kitchen kind of thing. I'm

sure many listeners listen listeners

have had this experience of like well

yeah I know I have these tickets to

build but like CI sucks. I got to fix CI

because it's too slow or it's like

there's too many different builds and

it's always breaking. like I'm going to

fix that and then I'm going to do the

end is just like Dex, I need you to stop

fixing the build pipeline and like do

the tickets I gave you. I'm sure you've

had this experience perhaps.

>> Yeah. And then and was this what led you

to either forward deploy engineering?

>> Yeah. So I like I really loved our

customers. Our customer our customers

are Hashi Corp, Data Stacks, Puppet, all

these really cool engineering brands.

TravisCI, CircleCI. I was like yeah I

actually love working with our

customers. Our customers are awesome.

And uh it was a great way to like get in

the trenches. a lot of really good

engineers who were solving the hardest

problem at the company which is like how

do we take this 3 to 5year-old SAS

platform and package it all up so that

someone who knows nothing about our

architecture can run it reliably in

their own AWS VPC in their own on-prem

data center whatever it was and so I

spent I was our first kind of customerf

facing engineer and it was in about

three months I we closed I met with like

every company customer that was like

kind of in the pipeline but wasn't

moving saleswise is and we closed like

12 deals in 3 months and the CEO was

like, "Holy crap, Dex. Like the the

investors are taking my calls again.

Like I don't I know you want to get back

to coding, but like I need you to go

hire three people and like build this

team out cuz I think you might have been

like born for this."

>> Wow.

>> Yeah. So I did that for about four

years, built that or to like 25 people

and then Zer happened and uh it got a

lot smaller and we kind of realized

like, hey, we have a product that's like

pretty good uh and we've been solving

what lots of early startups do is like,

okay, there's some usability issues. is

we'll throw we'll get a bunch of smart

people, throw them in the trenches with

our customers, great for sales, great

for retention, all this stuff. And it

was like, oh, we actually like the

margins on that aren't aren't good

enough. And so we basically were like,

cool, we actually just need to make the

product way more usable, do a more

PLG-shaped thing, make it productled

growth,

>> product led growth. Make it a little

more self-service so you don't need an

expert to teach you how to use it. And I

was like, cool. If that's the most

important thing, then I want to go be a

product manager because I have tons of

opinions. I've now spent four years in

the trenches with our customers. I have

a laundry list of roadmap things that I

think would make the product way easier

to use and adopt and implement and

deploy

>> and and now you went the full ar you

went towards a dark side.

>> Exactly. Yeah, I did. I was like this is

going to kill my street cred isn't it?

But uh I was really glad you know I

think a lot of engineers are afraid that

if they go do a customerf facing thing

they lose all their credibility and like

yes I wasn't coding for 10 hours a day.

I was coding for like three or four

hours on a Saturday for fun. Not uh but

I mean we were helping people build YAML

we were building CLIs. We owned a lot of

the tooling that customers use, but it

was like the last mile delivery side of

it, not the core platform. And like on a

more personal note, I had spent the last

like most of my 20s feeling like okay, a

little bit introverted, a little bit

like socially awkward. What I what a lot

of engineers I'm sure experience and uh

I had talked to my uncle's a music

producer. So he used to work with like

Randy Newman and a bunch of like really

famous musicians.

>> Oh wow.

>> Yeah. This guy Mitchell F. And he he I

was sitting with dinner with him at some

point and when I was I think it was when

I was still in undergrad, but he gave me

this lecture. He was basically like if

you want to be really good at something,

you have to make it the only thing you

do. The guy playing guitar nights and

weekends trying to get his band off off

the ground will probably never achieve

greatness. The people who become great

are the people who basically make it

like if I don't play guitar, I don't

eat. And you go and you sit on the

street all day and you play for 14 hours

a day or whatever it is. That's the only

way to become great. So, I said, "Okay,

instead of trying to like read self-help

books about how to be less introverted

and less socially awkward, like what if

I just made it my freaking job to just

talk to people and make friends and like

help people and solve their problems and

uh I think it worked out. I recommend

it. I think everyone should spend a year

or two at least doing something really

like customerf facing."

>> Did you do this because you felt that it

was holding you back be being

introverted or or like what what what

and I I know you got the motivation from

the whole musician motivation. I I get

it on one part, but what was it that you

said like is a customerf facing thing

that I'm I'm going to be doing it

because clearly you were pretty great at

like writing code by that point. You

could argue you were doing it night and

day. So where where did you find that

like I actually I think like customerf

facing or like getting this introvert

off of me? Did you feel that I was

holding you back or you just wanted to

be good at it?

>> It was just kind of a thing that was

like interfering with my like general

life satisfaction.

>> And it was also like I'm not a very type

A person. I'm very disorganized. is I

don't know if people call it like okay

I'm like ADHD now that's why I can run

30 quads in parallel or whatever it is

but it was like I was really bad at

email and calendars and spreadsheets I

just like didn't care about these didn't

understand them and so like another side

effect of this was like it just forced

me to be organized and keep a lot of

things going and so like I don't know

there's like weird benefits you get from

like stepping outside your comfort zone

and learning like industrial disciplines

that are separate from what you've been

doing and so the opportunity presented

itself and I was like oh I like working

I'll try this for a little bit started

going really well I'm like cool let's

keep let's see let's see how far this

thread goes

>> and then afterwards you're now in your

second startup. You you became a founder

and you also got involved in in AI

pretty early as I as it was even before

it was so obvious that it would change

how it would change how we develop

software, right?

>> Well, I would say I was I was later than

I could have been because we started the

company uh me and a buddy in Chicago

started a company in the data

engineering space in about 2020 November

20. We decided in like August of 2015,

>> this is Metalytics.

>> Metalytics. Um, technically still the

same company as human layer, we just

like pivoted the the the mission. But,

uh, yeah, the the the advice I got from

every angel investor that, you know,

people who just knew CTOs I'd worked for

before and stuff, they were just like,

look, hitting a lot of heads, wins. I

don't know if you know like the whole

DBT data engineering fiverr that whole

arc where it was like this huge party

and tons of investor money going into

all these different companies and then

within by like 2021 2022 there was kind

of the zer thing and just this general

realization that the TAM for those sorts

of tools is not as big as everyone

market

>> yes the total addressable market for

those sort of tools was was not as quite

as big as uh as we all thought it was.

Um so it was it was a hard place to

raise money. It was a hard place to get

customers.

>> Yeah. And then I I met you at while you

were at Human Layer NSF at an event. We

you actually talked and we chatted

afterwards. But by by that, this was

about a year ago, you were already you

you started to have some really strong

opinions on using AI. And one of them

was this now famous 12 factor agents

manifesto.

>> Is are we calling it a manifesto now?

>> I'm I'm calling it a manifesto. It's a

manifesto. I'm calling it. Let's talk

about this. This was 12 engineuring

principles to build reliable production

ready apps. uh how did you come up with

this and maybe we can also talk about

some of them.

>> Yeah. So um I'll I'll kind of like go to

like around August the co-founder I was

working with kind of burned out and left

and it was very we were on good terms.

It was very mutual. Um and I decided to

start messing with AI stuff and I was

building a AI agents and what was really

in fog right then was like the lang

chain the crew AI these like agent

frameworks. Um and it seemed like there

was a ton of you go you go in the crew

AI discord there's 10,000 people. It's

like, okay, this feels like the right

shape and this there's clearly this eco.

You go in every single one of those

projects, they have a Chroma DB plugin.

They have like a Composeio plugin.

There's like clearly like this is the

this is the shared interface that

everybody is building for. I say, okay,

what's missing from all of this? The

agents can call tools, but it's really

hard to like control which tools they

call. And if it's a chatbot, obviously,

you can show approved deny in the UI of

your application. But I kind of was

obsessed with what I would call like

outer loop agents or proactive agent.

Agents that would run in the background,

get triggered by events. I mean,

OpenClaw is basically like the biggest

manifestation of this of like you have a

heartbeat, it wakes up, it sees if

there's any work to do, it tries to do

stuff. And my thought was like, I'm not

going to trust that agent to do anything

meaningful.

if I can't get like a Slack message or

an iMessage or something when it wants

to do something and kind of guarantee

deterministically that I can approve or

deny that or deny it with feedback and

say actually no do it like this. So we

played in that space for a while and

talked to a lot of founders and founding

engineers and builders. We came into YC

in the fall of 2024 with this idea.

We're building out this API platform and

it was sort of like pedag duty but like

it wasn't who's on call to fix the

servers. It was like who's on call to

this like routing mechanism for like who

needs to approve this agent and can they

like escalate it or delegate it or defer

it all this stuff and we built it for

this ecosystem crew AI link chain fi

there's so many grip tape there was so

many in that in that time and then I

talked to tons of AI engineers who were

actually building really interesting

things and like actually making money

doing six figure contracts shipping AI

to the enterprise and all of them had

tried that stuff for like a month or two

and then they had thrown it out and they

were just writing all a API calls by

hand and they were building more things

that look more like pipelines and

workflows than these sort of like

hands-off call tools in a loop kind of

thing. And so I talked to a hundred

people and I spent a lot of time a lot a

lot of time hanging out with one of my

best friends uh Vib from uh Boundary. So

they build a programming they built like

this like protobuffs for AI thing and

they're I think they're about to launch

their like full fat like programming

language touring complete thing. But he

had this way of thinking about agents

and building with models and building

with inference where it was a lot more

about understanding what structured

output really is under the hood. And

every single step in your AI workflow is

just tokens in tokens out. And your job

as an engineer is figure out, okay, what

tokens do I need to put in to maximize

the chance that the tokens out are going

to be good. and kind of distilled all

these ideas into about 12 principles and

wrote about it on GitHub, posted just

like this like 12-page GitHub repo,

threw it on HackerNews, got like five,

it was on the front page for like two

days and it I think it really resonated

with a lot of people.

>> Yeah. So, I I'll just quickly read the

12 principles and and then let's talk

about like one or two that resonate. Sub

12 are natural language of tool calls,

own your prompts, own your context

window. Tools are just structured

outputs. Unify execution state and

business state. Launch pause resume with

simple APIs. Contact humans with tool

calls. Own your control flow. Compact

errors into context window. Small

focused agents. Trigger from anywhere.

Meet users where they are. Make your

agent a stateless reducer.

>> The stateless. Yeah, the stateless

reducer one was a little actually

someone hit me up on Twitter and uh

corrected me. It's actually it's

actually a transducer because there's

technically multiple steps in the

workflow, but there we go. But but but

of this one, this this was a year ago,

so like which is like forever in uh in

in how the tooling is is evolving. Which

ones still stick with you where you're

like, "All right, these were good that

that still seem to hold off." Yeah, I

think I'm going spent most of March

writing it, published this in April. Uh,

and then Swix hit me up from AI.engineer

and he said, "Hey, can you come? You

want to come talk about this." So, I

gave this talk 12 factor agents in like

June 6th, I think. And, uh, small room

maybe like it was packed, but it was

like maybe a hundred people. That was

the year at AI engineer where like the

lower physically like on on the on the

second basement floor was all the super

corporate stuff and you go up a level is

a little bit more and then like on the

top floor is all the like weird cutting

edge like startup stuff that like you

probably shouldn't care about yet kind

of thing. So we were up there on the top

of this like weird way of thinking about

agents. Uh and then about a week later

or two weeks later uh Toby Licki from

Shopify says I really like this idea of

like context engineering. And I'm like I

I wrote about this two months ago. This

is great. Toby gets it and then a week

later Andre Carpathi is like well I

really like I think what we should think

about is not prompt engineering but

context engineering. And I was like,

"Yes, that's my." Anyways, I don't know.

If you ask Gemini, depends what day it

is, they will tell you either me or Toby

or Andre came up with context

engineering. You can't really own a

word. Like I don't no one remembers who

invented the word prompt engineering.

But of all the factors, factor three of

own your context window. And basically

the only way you can whether it's

agentic or a single step at a pipeline,

the only way you can impact the quality

of your output from AI is by caring a

lot about what the inputs and crafting

them. Let's talk about context

engineering, which I am going to credit

you that you coined it. I I did some

research and like I think you were

earlier by a few days. So there we go.

You you coined it. We're adding we're

adding to the we're adding to SEO juice.

We'll have it in a transcript. Dex

coined context engineering.

>> Well, and and like a asterisk on that is

basically like I learned about context

engineering from talking to these

hundred engineers and founders. I just

kind of like what was the same about

what they were all doing and I put a

name on it. So like I didn't invent

doing it. I was just like I think we I

think there's this thing and like

vocabulary and names are really

important and having like clean ways to

talk about the problem especially when

like a lot of the content about AI right

now is so much hype and jargon that is

like meaningless. I was like okay I

think there's a word here that is useful

to builders that explains how they

should be thinking about building their

software. So what is context

engineering?

>> It's kind of like deabstracting a lot of

the abstractions that have been layered

on top. So you have rag, you have

memory, you have agentic history, you

have structured output, you have all

these things that are like different

ideas in the frame of agentic

programming. And at the end of the day,

they're all like different ways to pass

tokens into a model and ask it to

produce usually some structured output.

And understanding that is a lot more

powerful than trying to learn memory and

trying to pick some agent framework off

the shelf and some memory framework off

the shelf. I mean, those are these

things are all really good. If you want

to get to like 80%, you want to get a

really good demo. But when you have to

go from 80% to 95% or 99%. You need to

go down a level and think about what's

everything we're putting into the

context window. What order is it going

in depending on which model we're doing?

And all of this stuff matters. You have

all of these levers that you can pull.

And it just felt like the right

abstraction for thinking about how do I

get AI to do the thing I want as

accurately as possible. Why is context

engineering started to become more more

talked about? It it was about a year

ago. Was it did it have to do with the

the context the the context window that

we could pass on to LLMs pretty much.

Did it start to expand or did did we

just start to realize that we can do a

lot more by passing on from you know the

easiest one is of course system prompts

but of course whenever you build an LLM

behind the scenes you will pass

additional context as well not just to

prompt the user you will add a bunch of

stuff that's I guess a dirty secret of

any any LM but why do you think the

focus is moving on to like all right

context is important

>> I think it always was important I think

what had to happen is a ton of smart

people again like all these builders I

talked to a ton of smart people had to

like focus really hard on producing like

I want to make software that I can sell.

I want to make something that is

accurate enough that I'm proud of and I

can sell to an enterprise and they're

going to be happy with it. And there's

just like the the the the easiest way to

get to really high quality AI

applications is by thinking at that

token level. Thinking about a string of

different LLM calls like rather than

just tools in the loop and it's kind of

open-ended and very flexible but not

that reliable. thinking of agents as as

workflows, as pipelines, as some mix

between maybe a couple tools in a loop

versus just, hey, I have my tools and I

have my model and I have my system

prompt and these are the only levers I

have. And it's actually no, you have way

more levers. It's going to take more

work and you're going to have to like

understand the LLM with a deeper

intuition. But it was a thing that we

always needed and it just took time for

people to build with this technology to

figure out that like this is the layer

of abstraction that allows you to break

through the quality ceiling.

>> And how are cost and context engineering

connected?

>> Yeah. Um I don't know. I was I was

talking about this with uh someone this

morning um about like when you're

working with LM, one of the things I I

like to say is kind of like make it run,

make it right, make it fast. see if the

world's best LLM at the time I think we

did a podcast episode that at the time

it was like 03 see if 03 can solve your

problem and then give it to people and

see if they want that and then if people

want it and you use it a lot then go do

a bunch of context engineering because

your engineering time is always the

bottleneck like humans trying to figure

out and solve problems and build evals

and improve and try different dimensions

or set up jeep or whatever it is is

always going to be more expensive than

just using a smarter model until you

have millions of requests a A and then

it's like, okay, we're going to do a

bunch of context engineering, break this

up into three calls, and get it to work

on GPT40. And then we're going to take

two of those and make those two work on

GPT40 and using old model names. But the

point is like for a certain task in your

workflow, can you get GPT OSS 12B, which

is like 1/ 1,000th of the cost of Opus,

can you get it to solve parts of the

problem so that the tokens and the

things you're using the smartest

frontier models for are just the things

that you really need, that level of

intelligence? But you shouldn't go build

all of that and overengineer it until

you've proved that you need it that it's

valuable that it's like okay this is now

I mean we get to Eli Goldrat and like

what is the the he had this book the

goal right it was about how to model

your factory and I'm sure we'll get to

that when we talk about software

factories it was like what is the

bottleneck in your system and one day it

will be latency and cost but it's

probably not that when you first start

out and context engineering is how you

move from the you you add human effort

to the equation to improve the

efficiency the speed the price the cost

efficiency of your system.

>> Interesting. And then one thing that

came up more recently and a lot later uh

recently is harness engineering. What is

harness engineering? So I made a post in

like October I think about or maybe

November of of like hey there's this new

thing that I see is like I'm calling it

harness engineering. My definition that

I had at the time is not what actually

this guy Viv who's at lang chain now

does a lot of really good writing on

agents and how to think about harness.

He had written something called harness

engineering like a couple weeks before

me but I hadn't read it at that point.

And my take was basically like okay when

you build an agent you use use context

engineering. When you use an agent

because we gave this talk in August of

2025 about like how to apply context

engineering to how you use coding

agents. And that kind of evolved into

this idea of like how do you take a

harness like cloud code like codeex how

do you engineer against the integration

points of that harness. So commands,

MCPs, skills, how you organize your

codebase. How do you kind of optimize

the environment that the coding agent

runs in to like get the best results?

The same way with context engineer, how

do you optimize the inputs to every

single prompt? Well, harness engineering

just is like how do I raise the floor so

that every single turn of this thing,

the results are as good as possible. And

the term got super blurry and some

people think harness engineering means

building a harness. And some people

think hard harness engineering means

building around a harness. I actually

like what Martin Fowler came up with uh

as usual he's very good at naming things

and he kind of defined the you have the

LLM and then you have the inner harness

which is like the thing the the tool

definitions and the integration points

that like say like a cloud code or a

codeex or a amp actually exposes that's

your inner harness and then you have the

outer harness which is the stuff that

you the human do to customize that for

your specific needs your codebase your

languages etc that's the best definition

I think we have for harness engineering

>> it's interesting how naming is still so

so important, isn't it?

>> Well, it's like as soon as you name

anything, people are most people are I'm

actually surprised that context

engineering still means the same thing

to most people that it did a year ago

and that it's even still relevant. Like

that's honestly the the craziest thing

to me is like you wrote how many things

that were written about AI 15 months ago

still matter or still interesting um or

are still like have good advice baked

into them. Stuff changes a I think

context engineering has been so long

lived because it's it's grounded in the

fundamentals of how transformer

attention works and until we have post

transformer models or linear attention

or whatever it is which who knows when

that's going to happen context

engineering will be interesting and

important to anyone building on AI and

can we talk about the physics of of

context uh you you you had a you had a

tweet uh this this one the the context

reality check this is a graph of uh as

you get to 1 million context just the

the quality just drops it. It goes down.

What do we need to know about like the

context? Again, we we now have models

that do have a 1 million context window.

Maybe we'll have even longer ones, but

when you start to just put in more stuff

into the context, it starts to become

less efficient. Like what what do we

know so far in terms of from a practical

perspective of like someone who is using

the context window to add on a bunch of

stuff? May that be MCP, may that be

tools, may that be scales, may that be

all of these things. Yeah. I mean, so

the longer context windows are good. You

can talk to it for longer. Like they're

doing a good job. But at the end of the

day, like especially when you had like

Opus, it was like Opus 4.5 and then Opus

4.51 mil or 4.6 and 4.61 mil. You're not

actually getting a like smarter model.

like the intelligence of the model is is

what drives its ability to attend to all

of the tokens in the context window to

figure out on the next turn which parts

of this 100k or 200k context window are

the most relevant to making the decision

of like what is the next tool we call

and doing that over and over again in a

loop. So I don't know there was some

study that came out in 2025 which found

that and again these are old models so

like inflate your numbers but it was

like frontier LLMs can follow about

150 to 250

instructions before it starts to drop

off. Their ability to follow all the

instructions just like drops off pretty

quickly. And I think Lori Vos I haven't

actually looked at the data but they did

a study with like the next generation

models a year later and it looks like

it's like much better the number of

instructions you can get in. In any

case, you have like I split context

engineering into like two categories.

You have like the the most people think

about like the information budget of

like okay I can do rag and I can pull

out chunks of this document rather than

putting the entire book into my context

window. I can just go grab the pages

that matter. But it's also your

instruction budget is like if you give

the model too many instructions and

especially too many conflicting

instructions and that's in your initial

prompt and also like if you have a

conversation you start going down a path

and then you change your mind and you

start going down a different you

actually I don't want to do any of that

I want to do this. It's like a it's a

lot of computation the model has to do

to notice that it has to ignore that

whole thing. And when both of those

things are kind of far back enough in

the context window that they're only

half getting attended to, your

likelihood that it's like actually going

to like remember the exact instructions

you gave it 100,000 tokens ago is like

it goes down quite significantly. This

is all very interesting because as

engineers there we are expected when you

know when we're AI engineers which now a

lot of software engineer meaning you

just like use LMS to to build software

like underneath there's an LLM layer

somewhere you're an AI engineer

congratulations but it sounds like the

expectation is to be you know to be a

good to be a good software engineer preI

you need to understand you know how to

write good code and it helps when you

understand a little bit of the

underlying we didn't need to do that

that much over time but it it it never

hurts but sounds Like right now we're in

this phase that to be an engineer who

can write an efficient AI system that

use LMS. You need to understand the

dynamics of the context you need to

understand why stuffing your context one

way or the other can be compute can

introduce latency and all of these. It

sounds like it's kind of more of an

intuition and of course there's some

understanding but from talking to you

you're like well it it it does this

computation like I know you know cuz you

tried it out right?

>> Yeah like I'm not I'm not a PhD in

machine learning like I couldn't

actually go like draw a mathematical

proof of how this works but we know

attention is quadratic and the more

stuff you put in the more it has to

spread this attention out over

everything.

This just feels like an absolute new

area and like a little bit very

different to like what we're used to

like software engineering which is like

pretty kind of like black and white,

right? That compiles or doesn't compile.

>> That's true. I mean there's a different

kind of intuition. I was talking about

this earlier as well is like there's a

different kind of intuition that you

that you develop over years as a

software engineer and uh there's many

categories of it but the one I'll I'll

call attention to that is like a thing

that you cannot teach you cannot do you

cannot learn in a textbook. The only way

to learn it is like I know bad patterns

in software because I have debugged them

at three in the morning. This is my

buddy Jake from Netflix said this in his

talk at AI engineer code. It's just like

there's no better way to learn what is

good and what is bad and what works and

what doesn't than suffering through the

thing that doesn't work.

>> Well, speaking of suffering through the

things that that doesn't work, uh a new

paradigm uh that is spreading up is

loops. Loop engineering. The idea that

instead of writing prompts, just write

loops. Set up your loops. And this all

started with the Ralph Wiggum technique

where it it will just well it I I guess

that's an early version of loops that

were just loops around and now we're

we're hearing with some of the big

biggest labs talking about that they're

actually just doing looping. What is

your take on have have you done some

looping yourself? Have you set up some

loops? And what do you think is good

about it and what do you think is bad

about it?

>> Yeah. So I think of loops as I mean this

could I could ramble on this for 10

minutes. This is an entire talk, but

I'll I'll try to I'll try to lay out

some highle stuff and then we can dig in

wherever you think is most interesting.

We had Ralph Wickham. It was actually a

year and four days ago was the first

time I saw the Ralph Wickham demo and

like Jeff Hunley was just like visiting

SF and he just like came through and

like dropped everybody's jaws with his

like, "Yeah, I just ran Sonnet around

the clock and spent six grand in six

weeks and like I built an entire Gen Z

programming lang." Look at it compiles

and it has a stage two compiler where

the compiler for the language is written

in the language itself and all the

insane. And the core lesson from all of

that I think was the idea of back

pressure which is basically and I think

a lot of people were doing this for a

very have been doing this for a long

time which is how do I let the model

check its own work? How do I automate

the process of getting feedback into the

model? And there's lots and lots of

different flavors of this. You can have

deterministic llinters. You can have

unit tests. Like part of what made the

programming language easy to build with

Ralph is a programming language can be

infinitely verified. You write you write

the code in the language, you compile

it. If the compiler fails, you go fix

the compiler. You run the program. If

the program fails, you go fix the

compiler. Like it's like it's very very

verifiable. And I think the lesson in

loops engineering is like if you can

make a problem very verifiable, you can

kind of like treat it like a black box

>> and then have it loop because it will

keep improving itself because of the

verification loop is already there.

>> Exactly. And so like you can do this

with CI/CD is like I I do this every

time I'm doing a release. I'm like I'm

tired. The CI/CD is slow. Cool. Go

research the codebase, make a change,

make a pull request, run the test, see

if it's faster, try again. Run the p run

the test. push push to the branch check

again see if it's faster and so it's

like if it can verify its own work in a

loop instead of design instead of saying

let's try this approach or let's try

that approach or suggest and being

really back and forth you just say like

my goal is to make CI faster and you

tell the model here's the steps here's

the five here's the five steps you're

going to write some code you're going to

commit it you're going to push it you're

going to launch a sub agent to watch the

job until it's finished it's going to

tell you what happened then you're going

to decide what to do next and so that's

that's like the very simplest example I

have of like designing loops

>> and you just set the goal which is cloud

code and and I think codecs have both

chip/go goal which is you just set the

goal and it iterates until it reaches it

or or as long as it makes progress

towards it.

>> Exactly. And so it's like if it's

verifiable if you can measure this is

auto research too. Auto research is like

hey go make this model twice as fast and

like it's just a prompt that tells the

model to like go to it over and over

again and try things until it actually

has good results. So that's what I think

of loops engineering. I don't know. We

we do a very interesting kind of loops

engineering where like the the challenge

is like I think it's very easy to get

very excited about building the thing

that builds the thing or building the

thing that builds the thing that builds

the thing we talked about. Uh and so

people say, "Oh, we need to like redo

everything as this big like aentic first

factory, maybe even a dark factory." And

they're like redesigning their entire

thing to be their infrastructure for the

next 5 years. And I'm sure one thing we

know of in engineering uh and especially

uh pragmatic engineering is uh how can

you make this more incremental? How can

you make it more continuous? Uh and a

lot of people don't have the option to

just hey I ran a Ralph loop for 3 days

and it fixed every line error in our

codebase. Here's a 60,000line PR. Who

wants to review it and who wants to sign

off on merging and deploying it and uh

that there's not going to be any bugs?

Nobody. So I think the the thing I'm

most excited is actually like what we

call like iterated loops or like slow

loops where we basically have a cron

job. We have the loop the the the

structure of the loop is really easy.

It's like run this llinter fix one thing

commit and push and then we run that

every night in our GitHub actions and we

wake up every morning to one PR that

makes the codebase a little bit better.

>> I I like the slow loops.

>> Yeah. And it has two dimensions. So you

can add now we have a blueprint for it

and actually Kyle just shipped a skill

so that you can build these yourself.

you can add more like feedback

mechanisms. So, we have React Doctor for

the front end. We have another

anti-attern that has no deterministic

tooling, but Kyle's just like, "Here's

what good looks like. Here's what bad

looks like. Go fix one thing and bring

it back." It's like prop narrowing

basically. We have a bunch of optional

props and most of them don't need to be

optional. It's like here's how to make

the prop not optional so that you know

that the code just is like cleaner and

easier to reason about. And so, you can

add more conditions, more things of like

fix one thing. I want to wake up to a

PR. So, now we wake up to like four PRs

because there's four separate things.

And then the other dimension you can do

here is as you gain confidence, you can

increase the scope. Instead of fixing

one thing, fix four things. And so these

are like other ways to think about loops

where it's like something that's not a

human triggers it to start. Whether

it's, you know, an alert from Sentry,

whether it's a user feedback like

support ticket, whether it's PM writes a

ticket, whether it's a test is failing,

any of or it's a cron, it runs on a

schedule, but it's like the trigger

should be something that you don't have

to like press a button on and there's a

defined workflow and it makes everything

a little bit better. Dex just described

letting agents fix things without a

human pressing a button. But what if a

bug is too difficult not just for an

agent but also for human to reproduce

let alone fix? This is where presenting

sponsor anticysis comes in. I was

recently pairing with the antithesis

team where we did a walkthrough of how

they helped fix a nasty bug inc the open

source key value store used by

Kubernetes. This is a bug that actually

happened incd. The team noticed that the

linearization validation assertion

failed during the regular anticis runs.

This is not good because the

linearization guarantees strong

consistency. So this needs to be fixed.

So what the ETCD team did was run a

casualty analysis inside anticysis. This

generates this graph which is a bug

probability graph. Here the x-axis is

virtual time and the y-axis is

probability. Now we see that something

happened just before virtual time 24

that caused a huge jump in the

probability that the bug would occur.

Going deeper, we can look at the entire

set of timelines. Vertical lines going

down represent events branching off from

the same state and the purple dots are

where the buck happens. If we look

closely enough, we see that all of the

failures come from one parent branch.

Gotcha. This is such a useful debugging

tool. In the end, the team was able to

figure out that process pauses were

causing the bug using all these anticis

debugging tools. This non-deterministic

bug was diagnosed in a deterministic

way. How cool is that? Oh, and this is

an actual bug that then got fixed incd.

You can see the bug and the fix in

ATCD's GitHub repo. Honestly, the tools

that Antys is built for debugging feel

pretty darn futuristic, but they are

also really powerful. Head over to

antithesis.com/pragmatic

to learn more. I'd also like to talk

about our season sponsor, Sentry. Sentry

is a tool I use for application

monitoring on all of my projects,

including their pragmatic engine back

end. I've used it for 10 years now,

starting with when I worked at Uber. A

neat Sentry feature I'm liking is their

SE AI agent, which helps investigate

production errors. For example, here's

an actual error I had in my application.

I can just ask Seir what might be the

root cause and it brings context. And it

can also make a plan to fix it right

from the web interface. And a nice thing

is how Seir also works great from Slack

as well, not just from the web. One

place I find even more handy to use

Sentry is from codeex or clock code

using Sentry MCP. Also, you can set up

neat automations like when a resolve

sentry issue resurfaces. You can kick

off a cursor agent or GitHub copilot

agent to investigate the regression,

read the relevant code, and open a PR

with a suggested fix. I'm not a fan for

using AI tools just for the sake of it,

but I really like the practical

integrations where I can fix errors

faster and with more context. Check out

Sentry at centry.io/pragmatic

and start monitoring and fixing

regressions today. And with this, let's

get back to Dex and to Agentic loops

that trigger themselves. Now, you said

we can get more ambitious and we can add

more things to it, but I'm I'm going to

quote you with uh with one of your

tweets which says, "This may surprise

you that this is coming from me, but I

think we're in for a 1 to three year

period where stuff might break at 3:00

a.m. and you're relying on loops to fix

it and nobody understands what's under

the hood, and you're looking at an ex

existential threat to your company."

>> Yes. Uh yeah, that one was great. That

one did a lot of numbers. Uh [laughter]

>> it resonated. Here's the other side of

it is like I think that the today with

today's models, today's programming

languages, today's infrastructure, you

might get away with not reading the

code. Problem with loops is like at a

certain point you're going to generate

so much code that you can't read it

anymore. This is the strong VM dark

factory. This is like Ryan Leopo's like

harness engineering. Just spend as many

tokens as possible. We tried this. We

built a lights off software factory in

July of 2025 and by November we had shut

it down. I think it takes about three to

six months of you shipping all the time

with nobody reading the code before you

realize like, wow, this is getting way

worse and it's easier to start over than

it is to fix it. Like the models have

made the codebase so bad that it is

actually going to be easier to just like

rethink this from scratch. And maybe

that's okay because we have AI and it's

easier to rebuild things from nothing.

And like usually when engineers say

like, "Oh, we can't fix this. We have to

rebuild it." The feedback is like, "No,

just refactor in place. Just constantly

keep the codebase getting better." You

mentioned what I said. You'll notice

what I said was not use loops to ship

the features that users want. We use

loops to actually improve the codebase

quality and we read all the code because

we care about how it's architected and

we care not just about the system

architecture but what I would call the

program design which I think is

something people are going to where are

the interfaces where are the seams how

are we doing dependency injection all of

these things that like make your

codebase more maintainable over time and

keep you from falling into this trap of

like okay well now if I change something

over here I broke something over here.

This is the classic problem of software

engineering that like software

engineering was invented in the 1970s

because we realized we needed techniques

for avoiding that problem of like this

giant ball of spaghetti. And I don't

think the models are smart enough and I

don't think we actually have the

training and the benchmarking and the

eval techniques to get models to write

code that is more maintainable over time

versus they're all trained on SWE and

SWEBench looking things, right? All of

the benchmarks are basically like here's

a commit in Django. Here's an issue that

was filed around that time. see if you

can create the fix that the human

created. And it's Django and it's Apache

and it's there's a hundred repos in Go

and C++ and Typescript and Java and all

these different languages, but they're

all it's like the problem with training

models on maintainability is like the

cost function of bad architecture and

bad program design can't be evaluated by

running the unit test because it hits

you 3 to 6 months later when you're

like, "Holy crap, like no one can make

it's this software has become so hard to

change." Is this not similar to how

senior software engineers why it took

years for someone to become a senior?

Because typically and in some

environments you became a you can become

a senior faster typically fast moving

where there's a bunch of issues and you

have to keep fixing it. Sometimes you

know some people are working in the same

place for 10 years and they're still not

that level. The point was it it just

takes time for you to understand the m

the small mistake that you make right

now that snowballs into like something

disastrous later and you get hit by it

and you realize like okay things like

you know like testing matters

architecture matters tech depth can

actually be a killer you know we don't

talk about it anymore but we used to

talk about how techdub kills or slows

down companies so badly preai that their

competitors can overtake them or they're

just like stuck with a 2-year refactor

not shipping any new features and the

competition you shifts a bunch bunch of

other stuff and now they're ahead.

>> And I I will say like it is possible

that GPT7 will fix this, but if you are

turning the lights off in your software

factory and you're saying like, "Hey,

you know what? Like we're not going to

read the code. It's fine. The models are

smart enough. If we give it the right

feedback and just throw enough tokens at

the problem, it will keep getting

better." This is what led to this tweet.

that might work, but if nobody read the

code in three months and you replace all

of your all of your like code review

with loops of like, hey, if a user

complains, we give it to an agent. If

something crashes, we give it to an

agent. If a if a PM writes a ticket, we

give it to an agent. If a CEO writes an

obnoxious essay about what we should be

building in Slack, we give it to an

agent.

>> Yeah. [laughter]

>> And then you stop reading the code

because that's going to produce way too

much. Like, no one can read it. And like

the the the PR reviews become the

bottleneck. So, you replace that with

aentic testing and agentic uh agentic

code review. Uh, but none of these

things have intuition for software

architecture because we haven't trained

it in yet. And so you're going to wake

up one day and you're going to have an

issue with this happened to us and like

we got through it and at the time like

it was still worth it. It was like spent

3 weeks onboarding back into the

codebase that we had stopped reading 3

months ago because no matter how much

sophisticating expert prompting we could

not get Opus, I think it was Opus 4.1 at

the time. We could not get Opus 4.1 to

actually find the root cause. We had to

go spend several days digging through

the code and figuring out like, oh,

there's just actually a primary key

that's being routed through this whole

thing that needs to be changed to a

different type of object and it needs

it.

>> This actually happened to you.

>> This happened to us. Yeah.

>> And when it happened, I was like, you

know what? That sucked. That was

terrible. But we did it. We solved it.

And uh it's still worth it's still worth

not reading the code for most of the

time at the cost of every once in a

while I'm going to have to spend two

weeks fixing an issue by hand. And I

don't believe that anymore because I

think the amount of code we're able to

write now is actually like 10xed or

100xed and I think the problem's just

getting worse.

>> So let's talk about software factories.

Yeah.

>> In your mind, cuz I feel it's an

overloaded word, but what do you think

of a software factory before AI and now

post AI?

>> Do you know the first definition of

software factory the first time it was

used?

>> No. It was a NATO conference in 1968.

>> Oh, Grady Buch would know about this.

>> Yeah, exactly. Yeah, great. You should

ask Grady about it. They talked about

the idea of like, okay, you actually

need to build a system of steps and like

just like a factory floor. You have like

the coding part and the testing part and

the validation part and the integration

part. We had no CI/CD. we barely had

version control like but you needed a

factory and then it was adopted by like

um Toshiba and a bunch of companies and

then the the next moment was like DevOps

and you have like this idea of like okay

we're going to do CI/CD we're going to

automate we're chef and anible puppet

whatever all these technologies is like

instead of having dudes running around

data centers like resizing discs and

stuff or clicking around the AWS console

yeah exactly it was like cool we build

loops the server hits 90% disc space

that sends an alert to Nagios Nagios

triggers a chef front chefs makes the

disc the disc bigger feedback loops,

right? This has been around for a while.

And in 2018, I want to say this guy Nick

Chalane who was uh he was like the CTO

or chie ch ch ch ch ch ch ch ch ch ch ch

ch ch ch ch ch ch ch ch ch ch ch ch ch

ch ch ch ch ch ch ch ch ch ch ch ch ch

ch ch chief software officer of the air

force, he wrote this 100page essay of

hey the DoD needs a software factory,

>> the department of defense.

>> Yeah, the the department of defense and

the air force. And he called it the dev

sec ops factory. And he said we need all

the things that all of the good

enterprises are using. We need Jenkins.

We need like code quality scanning. We

need security scanning. We need CI/CD.

We need to be able to ship. We're

shipping once every three months or once

a year. We need to be able to ship every

day like all these other companies. And

the way we do that is we actually

embrace all these automations and

technologies so that engineers are are

90% of the issues are caught by

automations instead of people actually

like manually checking it or manually

reading the code or manually integrating

modules together.

>> Wow. Talk about forward thinking in in

the government.

>> I know. Oh no, as I was surprised like

oh nice like this is I mean and that was

part of it is like hey look we're

falling behind in like you know I don't

know exactly all the reason but I I

imagine also about like attracting

really good talent is like hey look if

we have like the modern software stack

and we're building things fast and we

care about efficiency and we care about

people's using people's time well we

care about them spending time on the

hard parts of the job not manually

looking for SQL injections like you

could automate that. So this was

software factories pre AAI.

>> Pre AI.

>> Now I've heard the term a lot more

because of AI.

>> Yeah.

>> Is it the same? Is it different?

>> So this is really hard to say without a

drawing, but I'll try to draw it out. At

the core of a software factory, you have

like a source of work. Most you you can

imagine a linear a Jira a the st source

of truth your object whether it's a

spreadsheet or whatever is you have like

what stages is the work in.

>> Yep. And prei you would take, you know,

you would maybe do some architecture

review planning. You would maybe do some

sprint planning and then people would

take tickets off the queue and they

would go build them. And then you would

make a pull request and people would

review it and you would run CI checks

and then you would send it to prod and

then it would make contact with your

users and your users would complain

about stuff and that would go to your

support team and back into your work

tracker and it would crash and you would

have issues and that would go into your

monitoring stack and that would go into

your tracker and that was your loop. And

then people would take stuff off the

tracker based on priorities. product

managers, engineering managers,

engineers prioritizing work and then we

go and do that and the first change is

is like this long wind lot lots of

phases and this is also why when like a

developer shifts a bug but by the time

it comes back to you it might be two or

3 months or even longer and by the time

it get fixed it might be a year or two

and you know this is why when you're

using a piece of software it's like that

annoying bug and you talk with customer

support but it's just a very like long

latencies at each each part of the the

factory if you will. Yeah. And the the

step where someone pulls a work item off

a queue and starts working on it is, you

know, couple hours to a couple days

before it actually gets integrated into

everything else and touches user. And

that's in a in a in a great world,

right? Sometimes you go build it and

then you merge it and then it actually

gets released 3 months later. But we're

going to assume we're in a fairly modern

like we're somewhere like the a Netflix

or a meta where engineers are capable of

shipping 100 times a day or a thousand

times a day, but it still takes 2 three

hours to do the work. And now with an

identic factory, what you do is you take

out that person building the thing and

you replace it with an agent building

the thing. And so you have orchestration

to trigger things. You have a sandbox,

you have an LLM, you have an inner

harness, you have an outer harness,

which is like the dev environment you

build for the agent. And maybe you give

it a browser, you give it a video

recorder if you use like things like

cursor background agents. They've kind

of built this outer harness around the

inner harness that is the coding agent.

And then you make PRs with that. problem

there is that like okay now now it takes

10 minutes to do a build instead of two

hours or two days and so now the

bottleneck is code review so okay let's

throw a bunch of AI agents at code

review and let's do agentic testing so

that like we can basically catch a lot

of the easy stuff and humans are only

focused on the most like important

critical core parts of the codebase and

then the next level up of your agentic

factory is you do the top it's like okay

then it gets deployed it goes to prod

and a user complains you just hook your

support queue right up to the agent

someone complains about something agent

tries to fix it and instead of looking

at a ticket and then saying okay go send

you just close that loop and instead

every time something goes wrong you just

get a PR and then every time something

crashes in Sentry or Data Dog or

whatever it goes into the tracker it

gets picked up by an agent and you get a

PR this is the ramp inspect thing this

is the the only difference is like then

you have so much code to review and

people say well let's try turning the

lights off let's just take all the human

testing and review steps out and we'll

say okay cool if users complain then

it's broken and if users don't complain

and it's working and we're not going to

read the code. We're going to use we're

going to treat the whole system as a

black box.

>> So, you said you tried this out uh when

it was like Opus Formula and you you

built the software factory was running

beautifully until it just blew up on

your faces. How do you think of this

model? cuz I I can see an ideal world

where it works, but clearly we're not in

an ideal world. Like where do you think

we are like and could some of this

actually work at some point or you know

like like what what progress are you

seeing right now and and what is the the

today the situation like how much of

this do you believe we can automate or

should we automate?

>> Yep. So if you know me, you follow my

stuff, you know I stand for three

things. Number one is like cutting

through the hype and the jargon and

going trying things and talking to

people who are using things and figuring

out which parts of this actually work

and are valuable. Number two, we talked

about words. I try to find and protect

useful bits of language because I think

it helps us all move forward. And when

you take a useful word like agents or

you take a useful word like software

factory and then you semantically

diffuse it, this is another Martin

Fowler word. You make it mean everybody

likes the word and it all becomes hype

and everyone starts agents means nothing

anymore. agents could be a chatbot, it

could be a Slackbot, it could be a

coding agent, it could be tools in a

loop, whatever it is. So, I like to

protect important useful words and like

help help us all like elevate the

conversation out of that hype and

jargon. And then I care a lot about

going one level down beneath where I'm

generally working. I think there's

always this is the same thing with

context engineering is like I was rarely

actually going and like building LLMs or

understanding or training LLMs but

knowing how they're trained how

transformers works informs how you build

at one layer up and for the software

factory my version of that is I spent

the last couple weeks going really deep

on uh reinforcement learning with uh

verifiable rewards RLVR which is like

this very productionized like it's not

like RH RHF is still like fairly

academic and pure RLVR are is this like

it's a machine in these labs of how we

train these models and I'm studying like

the benchmarks for coding agents and the

techniques for training them and how we

like give it a small problem have it

solve it delete the test changes it made

revert them apply a test patch see if it

passed and then even the frontier this

year we have like we can get into this

later but like frontier code and

marathon these new benchmarks that are

supposed to be like better at evaluating

models's ability to maintain a codebase

over time and write maintainable code um

and they are better But I don't think

they're sufficient. But it's basically

this idea that like the only thing that

made claude code good was reinforcement

learning. And the dimension along which

it got good was like we made a model. We

trained the model and the harness

together. And so the model got really

good at calling the specific tools in

that harness. Really good at reading

files, writing files, searching for

files, all this stuff through doing

these problems. And that was what made

it feel so much better than all the

other CLI coding agents that came before

it. And so people like, "Okay, that was

so much better." And they're just going

to keep getting better. But it's like it

got really good in one dimension. And

the dimension that they're not getting

better in because it's hard, expensive.

Maybe we need to like get a lot more

creative with how we design these these

verifiers and benchmarks is in how do I

make code that in three months is going

to like improve the productivity of

humans and agents, mostly agents, but

humans and agents in the codebase

instead of making it worse over time.

>> And so you think that part is just

missing? We haven't seen too much

improvement.

>> I haven't seen obviously no one knows

what the labs are doing internally cuz

it's all very secret. But I think if we

looking at where the bench the

benchmarks tend to reflect where the

labs are, right? If there is no

benchmark that can convey to me did this

model write code that is going to make

my codebase better or worse. The best we

have is I I think frontier code from the

cognition team is really interesting.

They have like did the test pass and

then they have like two layers of model

review. So they have a judge model that

checks okay is the patch the model made

similar to the patch that is like the

golden answer set. So even if the model

didn't write the exact code that the

benchmark was expecting did was it

functionally equivalent and the next one

is like a like code quality review from

another judge model and like that's

better but it's not it's not sufficient.

And this is why I also think agentic

code review is like yes it will catch

things and it will raise your floor but

I don't believe like the model writing

the code is the same model reading the

code and if you ask a model hey is this

code good it's going to be like oh yeah

it's great comprehensive it's got unit

tests you've tried this I'm sure and you

say okay review this PR that my coworker

wrote and tell me everything that's

wrong with it I was like oh it has this

problem and this problem and this is

this is sickopantic and they want to

tell you what you want to hear and so

like it's really hard for me to trust a

model to evaluate the quality of of of

code that's written And so I I I have

some ideas on like, okay, can you build

a benchmark where the model builds 20

features in a row and maintains the

codebase the whole time and it doesn't

know what features are coming. You treat

it like a real product team where you

don't know what you're going to build

next week until you get there and you

find out what's most important and then

can we try to evaluate like can we build

a problem like that that's hard enough

that most frontier models fail by issue

six or seven. Is it fair to say that you

know like we've had the software factory

like before AI it was just like lots of

loop it was like the the PM giving the

ticket to the dev the dev building it

deploying to production user customers

using it customer support getting

tickets and then you creating PM

triaging and it kind of goes around like

in this loop is it fair to say that the

software factory of how a company a team

builds and maintains software that is

changing because now everyone's

replacing some parts of it, you know,

maybe the the least advanced teams will

just be devs are starting to use cloud

code or codecs to write faster. They're

not spending as much time on there. Some

others are also having the deployment

the feedback. Some some actually have

the agents already oneshotting bucks. So

like is it fair to say that that the

software factory is just is just

changing everywhere maybe at different

speeds but everyone I think every team

who is building production software

they're like they're frantically

experimenting trying and everyone's at a

different pace. You'll have the AI

native starters where most of this will

have agents in them and you'll have the

the laggers who are or more more

cautious ones. They have agents in a few

places but not in the others. Well, and

I think that's the key is like if you

want to do loops engineering, you should

build one loop at a time and you should

keep them small and contained.

Basically, I think everything except

stop reading the code is really good

advice. Take support tickets and turn

them into tickets in your system and

then maybe turn those into PRs. Great.

The advice that I have and like what we

kind of like are chasing at human layer

is like how can I add another checkpoint

in that factory? So instead of having

one human re view point where you're

reviewing PRs and sometimes they're 100

lines and sometimes they're a thousand

lines but it's quite a lot of effort for

especially if it's bad especially if it

needs rework. It's quite a lot of effort

for a human to be like okay this is

wrong go change it in this way and then

you loop back to the agent and then you

come with another one and like doing a

lot of loops on there once once the

direction has been committed to it's

really hard to steer off like you're

better off just kind of restarting from

scratch. How do you build like controls

and mechanisms around that? And then my

take is like if you do a little bit of

human agent planning and like discussion

before you hand it to the impletor

whether it's I mean planning and specs

whatever you want to call it again this

is spec driven development is another

word that has become kind of very like

muddled as far as what it means but

basically how can we spend an hour

before we start building so that the PR

when we read it only takes 20 minutes

because the code is perfect instead of

not touching it just literally saying

every user reported issue becomes a PR

through the loop and then we read that

PR and it takes six hours because

there's back and forth and we have to

make changes and things. It's all I'm

all about like let's find leverage. And

so you basically you have three options

in the software factory world. If you're

going to go all in on aentic software

factories, you can turn the lights off

and just let everything flow and pray

that you don't create too much slop and

pray that the next generation of models

comes fast enough before you create a

giant pile of ash. you can slow way down

and read every PR and read every line of

code. Uh, and then you're only going to

really get modest benefits from AI

because that becomes I I think you

should expect maybe 30 to 50% lift in

productivity is kind of what I see when

we go into teams

or you can find the right leverage

points where humans can actually an hour

spent over here in planning can save you

four hours in in implementation in terms

of fixing and going back and and getting

the design right. And that's what I call

like seeking leverage. If you can find

the right leverage points for the agents

to guide the work, then you can actually

move like two to three times faster

while maintaining a like 99% like

accuracy to like if the humans were

carefully writing this code by hand, how

would it come out?

>> Now jumping a little bit back to ideas.

I will come back to this. This was

earlier maybe it was last year but you

had the research plan implement. Can we

talk about the original research plan

implement framework and then also what

you've learned about this approach? what

what you got wrong about it.

>> Yeah, sure. Yeah. So, um I mean the

first time we talked about RPI was in

August of 2025. Um and it was basically

like the research was this thing of

like, hey, before you go build anything,

go read lots and lots of code. Use a

bunch of sub aent sub aents in parallel,

understand all the code. It was this

technique that like worked really well

for hard problems in complex code bases.

You just ask Claude uh to do a thing

that that's it would read three files

and make a change. It would have no