HomeVideos

What 6 months of AI coding did to my dev team

Now Playing

What 6 months of AI coding did to my dev team

Transcript

378 segments

0:00

If you're building software in your

0:01

business, your dev team is changing

0:03

faster than you realize. Not the people,

0:05

but the work itself. I'm running a 20

0:08

person software team at Wu. And over the

0:10

last 6 months, I've watched something

0:12

strange happen. The bottleneck in

0:14

software development has changed. It's

0:15

not where it used to be. It's moved. And

0:17

if you're hiring developers right now or

0:19

you're trying to figure out why your

0:20

team isn't shipping faster, you need to

0:22

watch this.

0:25

Here's how software development used to

0:27

work. You hired a team of developers to

0:29

write code. You measured their output in

0:32

lines committed in tickets closed in

0:34

features shipped. The craft was in the

0:36

code itself. When I started building Wu,

0:38

that's exactly how we operated. We hired

0:40

people who could turn tickets into work

0:42

in features. Code review was the quality

0:44

gate. If it passed code review, it

0:46

shipped. But something fundamental has

0:48

shifted. We started using AI coding

0:50

tools, clawed code, cursor, and the

0:52

entire rhythm broke. The code started

0:55

arriving faster than we could process

0:56

it. So the job actually changed

1:01

three months ago. One of our senior

1:02

engineers came to me visibly frustrated.

1:05

He'd spent like 3 days reviewing pull

1:07

requests from an engineer, a junior

1:08

engineer that had used clawed code.

1:10

There was like thousands of lines of

1:11

code, but the application worked. But he

1:13

looked at me and he said, "I didn't

1:15

actually read all the code. I couldn't

1:16

read all the code. There was too much of

1:18

it. What do I do now?" And that question

1:19

made me think cuz I'd been feeling the

1:21

same thing, but I just couldn't name it.

1:22

Around the same time, I came across some

1:24

findings from a retreat called Thought

1:27

Works, which is basically senior

1:28

engineers from the world's biggest tech

1:30

companies brought together to find out

1:32

what happens when AI writes the code.

1:35

And they didn't leave with answers from

1:36

this retreat. They're left with a map of

1:38

fault lines, places where traditional

1:40

software development is cracking right

1:42

now. And reading through it felt like

1:44

reading my own history from building Wu.

1:46

You've got the cheating agent problem

1:48

where AI writes broken code and then it

1:51

writes broken tests to validate the

1:53

broken code. You got the productivity

1:56

experience paradox where your developers

1:58

are more productive but more miserable.

2:00

I actually seen this in our team. The

2:01

migration of code review from code

2:03

review to specifications. We' actually

2:05

started writing more more stricter

2:07

specification dogs without even

2:08

realizing why. And now we know what I'm

2:10

seeing and what we're struggling with in

2:12

our in our own teams. looks like it's

2:13

something that's happening also in the

2:15

biggest tech companies in the world. The

2:16

work is migrating. The skills that

2:18

matter are changing. And if you're

2:20

hiring developers or you're managing a

2:21

tech team and you don't see where the

2:23

work is going, you'll end up with the

2:24

wrong people doing the wrong things. So,

2:27

let me show you what's actually

2:28

happening.

2:31

Here's what nobody tells you about AI

2:33

writing code. The engineering quality

2:35

doesn't vanish. It just moves upstream.

2:37

Think about a normal user story. I want

2:40

to upload a photo. Your developers will

2:42

know what that means. you know, JPEG or

2:43

PNG uploaded to a site and you got a

2:45

progress bar, you know, cuz cultural

2:47

context fills in the gaps. But an AI

2:49

doesn't have that context. You need to

2:51

be really specific. I read a story

2:52

recently of a developer asking an AI to

2:55

write a notification system of some

2:57

kind. You know, simple request, it

2:59

worked beautifully in testing when it

3:00

was built and then it went into

3:01

production and started sending like

3:03

50,000 emails in a few minutes. Turns

3:05

out that there was no rate limiting set

3:07

up in in the specs. See, the engineering

3:08

rigor that we used to apply after the

3:10

code was written now needs to apply

3:12

before in the specs, before a single

3:14

line of code has been written. We've

3:16

gone back to techniques that felt dead.

3:18

Now we need to go back to structured

3:20

requirements, state machines, decision

3:22

tables, extremely detailed PRDs. It's

3:25

the kind of formal documentation that

3:27

agile was supposed to kill. But here's

3:28

the thing, all that documentation makes

3:31

AI incredibly effective at writing code.

3:33

When we feed an agent a state machine

3:35

that displays and shows exactly what

3:37

states are possible within the

3:38

application, the code it generates is

3:40

almost always correct. It's crazy

3:41

because the specification became the

3:44

product. The code is dispensable. Think

3:47

about it. If you've got a perfect test

3:48

suite and you decide to rewrite your

3:50

backend from NodeJS to Rust, all you got

3:52

to do is ask it. You just feed the tests

3:55

into the agent and you say, "Do the

3:57

rewrite from Node.js to Rust and make

4:00

sure that these tests pass." and the AI

4:02

will get to work and make sure that it

4:03

tests itself on the work using the

4:05

tests. So then the output will always

4:08

work. This is a complete inversion. So

4:10

if you're hiring developers right now,

4:11

the question is isn't can they write

4:14

clean code. The question is can they

4:16

write a specification clean enough that

4:19

an AI can't misinterpret it. Can they

4:22

write a set of tests for a test suite

4:24

that catches hallucinations before

4:25

production? You see, those are different

4:27

skills and most developers don't have

4:28

them yet.

4:32

There is a layer of work in my team that

4:34

doesn't quite have a name yet. It sits

4:36

between writing code and shipping to

4:37

production. I call it supervisory work.

4:40

Basically breaking down problems into

4:42

agentsized chunks. Knowing when to let

4:44

the agent run and when to step in,

4:46

fixing the output not by actually

4:48

rewriting the code, but by rewriting the

4:50

prompt. Here's what surprised me. My

4:52

team is currently split into two groups

4:54

primarily. Group one is the more senior

4:56

people who understand the whole system

4:58

architecturally and they're drowning

5:00

because they're spending the majority of

5:01

their time doing code reviews. And then

5:03

you got group two, the more junior ones

5:05

that are spending their time writing

5:07

code using clawed code and other AI

5:09

tools at 10x the speed that they were

5:11

doing it before. Basically generating a

5:13

lot more code. But that code doesn't

5:14

ship itself. It needs architectural

5:16

review. It needs to fit into our

5:18

structure. It basically needs check-in

5:20

before it can be deployed. So the most

5:21

senior engineers we have have become

5:23

traffic controllers too busy reviewing

5:25

AI code and other people's code to

5:28

actually build anything themselves. The

5:30

more junior developers, they're

5:31

thriving. No muscle memory telling them

5:33

to write code in a specific way. So

5:35

they're using AI tools like a teammate,

5:37

not a threat to their identity. In the

5:39

early days used to hire a junior and

5:41

they used to take 6 months or so of

5:42

draining the rest of the team for that

5:44

junior to become productive. Now a

5:46

junior can get they can be writing

5:49

useful code into production within a

5:51

week. But I think there's this this

5:52

there's this danger level of midlevel

5:55

developers. The guys that have a few

5:56

years of experience, they're used to

5:57

writing code in the way that they

5:59

normally write code before AI existed.

6:01

and retraining them to use AI in an

6:03

effective way is extremely difficult

6:05

because they need to change their

6:07

mindset around instead of focusing on

6:09

the syntax and the code that they're

6:11

writing around a detailed implementation

6:14

request on how they talk to the model to

6:16

achieve the result that they need. So

6:17

here's what I'm learning as a CEO hiring

6:19

developers and running a development

6:21

team. The job description has changed.

6:23

If you're looking for people that can

6:24

write code fast, you're looking at the

6:26

wrong skill. You need to look for people

6:28

that can architect systems, write

6:30

unambiguous specs, and supervise AI

6:34

agents. And that's a completely

6:36

different person to the old school

6:37

developer that we hired a few years ago.

6:41

Let me tell you a story. Last month at

6:44

around 2:00 a.m., one of our servers

6:46

broke. It was spitting out error 503.

6:48

Service unavailable. our own call

6:50

engineer at the time, you know, a guy

6:51

pretty sharp and uh really really

6:53

capable. He put this into AI to see what

6:56

he needed to do. The AI tool looked at

6:58

the error, read the documentation and

7:00

said restart the server. So our engineer

7:02

restarted the server. Uh and then a few

7:04

minutes later after restarting it

7:05

crashed again. So then he repeated the

7:06

process and AI said restart the server.

7:08

So he restarted the server again and

7:10

then he repeated the process again and

7:12

AI said restart the server. By the time

7:14

he'd escalated to a senior engineer,

7:16

he'd restarted the server six times. The

7:18

senior engineer looked at the logs for

7:20

about 30 seconds and knew exactly what

7:22

the problem was. Turns out the database

7:24

connection pool was full because of some

7:25

batch cron job that was running in the

7:27

background. You see, that's not

7:29

documented anywhere. That's tribal

7:31

knowledge. That's lived experience. And

7:34

an AI doesn't have that, or at least

7:36

ours didn't. It sees 503, it reads the

7:38

manual, restart the server. Typically,

7:40

that's what you would do. But without

7:42

that other knowledge, without the other

7:43

bit of information, you're just in that

7:45

cycle of restart the server, it goes up,

7:47

it crashes, restart the server, it comes

7:49

up, and it crashes again. This is why I

7:51

think all this hype about self-healing

7:53

systems is rubbish right now. Unless you

7:55

really got all the knowledge in the AIS

7:59

context. All the knowledge that a human

8:00

would have, all the knowledge that a

8:02

senior human would have. To make an AI

8:03

agent effective during an outage, you

8:05

need to build what the Thought Works

8:07

Retreat called an agent subconscious.

8:10

basically a knowledge graph of every

8:12

incident, every weird edge case, every

8:15

bit of undocumented institutional

8:17

knowledge that lives in your senior

8:18

engineer's heads. We're starting to

8:20

build this at Wu. Every time something

8:22

breaks, we document not just what

8:24

happened, but how we fixed it and what

8:27

would be in a senior's engineer's head

8:29

when fixing it. You know, that bit of

8:31

information that is just known by the

8:32

human. We document that. But then

8:34

there's a valid point here is that

8:36

there's another problem and that's that

8:37

AI agents are trained to be helpful.

8:40

They are yesmen or or woman you decide.

8:42

And the thing is during an outage you

8:44

don't want a yes man. You want somebody

8:46

to challenge your assumptions. One

8:48

engineer at the retreat said that we

8:50

need something called angry agents. Ones

8:52

that are specifically prompted to poke

8:54

holes in your theory because otherwise

8:56

the human and the agent will just agree

8:58

with each other while the server burns.

9:00

And here's the point. If you're running

9:01

a tech company and betting on AI to make

9:03

your team faster, you need the pre-

9:06

requirements first. you know,

9:07

documentation that captures how things

9:09

work. Seniors who can architect, not

9:11

just code, and a system for architecting

9:13

institutional knowledge before makes

9:15

people forget how things work.

9:19

So, here's what I've learned from

9:20

running a dev team in the age of AI

9:22

agents. The work isn't disappearing.

9:24

It's moving from execution to

9:26

supervision. The bottleneck used to be

9:28

typing code into a file. That bottleneck

9:30

is now gone away. Now it's decision-m

9:33

verification and starting off by

9:35

specifying clear intent. Think about

9:37

graphics programming. In 1992, engineers

9:40

hardcoded the maths to draw a single

9:43

polygon, calculating the exact pixel

9:45

positions. By 1994, the GPU arrived and

9:48

the hardware did the polygons

9:50

automatically. If you insisted in

9:52

handcoding polygons in 1995, you weren't

9:56

a specialist. You were obsolete. And the

9:57

graphics engineers from those days

9:59

transitioned to lighting engineers,

10:01

animators or physics programmers. They

10:03

stopped telling the computer how to draw

10:05

a triangle and moved on to telling it

10:07

how light reflects off a street, for

10:09

example. Nobody handcodes polygons

10:12

anymore. We all work in game engines. I

10:14

think software engineering is hitting

10:16

that exact point right now. So, if

10:17

you're hiring developers or you've got a

10:19

team of developers, here's what to look

10:20

out for. Don't look for people that can

10:21

write code. Look for architectural

10:23

thinking. Can they write an a spec that

10:26

is not open to interpretation? Can they

10:29

write tests and design a test suite that

10:30

actually becomes the product? And can

10:32

they debug a system that they didn't

10:34

write? At Wu, that's what we hire for

10:36

now. But here's what keeps me up at

10:37

night. In the past, code reviewing

10:39

wasn't just about catching bugs. It was

10:42

also about how developers learned the

10:44

system. So, if agents write all the code

10:46

and your team stops reading it, then

10:49

they become strangers in your own

10:50

system, strangers in your own code base.

10:52

When something breaks at 3:00 a.m.,

10:54

they're staring at code that was written

10:56

by a machine, trying to reverse engineer

10:58

the logic while your customers are

11:00

screaming. I think the solution is to

11:01

force AI to lay out all the

11:03

architectural decisions that it makes

11:05

when it writes the code and then arrange

11:06

for meetings with your senior engineers

11:10

to review these decisions that the AI is

11:12

making essentially. So there's like a

11:14

symbiosis between the architectural

11:16

decisions that the AI is making and your

11:18

team. So your team is fully aware of

11:20

those decisions. And all this has to

11:21

happen before the agents write the code

11:24

because you've got to schedule time to

11:26

understand your own software now. It

11:27

won't happen automatically. The speed of

11:29

AI demands this. And if you're running a

11:31

tech company or you're employing

11:32

developers, that's the shift that you

11:34

need to see coming.

11:37

So look, if you're running a software

11:39

team or you're thinking about building

11:40

one, the ground is shifting. Senior

11:42

engineers are drowning in code reviews.

11:44

Junior engineers are smashing out code

11:46

at a 10x speed at a rate that the

11:48

seniors can't keep up with. And the

11:50

mid-level guys are trying to get their

11:52

head around starting to write code with

11:54

AI. I think the companies that will win

11:55

are the ones that will manage to retrain

11:57

before it's too late. I am documenting

11:59

everything I'm learning building Wii U.

12:00

The systems we're putting in place, the

12:02

mistakes we're making, the changes we're

12:04

making, what works for us, what doesn't

12:06

work. If this is for you, hit subscribe

12:08

and head over to axelmiss.com

12:11

and join my newsletter where you'll get

12:13

weekly content like this. And if you're

12:15

building software right now in this

12:17

exact moment of transition, trying to

12:19

figure out what to do with your dev

12:20

team, you're not alone. The best tech

12:23

companies aren't panicking, they're

12:25

adapting. Be one of them. See you in the

12:28

next one.

Interactive Summary

The video discusses the evolving landscape of software development due to AI coding tools. The traditional bottleneck of writing code has shifted, with AI now generating code much faster than humans can review it. This has led to a need for new skills, focusing on architectural design, writing unambiguous specifications, and supervising AI agents rather than just writing code. Senior engineers are overwhelmed with code reviews, while junior engineers are highly productive with AI assistance. Mid-level developers struggle to adapt to this new paradigm. The speaker emphasizes that the future of software development requires a shift from execution to supervision, with a focus on clear intent, robust specifications, and effectively leveraging AI as a tool rather than a replacement for human expertise. Companies need to adapt by retraining their teams and documenting institutional knowledge to keep pace with AI's speed.

Suggested questions

8 ready-made prompts