There's a misconception about the way

you approach optimization. You run a

profile, you identify the big parts, you

make some changes, you measure

statistics. I've worked with many

extremely good optimization people and

that is not how it is done.

>> Assembly language is not all that

complicated, right?

>> All of the JavaScript libraries, the

DOM, CSS, React, assembly language,

maybe there's 20, 30 instructions you

might have to learn total. If you can

vertically center a div in HTML, then

you can probably learn assembly

language, I would say.

>> How are you using AI tools, if you're

using them at all?

>> We are not using them at all. The reason

that I want to program things in a game

is because I want to program them. If I

just wanted an AI to program them, I'd

just go get the Unreal [laughter]

Engine.

>> What are things that you think are

non-negotiable for someone to be a great

software engineer?

>> So, I find there's a lot of received

programming wisdom that's just nonsense.

Clearly, no one's ever tested it. And so

in order for it to be received wisdom,

you should

Why do most devs not care about writing

performance software and should we?

Today's guest Casey Moratory spent the

last decade arguing that we should. He

also says that most software out there

runs tens to 100 times slower than it

needs to. [music] Today we discuss why

the focus on performance took a backseat

across the industry and why Casey thinks

the tide is finally turning. why you'll

want to learn reading assembly if you're

serious about high performance code and

why it's less scary than it sounds. The

saying premature optimization is the

root of all evil, why Casey says that

the majority of people use it to avoid

thinking about performance when they

really should and many more. If you want

to get better at writing faster software

[music] and become a better engineer

while doing so, then this episode is for

you. And if you're one of the people who

hit a like on this post by Ryan asking

for this episode to be more about

programming than about AI, this episode

is also for you. This episode is

presented by antithesis. If you work

with agents, your job is no longer just

writing code. It's also specifying and

testing it. [music] Antithesis is the

most effective method of verifying

agentic code today. This episode is

brought to you by Sentry. You probably

already know what Sentry is because

you're a developer. If not, just ask a

dev and they'll tell you. I use Cry to

monitor the back end of the pragmatic

engine for any and all events and

errors. Of course, Sentry doesn't only

do errors. They also have logs, replays,

spans, profiles, metrics, and more

because they're all connected to the

same trace. One new capability Sentry

has I'm really liking is its ability to

fix errors. Let me show you. Here's a

list of errors on my admin back end.

There's a recent error on O that I want

to check out. Let's have Seir run an

autofix for us. Seir is Century's AI

debugging tool. First, it generates a

root cause analysis. It's finding some

problem with HTTP versus HTTPS URLs.

Cool. Now that we know what's going

wrong, Seir can create a plan on how to

go about fixing it. I could go and edit

this plan, but I'm happy with it. So,

let's create the actual code fix. Here's

a code fix that Seir generated. Assuming

it looks good, and in my case, it does.

Let's draft the pull request. And boom,

the PR is created, ready to merge. What

I love about Autofix is how Senu went

from showing a list of errors inside my

application to offering me a fast way to

fix it and close the loop while I stay

in charge of this bug fix the whole

time. Debugging got a whole lot faster

and a whole lot easier. Check out Sentry

at centry.io/pragmatic io/pragmatic and

start detecting errors, diagnosing your

root causes, and fixing issues and

regressions today. All right, Casey,

welcome to the podcast. It's so nice to

have you here.

>> Thank you so much. It's great to be

here. Thank you for the invitation.

>> Now, I want to go back when we start to

the beginning. How did you get into tech

programming, computers?

>> Well, I guess computers, it's like very

very early on. Um, my dad was a

programmer at Digital Equipment

Corporation, which is a company that

people will know if they studied

computer history, but would not know if

you just looked at the landscape today.

They're they're completely gone, right?

Uh, they got absorbed partly by Intel,

partly by Compact, I think. Uh, there

was, you know, they kind of got uh,

broken up. At that time, it was kind of

a really big computer manufacturer. You

know, computers like the PDP11, that's a

digital equipment corporation computer.

uh the vax like things that you may have

heard of in computers.

>> Oh, these were these ma massive main

frames.

>> Yeah. Uh mini computers as well. So like

smaller also sometimes than main frames

like the kind of next step down, right?

And so in general that that era, my dad

was a programmer there. He would later

end up at Intel because, you know, like

I said, parts got acquired. He never

actually left his job. He just ended up

at Intel through kind of uh digital's

eventual demise. But as a result, we

always had computers at home, even

though at that time, you know, that

might have been a little bit odd. Um,

you know, I I learned to program when I

was seven, which would have been in

like, you know, 1982 or something like

that. And so at that time, you know,

maybe you might have an Apple or

Commodore kind of computer at home, like

some kind of early computer. I don't

remember the exact line dates of those

computers, but most people didn't. And

it was only until a little bit later

that you would and you probably wouldn't

have had a programmer in your household

to teach you more importantly, right?

So, uh, so I learned really early on and

that's when I got into computers. How I

got into games was, uh, I ended up

randomly interning at Microsoft and I

met people there and like I went and,

you know, sort of went off into games

through them. That was that was how that

happened if that makes sense.

>> But wait, how is the Microsoft/game

relationship? That's not kind of a

given, right? Microsoft is not

They [laughter] have one game, right?

It's Flight Simulator.

>> Uh yes, at that time it would it would

have been very weird. Uh the reason that

happened was a guy called Chris Hecker

who uh a lot of people don't really know

his history because there was sort of

two waves of bringing games to Microsoft

Windows. Um cuz I mean now it's it's

funny to think about now because people

think of like what platform are you

going to play games on if you have a PC?

You know, Windows is like the default.

Linux is now an insurgent to that. But

Windows is the default. And so you think

about how did that happen? Be you know

uh because that wasn't the case if you

were back in the early days. Uh

Microsoft Windows not a gaming platform.

It's almost nothing on it. It's like

solitire and mind sweeper and a few

other uh sort of games.

>> How did it happen?

>> So uh what happened the reason for this

this kind of gets into a technological

reason. The reason for this is that it

was very hard to actually produce images

that could be displayed on the screen

quickly.

And to understand why this is is like

its own kind of topic, but in general,

you can just imagine you have these this

operating system running, which is

Microsoft Windows. It's controlling the

graphics card. It's often running at a

fairly high resolution uh compared to

what a game might want to run at. You

have to negotiate with it in order to

display your bit map in some way that

won't destroy all the things that it's

trying to display. So on so forth. early

versions of Windows uh up through like

Windows uh three uh Windows for workg

groupoups let's say if anyone remembers

that name.

>> Was it after 3.1?

>> It was 3.51 I think is what it's called

or maybe just 31. Yeah, there's NT351.

No, so it's like I think you're right.

31 3.1 I don't know something like this.

Yeah, Windows for workg groupoups was uh

you know around that time. that version

of Windows, which is in the 3 series,

didn't really have a way to quickly use

the CPU to fill pixels, which is what

games need to do, right? There's there's

no GPU acceleration really at this

point. We there's a little bit we could

talk about, but it's not mainstream.

It's not in consumer. So, they they need

to be able to to do this sort of thing.

They need to be do it in a double

buffered way so they can draw to a back

buffer and then show it to the screen.

And that has to happen very quickly. And

there just wasn't a way to do this in

Windows. In Windows, you had to kind of

go through uh this API where you would

produce sort of a bit map that wasn't

necessarily in the right format for the

display you were using and then it had

to do a translation from that bit map to

the other one when it displayed it. All

this sort of stuff. So games on Windows

like things like Doom, they're not

coming to Windows, right? Uh that kind

of future was not in the works uh for

Windows. And that was kind of what some

people in Microsoft wanted to change.

And one of the people who brought this

change about was a guy named Chris

Hecker. and he was like, "Okay, we could

actually just make a library that did

the fast blitz to the screen so that we

could have a way that people could do

these draws and get them on the screen

quick enough to make gaming viable.

Would it be 100% as fast as DOSs?"

Probably not. But could you actually run

uh you know, some of these new games?

You know, Wolfenstein 3D I think would

have been out at this time. Doom was

kind of on the horizon and that sort of

thing, right?

>> Yeah. And so he started this project

that he was not supposed to do. He did

not have the authority to do this called

wing,

which I believe at least stand for like

win graphics or something like that.

Total skunk works project. He had cover

from his manager. I'm I'm assuming I can

say all this stuff now because it's

ancient history. He had cover from his

manager. His name guy's name was Michael

Edwards. And Michael Edwards basically

just kind of ran cover for this, which

is a thing that probably wasn't going to

fly because they were in a division at

that time which would have been

Microsoft Research today kind of. It was

called at Advanced Technology was the

early version of Microsoft Research. You

were not supposed to be shipping core

libraries for Windows like it had

nothing to do with that. Long story

long, what ended up happening is that

product did ship. Win, I guess you

wouldn't call it a product. It's an

add-on for Windows. did ship and it was

the first step towards DirectX.

>> Uh people forget that Wing was the first

way you did this and then eventually we

had DirectX and DIB sections and Win 95

and all that sort of stuff. When I went

to Microsoft, amusingly the person I was

supposed to be reporting to as an intern

was Michael Edwards. When I showed up

first day along with the other interns,

uh guy named Rudy, a guy named Rajie, uh

we were all supposed to report to him,

right? because you get a couple interns

in under one, you know, sort of manager.

We show up and we're just taken aside,

you know, after some kind of, you know,

stupid HR orientation thing that was,

you know, lame as it always is. And and

we get taken aside by somebody. I don't

remember the lyrics. They're like,

"Look, bit of bit of a problem. Uh the

person you interviewed with and were

supposed to like report to, uh they they

aren't here. Like I don't remember

exactly how they put it. So, uh, you're

going to go talk to this other guy and

they'll find something for you to do.

Turns out the wind thing had blown up

the previous week and Mike Edwards had

stormed out of the building and had not

been seen since. That is what actually

happened. [laughter] So, that's I arrive

at Microsoft, the person I'm supposed to

intern with, he just he just flamed out

and left. No one knows when he'll be

back. He did end up coming back like a

week after I got there. Uh, but he was

kind of moved over to a different

division. like they you know there was a

bunch of like triage work done there to

figure out what's going on. So that that

was my experience but suffice to say it

meant that I was right in sort of like

ground zero of games on Windows. So I

met Chris Hecker. I got to talk to a

bunch of people there. He actually took

me out to see one of my childhood heroes

Ron Gilbert who is the guy who did the

scum engine. Yeah.

>> He knew all these guys because working

on WG he had gone out to see a bunch of

game developers and like work with them

and this sort of thing. So, I got to go

to Humongous Entertainment, which was

Ron's new company. He gave me a a secret

of Monkey Island, uh, mouse mouse pad, I

remember. Uh, so anyway, it was it was

really cool. And that's how I ended up

getting into the game industry was was

was through Chris Hecker. And he's kind

of an unsung hero of getting games on

Windows because, you know, he he just he

wasn't out there making his story known,

but but, you know, I am now, I guess.

>> But it's it's so interesting to hear

these stories. Obviously, now you can

share it, I'm sure. you know, like for a

while this would have been like only

with within the inner circle, but the

fact that you know, of course, DirectX

was a huge

>> success and and it it did like as far as

you know, from my vantage point, huge

reason why games are big on Windows, but

now here's someone who just ignored

wasn't asked anything, just was doing

something, got into conflict, fights,

and just like pushed an idea. It's more

than you think because there were three

people who really were the core people

who pushed DirectX, meaning

institutionally pushed it. There are

tons of programmers like Todd Laney who

did, you know, really important core

work. It never would have shipped

without people like him. So, not on the

program side, I'm talking about

institutional side. It's Angstrom and

Alex St. John. I don't remember which

one is either or Angstrom

was the tester on Win. So he came from

that team. So the start of DirectX, one

of the core members of DirectX was on

the Windy team. So it's a direct

lineage. It's not even like an unrelated

push. So WGI really was the start of it.

And then DirectX was kind of the actual

full blossoming into an org with

Microsoft's blessing at that point, you

know, that that actually became powerful

internally. And then after after this

Microsoft internship and getting exposed

to all these folks, you actually went

and and built games tooling, you then

started your own studio as well, right?

So like how how did that sequence?

>> I guess there's a couple of steps in

there. I worked at a startup with Chris

Hecker that didn't end up doing anything

interesting. Then I went to a company

called Gaspowered Games, which was

actually a Microsoft guess they're not a

Microsoft studio, but their publishing

deal with was Microsoft. and uh they did

a game called Dungeon Siege which is

kind of a you know I don't know it's not

a particularly well-known title. From

there I went to Rad Game Tools and

that's where I stayed for quite some

time. Uh I did their character animation

system that was a very popular product

that ended up getting used in in lots

lots of games. It's still used to this

day, much to my surprise. Uh because

that's a very very I haven't worked on

it since 2004, [laughter]

but I guess other people had maintained

it and some studios just kind of

integrated into their you could get

source licenses. So I guess some studios

just integrated it into their pipelines

and have never removed it. Uh and they

just maybe keep updating it um to to

keep it working the way that they want.

So anyway, uh that's what I did there.

And then afterwards I' I've been

independent since then. I just have a

company called Malle Rocket uh where we

do various stuff. I've done you know

contract work for for people through

that. We now do like the Substack

through that where we do educational

materials. So I've kind of just done

random stuff uh since then. Although I

have done some work on games. I noticed

your checklist you were talking about

the witness.

>> Uh obviously that one was an actual

specific title that I worked on but that

was that was mostly just because it was

a very big project and I was you know

I'm friends with John so I was just

trying to do some helpful programming on

the side. I did some stuff on how the

movement system worked. I thought there

were some interesting problems that we

could solve there for games. And so that

was a that was a really fun project to

work on.

>> Yeah. And then today you're you're doing

educational stuff on a bunch of

subprogramming performance on your

substack. And what else are you busy

with?

>> So we do actually have a an unannounced

project that we've been working on uses

up sort of the rest of the time

[laughter] that that I have if that

makes sense which is not always so much.

um and that we will we're hoping to

announce it sometime soon. Um but it it

it's not quite out yet. Believe me, I

will I will you will hereby I will send

you an email as soon as we have an

actual announcement. But uh given the

fact that it is kind of like a split

time sort of thing for us because we're

pretty focused on making sure the

substack is good and all that sort of

stuff. We're we're trying to keep it

fairly tight lipped until we actually

know we're mostly done because we don't

know how much time we can always devote

to it if that makes sense.

>> Yeah. No, it's it's it's it's pretty

typical games related things, right?

Like tight lipped until you have

something for good reasons.

>> Well, sometimes people play the other

game. They go like, "Look, we're going

to day one we're going to be very loud

about this and try to build a community

around the development of the game and

all that sort of stuff." And that's

great. Uh so, you know, that's another

route you can go. But if it's not your

full-time thing, if you have, you know,

other responsibilities, that doesn't

seem great, right? because you you don't

have any insight into how much time you

will actually be able to devote to it,

right?

>> Yeah. And on on Substack, it's called

computer enhance and you started it with

performance related topics and that's

how we started to talk I think about

three years ago when when you already

had a Substack and we we had a direct

message conversation. I remember you

messaged me saying, "Hey, Gerge, why do

you think in the industry

people software developers just don't

really focus on performance?" I I think

you specifically wrote

>> I did

>> uh you were saying how there's there's

little emphasis on performance. There's

even though there seems to be

overwhelming evidence that performance

is critical to the bottom line of of

most software and I wanted to ask you we

we've had a good back and forth on this

and actually I think initially I told

you like oh here's why you don't need to

care about performance when you're

building I don't know distributed

systems or like but since then what have

you learned? Why do most developers

don't care or even most companies,

teams, engineering teams not care about

performance all that much? It's a really

good question and I think I do think

your answer at the time if I remember it

correctly is certainly an accurate one

for some subset of of industries uh

which you know you said something along

the lines of look a lot of these pieces

of software that you're seeing the user

isn't the purchaser right like you were

you were like this is some kind of thing

where you know somebody very high up is

going to look and say we need software

for managing HR they're going to look at

the cost of the software they're going

to look at the compliance terms terms of

the software, the legal liability,

whatever, right? And then they're just

going to make a purchase decision on

that sheet. They're not in there looking

to see whether it takes like, you know,

whether there's a 30-cond pause every

time you want to try and access

somebody's record, right? And I think

that's very true. Like unfortunately,

the situation for a lot of enterprise

software probably is that way. So maybe

an individual might well be upset about

the performance of that software. And I

certainly hear from people all the time

who are upset about the software that

they use. They might not be in any

position to change it. I think that's

one thing. There's thing two which is

that in a lot of cases you simply have

monopoly effects. Uh you know people

aren't right now realistically going to

challenge the social networks that

currently exist. For example, people

have tried. It's very hard. you know,

Blue Sky and Threads have tried to

assail X and you know, you've got

Facebook and Instagram and Tik Tok and

they kind of just own those spaces,

right? And it's very hard to push into

those because of these like network

effects and maybe performance could be

part of a package where you try to take

on one of those players. like, hey, look

at how much more responsive our thing is

than theirs. Might be a nice plus, but

that's not going to be sufficient. If

you just show up with no plan for how

you get adoption, no plan how you get

big influences over there, all that sort

of stuff,

>> it can't sell a product on its own into

a monopoly space, right? If you're just

talking about apps that someone can

choose to download, maybe you've got a

shot there. But those are just they're

forming a smaller and smaller subset of

what software is, right? and this bigger

and bigger subset is like these monopoly

platforms you go on uh to to sort of

work with. So I'd say that's another

thing. Thing number three is I think now

people sort of are caring about

performance more. I think over the past

decade uh the people uh including myself

but many many other people who have been

saying that this is a problem uh have

actually had some effect like I don't

think that it was a waste of time. I'm

seeing a lot of new emphasis on

performance, people talking about

performance, people posting benchmarks

on things. And so I actually think that

the third thing is well actually it kind

of does seem that pointing at this issue

and saying this is something we should

be doing better has not been completely

uh a waste of time. I I do see things as

sort of starting to turn around a little

bit. I also see people attacking major

product categories now with

performance-based pitches. Things like

File Pilot or the Blick video editor,

like things like this that have been

coming out lately where it's like, oh,

really performant software to try to

take on uh incumbents in a space and

they've been getting traction. So, I

think that's also a really good sign.

>> Yeah. And I guess on on this last

category, a really good example in the

developer community is bun versus npm

where bun just said like okay we're like

10x or 20x or 50x faster and devs are

like what this possible and then it was

so they there was this outrageous claim.

I almost I I wonder if like you need to

have these outrageous claims cuz devs

started to pay attention cuz it was 10x

faster in many you know categories. You

know, linear versus was gyra is also a

good example where linear have this

benchmark of okay, they have 300

milliseconds for any action and gyra of

course we know is is just slow because

they have a bunch of complex you can

explain why but it's slow. They it was

never built for that.

>> Yeah. And you know if you think about

something like a 300 millisecond budget

for an operation 300 milliseconds is

like an eternity in computing, right?

And so if you're talking about like our

pitch is that we're not more than 300

milliseconds, that just shows you how

the bar was so far past where it

probably should have been for something.

And you see it everywhere. You know, I

you go on to programs and you're waiting

sometimes seconds for an operation. And

I don't think people realize just what

an eternity a second is in modern

computing. Uh especially when you're

sitting on networking like that has, you

know, sub 10 millisecond ping times.

Sometimes you're talking about this, you

know, the actual packets had to travel

physical distance to get to this data

center and that was being done far

faster than this very simple operation

that you were failing to do in a

reasonable amount of time. It's just

like we are massively underperforming

and people don't believe it when you say

10x 100x but it's actually true and

we've seen a lot of proof of it as you

point out. I do wonder if one part of

the not really much focus on performance

is that a lot of developers don't know

the the baseline thing. And I I'm

reminded by Simon Ericson, the founder

of Turbuffer, have this has this project

called Napkin Math where he did a list

of mostly networking operations. How

long does it take to transfer one bite

between like two AWS data centers? How

much does a gigabyte take? How much does

a terabyte take? how much does it take

uh to to write an SSD to an N an MVM an

N N N N N N N N N N N N N N N N N N N N

N N N N N N N N N N N N N N N N N N N N

VME and so on. And so he had he had

these numbers and he said that what he

found is whenever inside Shopify they

were deciding do we choose vendor A or

vendor B as a database they would just

run uh a benchmark that they would write

themselves and they would get like okay

like I don't know storing this and this

it takes 2 seconds on this one 10

seconds on that one we will choose a two

second one and he looked at it and said

like hey like this doesn't make sense

like the amount to to store in a file

system like here's a theoretical limit

which is I don't know 100 milliseconds.

Like there's no way that's going to be

10 seconds. And it often turns out that

he found that the benchmark was just

wrong. They were benchmarking the wrong

thing and they were making decisions. So

I wonder if there's a thing where many

engineers, developers are maybe just not

aware of how truly devastatingly slow

this thing is versus the resources you

have.

>> Uh that is the entire point of like my

substack, right? So what you just said

is exactly true and it is the thing that

I hammer home on the substack through

all the parts of like the courses on

there which is that in general there's a

misconception about the way that you

approach optimization in uh like in

computer science or in whatever software

engineering let's say and that

misconception is that what you do is you

run a profile you identify where the

like you know big parts of the profile

are you make some changes to those and

you you measure like statistics the

better the statistics you know the the

the more uh statistics you can get the

better and you look to see if those

statistics improved if they have that

was a good change and you proceed as

such and this is completely not correct

that is not how anyone has ever you know

I I've worked with many extremely good

optimization people and that is not how

it is done the correct way to do

optimization is very much like what you

just said you first go what are the

operations that this system has to

perform form. What is the underlying

hardware capable of doing at its

theoretical peak? And then you measure

the delta between that theoretical

maximum and what you have achieved. And

then your goal during optimization is to

shrink that gap to something that you

think could be plausibly explained and

hopefully come up with explanations of

why you aren't at theoretical. Because

often times you can't hit theoretical.

That's why we call it theoretical,

right? And it's crucial that you do this

because otherwise all you're doing with

that other method is finding a you know

with with the with the I'm just going to

you know make something I think might be

an optimization and look if my

statistics improved. All you're doing

there is finding a local minima.

>> That's all you're doing. You're just

you're just you know you've got this

this shape of your performance and

you're finding some little spot and

you're sitting in it. That's not

optimization. That's improvement. But

optimization means to make optimal,

right? means we're going to find what we

actually should be able to get this

machine to do. And so, uh, you know,

that's why I emphasize that approach

because it's the one that I I've always

seen great optimizers take. That is how

they get good performance is by knowing

what the maximum could be. In addition

to that, it also is what lets you become

better at optimization. Because no

matter who you are and no matter how

much you already know, when you go to

tackle an optimization problem, there

may well be some things in the new like

way that the system is laid out that you

don't know about, new uh things that

people have not figured out about modern

CPUs, new things that are different

about the network backplane, new things

that are different about the GPU

drivers, who knows, right? And if you

don't have some theoretical maximum to

look at and to measure your delta from,

you don't know if there's some serious

anomaly there. And we you would be

surprised at how many times we find

anomalies like this things in CPUs that

no one knew about. And we, you know,

like I've literally had them in the

course of making the Substack. I've been

like, "What is this thing?" And I look

into it's like, "Oh, there's this new

renaming this this new rat table thing

that Intel chips seem to be able to do.

We didn't know about that." And that's

like a new thing we have to model when

we talk about how to do performance. And

so the that's the other crucial part of

I guess what you were calling napkin

math. I also call it back of the

envelope. That's the term I've heard

used for it. Often times they're kind of

interchangeable, right? Knowing what the

theoretical is is how you learn as well.

Uh how you learn about new hardware and

new performance op options.

>> Interesting. Plus by doing this you're

just learning. You're becoming better

professional. you understand more about

given hardware or or the inner workings

of of your computer or or software stack

or kernel, you know, all the stuff that

I guess goes way beyond the the the b

the vanilla programming language like

cuz you can say I'm an I'm an engineer.

I'm a software engineer because I know

how to use this programming language,

but I'd argue you're probably an

engineer if you can go down the stack

and you have that ability and you have

like a good good understanding of some

of it at least, right? And you can learn

the rest. And I would also say that one

of the other things that uh we do in the

class is teach how to read assembly

language. And people often ask what like

why like you know assembly language what

would I ever need that for? Uh there's a

very good reason for it. And that is

that everything else that you might use

doesn't tell you anything about what the

CPU is actually receiving. You know if I

look at a Java program if I look at a C

program if I look at Haskell OAML

whatever right uh Rust all I'm seeing is

input to a compiler. I have no idea what

the CPU is actually going to be asked to

do. If I look at the assembly language

output from that compiler, I know

exactly what the CPU is being asked to

do. And it's not that hard to be able to

learn to read assembly language so that

you can see very quickly is the CPU

being asked to do the things that I

think it should be asked to do them. And

in that way, right, you don't have to

write it hardly ever. Uh it's very rare

that you have to write assembly language

to do anything. um other than sometimes

for test purposes it's easier to do that

so you don't have to try and convince a

compiler to output something. So if

you're just testing something, sometimes

it helps to be able to write some

assembly language. But if you're just

talking about the uh vast majority of

tasks you might do in optimization,

writing it, no reading it essential. And

it also unlocks this sort of uh huge

world of possibilities to you because

once you know assembly language, you can

now do things like read those CPU

diagrams like you know when they

announce a new processor, they put up a

little diagram. That diagram tells you

stuff like the fastest this thing could

do multiplication and stuff like that.

It tells you that if you know a semi

language, you can read it right off the

chart, right? If you don't know a semi

language, you look at that chart and

like I have no idea do what I'm looking

at, right? Like it's just this weird

flowchart that doesn't really tell me

anything, right? And so one of the

really great things about s unlocks all

of this knowledge for you because it's

the it's the it's the actual uh input

language to the machine and it allows

you to figure out how it's operating.

>> Plus, I guess we should add that

assembly language is not all that

complicated, right? Just just by by by

nature, it's a far simpler language.

Okay, it's harder to read if you've

never seen it, but it's in terms of the

number of operations. It's so barebones

because you know that's what assembly

is. Like every single higher level

language will have way more keywords,

structure, whatever you name it, right?

Than assembly,

>> massively more uh and especially when

you consider the subset that are

actually used. If you look at the subset

of constructs that you would need to

understand to be able to understand um

say just a website from today, all of

the JavaScript libraries, all of the

JavaScript syntax, all of the DOM, you

know, all of the behavior that's going

to go on there, CSS, [laughter]

>> React, CSS, right? all of that

>> assembly language, you know, maybe

there's 20 30 instructions you might

have to learn total because most

assembly most things in legacy assembly

like x64, most of them are hardly ever

output by the compiler. So you only need

to learn a very small subsets. That's

the ones that's actually going to be

that you're going to be seeing in 90% of

the cases. It's so much simpler. And

also when you're looking at performance,

you're typically only looking at a very

small part, right? You've kind of

understood roughly what's going on. and

you've you've seen the basics layouts of

your program. you've identified what's

supposed to be happening and you're just

looking to see like wait why is this

part which I don't think should be

running this slowly why is it running

this slowly it's just a very small piece

you typically end up having to look at

as well so it's really much easier if

you can understand how to center a div

as they say if you can vertically center

a div in HTML then you can probably

learn assembly language I would say

>> okay you're super passionate about

performance optimization you also have

really good educational materials both

both free videos your paid substack the

free parts of it, etc. But let me just

play devil's advocate. There's this

saying that premature optimization is

the root of all evil. And we typically

use it or I typically use it so many

times. We're like, oh, should we make

this performance? Should we optimize the

thing? And like, nah, let's not do that.

Let's first build it. Let's see if it's

good enough for our customers, for

ourselves, and if we need to, we can

always optimize it. I mean, you know,

like it's it's not the hardest thing in

the world. Okay, maybe not as good as

how you mentioned cuz maybe I don't read

assembly but and that's that's kind of a

thinking of building you know like I

guess SAS software building software at

big tech. What is your reply to that cuz

I I'm I feel really good that I made a

really good argument here.

>> So I guess what I would say is the

important part about that and I guess

I'll divorce it a little bit from the

saying. I have an entire lecture on that

saying by the way. It's it's like two

hours long and I I gave it at better

software conference this year and I

believe the VOD we're linking that in

the show notes below.

>> Okay. It'll it'll be like a week or two

I think till it's up. So it may be right

at the same time as this. Uh but so if

you want to find out the history of that

phrase you can go look at that. But I

wanted to talk about the idea behind it

because I think there's uh I don't want

to dismiss it entirely because it's not

entirely false. And the idea is that

well I'm just going to delay

optimization work. I'm not going to

think about that and then I'm just going

to make whatever I'm going to make and

then you know either myself or maybe

I'll just hire some performance person

to come in and clean up the mess later.

Right? So here's the positive side of

that first. The positive side of that

first is for some types of code that

will work. If you happen to have written

some operation poorly where the

optimized version of that operation just

looks like someone taking a loop and

changing the loop from your really like

you know naive version to a really well

optimized version.

>> The the the typical of like I wrote a

bubble sort we can later optimize that.

>> Who knows right? Anything of that form.

Okay maybe we can just do that. So there

are certain times where you do in your

head want to be doing this where you

want to say okay I could go spend a week

researching the fastest hasht

implementation here but part of software

engineering is being smart enough to

know it won't matter if I do that now or

later the architecture around this piece

won't have to change I'm quite certain

because I understand the problem well

enough so it's okay I can defer that to

later maybe it's never too slow with the

naive when put in there and then we

don't have to do any work. Maybe it's

too slow later. That's okay. I just

target this one hasht implementation and

we'll get as fast as we need. Right? If

you're doing that, if you're applying

that true engineering mentality to it,

you don't have a problem. The problem

comes when you don't know if the choice

that you're making produces that kind of

optimizable hotspot. And I'll give you a

very simple example that usually people

have had experience with. A very simple

example would be we write our entire

software thing like we just whatever

this massive thing that we're imagining

doing where we're going to ignore

optimization. We sit down and we write

it and we use a paradigm where we ask

the server for something. We have like

some API, you know, that we've built for

asking servers for things. We ask the

server and it returns to us what the

server's response was. And we that's

like kind of how we architect this

thing. So everyone writes, you know,

hundreds of thousands or millions of

lines of code and they all look like ask

the server something, do some

calculations, ask the server for the

next thing, do some calculations, right?

Then at the end you find this is way too

slow. But that's okay. You weren't

worried about that cuz you're like when

the end you call in some performance

experts. They look at and they go,

there's nothing we can do for you.

Sorry.

>> Why? Well, the reason is because you

created a serial dependency chain. All

of your code looks like wait for a

network request to come back, do

something. wait for a network request to

come back, do something, wait for and

that serial dependency chain can't

really be shortened without just

rewriting it. If instead you had made

the paradigm and told your programmers,

look, here's what you need to do. At the

top of every operation, you need to

figure out all the things you might want

to ask the server for, you ask them for

all of those things, right? And then you

do all of your processing there. And you

only create a chain of dependencies if

you absolutely couldn't have determined

what it was you needed to ask a server

for. Now you're just in this situation

because you didn't tell them to do that.

You have to rewrite all your code.

Everyone is now going out rewriting all

the code if they even can. If it's even

possible to really do that in a way

that's not slower than just rewriting

the thing, right? So what happened

there? Well, again, we talk about this a

lot on on the Substack, but there's this

idea of a serial dependency chain. It's

when you stack things in order, right?

And the performance of your software is

generally determined by the longest

serial dependency chain because it's

something that cannot be parallelized.

If I have thing A that then B depends on

that then C depends on that we cannot

shorten that because it has to go in

order. Everything waits for it. We can't

multi-thread it because it's dependent.

We can't uh you know make it run wide.

We can't you know uh amortize the

network request whatever. that kind of

thing can be pervasive in the

programming and we can't cheaply remove

it cuz it's not a hot spot. It's a way

that you did things. That's the part

where that kind of thinking breaks down.

If every uh software engineer knew to

watch out for false serial dependency

chains, things where they were creating

series of dependent operations that

could not be optimized away or other

sorts of architectural problems like

that that cannot be easily fixed, then

the world would look more like just wait

and optimize the hotspot, right?

>> Yeah. So this this is the architecture,

the planning, right? Like if if in that

phase you're like okay like as this

thing grows like what will get in the

way of performance what will slow it

down or you can ask all these questions

or like different flavors of the

questions you know or from the other

side and so on.

>> Yeah. Another way to think of it is cuz

hotspots is the way that people talk

about that like oh it's it's going to be

hotspot optimization. We just got a few

spikes. Someone will come in and clean

up those spikes and we're done. Right.

The way to think about it is your

codebase will not end up that way by

accident in most cases anymore. you have

to engineer upfront for a [snorts]

hotspot codebase that people can then

optimize, right? And so that's the

crucial takeaway is everybody on your

team who is making architectural

decisions, those people must know

performance and they must make decisions

that will allow the other people

downstream of them to use an

architecture which can be optimized

later. If you don't do that, you're just

rolling the dice. Casey just talked

about how engineers making architectural

decisions should know about performance.

This is also true when choosing your

dependencies like which database to use.

And this is where I want to mention our

season sponsor Turbopuffer. You already

know how Turboper is a vector and full

text search engine. But here's an

interesting story from Linear on what

happens when you stop thinking of

Turbopuffer as a search engine and start

using it as a primitive to reduce

latency. As context, Linear is a local

first app. So each client keeps a local

database and when that client goes back

online, it needs to catch up with what

happened and do it fast. Their biggest

workspaces generate around a million

sync actions per day. Doing catch-ups by

reading from post was getting slow for

large reads. So the tail latency got too

large and adding more replicas did not

help either. Linear solved the problem

cleverly. They started using Turbopuffer

as a serving index for each client. This

is because Turbopuffer itself is built

on top of inverted indexes. So for every

index value, it stores the documents

that that value can be found in. The

lookup cost for such an index is

constant. So linear took this structure

and had each client's index point to the

changes that they needed to sync. As a

result, not only did they reduce

latency, but they kept it constantly low

matter how long the change list is

synced to the client is. Linear

published a blog post about this

refactors titled rebuilding linear's

delta sync read path. Check it out. I

love this story because it shows how

important it is to choose the right

primitives and how good primitives can

improve your system. If you're building

systems where you store a lot of data or

serve a lot of data, Turbopuffer can

probably speed things up or save on your

costs. Learn more at

turbopuffer.com/pragmatic.

I'd also like to talk about a presenting

sponsor and this is while Casey

deliberately does not use AI coding

agents for his work. Most of us do. And

when you work with coding agents, your

job is no longer writing code. It's

specifying and testing it. Antithesis is

the most effective method for verifying

agentic code today. Let me explain how

it works. Antithesis runs your whole

system in a hostile simulation. By doing

so, it finds every bug before your users

do. And because the simulation is fully

deterministic, and this doesn't only

find bugs, it gives you a perfect

reproduction of every issue. To create

such a tool, the anticys needed to

invent new kinds of debugging tools as

well. For example, here's what's called

a bug probability graph. The xaxis is

virtual time and the yaxis is

probability. As anticis runs a hostile

simulations. It plots time frames when

the bug probably increases which greatly

helps with finding the root cause of the

bugs and antithesis also has a log

visualizer. Vertical lines going down

represent events branching off from the

same state. And the purple dots are

where the bug happens. Antithesis is as

good as it gets in being able to ship

agent written code. It's what teams at

Jane Streetfly.io and the CD community

used to ship with full confidence. Head

to antithesis.com/primmatic

to learn more. And with this, let's get

back to Casey and how if you don't

design an architecture that can be

performance optimized later, you're just

rolling the dice. And we've seen so many

projects. I have an entire video where I

go through like look at all these blog

posts of people who like say we, you

know, it's Facebook, it's Uber, it's

everybody. They've got blog posts of we

had to rewrite this whole thing because

the performance was bad. If it was

always hotspots that made your

performance bad, you'd never have to

rewrite the whole thing. So, we know

that that doesn't work anymore. Uh why?

Because of the things I just said. I was

at Uber where I I was not making the

decision but the teams next to me were

and I saw or I I kind of understood why

they were making but typically and right

now it's happening with AI companies

oftent times it's like we chose this

technology which is Python and it's

single threaded and it made sense at the

time on the web server but now we're big

and this happened at Uber it was it was

Python and NodeJS and then they went to

go and Java on the back end and now with

AI companies it was Python open AI and

Tropic are both going through this right

now uh They're both either public about

it or I I've written with Antropic. They

they they share with me with with me,

but I I put it out there. They used

Python because data scientists or AI or

machine learning engineers knew Python.

They put on a bunch of web servers. They

had their API run on it. Initially, they

just, you know, scales horizontally, but

now they're like, well, if we move move

over to Rust, they right now they're

choosing Rust or or Go, but I think it's

Rust. Well, we can actually have

multi-threaded and the same machine can

actually handle more connections. So,

cool. I came across a lot of that

because I think that's easy and safe to

communicate because it doesn't look bad

on you. But you're right, a lot of times

I don't think on engineering blog post

you'll get the real reason that these

companies put out there like when it's

kind of a very kind of you know easy to

own mistake or not mistake but just the

decision which made sense. they'll tell

you. But if it's something that was an

oversight, you're not really going to

get that on a on a public facing

engineering blog post, except for maybe

some startups who are really there. But

don't forget like a lot of those blog

posts are going to help someone get

promoted or get recognized and they will

always be way more positive in

especially when there's a content writer

team which large companies do have. So

it's it's not quite PR but it's

somewhere midway in between. And I mean,

yes. And also, I would just point out

the fact that like the fact that these

things are happening though is all we

really need to know for the signal,

right? Because in general, this should

not be happening. If the the ideas about

activation were true, you'd never have

to rewrite something in a language in a

different language unless you just

preferred that language. It would just

the story would just be we rewrote it in

this language because we wanted to use

this new language. would never be um or

for Rust it might be just memory safety.

We see those blog posts, right? It's

like why did we write into Rust? It

wasn't performance. It was just we

wanted the memory safety or something

like that.

>> If I'm a software engineer, programmer

and I'd like to just get better at

writing performance code, I'm

interested, you know, maybe after this

podcast or or looking into some of the

things that you did. What is a learning

path you would follow outside of your

substack where you cover a lot of these

things, but what are areas that you

think are kind of like you need to

understand these things to like get

better at writing performant code? I

think it's actually very simple and

perhaps a little bit counterintuitive.

So, I'll start with the uh the very good

news about learning to write uh

performance software. The good news is

that optimization of the kind that we

sort of talked about, the like hotspot

kind where it's like somebody's going to

go in here, maybe they're going to even

rewrite this routine in handcoded

assembly or something crazy like this,

right?

That's very rarely necessary these days.

One of the reasons that you don't see

hotspot optimization as a thing that

really matters that much anymore and one

of the reasons I advise that

architecture and and not making bad

decisions is much more important is

because a lot of libraries already have

been optimized for you that you might

use. CPUs are incredibly good at taking

bad code and running it quickly and so

on. So, typically when we're talking

about the causes of performance, uh,

negative performance that aren't

squeezing every last little thing out of

the hardware, but rather just making

sure this thing isn't running like a

hundred times slower than it should be,

usually it's more just about having an

awareness again of what the computer

should be able to do and making sure

you're making uh software architecture

choices that allow it to do that. And if

you do those things, you will generally

be within, you know, 2x or something,

which is 50x better than the people who

are 100x [laughter] away, right? So the

good news is in order to write software

that's much better than a lot of the

software you use today, you don't have

to know that much. So what do you have

to know? What I argue and what we focus

on the substack is I think you just have

to go through the experience once of

learning reading the assembly language

seeing how the CPU works seeing the

difference seeing why Python is slow

which we show on the sub. So one of the

first things I show is uh I walk you

through the assembly language necessary

to execute uh A plus B in Python and

it's so vast that you know I have to

skip most of it. It's it's massive right

it's like this huge and whereas you know

if you have the equivalent function in C

it's one instruction add right so you

know uh understanding basic things like

that if you go through learn to read

assembly language learn to look at some

code learn to do some CPU timings and

you just have that experience just spend

you know uh a month or two of nights or

whatever you want just understanding

some performance stuff and going through

a few examples where you play with it

and you see the difference

>> and and just so I understand you're

you're saying do this not because let's

say you're doing iOS development or or

web development read what React like you

you will not look at the assembly that

the React does but if you do this on a

project you will be able to

conceptualize what is likely happening

what the layers are and you you might be

able to decide like do I want this layer

or do I want to use let's say WebGL If

you're a React engineer, you probably

haven't touched it. But again, you can

skip a bunch of those things and it

comes to trade-offs with

maintainability, yada yada, but that now

you you you will know like kind of what

you say by keeping this layer or not

keeping it and so on. Is it do I get

that right?

>> Essentially, yes. And like uh you know

the simplest example is the Python

example. Most people have never

internalized the fact that it takes, you

know, maybe on the order of a hundred

more uh CPU instructions to do an ad in

Python than it does to do it uh in an

equivalent language like C for the same

piece of text, just A plus B compiled in

two different, you know, in two

different languages, right? And so just

understanding even just that is enough

for you to kind of go like, "Oh, okay.

A, now I kind of understand why if I'm

using Python, I kind of have to use

libraries to do things." And those

libraries were written in C. Because

it's like if I'm ever going to do any

operations on a on a large number of

things, I can't do it in this language

because the amplification factor is so

high on each operation that it just, you

know, kills the performance immediately.

whereas these other languages don't have

that, right? And so understanding those

orders of magnitude and what's actually

going on, I think that allows the

programmer to know, okay, if I think

through what I'm doing right now, can I

afford the super slowdown that I'm going

to take? And usually I don't think you

have to be a performance expert to make

that decision. You could usually know

like, okay, is this a part of the code

that can afford to be 100 times slower

than it should be or not? Right? And uh

you know most people can I think make

that decision fairly logically. And if

it's not then now you know like oh okay

if I'm in Python then what I got to do

is either I got to go find a library

call that will do these sorts of things

and structure around how that library

works or I should maybe get something

like Syon or something where I can do

compiled stuff inside my Python and make

my uh code work around calls out to that

kind of code. You know, you now have the

tools you need upfront to make sure that

when you write the program, you've put

the parts that needed this and you've

structured the code in such a way that

you are only paying the 100x on things

that you know are very infrequent or

happen like only uh you know once per

every so often things like that right

that's I think the biggest thing is just

the knowledge and once you know you can

start to make much better decisions in

any language because it doesn't take you

very long you know a simple le search or

you know asking an AI or whatever is the

common practice that you're going to do.

A simple bit of that once you know what

you're asking for will get you this

information back very quickly. Right?

You just have to know that you should

have been thinking about it.

>> Now, as a software, you mentioned it's

good to understand how the CPU works. As

a software engineer who is not a games

developer, I'm not doing low-level

stuff. What does that give me? because

for the most part even in academia or or

in computer science you know there is

some level of of some basic CPU theory

taught but usually we just kind of we

stop at the code okay maybe you look at

the assembly but you rarely go further

than that the folks who you know you've

you've taught and they they learn these

things what do you see them get out of

this that they wouldn't otherwise

>> so you're talking about specifically the

knowing the CPU part

>> knowing about the CPU, knowing about the

details about a CP because because you

mentioned that that's also part of it,

right? It's not necessarily just

stopping at assembly.

>> So the reason for that is more the other

way around the the reason to learn the

assembly language is so that you know

what the CPU is doing. So it's the CPU

part that's actually important and the

part that's important about it is that

the CPU is basically you can think of it

as a little machine whose internal

gearings we are not privy to because for

the CPUs that we care about. So you know

an M series CPU in a Mac a Zen core CPU

in a server uh or in a laptop or an

Intel you know core series those sorts

of things. These CPUs are not documented

at the level where you're going to be

thinking about how each little

individual part works. And to that end,

it's unclear that you would have time to

do so anyway because these are massive,

very complicated machines that we're

talking about, right? But from a high

level, from a more blackbox perspective,

they are machines that we can think of

in relatively straightforward ways once

we know kind of what their core

instructions are that they tend to

execute.

And they break down into a couple

different categories. There's how does

data move into and out of a core. And

this is basically how like load store

units work, how the cache levels work,

L1, L2, L3. Some we have like we have L0

now sometimes things like this. How does

that work and why? What is the

granularity of it? What is the policy?

How does the CPU go about actually

working with those things? Understanding

that part of the machine is crucial

because when you're working with a lot

of data, the the difference can be

massive if you structure it in one way

versus structuring another way, right?

Again, architectural decisions that have

nothing to do with hotspots, they're how

all the data is laid out and what the

access pattern is, right? Things that

are very hard to change sometimes. So,

that's one part of the machine you want

to understand. Another part of the

machine you want to understand is how

the instructions flow through it. And

you, you know, a lot of people have

heard about like branch misprediction or

things like this. eye cache misses.

There's words that you might hear, but

you're not really sure what they mean.

They're all actually pretty simple to

conceptualize. Sometimes they're they're

harder to pin down exactly how they work

because branch predictors are, you know,

getting more and more complicated and so

on, but you can still categorize the

behavior of them and understand how code

flows through it and when you might care

and when you won't. And then finally,

there's the execution unit scheduling

part, which is about knowing what's the

raw sort of throughput for any

particular type of operation.

floatingoint multiplies, integer

additions, division, whatever it is that

you might want to know. Right? Once you

learn a little bit of assembly language,

you understand what it's reading, you

understand how it turns those assembly

language instructions into micro

opterations, which it actually does, and

how they get distributed through that

machine. That flowchart that they put

like basically up on the we've announced

the new Zen core, that flowchart, you

can look at it and go, I know the

performance of this machine roughly,

right? Not exactly because like I said

there's all these little edge cases that

if you really want to be a crazy

optimizer which again I don't really

advocate people do. I don't think it's

important that they be like crazy hyper

optimizers. If you like to do it great

it's a lot of fun but it's not the

important part. Just look at the C like

okay I see what the CPU should be

getting in terms of like what I could do

with this size data load that size data

load. This is what I could probably get

out of it for if I was doing a bunch of

like how to do a bunch of like math ops

on it you know. And and I think that's

that's just something that should be

kind of par for the cars to software

engineering. You go to school for four

years to learn this. There's no reason

you can't learn this in a few months.

It's not that hard.

>> Yeah. Plus, I guess just from a

craftsmanship perspective, like we

should know our tools. We should know

the machines that we're programming.

Obviously, we know that our code will be

if you're doing web, it'll be running on

all these different things or if it's a

if it's mobile on all these different

phones, but from a conceptual point of

view, like we should be able to know

what's going on. So I I feel there's a

bit of a pride as well. Like if if

nothing else, you would learn a bunch of

stuff. Like I I know some of it, but I'm

now getting motivation to learn more

about it.

>> I do think there's a craftsmanship

angle. I think there's a a large number

of people who maybe don't feel fulfilled

when they I' I've certainly heard from

lots of people who when they write

something and it's just kind of this

amorphous highle thing, they they don't

get as much satisfaction out of it. And

then when they learn how they can look

more deeply at what's going on, they

feel much more satisfied. even if they

didn't change what level they were

programming at, they feel much more

satisfied that now they know what

they're doing, right? And it's like, oh,

I see and I understand why this thing

was happening this way and this thing

was happening that way, that's very

satisfying, right? So there there's an

aspect of that. I also want to emphasize

another part which is that it's a

percentages game. If we convinced enough

library maintainers that this stuff was

important and the libraries all get a

lot faster, all of a sudden all the

people using the libraries code gets a

lot faster and so on and so forth. If

the APIs start changing to make it

easier to optimize the libraries because

people now thought that through, right?

Like it's infectious. The more people

are doing performance, the less people

need to do performance, [laughter] if

that makes it kind of paradoxical,

right? Well, plus plus I I I I do think

that right now there's still an edge in

in just being performant again and you

mentioned but there are categories of

software that is just winning by being

much faster and to do that you need to

do this and if you know how to do this

maybe you're going to spot opportunities

as a software engineer maybe right now

you're not as happy in your position but

maybe start something or do a side

project that turned into a full-time

thing and so on and so forth. So like I

feel there's like and worst case you

just learn like net new knowledge which

will probably not be as outdated with AI

which we we'll get into later but this

stuff it it feels it just feels very

interesting right like kind of it moves

your brain. It does and thankfully like

uh it's also not that hard to update

your knowledge because again you get

these presentations that that the CPU

companies give and they're like here's

the changes we made. So you kind of are

aware every time a new thing gets you

know there's little tiny things that

creep in that you don't that you know

but again there's people out there who

are running lots of microbenchmarks that

you will find out about and they often

uncover these for you as well right uh

so so I I wanted to talk about games

you've been you've built games for like

decades at at this point can you give an

overview for those of us who are not in

the games industry how is a game

typically built from the games that you

know of that you've observed or or you

worked on especially trying to compare

for you know like in typical SAS or

distributed system or or something we're

building a a website or service is kind

of you plan this stuff you you know

we'll we'll do an estimate we'll build

it in a few months or a few weeks we

deploy it and then we monitor it and

then we keep tweaking it and then you

know fast forward 5 years later it's now

this like gigantic thing with

microservices but but it keeps evolving

right like it's we do this like a lot of

prototyping thing for games It's pretty

obvious right from the get-go as we're

talking like there there will be a

launch but can you when you're inside or

when you when you join a game studio

like what would you observe there in

terms of what the process is like and

how it's different or or how it feels

weird compared to like this I guess I

don't know traditional software SAS

software whatever development that is.

So, I guess what what I would say is

unfortunately I'm probably the wrong one

to ask because my knowledge is outdated

at this point because one of the things

that has happened to games recently is I

feel like they've moved closer. I don't

want to necessarily say entirely in

development practices but at least in

terms of the nature of the product has

changed somewhat dramatically to be more

like something like SAS where you know

if you take some of the most popular

things that are being in terms of

dollars let's say so I guess maybe

popular is kind of nec might be hard to

say specifically but let's just say

revenue generating so if we were to

measure the total games industry revenue

and you look at what are the largest

slices of that, you're seeing things

like Fortnite, like Roblox, like Grand

Theft Auto 5 online, etc., etc.,

Minecraft.

These things are starting to look a lot

more like an always on live service kind

of we ship incremental features to our

customers uh kinds of things. And so I

would actually say that, you know, I I'm

I'm the wrong one to ask about what that

actually looks like from the inside

because I haven't actually gone and

worked at one of those companies. I have

friends there, so I hear things, but I'm

not the right one to like give it an

accurate picture of it. But I just would

point out from my perspective, the game

industry practices look different today

than they did when I have a more

intimate sort of experience with what we

were actually building. But c can can we

talk about it when you were building

games which was you know 10 plus years

ago when I understand these were the

games where they were built they were

launched you know maybe they got a patch

or two and then they were kind of you

know the team moved on they were

disbanded. It was this time box thing

that was a lot of development a big

launch and and either it it was it went

big huge hit or you know a huge failure

right and then the studio goes bankrupt.

So how did that work? because I feel

that's a world where okay today a bunch

of games don't have those constraints

but it has a bunch of constraints and

I'm interested in in what worked in

those constraints. So in the early days

you didn't have licensable engines. So

up until sort of the point like nowadays

like this is why I say it's a lot

different now than it used to be. You

know nowadays you think of web

development like I'm going to go go grab

like a thing like react and I'm going to

make this thing or whatever. I'm going

to go grab an off-the-shelf database

thing, Postgres or Oracle or I don't

know like what would be the thing of

choice, right? But

>> by the way, just definitely Postgress

and not Oracle for most people.

>> Okay, sorry. Sorry. [laughter] Postgres,

uh I didn't want to slight anybody

there, so I apologize. So, okay,

definitely Postgres, sorry, Oracle.

Yeah.

>> Um, so you're going to go use some kind

of a variant of of one of these offshelf

databases and so on. That's more, like I

said, what people might be doing

nowadays, too. Like they'll grab the

Unreal Engine. They're not going to

develop an engine on their own. They'll

grab backend server stuff from people.

It might even be some Postgres in there,

right? Like, who knows? In the earlier

days, none of this stuff existed for

games. I actually worked, like I said,

in middleware at the time. So, I was

actually sort of one of the people who

was working at the time on maybe

changing that a little, like producing

code that would get reused throughout

games, which was actually fairly rare.

>> And and and so and so back in the day,

every game built their own rendering

engine, for example.

>> Correct. And so,

>> yeah. Yeah. Yeah. Uh and so really early

on, right, if you if you rewind the

clock far enough, uh then yeah, the the

degree to which people were reusing code

for their thing for like their rendering

engine, it'd be like cuz I got some code

from like Dave or whatever who was or we

were both at Atari and somebody wrote

this good routine so we used it, right?

There was that kind of thing, but there

wasn't like this sort of set uh engine.

And the time when that sort of maybe you

could say first started happening a

little more widespread was with things

like what ID Software did where they

sort of started having like oh you know

like someone's going to build something

with the Doom engine or someone's going

to build something with the Quake

engine. There was also the uh the build

engine at the time uh made by Ken

Silverman and some things like that. So

there were there were some early cases

where a few people would make a game,

but they were making a game very much

like that. Like if we did the Doom

engine, we're going to make a game very

much like Doom. So it really was the

case that for most games, people were

rebuilding most of the things from

scratch, at least for their studio. And

studios often their existing code base

was kind of part of the value of the

studio, too. Like if you are uh think

Blizzard and we just built Warcraft 1.

Well, rolling all of that knowledge and

code into Warcraft 2 is a huge advantage

for us because everybody else who wants

to build a competitor to Warcraft 1 has

to do all of that from scratch. They

have to make the path. They have to make

the level editing tools. They have to

make the rendering. They have to make

whatever. And so, you know, that was how

things were traditionally done.

>> No wonder the games industry is so

secretive compared to the rest of the

software engineering industry. Like

seriously,

>> used to be

>> used to be at least. Yeah. maybe now

it's changing

>> and so there were two really big risks

typically uh when you started a game

project in in those days one was the

engine risk would we be able to make

something that would be technically able

to do what we need to do for this game

and that risk comes in a lot of flavors

one will it happen at all two will it

happen fast enough for us to actually

reliably build the game on it right one

of the things I mean I don't how

detailed an answer you're looking for

for this question. So stop me if I'm

going down too many tangents, but one of

the things you also have to remember is

that at that time, and this is sort of

still true today, but at that time it

was very important, there was no way to

really buy something all that much

faster than what you had. There was not

a huge strata of PCs that you could, you

know, buy or anything like that. So the

rendering engine, there wasn't like a

way your level designers could like be

playing on a faster thing than the

consumer would have really. you can only

have the machine that you have now and

if this game comes out in a year that's

sort of roughly what the consumers might

have or a little bit but you know so

there are people who started doing

things like buying SGI workstations

because those were actually faster

enough right and and things like that is

what you know you kind of had to do uh

and so on so anyway so there was a huge

engine risk and some games just failed

because they couldn't produce a thing

that could technologically do what they

needed you saw houses who survived on

technological prowess you had ID

software that was kind unrivaled at

making those kind of first-person

engines. You had Bullfrog who had this

engine that the pseudo3 engine that they

did for like the racing games, Magic

Carpet, Dungeon Keeper, like they were

all based on this, you know, one core

tech and all the sorts of things.

There's that engine risk that was huge.

And how do you mitigate it? You didn't.

You were just grit, right? Because there

wasn't a way to buy one off the shelf.

So, you were just kind of uh gritting

your teeth. The other big risk, and this

one is still somewhat true today, but

it's just much less because you can

start, you could do prototyping early.

The risk is, is the game any good? Like,

what are we building? Is it interesting?

Is it fun? And when you think about this

problem of we can't even really run the

game as it will be cuz we're just

building this engine and we we don't

even really have a way to test the game

super well. And we can't really build

much of a final level because we don't

have level editing tools yet. Those are

just coming online. trying to guess what

you are actually going to be shipping in

terms of gameplay is incredibly hard and

there are games famous games um I want

to say like Thief the Dark Project a

very famous game looking glass game uh

it was formative in the stealth genre

launched a franchise which was Thief you

know I want to say uh everything I heard

from people on that team was that like

the final core gameplay only sort of

came together like right at the end

right and so the game just could have

been a lot more could have been polished

a lot more but it just the timing of

these things coming together was so hard

and so it really was an incredibly

different thing and nobody really had a

way um around it. Eventually, there was

sort of this push towards something.

Well, it was basically early vertical

slice prototyping where as games started

getting bigger and people were like, we

can't keep doing this, especially if

we're going to be putting millions of

dollars on like this is not like an

option, right? They started to move

toward this thing about look what we're

going to do is we're going to focus the

entire studio on building one vertical

gameplay slice as fast as we can, as

hacky as we can. Whatever we have to do,

do that. prove that that is engaging to

play and then start building out

everything else because we simply can't

afford to not know what that thing is

and then we can start building like

spreadsheets that'll schedule what are

the assets we need cuz now we actually

believe in the thing we can see it

running and it fills in all those

details right and that I believe you

know I'm not a game historian so take

what I'm saying with a huge grain of

salt that I believe was a pretty big

paradigm shift for the industry when

they started going okay we got to

actually know and it became much less

seat of the pants after that if you

will.

>> Now I'm interested in your observation.

I know you're not a game historian but

you were in the industry and you still

remain connected to it. What happened