I describe microservices as being an

architecture of last resort.

>> Really?

>> Yeah. Yeah. Absolutely. What a lot of

people get wrong, I think, is they adopt

microservices and they think other magic

stuff's going to happen.

>> You talk about the three rules of

distributed systems. What are these

three rules? The first is that you can't

send information instantaneously from

point A to point B. Rule number two is

sometimes the thing you want to talk to

isn't there. And the third rule is that

resource pools are not infinite. The

vast majority of the time when you have

a system outage, just down to that, some

resource has been totally saturated

somewhere. What are the things that we

know that we do not know about AI?

>> I think the tech world in general is

quite naive about AI and fundamentally

misunderstand.

Sam Newman is the author who wrote

probably the most widely read book on

microservices. And yet, as you heard, he

said that microservices are an

architecture of the last resort. But

why? Sam was in the room when the term

microservices was coined. I find him to

be one of the most thoughtful engineers

on this topic as well as on resilient

systems and his latest book building

resilient distributed systems is out

now. Today we cover specs versus code as

a source of truth and why Sam says that

production is truth. Item potency keys

versus fingerprints and why you should

avoid item potency keys when possible.

the four dimensions of resilience,

robustness, rebound, graceful

extensibility, and sustained

adaptability, cognitive depth and

cognitive surrender, and Sam's advice on

how to better use AI to avoid these and

many more. If you want to understand

what it takes to build resilient systems

from an expert who has been needing this

topic for 10 plus years, this episode is

for you. This episode is presented by

Turbopuffer, the AI retrieval layer

built on object storage. It's fast,

cheap, and extremely scalable. Andropic,

Cognition, Notion, and many others are

happy customers. Before we start, let me

mention Tire, our season sponsor, and a

product and team I'm very bullish on.

You probably already know that Git

hosting has a problem. AI agents. Git is

increasingly becoming a bottleneck for

modern agent heavy software development.

Engines like you or me are creating more

code with agents. Those agents push more

code and most of us are running parallel

agents, which push way more code. GitHub

is clearly struggling to keep up and has

frequent outages. So, what's the

solution? Entire was founded by GitHub's

last CEO, Thomas Dumpka, who by the way

was a guest on a podcast when he was

still at GitHub. And Thomas revealed Git

hosting for the Agentic era from

scratch. Entire was built to be very

fast and to have your repos regionally

close to you to reduce latency. And this

allows for fleets of agents to push in

parallel. And Entire's performance is

next level. The repo can handle 418

pushes per second, which is up to 89

times faster than every competitor on

the market. When GitHub is down, you can

still keep working and you don't even

need to migrate away from GitHub. You

just sign up to Entire and the platform

mirrors your repo. And here's a neat

thing. Have you ever wondered what

prompt resulted in this specific code

being generated? I find that the prompt

and conversation with the agent carries

more information than the PR itself.

Often times, entire captures all the

prompt history with your agent right

inside of the repo easy to check back.

And it's got a pretty innovative UI to

show all of this. If you're looking for

Git hosting that works even when GitHub

is down, head to entire.io/pragmatic,

install the CLI and mirror your repo

with a click. I've already done it and I

love it. Oh, and did I mention that it

works with any agent and it's open

source? Sam, it's great to have you on

the podcast.

>> Thank you so much for having me. Happy

to be here. Really happy to be here. Big

fan of the show, as they say. I'm not

going to lie, I've become a bit of a a

millennial and I think I watch a lot of

shorts rather than watching the whole

things now and becoming my son. But

>> yeah, Millennial is a new boomer

apparently I've learned. So

>> God, I was I was happy enough being Gen

Y. Now I'm wherever I am now.

>> Well, so so let's go back to the the

beginning. You've been known for a lot

of your books, your takes on resilience

system, microservices. Before that, you

worked at Thought Works, but let's

rewind to the very beginning. How did

you get interested in tech and

computers? I'm quite fortunate living in

UK. We had at places an access to

computers quite early on. We had an

Acorn Electron at home, which was sort

of one of the generations of the BBC

computers that came after the BBCB. And

then I got a Spectrum 1283,

which is the Spectrum that had the

built-in tape drive.

>> This is way before the Commodores.

>> I think some of this was the B. I think

I'm pretty sure that the Acorn Electron

was pre-Comodore, but the Spectrum would

have been around the same as well. The

UK was a bit more of a Spectrum and BBC

market, whereas the US, the Commodores

and the Ataries and things were a bit

bigger. So that's that sort of got me a

bit interested and then I always stayed

interested in it. Dad had a computer. I

did the very boring thing. I went to

uni. I did a software engineering degree

at a place called Deontford University,

which the running joke in my family is

it's the in the top five universities in

the Leicester area. There are only two

universities in Leicester. It was a I

mean it was I loved it actually. It was

a great class. It's basically MIT

syllabus. And then I just did the work

after I got a great um back then we did

these things called uh sandwich courses.

>> Sandwich course.

>> Yeah. And so basically the idea was you

would do two years of study, you do a

year in industry that was paid but you

effectively going in as a student and

then you did your your final year.

>> Oh, nice.

>> Which is actually fantastic cuz I kind

of was drifting a bit and when I did my

year in industry

>> so many things clicked. It meant as much

focus going in and I got to do work for

a company called back then they were

called GC GC also now that was called

ALSTOM and we did software for the

European Space Agency and gas and steam

turbines. Yeah, it was really cool. I

joined about the first time at that

about and then I went back after and

when I went back it was just around the

time the first com boom and so everybody

left. So I went from being the person

that was doing like nothing to person

did everything. But I remember my first

week there I was given a 4R77 codebase.

So forran77 was released in the year I

was born and every single for loop had a

break condition because we were

basically doing analysis on these Unix

machines and this if the models and

these the time on the machines was

expensive and we were worried about

accidental infinite loops. This is I was

trying to solve the halting problem

basically. So every for loop across

these 8,000 files had a break condition

of I think it was 99,999

loops. And if you went over that, the

program basically broke and aborted. The

models had got bigger. The computer got

more powerful. Go through and change

them all by by hand back then.

>> By hand. Well, this is one of the nicest

things that anyone's ever done to me in

a professional career because for 77 is

based on the old um punch card

limitations to an extent. So your lines

can't go very long. So it's all a bit

fiddly. You can't go and put a nine. And

my boss said, I reckon I reckon it will

take you about two weeks if you do it by

hand or and he slid a book across to me.

You can read this. And he said he gave

me a copy of the O'Reilly said an orc

book. And so I spent two days learning

how to do regular expressions. And I

wrote a refactoring using regular

expressions, accounting for line lengths

and everything else. And it's I think

the closest I've ever come to feeling

like a god. I did it and I thought

that's the thrill. That's the absolute

I'm not sure if it's a joke or a story,

but you know, the two the two um

builders who need to build a wall and

and one of them starts to build the wall

one by one by one and the other one is

kind of drawing making plans and like I

don't know like it takes three days to

build the wall and first two days this

guy does nothing and the builder

building it feels really good and then

on day three this builder like builds a

machine that builds or or like puts

together a machine that builds the wall

for them and then

>> yeah but sometime and sometimes of

course the wall falls down the moment

you set the machine I thing is I love

that that story that story was all true

and it it's one of the earliest times I

can remember learning something and

having such a vis visceral reaction and

I remember this has been just towards

when the end of last thought works some

grads in and they asked to tell some

stories about like what had happened in

previous career and I told the story I

said I added the nine made the models

bigger and then one of the one of the

grads said why didn't you just replace

it all with a constant so you could just

change the number once anytime in the

future. And I was like,

I thought I'm learning once again. Yeah.

So that that's also why I love it. It's

like you you get humbled all the time as

industry. You get to learn stuff. Yes.

There's reinvention of the wheel going

on, but you we're inventing better

wheels the whole time. say and I think

it's just nice to remind us every now

and then I'm sure everyone who's built

code and programs like at some point you

have the realization that you do feel

like a god that you can do things with

these things which is just so difficult

to do by hand right like there's

>> I think we forget about that when it's

kind of dayto-day and we're just like

doing our stuff or even like right now

we're using LMS which also do incredible

magical stuff that would have been

impossible but like it's kind of cool

that and and we are in charge we can do

amazing stuff you get that little fizzy

dopamine hit. And I think that's

something we're struggling with a bit

now is as people are getting further

away from the code, they're trying to

work out like how do I still get that

kind of dopamine hit in the same way.

>> Um it's not as much an issue for me

because I' later on, you know, I've been

doing this for almost 30 years now and

I've already been mostly working at the

area of architecture for and sort of

architecture and infrastructure for a

while. So I feel a little bit more

removed but it's uh yeah talking to my

son who's a programmer. He's got into it

around an interesting time right now.

>> And then you eventually made your way to

Thought Works where you spent a bunch of

time. What kinds of things did you do at

Thought Works? And what are what are

things that you learned there?

>> I did anything. I mean it was I joined

when there were probably just under 400

people left with about 4,000 and I

worked in you know got to go all over

the place. I joined at a good time

because a lot of the I mean we're very

much an XP shop and I think that the

core extreme programming. Yeah. I think

the core of the business still kind of

primarily is and I speak to people there

as well.

>> By the way, XP like was that influenced

by by Ken Ken Beck's XP?

>> Yeah. Know that the extreme programming

is very much it's it's it's Kent's

extreme programming. There was a bit of

a London flavor. Um so the kind of

London school of test development and

extreme the extreme Tuesday club XDC

around here. So what what's what's the

London flavor of XP and TV? It's sort of

like if you want to know you probably

need to read Growing Object-oriented

software guided by tests one of the

worst titled books in existence written

by a couple of friends of mine um Steve

Freeman and Matt Price and I think that

was a lot there's a lot of people in the

community around here that knew that

there's a rich meetup scene and so I

kind of got introduced some people

through there and came in and I jo I was

same time I was offered a job by

somebody else was a bank the bank could

tell me exactly what I was going to be

doing and thought works basically

the recruiter at the time, he's now one

of my very good friends, one of my

oldest friends now, she said to me, I

don't really know what you'll be doing.

And I'm like, that sounds like a job for

me.

>> I went for that one. And this is at a

time when like it, you know, we talk

about continuous integration, but the

tooling wasn't there and talked about

TDD and the tooling wasn't there. So, it

was in, you know, it was around that

sort of time that we started I started

building the build pipeline stuff in

about 2004. And then I worked Dave Dave

Farley was my first tech lead. Uh worked

alongside Dave Farley.

>> Yeah. Yeah. He was my first tech lead at

Thoughtworks. I I was one of the people

that interviewed Jez Humble to backfill

for me on a project at No way. So your

project at AOL of all places and I we

needed a backfield because I was like a

I'm sort of half admin halfdev really is

how I see myself. So we we needed

someone to take over my job and so I I

interviewed Jez. Um, and so I was around

when a lot of interesting things were

happening. So I was fortunate enough to

be there while we were sort of kicking

those ideas around. And obviously people

like Dave and and Jez put those ideas

together really really well. You're sort

of surrounded by everyone talking and

thinking about these ideas. I think you

got that I got the idea very early on

that this stuff's all up for grabs.

These things aren't set in stone that we

can push. And when you're saying these

things like you mentioned continuous

delivery, you mentioned what what other

>> um test automation.

>> Yeah.

>> Um build pipeline optimization, uh

infrastructure automation. I went over

to the US in 2007 actually to work in a

big I actually worked at Google for like

a year and a half for as the

thoughtworks employee.

>> Wow. In Google is we were doing we were

there doing a project um this all been

talked about publicly. It's a great

write up by Mike Bland all about our

experiences. It was me, a few of me, a

few of my Thor colleagues, some people

from Industrial Logic and some people

from Pivotal, the old Pivotal pre, you

know, pre-acquisition.

>> They were known for their really

excellent engineering culture

>> back when it was just Robbie's company.

And we were there to teach Googlers

primarily how to write automated tests

and how to write code that was testable.

And so I got out there and I was talking

to people out in the valley and stuff.

It's like we were going into these

enterprise companies doing test

automation using stuff like Selenium.

And you go around talking to people and

it's like people didn't I I remember

having conversations with people at like

Yahoo and even a big chunks of Google

who did not think you could do

functional automated testing of like

websites and stuff like

>> I guess cuz they just never did it

right.

>> Yeah. Absolutely. And for me like I hate

it whenever I haven't pushed myself or

like found a better way of doing

something. And I I was lucky enough that

early days in ThoughtWorks, that was

just a constant churn of ideas and

reinvention. I mean, goodness knows how

many mocking frameworks and inversion of

control things I was I I built two

different replacements for cruise

control myself, right? And so, but like

I you know, it was at that time I

created thing called DB deploy my

friends Nick Ashley and Graham Tapley,

which was like the first database

refactoring tool and all these sorts of

things because they didn't exist. needed

them and it was like if you if you want

to do it do it that was the mindset and

so I think there was always this idea

working at Thoughtworks was always like

what's next and I was quite fortunate

that cuz Thor works was growing so much

there was always things needed to be

done and my default was normally just to

say yes to everything so I got to travel

all over the world was fantastic met my

wife through works most of my best

friends have all come from Thor works

now still to this day so although it's

not been there for a long time it was

kind of really formative I think in

terms of that and it It's really

interesting because Thoughtworks has a

alumni pipeline people who used to work

there who are now really well known in

the industry for their ideas for for

think differently. Some some of them

educate people others uh have created

businesses to to build software

in better ways in certain dimensions.

And it's just interesting because like

usually I wouldn't think of I I'll just

like put it in a box like you know you

have product companies like like Google

like Meta like they have they build

their own products and and you have like

different tiers of these companies some

can afford the best people and they have

an external engineering culture again

Google was known for this Meta to some

extent and there's some companies maybe

who are like lower in the tier and then

you have the so-called consultancies who

are you know they don't build their own

stuff or they know do too much of it

they come in and they help companies

build or you have a problem and you hire

a group of experts Thoughtworks

technically is in this group and usually

you would not think that there's too

much innovation going on in these places

but with thought with thought works

proves to me yourself and other people

it's actually you can have like when you

are the best in what you do globally you

can actually have a lot of innovation

happening there as well right

>> yeah and it was innovation with a

different set of constraints I mean I

would say there aren't that many people

that have come from Thoughtworks that

were really great startup product people

I mean a lot of you know a lot of

thought workers make excellent scale up

people once you're all together actually

very few real entrepreneurs that came

out of the Thoughtworks machine. I I

know a few of them. There aren't many of

them. But I think there was always the

opportunity

to innovate even within a constrained

enterprise environment. So much of our

working conditions as a consultancy who

mostly did co-source delivery at that

time. We were doing kind of co-source

delivery, a bit of enablement through

delivery. Your working conditions are

often based on what the client does

>> and unless you're there explicitly to

change how the client builds software.

So what what you were what you got. But

that was self-perpetuating because we

were already used to sort of challenging

and pushing and trying to do new things.

That spirit carried on. And so I think

we grew quite aggressively but not that

aggressively rotating people through and

that spirit kind of carried on. I mean

look we had to start most engagements

with convincing people why testing was a

good why pairing was a good idea let

alone TDD.

>> Yeah. And I imagine how the conversation

went like a manager going like well what

would you pay then I have like you know

like eight engineers and now you're

making it four which sounds like a

terrible thing to do right

>> which and then this is I mean I had good

fortune because because of the people

because of work at thought as I got to

know Martin Fowler quite well and you

know he supported me a lot when I read

my first book I still remember when he

said well programming is not typing and

fundamentally it made it click for me

was having those conversations with

clients if the client thinks that

programming is typing

and That's why they don't want you to

pair program. Well, then you've got a

fundamental problem that needs to be

dealt with. And it's not a conversation

about pairing. The conversation is what

do you think programming actually is?

Well, but let's bring this a little bit

forward because right now we're having

the same conversation around right for a

long for a lot of CEOs, even founders

are thinking who have been building tech

companies, you know, they all look at

what what is the most timeconuming part

of what we do in building software. 50%

of the time has historically been

writing the code. So a lot of people

think you know like programming is code

and LMS are really good at turning out

code. So what is programming? Okay,

there's the there's programming I think

it's Peter no's article the guy who came

up with barkers nor form well he's no in

barkers norm he wrote his paper years

ago which you won the chewing award for

and he sort of describes

like the the program as being a theory

that lives in the mind of the developers

that it's actually us together working

together collectively who builds a

program

>> the fact that we happen to make that

program work through code the code is

really a side effect of how we work

together and I think product development

is just that extended right what we're

building is collectively we are building

a system together and the code is one of

the mechanisms we use to deliver the

system it's not the only thing so I

think holistically that's what it's all

about it's just that we focus on the

code because that's historically been

where we sort of spend the time but I

mean this is also classic theory of

constraint stuff though, right? If you

speed one bit up, you get constraints

elsewhere and it exposes friction points

elsewhere. I would also argue that a lot

of the time coding was not the problem

or not the bottleneck. It was how the

coding was done,

>> delayed feedback cycles and things. But

I do come back to that Peter piece again

because it's like Margaret an story

referenced it recently in her term pay

debt. But it's like building of software

is a collective ex it's a collective

endeavor. It's people working together

who build software. uh Kent put it in

his 21st book that software design isn't

exercising human relationships, right?

AI doesn't change that. And I think all

that's really happened over say the last

15 years in my experience is that we've

got a bit better at understanding that

we need better working relationships

like people like product owners for

example. But so that's what the pro the

programming is the art of creating. It's

the collective art of creating software

and it's not just developers that do

that. Never has been.

>> And so after thought works you've you've

gone independent, right? I did three

startups. None of them worked out for

different reasons. So don't ask me for

advice on how to make a successful

startup. Um and then I went independent.

Yeah. Moved back. I was living in

Australia at the time and moved back

from Australia and yeah, I've been

independent ever since. So doing kind of

a mix of stuff really ever since. I'd

written the first book of about

microservices. I thought my spare time

at Thoughtworks and then uh that book

came out and I'd been at Thoughtworks a

long time and I realized that there's

always a danger becoming

institutionalized.

You get to a point with a company, it's

like I think there's things as companies

do to change and I've tried changing

those things and they don't want me to

change them. So either I'm bad at

changing them, which is probably true,

or they don't think it needs to be

changed, which is also probably true.

And it's like I could hang around and

get bitter. I just go now. I go now. So

I went off and I thought I went

independent. It's been really

interesting ever since because it's sort

of on one level the shackles are off.

Um, the problem with working for

yourself though is that the boss is

frequently an

>> So,

current Sam frequently is cursing past

Sam for the commitments he's made.

>> Um,

>> that's one lesson, I guess.

>> Yeah. Oh, yeah. Yeah. Yeah. Yeah. I

mean, as I say, I'm I'm almost 15. I'm

still not learning the lessons of that,

right? And I've you've made those

mistakes for enough years now.

>> So, I wanted to talk about

microservices. Uh, you you have written

the book building microservices uh in in

2015. And you started to write it in

2014 and as reference Uber started to

talk about how they have thousands of

microservices in 2015. So you were kind

of writing this book right as yes

>> the concept of microservices which was

starting to go mainstream across

startups. Do you remember how and when

microservices took off and where was it

even coined? Was was it Martin Fowler?

>> No, it was James Lewis. So going back in

the day so you got to remember again

it's like something in the water. It's

like we have a bushfire. there was a lot

of buildup before the bushfire kicked

off and uh so this would have been 2011

to 2012 James was sort of talking to a

bunch of uh different companies who were

building what would have classed back

then as serviceoriented architectures

and early there'd been this kind of what

we refer to as the gorilla sa movement

which is an aim to simplify

serviceoriented architecture you know

shift over to rest work done by people

like yeah the original serviceoriented

architecture had like this this XML

format. What was the name?

>> Oh, wisdom and simple object access

protocol. Never has the word simple been

less well used.

>> Samap soap.

>> You had soap.

>> Soap. Yeah.

>> Yeah. Start with simple and the most

complex thing ever.

>> Exactly. So you had this whole heavy

weight and then people like Ian Weber

and and Ian Robinson and Jim Weber had

like been pushing centralized had

managers like make the endpoints smart

and the pipes dumb to keep things

simple. Put all the smarts in. So this

is all background. We were also at this

time doing a lot of work around like I

was running the cont practice for Europe

at the time and so a lot of my

engagements were like optimizing the

final mile. So you've written the code

and it's like how do I get it here into

production as quickly and effectively as

possible. So basically taking the ideas

of continuous delivery. So I was around

when JZ and Dave started working on

those in 2006. I' I'd had the

conversations with them. So that was all

happening. James was talking to

companies that were creating service

oriented architectures where they could

replace the services quite quickly. And

so basically you'd look at it and say

it's a finer grained form of S SOA. So

that was all happening. So James went we

had this architectural symposium up in

the Lake District and James pitched this

idea. He was called the micro apps at

the time in this room and somebody else

in the room said they're more like

services. Don't remember who said it

there about 10 of us in the room. most

of us were hung over and then James and

Ian so James and Martin wrote the first

post. So the first paper on Martin's

website was written by it was sort of

James' original kind of discovery of

this pattern that he called

microservices. At the same time I was

doing a lot of stuff around

infrastructure automation and so I was

coming in it from a different lens. So I

think James was coming at it more from

an architectural lens and I was coming

into it from a continuous delivery lens.

I spent at this point almost 10 years

with that being my focus. Yeah.

>> And so I was starting to see the

architectural impediment.

So if I want to change things

frequently, but the things I want to

change are integrated with other things.

>> Yeah. Like like like a monolith now. Now

that's how we would describe it.

>> Exactly. Right. I mean I I did a talk

called designing for rapid release in

2012 which is sort of touching this. So

basically James had this sort of had put

a name to this stuff and put some

patterns on it and I sort of thought

hang on a minute that stuff I was

talking about over here seems these

things seem quite sensible and I'd

always wanted to write a book. I mean

honestly I wanted to write an animal

book. I wanted to write an O'Reilly book

because that Zenor book I was given book

y

>> and so I just started writing and I I've

always found writing quite I'm quite

lucky that I find writing enjoyable and

quite easy. So I just yeah sat down and

wrote the book and that's it. But for me

it was an extension of continuous

delivery and it was a a style of

architecture that was only possible

because of continuous deliver because of

DevOps. We'd had these ideas before. I

mean Erlang has is is an representation

of these ideas but making it real and

practical and attractable thing. But I

mean really fundamentally it's just a

the whole thing has been an exercise in

smuggling through um concepts from the

1970s which is it's fine by me right if

I get people to pay attention to

information hiding and uh coupling and

cohesion that's all right but then as as

you said like this was there was a spark

before the bushfire and then the

bushfire did start like suddenly again I

think Uber was a very notable company

starting to talk in public about having

thousands of microservices on on

conferences like infoq and it got

people's attention and then it kind of

seemed that overnight a lot of stars

started to say like you know what let's

just do microservices

>> there was a bit of rebadging going on I

mean I was lucky when I wrote the first

book that I met uh Ben Christensen

fairly early on through the Yao

conference Ben was and Ben was looking

at um looking after performance and sort

of reliability at Netflix and I also

knew people at REA and these were two

early rea.com.au EU in Australia and

these were companies that were very much

pushing the envelope in different ways

and so I had two very good real world

case studies to draw upon for my book

>> but most other people weren't talking as

publicly around that and also before

Netflix referred to their architecture

as microservices they just referred to

it as fine grained SOA and to an extent

you know my understandings that's what

happened at Uber and elsewhere we didn't

have a different word for it then of

course what happened was that everyone

just called their S so SOA architectures

microservices which is sort of to be you

know semantic diffusion is a thing I is

to be expected so I just kept fighting

the good fight for like it's just

independent deployability and you know

it's an I've always positioned it as

it's an opinionated style of S so SOA

>> you may not agree with the opinions

that's fine do something else then but

there are some opinions

>> so then what is the difference between

microservices and what was it or service

>> serviceoriented architecture

>> service oriented arch what are the

differences

in in in your very

>> now to be clear right I have my

definition you might have your own

definition but I'm the one that wrote

the books on it so my definition wins

that's how that's always be my argument

right your own damn book um I've always

presented it as it's there are two kind

of very strong opinions I think we hold

with microservices the first and it's

most important is that your

microservices need to be independently

deployable

>> so your unit of deployment unit of

change needs to be a single microser

boundary a single service boundary

>> um and you know rather than undeploy

deploying everything together that's

three services at the same time etc.

even mini or less right and it's a

single service of time that's clear

>> that's it and the second one which is

which I think there are can be

exceptions to is that your kind of

default way for defining your service

boundaries is around your business

domain this is taking the concepts of

you can use domain you don't have to but

rather than a lot of sa architectures

were dominated to an extent by technical

decomposition the classical ender

architectures

layered architectures where you've

picked your service decomposition based

on the nature of

rather than thinking about the business

functionality. That's a bit softer.

Independent deployability is the main

thing though.

>> Okay. Well, because I'm now thinking

back at Uber. I was at Uber at the time

when your book was starting to become

really popular. I was there from 2016

and we had independently deployable

services. We didn't really think too

much about the businesses or or domain.

They they they had obviously their their

interfaces. They had contracts that they

offered uh increasingly strongly typed

and we we used

their voice to do it. But it was just

more like I mean as you can imagine

there was just devs often times not very

experienced devs who were like all right

like let's build a new service here's

the interface here's and then anyone can

use it uh which actually like there was

these things where a dev or even a not

very experienced dev deploy the microser

and then others start to use it and

suddenly it became you know years later

there would be these exercises to figure

out the hot path and and label the

services and then this like lone service

was on the path of like some really it

needed to be really accessible as a

really high tier service cuz a lot of

others were using it. I mean there was

always I mean also those sorts of

environments when I was chatting to

people at SoundCloud similar about that

it's something I mean and this is also

like about a corporate culture and like

what was acceptable like in that kind of

environment just go and do stuff it's

absolutely fine and that I think that's

that's why Uber for example went to I

just talked with Tuan Palm the C2 at the

time about like what was the real deal

about microservices and he said Uber

like he never really wanted that many

services but it was just more of a

constraint because Uber had a monolith

the story was like which he told on a

podcast, we'll link it in the show notes

below, that Uber had this this big

monolith and it was really slow. It was

slowing delivery down and they needed

they needed to move fast because all the

other competitors were out there. So

they said, "All right, like they wanted

to get rid of the models and put

together smaller services, but they knew

unless they had a strong mandate that

you cannot add new stuff to it. This

will never happen." So they said the

mandate was everything new needs to be

your own service. You take care of it.

We don't care. And suddenly so many

independent services were born, which

was not the goal. and years after the

fact that when Uber decomposed the

monolith, they're now looking at maybe

sensibly merging based on business

domain etc. But this story wasn't really

told at the time. It was just oh look

like Uber has thousands of microservices

almost one or two per engineer right and

this is what the world saw.

>> I mean it's also though but this is the

reality of microser adoption is like

it's just a tool toolbox. I mean I

describe microservices as being an

architecture of last resort.

>> Really?

>> Yeah. Yeah. Absolutely. because it

pushes you quite naturally to a more

aggressive distribution of your of your

state and everything else. And so you

have to have a good reason. One of the

best reasons to use microservices is if

what you actually want is an

organization with a high degree of

autonomy. If what you want is teams who

are able to act independently from each

other, the microser architecture is a

fantastically good fit. But then really

it's a means to an end, right? So that's

often what a lot of people get wrong, I

think, is they adopt microservices and

they think this other magic stuff's

going to happen. No, if you want

organizational autonomy, that's what

you're working towards. And this is just

one of the mechanisms to get you there.

And most of the enterprise

organizations, the vast majority of ones

I've worked with in this space, that's

what they're after. They've got huge

numbers of developers, lots of middle

management, they're not getting much

work done. They're thinking, what do we

do? organizational autonomy becomes, you

know, the solution to all problems.

Again, I don't always get to have a say

whether or not it's a good idea and I

think that's where microservices can be

especially useful. I mean, there are

other exceptions as well. And so, an

organization like you know Uber uh I I

mean I say I know at Netflix as well

things like domain design weren't talked

about Netflix but the teams were already

fairly well scoped around areas of

product already. Yeah.

>> And so you had those natural high level

boundaries around the business domain.

those things flowed naturally and so

that for me was a logical kind of

stepping stone. You also have the

difference and this is something I found

realized quite early on when you go and

do the work in the dotcoms versus the

enterprise organizations. When you say

dotcoms, you mean the digital companies,

tech companies,

>> tech companies. I'm old enough that I

still talk about docoms. Um, but the

tech companies, I mean, fundamentally

algorithm, you know, domain from a

domain concept are incredibly simple.

The domains are very, very simple

compared to enterprise organizations by

and large. Domains in there are much

much more complicated because they're

based on business processes that have

existed for 10, 20, 30, 40, 50 years.

>> Oh yeah. So, so what you mean in a lot

of tech companies, a product is pretty

much like here's our product, here's how

it makes money and that's kind of it.

And we have another product in Uber.

inside Uber like a simple thing is

there's rides which it makes money like

this and then there's Uber Eats which is

a separate business over time by the way

it does get more complex but but when

you start it's like really really simple

and it's really separate and it's clear

like where one ends versus where the

other begins and then you do some hacks

when you need to launch Uber Eats really

quickly and you build it on the right

infrastructure and then Uber Eats order

has like two trips attached to it

because anyway but but but but as as you

say like early on it it starts very

simple

>> but then also So in those environments

your business processes and the digital

implementation of those business

processes is evolving together. A lot of

enterprise organizations didn't have

that. I mean their first generation of

digital products were codifications of

paperbased processes.

>> Oh so so you're talking to these

companies as said before digital let's

say an Adidas or let's say a shoe

company virtually all banks.

>> Virtually all banks and and now they're

like we have all these processes and now

we need to build a software that mimics

what we do which of course is

>> that's how you start off. I mean that's

what the whole domain design book came

out 200 Eric it was danger design came

out in 2004 and it was a reflection of

the fact the vast majority of software

development was the process of us trying

to model existing non-digital processes

and how you do that I wonder if we will

have a different or but similar problem

of the same thing now that you know we

know AI tools are here we know that for

example they're great at generating code

and we have two types of companies we

have a company a digital company that

has existed preAI they're used to you

know writing product requirement

document deploying developers product

managers designers QA they all have

their roles and it it works like a

really welloiled machine and now you

have these so-called AI native startups

who just again they have a few people

they're just I from the start they're

using agents for everything they have a

very different process and at some point

when this whole thing will mature and we

will have best practices we'll have the

existing companies who like the previous

one they will try to retrofit you know

this this tooling to how they're working

today.

>> It's already happening. I mean, and the

thing is enterprise companies will

rarely be successful at doing those

first gen things because ultimately the

corporate DNA will just crush that stuff

from within, right? And it will take a

while, but I think the whole industry is

getting is grappling with the idea of

like are we now moving to a new level of

abstraction, right? So software is all

about abstractions, right? But you know,

we went from doing machine code to doing

assembly, went from assembly to general

purpose programming languages. And

you're coming back to the idea of the

program being the mental model. We would

evolve that mental model at the code

point of view. That was the most

effective place was us working at Java,

you know, Java, you know, C, whatever it

is, rather than working at assembly

level, right? And it's like, okay, well,

is actually what we're talking about

really. We're stepping that level up.

We're still actually building a product

together. We're still thinking about

what the software is. we're still

building that mental model maybe now

more collaboratively involving you know

more product people for example in that

conversation when we're writing our

specs and so the question for me is are

we ready to make that step and the

answer is no not everywhere and this is

a really good point because if if you

think about it like uh you know like one

of the the mantras inside of Facebook

used to be that I think some something

along the line that code is code is

truth or something like that but in

general in a lot of tech companies if

you want to know like how things work

like where the truth lives is it in

documents or is it in the coded business

logic it's in the code in the business

logic and if if it's well-maintained

code often well commented or structured

you can actually see the business logic

and the rules for I don't know for

example see I I feel like I've got to

channel my friend charity at this point

and say the truth is production

>> that that's everything else is is is the

lies we tell ourselves production is

truth

>> um but I think in terms of like a

definition of what the system should be

doing. There is this ongoing debate

about where is that definition? Is it

the code or is it the spec? Brigitta

broccoli did a nice article on Martin

Fowler's website recently about spectrum

development and she talks about the idea

of some companies using uh being spec

first or spec anchored and a lot of it

comes down to where they see the source

of truth being. So a lot of companies

are using things like Kirro or Spec Kit

to define a feature and then the LLM

executes that spec and you can iterate

on that process, right? And so a lot of

people they start off they use the spec

to write the code and then they throw

the spec away and then you get other

companies that keep the spec and then

when the feature changes they just

update the spec. And so they're sort of

seeing a bit the code and a bit the spec

as a source of truth in terms of what

the system should do. And then you're

start now starting to get companies

where they don't even ever look at the

code and they just look at the spec.

>> Yeah.

>> And so we're kind of going on this

spectrum. She had some great terms for

these. And I think for me it's like is

that the right path? I don't know. I

think there's a lot of questions about

what we would need to do to to get that.

we have a degree of confidence about

what happens when I execute a line of

Java code beyond the safety of the

compiler and the runtime and you know

all that stuff. There's a lot more

questions to answer I think before the

whole world is ready to say the spec is

the source of truth of what our system

is supposed to do. Production is always

going to be the real truth though just

to be clear.

>> But but I think it's a good question to

just ask yourself what however you're

building and however you're using agents

of where is the truth of your system

right now. Is it more in the code? Do

you have a spec? Is it scattered between

the minds of people? Uh- which goes back

to the the collaboration? Because again,

a lot of the reality of a lot of

startups are is a few people hold these

things in their head, right? Which is

okay in in the beginning. But of course

as you scale one of the reasons we used

to love code review which is now going

out of fashion but we used to share

information and when someone left the

company or or got ill or you know the

the famous bus factor it was okay cuz

other people understood because they

reviewed they were taking part in that

part of it but now we're with with AI

code review is some some some companies

are like well we're not going to do it

because of it's it's with too much code.

That was always one of the superpowers

of pairing, right? Pairing you if you do

if you do promiscuous pairing, you

rotate every day. When someone leaves,

everyone's got the same shared context.

That was always something people missed

around one of the benefits of pairing.

>> I think there's a lot to be said around

this. I was chatting to Chris Ford who's

sort of head of software head of

technology for Thoughtworks in in Spain

and he said the biggest problem we've

got around our around AIdriven software

delivery is tacid knowledge. There's a

lot of knowledge in our heads that we've

built up that we don't necessarily know

that we have. When you use something

like a code review, you've probably had

looked at that and that doesn't really

feel right. I wouldn't do it like that.

And sometimes it's subjectivity, but

sometimes there's something about that

that you can't quite pinpoint. Maybe you

can unpack it, maybe you can't. But

there's also like you have bunches of

things like that in your head about how

software delivery should be done, about

how the business works and everything

else. If we want to step up that level

of abstraction to work at the spec

level,

we actually need as human beings to be

able to unpack that tacet knowledge that

we have and make sure it's in the spec

if we really want to be be hands off. I

also think there's a fundamental problem

that we've got in the current way a lot

of people are doing software deliveries.

You've got I mean this comes back to the

cognitive debt stuff which is you've got

a lot of people running off in different

directions. I've seen a few people

talking about pairing with their AI

assistant. not a human. Stop pretending

it is. And so you're all going off.

You're doing more in a different

direction. That shared mental model of

the program is breaking down and there's

gaps in between. AI was supposed to free

us from drudgery, right? It it isn't for

most software developers because we're

doing more work, we've got more context

switching, we're losing that big

picture. So I think if we can work

collectively on the specs, we can

rebuild that model, then that absolutely

could be the sh source of truth. I still

think it's going to be parts our brains

and part the spec, right? But for that

to work with there does have to be the

collaboration piece. Whether or not

that's our dev teams go from being 15

people all in to about five people or

six people all in, don't know. But

collaboration is really important. You

know, Chang proper company, right? They

do proper stuff. They're excellent firm.

They recently announced they were

getting rid of any human-based code

reviews and they said we're going to

spend more time working on things like

effects working at design doc level and

actually I'm okay with that because

firstly I trust that they know what

they're doing but also there's something

about I mean I think a lot of companies

are making a mistake of not doing the

thinking whereas Shanggard is saying

we're going to spend that time doing

more important thinking right and I

think that's sort of how we should view

this I'm okay you know not looking at

the code as long as you're doing other

things to make sure the

is operating correctly and that you're

thinking critically about how the system

needs to change and evolve. If you're

just not looking at the code and you're

not caring about those other things,

well then you deserve everything that's

going to happen to you.

>> Which also goes back a little bit to

what Charity was talking about on how

you want to have systems that that value

that works about having empathy for

example for for the work that QAS and

and uh SR folks have had to own code in

production that they did not they did

not write. they don't necessarily read

or even understand, but they know how to

operate. They know what good looks like

in production. They they know what what

it looks like when it goes goes poorly.

>> I mean, that's also a bigger thing that

now more developers are having to

struggle with because if you want to do

spectriven development, you have to know

what good looks like. I did a I I run

these like sort of half day conferences

for O'Reilly online. And we did one

recently on software delivery and a load

of speakers at um and about six speakers

and four of them were talking

specifically around different different

varieties of of spectrum development

effectively and all of them came back

says you can't do this unless you can

define what good looks like and if you

can't don't bother and you know some of

those people were much more bullish like

Addie is very bullish on all of this

stuff but one of the presenters who was

doing they did they've got like a small

software fracture only 15 or three

people but like they're active ly

pulling in their observability data

their hotel stuff right into uh into

their their agent to understand how it's

changing and I think that's exposing I

think more developers into understanding

I mean this comes back to the acceptance

criteria stuff people do on stories they

always narrow around functional

requirement stuff so now it's like I

think we have to be more grown up about

those things the discipline is

absolutely important and the expertise

is still needed it's just needed at a

different place

>> Sam made a really good point that you

cannot hand your work over to agents

unless it does these two things. One,

define what good looks like. Two, prove

that your system does what this good a

definition looks like. You already know

who I need to mention when it comes to

proving that your system works as

expected. Antithesis, our season

sponsor. Agents are writing more of the

code. So, how can we trust that our

system works correctly without bugs?

Code review is increasingly naive and

ineffective approach to do so, which is

why antithesis goes well beyond this

practice. Antithesis runs your whole

system in a hostile simulation. This

simulation includes both targeted

testing and fuss testing. By running the

simulation, it finds every bug before

your users do. And because the

simulation is fully deterministic, it

doesn't only find bugs, it gives you a

perfect reproduction of every issue,

which makes it a lot easier to fix

issues. The first thing I thought when I

heard about antithesis and that it's

automated bug discovery and fully

deterministic testing that it sounds

like science fiction, but actually it's

hardcore engineering when you look under

the hood. Jane Streetfly.io and the CD

community ship agent written code with

full confidence because they know it's

been verified by antithesis. To see more

case studies and details, head to

antithesis.com/pragmatic.

I'd also like to mention our presenting

sponsor, Turbopuffer. We know it's a

database that's ridiculously fast and

extremely cheap and they are doing

something amazing. They are completely

redesigning their storage architecture

from first principles to make search

faster, cheaper, and more reliable at

scale. If you follow Turbuffer, you know

that their storage architecture was a

massive part of their early success.

Redesigning a winning architecture is a

big deal. It's one thing for query plans

to pass unit tests, which is already

happening. It's another to bring each

query plan to performance par or better

while maintaining correctness and

reliability in production. And the team

is sharing the messy reality of the

rewrite and how the v3 version is

currently a lot slower than the current

production one. For example, we can see

how coldtech search is 11 times slower

than prod and hot full text search is

126 times slower. But we can already see

some wins like hot and cold vector

search is at par or better. I really

appreciate this work in progress view

and the real-time logs and what it takes

to do a successful storage architecture

rewrite. The Turbopuffer team is keeping

on the worklog of their journey and you

can follow along at turbopuffer.com/v3.

I definitely will do. And with this,

let's get back to Sam and what QA and S3

folks already know about running code

that they didn't write. So, I I I want

to talk to you about your latest book

that is just coming out, building

resilient distributed systems.

>> Yeah. What is the book about? And why

did you feel this is a good time to

write it?

>> Years and years and years ago, I went to

Velocity which was quite a well-known

conference.

>> It's a conference. Is is it still on?

>> No. No. It was the O'Reilly conference

and it was like the conference where a

lot of the early people went to talk

about how they made their system

scalable and stuff. I got to go. It was

been like one of those career ambitions

to go to. I got to go after I' written

the first book and I got chatting to

John there who I still chat to every now

and then. he now works in this

resilience engineering space. pointed me

I don't know if it was him or one of

someone else but it pointed me at this

paper by David Woods that was all about

resilience engineering and the paper was

utterly baffling to me in absolutely no

sense to me but it was all because it's

not yeah cuz it's just it's not written

for computer scientists to read and I'm

not even a computer scientist so I was

really in trouble

>> but it was really fascinating seeing

this whole other world about how we make

how how people viewed resiliency and

then what we do in software world and

people like John were trying to take

some of those ideas in a little bit but

I found it very difficult to take some

of those ideas from sort of safety

critical systems and that type of

thinking and make it in any way, shape

or form palatable for normal people like

me. And so I kind of this was being back

what 20 2016 maybe 2015 and so I'd go

back and try and reread that paper every

six months or so and I'd read bits

around the edges and see other smarter

people than me that summarized it data

better. And then I was thinking okay

these ideas now are starting to make

sense. And then also I kind of realized

that despite my best efforts, lots of

people seem to be building microser

architectures. Even though if you've

read my books, most of my book is about

telling you not to do it.

>> But you have to read the book, which

which is called microservices.

>> I mean, let's be clear, as long as they

buy the book, that's the most important

thing, whether they read it or not.

That's just, you know, but and so then I

realized that there are more people

exposed to the the the horror, pain, and

suffering of a distributed system. And

so it was like, okay, so I've got this

stuff happening. I've got this like my

you know my my son's a developer now and

I always thought that if somebody found

themselves building a distributed system

working as part of building a

distributed system what was like the the

kind of equivalent of a rubber ring I

could give them say stay a flow and

don't drown and so the goal of the book

is to start you off with really easy

foundational simple kind of engineering

concepts about how you can start making

dealing with the problems of distributed

system but to also take you on a journey

into kind of the whole concept of

resilience engineering and so it starts

off with like really easy ideas and then

you study it gets more and more

complicated and you know so I introduce

kind of concepts like what a

sociotechnical system is the difference

between safety management and resilience

engineering these are all like big

concepts but the second chapter after

the introduction is what's observability

let's talk about hotel the sec the third

chapter is what's a timeout right and

these are all I think quite tractable

things talked

>> but they're building blocks right the

primitives for Yes.

>> For these systems and the vocabulary as

well.

>> Yeah. Absolutely. And to give, you know,

to go through and sort of distill some

of that knowledge down

>> and also to be a bit of I'm quite happy

being a bit opinionated on these things,

but I'm also lucky now because partly

because my early work, I get to chat to

more people. So, I can actually talk to

people who this is their full-time job

and try and distill down some of that

knowledge a bit. And so, I wanted that

book. Partly, it's about being

evergreen, but it was also about saying

there are lots of people building a

distributed system. Let's put what type

of systems you've got off to one side

for a moment. There are lots of people

building them whether they should or not

and they're really important to us now.

So, how do I how do I help people start

and grow? And that's that's what I

wanted to do. I mean, I enjoy the

process of writing anyway, but this is a

book I wanted to be able to give to

people. That's always been why I wrote a

book. I want the book that book every

now and I've gone and sat down to read a

book to write a book and I found someone

else has done done it already. It's like

well that exists now. So this for me was

the book that I wanted to exist but

didn't. That's kind of why I wrote it.

>> Yeah. And I I think it's a book I I I

certainly could have used like it got

dropped into a place like Uber where we

did have what I realized were

distributed systems. You don't just

large systems handling a bunch of stuff

lots of services dependent on on one

another cuz even a simple thing of like

knowing like oh you should pay attention

to observability that I mean you figure

it out after a while.

>> Yeah. Yeah.

>> But it takes a while.

>> Yeah. And I think that's I mean again I

mean maybe this is this is my attempt to

unpack some of that tacet knowledge to

an extent. I've made so many mistakes

and I keep making mistakes. I think

making mistakes is useful but I think

also creating opportunity where people

can learn from other people's mistakes

first is a good idea too. And so in the

book I also try and pull in like real

world examples and sometimes from big

companies sometimes from my mistakes to

kind of make this stuff seem real. And I

think the the other thing I think I

really try and get across is that things

go wrong. Right. I mean, resilience

engineering is all about making sure as

many things go right as possible. It's

not about making sure everything goes

right.

>> Yeah.

>> And there's also the only way you can

build resilient systems in general is

also by being honest about mistakes that

have been made and talking about them

and sharing them because that's how

everyone gets better. And if you don't

do that, you can't build a resilient

system in your company. And so this

again is my attempt to sort of highlight

all the awesome work that's gone before

the fantastic post instant reports of

people that some companies share like

when there's a system that we all use or

many of us use has an outage. I mean,

it's it's annoying, especially when

you're using it. But I always greatly

appreciate when the companies because

they don't have to do this, but when

they share a public postmortem or

incident recap or whatever that is, they

go through what happened. For example,

GitHub had an outage and they shared a

really interesting one on what all the

things that went wrong and and in what

order. And then you also have companies

where something like this happens and

they don't do anything. So, of course,

you're not obliged to in any way, but I

always have a bit of a lesson. they are

obliged to now a lot of them regulations

are now making them that they have

>> which I which I I think is great

>> yeah things like Dora and stuff in the

UK in Europe rather and I think there's

also we should also separate around that

post instant report stuff that there's

marketing and PR versus have you done a

proper post instant review now I do

think you've got duty of care really to

let users know as part of a proper post

instant review but those companies maybe

say nothing they might do a fantastic

process internally there's also the fact

that some companies that put those

things out you read them and you're like

this does feel more marketing.

>> Yeah,

>> I think a great if you want a good a

company that puts out some of the best

self- flagagillating post instant review

writeups, go for Cloudflare stuff. They

absolutely go for themselves in those.

It's like I feel like guys go easy on

yourselves, right? Everyone makes some

mistakes, but I quite like

>> and and do it they do it rapidly within

24 hours and they're a publicly traded

company. Yeah. which which I think just

goes back to the myth like some of them

are like we're publicly traded we don't

want to do it because it would give away

information for shareholders yet no like

cloud for does it and and they're fine

>> exact exactly and and and also it's

>> and it builds of trust

>> it is I mean a lot of a lot of how an

organization is perceived is is down to

how you deal with mistakes when they

occur right I mean you know nothing ever

goes perfectly for every company

>> so in in your book you talk about the

three rules of distributed systems what

are these three rules

>> so I really tried to distill this down

because I could never remember the the

eight the rule of eight that came out of

sun the fallacies of distributed

computing I always forget what those

eight were there's loads of complicated

stuff but I realized when you strip it

all down there's really only three

things that really cause all the other

problems right the first is that you

can't send information instantaneously

from point A to point B

>> it's physics I mean now it turns out

when you get into it and start talking

to people about quantum physics they say

aha

>> and they go there's a thing called

quantum entanglement which does allow

for theoretical instantaneous read of

electron spin states across arbitrary

distances at which point I point out

that a that only works for reads as far

as I'm aware and b not a networking

protocol. So like but it takes time,

right? So I want to get from A to B and

I that I can't control all of that. I

can control bits of that as a developer.

>> Y

>> I could control how much data I send. I

could control the the the payload size

technology I use to serialize and

deserialize.

>> I'm unlikely to be specifying the

network cabling or the equipment we use

or the distances between them. So not

really under my control. And the moment

you go over the public internet,

>> BGP has a bad day and your packet has

gone down the wrong trail leg of the

internet. Right. Yep.

>> So that's the rule one. It takes time.

Um and that's important to recognize

because sometimes it looks

instantaneous. It's not. Second thing,

sometimes the thing you want to talk to

isn't there. Sometimes that's under and

we do things to make it under our

control, right? Like we would run say

multiple copies of a service instance

behind a load balancer. What happens?

The load balancer fails. But I mean like

you can't control for everything, right?

I've had services taken out by a fire in

a data center. a sand caught fire and

specifically one bit of a sand. The sand

had a caching storage layer with SSDs.

These two pairs of SSDs, one of the

pairs of SSDs caught fire. The other

pair the SSDs were working fine. And so

we had these weird latency graphs coming

out of the sand like perfectly fine if

your load went through the ones that

weren't on fire, but the failovers in

the other one were rooting to be fair.

It was rooting around the bits were on

fire. I had a system uh where we had

ducting between the office buildings and

rabbits moved into the ducks and ate a

networking cable. I can't control for

rabbits, right? And sure, these things

are like they're real. Those are two

things, right? Yes, you can control for

that and you can reduce the chances, but

you can't limit those chances, right? So

sometimes, so rule number two is

sometimes the thing you want to talk to

isn't there.

>> And the third rule, I'm very grateful to

an old colleague of mine, Ben Cole, for

telling for for saying I've missing is

that resource calls are not infinite.

meaning

>> you can run out of computing resources,

CPU, memory, IO, whatever.

>> And basically your load balancer could

be overloaded and then just not respond

even though it's there and we know the

latency, but it maybe a skew fills up.

>> And the vast majority of the time when

you have a system outage, it's down to

that

>> some resource has been totally saturated

somewhere. And it might have been that

saturation might in turn have been

caused by the fact that something is

down and you're in a retry storm which

is in turn causing you know connection

starvation which is of course resource

saturation which is in turn yada yada

yada y

>> but that that's the killer and and again

I think we understand that you know sort

of intellectually all of these things

but you can go to all the concept

complicated concepts and distribute

systems and sort why is bisonantine

consensus protocol a thing well let's

just go all the way down to this bit

what about cap and packel and I just

yeah let's put that on one side remember

these three rules first and then most

problems end up becoming a little bit a

little bit easier to deal with

>> and and in the book after you introduce

these rules you start as as we mentioned

the second chapter is observability to

me that kind of suggests that this is a

really really important part for

distributed systems why is observability

so important

>> I mean the way to think about

observability is it's a property of a

system right so the more observable

something is the easier is for me to

understand how that thing is behaving

based on the external signals. When you

run a pro a system which is a single

program, you actually get some pretty

high quality signals coming out of that

program. Therefore, give you a high

degree of observability.

I can get a lot of information out of

that. I can attach a debugger to a

process. I could generate a core dump

>> and run that in. I could do a thread

dump on J on the JVM and work out where

my semifalls are blocked. If this is my

whole system, like knowing the CPU is

stuck at 100% is actually really useful

information because that's probably a

bad that's a problem. Right. So this

basic information that we pretty much

get for free with a single program gives

us a high degree of observing program

running on your machine that you have

access to.

>> Exactly. Right. Yeah. But but now when

we go to a digital system, you don't

have access to the computer in the same

way. You've got multiple computers. And

so those basic signals are harder to

get. You can't correlate them unless you

do extra work. And those individual

signals by themselves don't really give

you a proper sense of how the system is

behaving. So, so you have to do more

work to make the system observable and

as as annoying with these sorts of

things, it's much easier to do that from

the beginning than it is to retrofit it.

And then, you know, luckily we've got

things like hotel now, the open

telemetry um protocol and specification

because that at least gives us an open

framework that we a vendor agnostic. Oh,

honeycomb are the best. Charity pays me

in stickers. Just to be really clear, I

have to be I always think I work

independently. I have to be very clear.

Charity pays me in stickers. But like we

get a vendor agnostic open standard for

this now. It makes it so much easier for

me to have the conversations about this.

But if I don't get the signals coming

out of the system, if I don't get the

that stream of structured events telling

me what the hell's going on with my

system, I'm flying blind. Like how do I

know what appropriate timeout is for

from a client point of view talking to a

service? Well, I'm only going to know

that if I have a sense of well, what is

its normal uh response time um

distribution? So, I'm going to need

latencies, know the success or failure

of those latency. It's probably going to

be a plot on a histogram maybe to take a

look at it. If I don't have that

information coming out, I'm not going to

specify a timeout correctly.

>> And then we're talking about a low-level

thing like latency and and things that

we can measure at the machine level.

But, of course, is is this observability

when I'm talking about like does it work

for the user or the customer? Does it do

what they do when it they click click

the button to submit? Does it submit? Do

you look at that because we know

observability is usually we talk about

these kind of latencies the the statuses

the 500 responses that's kind of bad

that's means it's down but there's

always a layer that I always wondered

and I found it hard even when we're

working in teams to okay like we have

the the easy observability like this is

kind of text not textbook

>> it's the metrics it's the metrics but it

looks but then there's the stuff of does

it work does it do what people care

about like again in the case of like an

Uber Like when people hit a button to

request a ride, like does it actually

request within within usually 5 seconds

or something like that or is it stuck in

an infinite spinner which is which is

bad. Even your locks can be as healthy

as can be but people are upset, right?

>> You know, you you treat the raw data of

making the system observable as as a as

a stream of structured events and then

from that you build up the higher level

abstractions and the views, right?

>> So use as primitives.

>> Use them as primitives. So a trace,

right? where we get a sense of for a

given click of a button what happened in

the production system and this is a big

blind spot right most organizations had

to build their own tracing systems. So

this is where things like the open

tracing stuff came from and things like

zip kit and things like that were built

as an attempt to recreate like I mean

most people were just trying to recreate

what Google had already built there and

this is an attempt to say okay can we at

least look at a flow of calls what

happened during that flow of calls where

did these things go

>> well that requires me to get a bunch of

spans

ac

>> and a span meaning

>> a span meaning an activity basically

something I did

>> um and then a span can be gapat more

spans So I can say my first span might

be a call has hit my my server, right?

So I've now got a span that starts

within that span I might talk to my

database and then do other operations

and I can put additional information on

those spans and then I call to another

service and then you a trace ID that's

shared across those spans and you've got

timing. So now you get effectively a

view of what's happened across those

courts but that's all just based on

individual events been stitched together

and then you also though start getting

into system situation which is is my

system behaving correctly and you know

this is going away from again as I think

half of just saying what charity would

say um you know again going away from

the vanity dashboards this is moving

into sort of the world we call the

SLOs's right the service level

objectives

>> and SLI

>> and SLI so an SLI service level

indicator that's our way of saying at

this given point in time for this thing

is this thing good or is this thing bad?

And we use a collection of those S. So

one example of that might be a

successful percentile response time.

Right? So that would be is it a good one

or a bad one? It could be a bad one

because it's an error. It could be a bad

one because it's slow, good or bad. And

I collect those together and that helps

me know if I'm meeting my SLO. So I

might have an overarching SLO which is

uh I should be able to uh request a ride

with 99 percentile successful ride

request within uh 300 minutes.

>> Yeah. And that would be your statement

and that would be a commitment. The SLO

is supposed to be a commitment. That's

something our team is is working

towards. You then get this raw data. You

calculate your SLI on the individual

things and then you use that to work out

what your SLO is looking like. If it's

green, everything good time. If it's

red, people are unhappy. I mean, you get

a bit of red. There's there's way more

nuance to it than that. I'm to give a

very bad introduction to this stuff, I

think. But without those building blocks

cuz I I talk about in the first chapter

of the book, nobody says I don't want my

system to be resilient.

>> Yeah. No, no one.

>> But at the same time, resiliency is not

a binary property. It's not like off or

on. It is a bit like observability. It's

a degree. So then the question is how

much do you need? Because am I having a

conversation with you about multicloud

and you know doing complete rewrites of

systems for critical standings like

Monzo do for example?

>> Yeah they have a they have a standing

bank when they go down so it's it's

sitting there it's costing them money

for the most part but then just a few

weeks ago their main application went

down for a short period of time and they

activated that bank which I'm assuming

it cost them millions of pounds or

dollars per year just to keep it there.

It's not even that slight sidebar, but

so firstly there's their standby bank

has a real stripped down list of

features. It's much more bare bones, but

it's all the the core ones like you can

freeze cards, you can take money out,

you can check your balance. It's a

complete rewrite, so it's no shared code

because they're worried about a bug.

Different cloud vendor. It's also by

design significantly more cheap to run.