It's kind of funny. You know, we talk

about Kubernetes being cloud native. The

reality is a lot of customers actually

use Kubernetes for running on premise. I

was talking to another one of our

customers actually just the other day.

They've got Kubernetes clusters running

on research vessels.

>> Research as in like

>> as in boats

>> on the ocean.

>> They've got Kubernetes clusters out in

the open sea.

>> What is GitOps?

>> GitOps is potentially not necessary for

all teams. Some of this absolutism that

sometimes exists may not be necessary.

>> I don't hear too much chatter about roll

backs.

>> Roll backs. This is always a spicy one.

Customers can go, "Yeah, we roll back

all the time." And then when you ask

them what do you do if you've got a

schema change they kind of stop and

realize that it's just sheer luck that

they've never run into that. You want to

avoid ever talking about roll back. It's

always roll forward.

>> When it comes to CI/CD systems, what are

you seeing changing there because of AI?

>> This is the elephant in the room to be

honest. It's

[music]

CI/CD remains one of the hardest things

to get right [music] in software

engineering. But why? Rob Eris is a

CI/CD expert having worked in this field

for more than a decade. In the early

2010s, we were teammates on the Skype

for web team and then Rob joined Octopus

Deploy as one of the first engineers 10

years [music] ago. In today's episode,

we cover progressive delivery in

practice, Canary deployments, blue

green, and why feature toggles are often

still better. [music] What is GitOps and

why it's not about Git and where the

everything in Git mindset breaks down,

why you should prioritize roll backs

less and focus on roll forwards and many

more. If you want hard-earned lessons

about CI/CD, progressive delivery, and

what's coming as AI changes, how much

code we ship to production, then this

episode is for you. This episode is

presented by Inticus. Verify your

systems correctness without human review

or traditional integration tests and

avoid bugs or outages. Today's episode

will be about CI/CD. CI/CD at scale is

one of the hardest infrastructure

problems to get right, and the teams who

nail it know that the details very much

matter. This is where I need to mention

our season sponsor work OS. Work brings

the same rigor as many of us use with

CI/CD at scale to enterprise o SSO skim

areback production ready battle tested

and built to handle real load and real

compliance requirements. To add

enterprise o without the infrastructure

project visit workwest.com

it's awesome to have you here on the

podcast.

>> Hello go. It's good to be here. Yeah,

I'm loving loving Amsterdam. Yeah, it's

it's been like what 11 12 years since we

walked worked together.

>> Yeah. Yeah. I think uh 2015 2014 2015 I

think I left uh UK. Um yeah, a while

>> and Skype when there was still Skype.

Our team somehow inherited the

outlook.com plugin which had like 400

million users per month or something

like that.

>> It was crazy the amount of

>> scale. So this was an interesting

interesting job. deployments were very

much a case of, you know, you ship once

a week and you have to go to a um a CAB

board, you know, a change advisory board

and you have to get sign off and

approval.

>> Yeah.

>> And I always found that really weird,

right? Like we're building this piece of

software. It runs on the web. We can

ship it whenever we want. It it was

running on Azure at the time. And so,

you know, we've got full access to push

whenever we want. And we make these

changes through the week, but we'd kind

of have to hold them back, I guess, to

Abdullah, our manager, both of our

managers at the time.

Um we kind of I guess worked around the

system. When the the code was ready,

we'd build it and ship it through

through the week. Um and I was really

sort of impressed and proud at this

process that the whole team had kind of

put together, right? Where we'd, you

know, commit the code. Uh the test would

run several kind of layers of testing,

it would go to staging, etc. And then it

would get shipped to production.

Um so we're kind of I guess executing a

form of I guess you know continuous um

delivery at a time. And [clears throat]

we would then ship ourselves, you know,

once a week. Um I kind of always like to

tell this story that um at the time, you

know, when when we'd have a build ready

to go, you know, we we do a form of

canary deployments. And so this is where

you kind of roll out to a small

percentage of your customer base. And we

always found that the the customer base

that would be our test subjects was New

Zealand. So New Zealand was always our

our canary.

>> Yep.

>> A bunch of reasons for that. you know,

they're in the the you know, the the

first country to kind of reach this, you

know, new date. So, they're always the

first ones to kind of roll out into a

into a time

>> when the it comes like, you know, like

midnight passes. It's like 1:00 a.m.

First country is New Zealand.

>> Bang. Exactly. So, the first country

that's of, you know, significant size.

Um, they speak English, so if there's

any bugs or issues or reports, it's kind

of easy to understand. But, you know, to

be honest, New Zealand is small enough

that no one really cared if we shipped a

bug and had to fix it quickly. So sorry

to all the New Zealanders listening.

Yeah, I I I think that's um kind of this

this good example of using um a

continuous delivery technique to um you

know ship the code faster than what we

otherwise could have if we had these

kind of big bang releases. And this

whole process I guess opened my eyes to

you know what what progressive delivery

what good CI/CD could be. And yeah, I

guess from there I spent a few years

there at Skype and eventually wife and I

decided it was sort of time to come home

to Australia.

>> And then back in Australia, you went to

start to work at Octopus Aloy.

>> Uh yeah, eventually I I came back and

actually um worked at a a place with a

friend of mine just for a little while

just to kind of get back on the feet. Um

and I remember they were using Octopus

Deploy there. And so Octopus Deploy for

those who don't know was is um a

deployment tool was built and and sort

of developed originally in Brisbane. So

there was a strong kind of Brisbane

>> attachment

>> attachment to it. Yeah, that's right. So

when I found out that they were hiring,

I thought okay, why not? I'll give it a

go. I like CIC CD. I like this space. I

think there's, you know, a lot of

interesting problems in this space. Um

so applied and joined and um at the time

I was employee I think employee number

eight or nine or something like that. So

it was very much still a bit of a

startup culture. Definitely not startup

in the sense of, you know, Silicon

Valley wild parties and, you know,

ridiculous spending, but startup in the

sense that everyone who I worked with

was an engineer. Now, even Paul Stella,

CEO, he's he's an engineer. This is kind

of where it started from. And so, we'd

all be working on on code together. Um,

you have someone had an idea, you'd have

a bit of a chat about it and ship it.

So, we were the marketing, we were

support, we were done a bit of

everything. Um, and yeah, obviously the

company has grown a lot since then. The

company was focused Octopus deployed

from the start they were focused on

deployments right can we talk a little

bit on whenever I think about deployment

I always say CI/CD continuous

integration continuous delivery why was

there a focus on deployments and is that

the same as continuous delivery

>> yeah interesting so um you're right like

quite often people talk about CI and CD

as this kind of interchangeable they're

either interchangeable or the word is

CI/CD is like the name

>> it's just attached to itself

And it's hard for me to imagine a CD

without a CI, continuous integration.

>> That's that's right. And I guess the way

to look at it is um you know, you've got

sort of multiple stages of maturity of

of um software teams as they kind of

move on their way from um you know,

initially uh CI which continuous

integration. This is the idea that

>> well initially it's YOLO.

>> Initially it's right initially. You just

deploy to prod or SSA like machine.

We've all we've all worked in places

where we've done that and that that's

the starting point. So you're right,

yolo is the first stage. The second

stage is you know continuous integration

and so this is this idea where um you

want to keep you know integrating

merging your code changes into a single

um a single branch and you want to be

continually running tests against it.

Now continuous delivery is kind of the

next stage where you know we talk about

testing our code and there's you know

unit test and integration tests etc. But

what you also really need to test is

your your deployment process itself.

Right? So continuous delivery is this

idea. Okay, you want to make sure that

at any point in time when I click the

button to deploy I want it to go to

production once we kind of get to this

place. Um the next stage beyond that

which you know not all companies

necessarily reach is uh continuous

deployment. Right? And so this is the

idea that not only are your your changes

being merged and merged together at the

same time and ready to go, but they're

also being shipped to to production

essentially.

>> So the stages we have is first yolo,

then continuous integration, then

continuous

>> deliver

>> delivery and continuous deployment.

>> That's right.

>> What is the difference between

continuous delivery and continuous

deployment?

>> The big difference I guess is um the

question of do your changes go out to

production automated? does it kind of

flow through without any um intervention

I guess

>> and then for continuous delivery they go

out but not necessarily to production

right

>> that's right and so that's why you'll

have environments like you know dev

environment or testing or staging or

whatever um now it's possible that you

know some parts of that process may also

still be manual you know maybe you only

update the test environment once a week

so the testers can play around with it

again um but the key principle is that

you you could you can kind of push it

through sort of automatically the whole

way through if you want

>> and what teams would not want to do

continuous deployment, right? Cuz it it

seems to me continuous delivery you kind

of want to get to because then you just

get more and more feedback, right? But

then it is a kind of a good question

like should it go out immediately? This

is this is the question you know

everyone always sort of asks like it's

almost ready to go out why can't we just

push it to production put as engineers

you want as soon as possible it's ready

right the reality is it doesn't really

suit every every every company right so

um you know it may be the case that you

know some some companies really do still

have you know review boards where you

need to validate is this good to go out

um particularly if you're in an industry

that has a lot of um regulation and and

compliance problems problems, compliance

requirements, and they need to make sure

that when it does go out to production,

it's it's sort of done at the right time

with the right people available, etc.,

etc. It's not necessarily true to say

that everyone should be going to

continuous deployment. Um, because

that's, you know, sometimes just not not

viable for various reasons. But if you

at least got to that point where you're

sort of continually seeing your changes

go through all the testing, uh, you

know, you're promoting it through the

different environments, which is, you

know, you're therefore testing the

process itself. If you can only click

that button to go to production once a

week or whatever, okay, that's fine. You

know, you've done a lot of that hard

work. You've mitigated risk, which is

what a lot of this process is about,

right? Is is fill the pain as soon as

possible. Um, and and derisk anything

that could go wrong right up until that

that last point.

So I know you're deep into CI/CD uh

where continuous integration, continuous

delivery, continuous deployment. You've

been doing this for like what 10 plus

years now, but I was pretty surprised to

see that when I checked Octopus deploy,

it said deployment. It says continuous

deployment, continuous delivery, but it

also says Kubernetes. How has Kubernetes

kind of arrived in the topic of CI/CD

and in general infrastructure? What

happened there?

>> Yeah. Yeah. Kubernetes is is the the

platform of of the moment. If we take a

bit of a step back, uh, Kubernetes came

out of, um, the, you know, Google, I

guess they originally had Borg, you

know, they were using it to to host and

run their infrastructure. They ended up

releasing Kubernetes, um, partly um, I'm

not going to, you know, pretend I can

read their minds and know exactly why,

but, partly as a way of helping to level

the playing field between them and some

of the other cloud vendors.

>> Yeah. So, like before Kubernetes, AWS

was a clear leader. And I I talked with

Kat Co's growth who came to the podcast

uh who's uh who works on the Kubernetes

team. And again, she speculated that by

releasing Kubernetes, it was a lot

easier to to move workloads from between

AWS and Google Cloud. So, it kind of

leveled the playing field and now there

was a reason to That's right.

>> like choosing Google Cloud was no longer

as big of a risk or choosing Azure was

not as big a risk and so on.

>> Yeah, that's right. But it kind of it

made it simple to move between vendors.

And so um as a as a customer of one of

these um platforms, if you wanted to

move to AWS and you're using containers,

no problem. You're just sort of putting

it in a new place. And so Kubernetes

came along at the time when um there was

a bunch of plays in the field for um

container orchestration. Um so you know

you had um you know Nomad even dock

>> from hashpform

a bunch of other options are out there

because

>> at core o Kelsey high tower was just on

the on the podcast they they built fleet

which was another container

orchestration and this was all around

like 20 2012 2013 2014 and then

Kubernetes came out and somehow it

started to win market share.

>> Yeah. Yeah. Yeah, I mean they I think

the the um some of the um mechanics that

it provided um kind of really appealed

to to engineers and and I guess DevOps

teams out there and eventually I think

particularly because it was so easy to

um use crossplatform

and because some of the cloud vendors

then did end up picking it up it kind of

has ended up now being essentially the

winner in this space. I know even back

then even non-container orchestration

tools like um Azure service fabric was

another kind of attempt to to handle the

the fact that you know this world

everyone's building microservices and

they want to host them in a single

platform and how do you orchestrate that

and deal with dependencies etc. uh but

Kubernetes has become the clear winner.

>> And when you say winner, I understand

that for example, when you have a bunch

of backend servers on a service like you

know you have a website, there's a large

back end. Okay, I'll use Kubernetes for

that. But you're talking about you're

talking about infrastructure, right? Or

you're talking about even things like

build servers.

>> That's right. So it's kind of funny, you

know, we talk about Kubernetes being,

you know, cloudnative. This is what this

is the term you always hear. It's cloud

native.

>> Yeah, that's that's what they say,

right?

>> That's what they say. And you know you

look at the vendors that picked it up

it's it's Azure and AWS and kind of the

made it available on their platforms.

The reality is a lot of customers

actually use Kubernetes for running on

premise. Um so you know a

non-insignificant number of our

customers who are doing Kubernetes are

running on potentially their own VMs on

their own server farms or maybe they're

running VMs in AWS or Azure but they're

maintaining Kubernetes itself. Um the

idea being that they have a lot more

control then over exactly what's

running. Uh it's particularly common

you'll find in things like financial

industry and things like that where

again wanting to fully sort of control

the process and and um uh manage the the

whole sort of piece of infrastructure

from end to end is kind of one of their

goals but they want to leverage the

capabilities that Kubernetes provides by

you know allowing the the application

team and the ops teams to just build and

define kind of in that declarative

fashion that Kubernetes provides um

exactly what runs and and how does it

run etc. So they they chose Kubernetes

because this is the best tool they can

manage their on-prem infrastructure and

say like okay I have like these physical

machines and I want this many virtual

machines and I want to run a database on

this many nodes and a internal web

server or like whatever. So it it just

won this area as well.

>> Yeah. Yeah. I mean so there's around the

same time that Kubernetes came out or

actually before that there was a lot of

these other kind of declarative type

tools right. So you have uh you know

terraform which is a really popular one

you can define kind of exactly what

infrastructure you want and and what

you're doing is you're essentially

defining the the desired state and then

the tool kind of applies it and you know

you've got puppet etc. And so Kubernetes

has this similar concept right where you

define what you want your sort of

infrastructure to look like and the

internal Kubernetes controllers and

operators will basically ensure that

whatever you've asked for always

applies. So if you say you want, you

know, three replicas of something, it

will ensure that there's three like

replicas of something. And so if one of

those pods dies, for example, it will

spin another one up. And so it

simplifies this process of being able to

find declarative declaratively kind of

exactly what what you as a as a sort of

application team uh need to run your

your system.

>> It's fascinating because I I always

assume that Kubernetes has won the the

cloud native space and and

Skyperscalers. Can you tell me a bit

more about how it's being used on prem

like some some interesting stories? You

must have seen some because you said

that you're working with companies who

are managing like large on-rem

Kubernetes or like interesting

situations.

>> Yeah, this is one of the nice things

about working at a company like Octopus,

right? We we talk to and deal with so

many different customers and you know,

everyone's doing things a little bit

different or they've got slightly

different needs and requirements. Um,

and you kind of get exposed to a lot of

different uh, you know, problems and

patterns. Um, and it's easy to sometimes

to get lost in, you know, what people

are talking about in conferences and

everyone's saying it's all about cloud

and this is the, you know, best practice

and you should be doing this. And the

reality is, you know, everyone's kind of

got their own little problems and they

just want to solve them the way they

kind of need to solve them. And so some

of our customers, in fact, a lot of our

customers will run Kubernetes kind of,

you know, quote unquote on premise. Um,

so for example, I was actually talking

with one

>> when you say on premise, can you just be

a bit more clear? Is this a data center

where they're like renting and

collocating? Is this actually like I

have my own data center or is this like

I actually have my own machines in my

closet?

>> Yeah. Yes. And yes, I guess. So,

>> what even in a closet? That mean I was

trying to joke there. [laughter]

>> I'm I'm sure there are still uh teams

out there that are running, you know,

the the the core um accounting tools and

etc. you under under Steve's desk. But

even when we talk about, you know, um

small computers, some of our customers

have um Kubernetes clusters basically in

their point of sale systems. So they

have hundreds and hundreds of stores and

they have little um Kubernetes clusters

that essentially run in them and each

one's independent and they you know run

into their own problems with that

because particularly at scale when

you've got you know thousands and

thousands of clusters um and you know

these these um customers are you know

following various GitOps practices etc

where they're pulling the the actual

state from from a git repository so the

git repository itself becomes the

bottleneck or they start getting

throttled um and so they have to sort of

resort to other mechanics to try to um

sort of mitigate and work around that. I

was talking to another one of our

customers um actually just the other day

at at KCOM there who they are deploying

they they've got Kubernetes clusters

running on research vessels and those

research vessels

>> research as in

>> like boats as in

>> ships on the ocean.

>> That's right. Um I'm not going to

pretend to know exactly what they're

doing on those ships. We didn't quite

get into that detail, but they've got

Kubernetes clusters out out in the open

sea, right? Which is apps given

Kubernetes name. The problems they run

into though are a little bit different,

right? So for them, um, you know, those

boats might be out at sea for, I don't

know, weeks, months at a time or

whatever that might be. So when you want

to do a deployment that the ship's not

available. So when that ship comes back

into port, it needs to get the update,

right? So they'd be talking to how you

would how you'd achieve this, right? And

how that process would work. This is

super interesting and I love how you

kind of get a peak into so many

different types of teams through the

fact that you know like you're talking

with them about how they do the

deployments but you're you probably see

some other things that they're doing or

things they're struggling with. What are

some trends you're seeing across the

industry in terms of the this wide range

of companies you work from startups to

like finance companies to like these

research vessels?

>> Yeah, I guess one of the one of the big

trends these days is a lot of focus on

on GitOps. So GitOps is this what is

GitOps?

>> What is GitOps? That's a that's a good

question. Go G. Let's take a step back

for a minute. So you know we mentioned

we talked about Kubernetes earlier. We

talked about the fact that it's kind of

got this internal continuous

reconciliation process where you say to

the cluster uh please spin up you know

five pods and um it takes that desired

state and it ensures it always sort of

is true in the in the world. And so

there was a lot of products around there

that were doing similar thing. You know

Terraform does that for infrastructure

etc. Um and a bunch of people started

wondering why can't we sort of take that

process and pull it back further so that

um not only is Kubernetes just dealing

with desired state but we can pull it

sort of directly out of git um and so

you know I can as as an engineer make

changes to that uh that git definition

that um desired state and I'd have some

process that essentially pushes that to

the cluster and and ensures that it it

remains um in in line with what I'm

asking what I'm expecting and so the

term githops was coined by um by Weave

Works in I think it was 2017 or so and

as a as a general practice it sort of

started picking up steam particularly in

in tandem with Kubernetes because at its

core Kubernetes is is very declarative

right later on um sort of in the early

you know 2020s

um it was kind of formalized a bit more

and there was sort of four key pillars

of of GitOps the first being um uh

essentially declar you want your state

to be declarative so this is the idea

that um you want to define what you want

the state of your infrastructure to look

like. This is to basically make things a

lot I guess simpler to to understand

what the state of the world is going to

be when a deployment takes place. So if

you think about um deployments that are

a bit more imperative that has sort of a

process, the end result is sort of the

result of of multiple steps. Um but when

you're wanting to just update some

infrastructure, that desired state kind

of works really well at um particularly

in in the Kubernetes space. And then in

GitOps the desired state will be just

describing like how many nodes I want or

like how many I don't know replicas do I

want on a database or how many web

servers or like load balancer how to be

connected that kind of stuff.

>> That that's right. Yeah. So it's it's

basically a way of being able to say I

want my infrastructure to have whatever

state it is

um and then the the GitOps um agents the

GitOps um products basically ensure that

remains the case. So they'll keep

applying it to Kubernetes. So you've

kind of got this this um situation where

Kubernetes keeps its internal status in

sync with reality and now you've got

these GitOps tools that take the

declarative configuration in in sync

with what Kubernetes is.

>> So so they will take the whatever I put

in Git and whatever format I use and

they kind of translate it into something

that makes sense for Kubernetes and now

Kubernetes can apply it.

>> Yeah. Yeah. I mean, ideally, you want it

as close as possible to what um sort of

um I guess Kubernetes is expecting cuz

>> allows you.

>> That's right. And so what you're

describing there, I guess, is the

continuous reconciliation. And so this

is the idea that um these these githops

apps will um essentially as we said sort

of take that state and apply it and if

there's any drift from kubernetes side.

So for example someone um you know runs

cube control you know delete pod or or a

delete deployment or whatever the case

might be

>> because your desired state is now stored

in in git in this case that will kind of

self-rep. Um the second uh pillar of

githops is that that desired set you've

sort of defined um should be um stored

somewhere that's uh immutable and

versioned. And so this is the idea that

um once I say that I want to have this

state, I want to have sort of something

I can point to a pointer and that might

be a tag or a commit show or whatever

and I want to basically use that to

define what what that actual state

should be. And I don't want that to be

able to change, right? Because otherwise

that kind of defeats half the point. Um

by having it versioned and immutable, it

also um makes things like auditing a lot

simpler, right? You can see the

transition of that that that desired

state over time. What's interesting

though is a lot of people will point to

that and go yes um version and immutable

I know what that is that's git

>> I was about to say that because git

gives you it definitely gives you

versioning or it gives you commit

history I'm not sure if it gives you

versioning and immutable in the sense

that I mean the past cannot be changed

>> that's that's right

>> or actually can it because you can

rewrite

>> history you're right so you depending on

how you sort of configure your githops

agent um you know you certainly can

rewrite history if If you have it

pointing at a tag, for example, you can

change tags. And so that's why there's,

you know, best practices around that.

Um, I guess kind of, you know, wiggle

the finger a bit if you're using tags to

to manage that sort of state. Uh, but

what's interesting though is really

nothing in the in these pillars. Um, and

very quickly, the third one being uh

pull versus push. And so this is the

idea that your um your GitOps agent will

pull the state from GitHub and put or

git, I should say, and put it into the

cluster. and the fourth being continuous

reconciliation. Uh but nothing in any of

these sort of pillars actually talks

about git. And I think that the naming

of githops is kind of um kind of gets

people to to already have this

expectation that everything has to be in

git.

>> I mean why would you not have that

expectation? That's what I assumed.

>> That's right. I think the the problem

though is um not everything should be in

in Git, right? So you've got this

constant kind of conversation within

that community about you know where do

you put secrets for example? So no one

not g no no no no we know that right is

that do not put it in git

>> and so that's the thing so you know

there's been all these solutions to try

to put it in git so there's things like

sealed secrets where you encrypt it and

put it in git um

>> sounds like a terrible idea

>> but I guess what's really this is

highlighting is um the reality that some

things don't need to be in git right as

long as you can um have this sort of

control over the versioning or

immutability of it um then that's that's

completely fine

>> and then the trend around githops is is

what you're seeing that a lot more infra

teams are moving from okay a few years

ago they might have just like made

definitions for Kubernetes and now

they're moving over to GitOps so saying

okay we'd like to control infra in in in

a tool in a way that's that's described

that's in version control is that the

trend or what is the trend around

githops

>> um I I guess it's more just the trend of

of the growth in general of GitOps in in

um enterprises right so not every uh

company out there is using Kubernetes

today um and as they sort

approached Kubernetes and they're

looking at well how do I how do I um you

know perform the deployments how do I

manage that process gitops becomes the

sort of deacto process and to some

extent it is um giving rise to this idea

of using it to manage um other things

outside of kubernetes and there are a

few examples of uh projects and

experiments that will use things like

terraform and there's a a continuous

reconciliation service that keeps your

your actual state um outside in in sync

at the moment it's really about the

focus is on I guess Kubernetes is the

core core place where it lives. Um and I

guess it's more the growth of Kubernetes

itself means that GitOps is is coming

along for the ride.

>> And you mentioned enterprises which

means like these large companies with

often times thousands of people or in

regulated environments that's what I

think of enterprises. Are you also

seeing smaller teams pick up uh things

like GitOps? Is it like everywhere or is

it more there's certain types of teams

that seem to be just more interested in

that?

>> That's a good question. So um I guess

sometimes what we see is a lot of people

go go to conferences or they read blog

posts and they hear that GitOps is what

you should do. So I guess what I want to

point out here is um GitHubs is

potentially not necessary for all all

locations, all environ all teams, right?

Um there's certainly a bunch of benefits

to it, but the reality is there's some

things you need to do outside of just

GitOps. You might use GitOps principles

in parts of your process, but some of

this absolutism I think that sometimes

exists uh may not be necessary. So

there's often a bunch of other processes

you do around your actual sort of um you

know, quote unquote deployment. So

things like maybe you run smoke tests or

maybe you want to send a notification

when it's complete or maybe um you want

to do a a database update or something

like that. These kind of steps don't

really lend themselves very well to kind

of this declarative everything is is in

git kind of process, right? And so

that's why you get things like um Argo

workflows and and and roll outs and

things come out to try to kind of get

opsify this this process. And that that

works for for some people. But the

reality I guess is that um um I think

some people get really hung up on this

idea that that everything is git so

therefore they they found the tool and

you know and so therefore everything is

a nail.

>> Yeah. I think that's

>> and this is the thing like talking with

with customers when we go through this

process of you know you can use GitOps

in Octopus um and you know we've got a

bunch of support for various um

mechanics that integrate well with

Kubernetes and Argo but there's a bunch

of other sort of operations you do

around that process that that doesn't

need and when you talk to them about it

you know they realize that what they're

trying to do is ultimately just ship

software so again that difference

between what you um hear when you talk

at conferences and things where you know

everything is everything is git and

everything must be you know in in this

particular format or whatever the case

might be. The reality is for most

customers they're just trying to ship

software right and they don't care what

name you give it. If it's GitOps and it

works end to end and solves everything

good. If they want to use GitOps as part

of the process but then have other

mechanics that are more sort of

imperative um then then good. It's just

sort of the reality of, you know,

there's there's tens and tens of

thousands of companies out there in the

world that are doing software delivery.

Um, and not all of them that are at

conferences and not all them are at the

the forefront. I guess

>> as Rob says, most teams don't care

whether you call it GitOps or anything

else. They just want to ship software

and know that it works. Our presenting

sponsor, Anticys, does exactly that. Is

let you ship knowing that it works.

Antithesis goes beyond code review. It

runs your whole system inside a hostile

simulation. By doing so, it finds every

bug before your users do. And because

the simulation is fully deterministic,

antithesis doesn't only find bugs, it

gives you a perfect reproduction of

every issue. I know this sounds like

science fiction, but it's actually

hardcore engineering under the hood.

Jane Streetfly.io and the CD community

ship agent written code with full

confidence because they know it's been

verified by antithesis. To see more case

studies and details, head to

antithesis.com/pragmatic.

That's antithesis.com/pragmatic.

I also want to mention our season

sponsor, Turbopuffer. Turbopuffer is

exactly the thing that just works. A

vector and full text search engine built

on object storage. Fast, cheap, and

extremely scalable. No exotic

architecture required. Here's something

I find interesting. The teams building

the smartest AI products out there,

cursor, notion, cognition, and tropic.

They all run on turbo buffer. But why?

Let's think about it. and LM without

context, it can feel pretty dumb. I can

still remember shortly after CantBT

launched early 2023, how it felt both

incredibly smart but also frustratingly

stupid. If you asked it a question that

was outside his training data, it just

made things up. Fast forward to today

and the models and their tools for

retrieving context are much better, but

hallucinations still happen frequently.

Here's a typical way to integrate

Turbuffer with an LM. Plug it behind

your search tools, get fast responses

and relevant context back. The neat

thing is how it gives you the perfect

blend of lowcost storage, fast

retrieval, and a bunch of different

search tools like vector, full text, and

filtering. You can afford to index

billions of documents, and then you can

query it with different tools to get the

most relevant handful of documents in

milliseconds. This is how your LM feels

smart. It gets the right context really

fast without blowing through a bunch of

tokens. Of course, you can use

Turbopuffer to search anything, not just

code. If you're building AI products,

check out Turbopuffer at

turbopuffer.com/pragmatic.

With this, let's get back to Rob and

talk about progressive delivery.

>> Yeah. Another trend that we talked about

just before is the rise of platform

teams. Can you talk about what you're

seeing?

>> So, platform teams are kind of I I guess

in the past several years, they've

become this sort of new standard

organizational structure to help teams

manage their I guess deployment

workflows, the a bunch of the

infrastructure around it. And it's kind

of come out of this evolution of of de

DevOps, right? So, you know, we

mentioned before in the old days, you'd

write a bit of code um and you'd throw

it over the wall to the ops team. So, it

was dev teams and ops teams and you

>> this was like in the 2010s, 2000s

>> back in the back in the long long ago.

Um and then DevOps became, you know, the

the practice that everyone sort of

realized that actually we want to have

the the engineering teams be involved in

and have ownership of part of that that

operational um process. idea being, you

know, you get faster feedback loops. You

are able to kind of if you feel the

pain, you you sort of fix, you know,

it's that saying, you know, you fix it,

you ship it. Um, you know, we've all

kind of heard that. And so, um, a lot of

teams, you know, took that to heart.

That's that's good, great, uh, good

practice. But as things start to scale

up, what you'd find is that there would

end up being like a DevOps team again,

and sometimes sometimes they're separate

to another ops team. And so there'd be

this separation of of um development and

DevOps. And it kind of goes against some

of the principles of what DevOps was,

you know, trying to destroy. But not

only that, these teams then um end up um

having lots of different ways of doing

their um deployment. So you know, you've

got every single, you know, a whole

bunch of application teams and they've

all got slightly different requirements

and they're all building it from

scratch. And so you'd end up with these

these teams either whether you had the

DevOps teams or it was still within the

application teams where um there was

just this this um large number of

different ways of doing things, right?

And that becomes difficult at scale. So

um you know you can't really move

between teams

>> and and by scale you mean typically when

there's a lot of teams, right? That's

the easiest.

>> Yeah, that's right. If you got if you

got lots of lots of teams and each one

is kind of owning that process end to

end, you know, you sort of get this

bifocation of processes and not only

that the application teams themselves

start kind of getting this this um

context overload, right? They now need

to think about what's be best practices

of of the different cloud um tools.

>> Yeah. And there's of of devs rarely want

to configure the deployment scripts and

test them and testing is hard. It's you

can't it's not really unit testable. So

it's it's now a different job. I

remember when I was on earlier teams

where you know like typically on a

mobile team like you have a you have a

mobile team of five people and one of

them one of us had to kind of specialize

in Jenkins configurations because

Jenkins is often times the or used to be

the mobile CI/CD and it's kind of like

half a person dedicated to that and it

was more like draw you like we had to

draw a stick on who's going to do it

because we want to build

>> you want to write code right you just

want to focus on writing code and so if

you're spending a bunch of your time

sort of managing infrastructure and

pipelines and things um you know that

that's no fun for anyone

Um and so platform teams have come about

as a new way of solving that problem

where it's different to kind of you know

this idea of a devops team or ops team

that kind of own the whole process. they

they more sort of define best practices

and they provide a um ideally a

self-service mechanism where application

teams can um essentially use um often

what it's called as an IDP an internal

development portal and they'll be able

to essentially self-service and you know

maybe they want to spin up a new project

and they're able to use a template that

the platform team have generated and so

the platform team are able to sort of

create these standards throughout the

throughout the company and they can be

responsible for sort of I guess the

definitions of those pro processes and

the best practices and how to achieve

that. Um, but the ownership of the the

actual running operational sort of

element is still within the teams,

right? So, they still get those benefits

of of of, you know, DevOps, being close

to the close to the real code and and

feeling the pain if there's a problem

and etc., etc., etc., but they don't

need to spend all that time becoming

experts in um, you know, all the

different ways that you can deploy the

software they've got. And so this has

become um really common now where

particularly as you sort of get to a

larger size platform teams are a great

way of of solving that problem. Now um

that's not to say that every company

everywhere should have a platform team.

You know if you're a smaller company

sometimes it's you you've just got the

apps team and and they sort of are doing

you know quote unquote DevOps. Um but

this is certainly something that as as

you sort of start seeing larger

organizations with multiple teams or

multiple projects, these platform teams

are a way of basically bringing some

some sanity and control and focus I

guess to to the whole space.

>> One trend across the industry of course

is AI. Everyone's it's hard to see any

teams where devs are not using AI agents

specifically to code. you know product

managers will will be using these things

and of course we have a lot more code

produced as a result when it comes to

CI/CD systems what are you seeing

changing there because of AI

>> this is the this is the uh the elephant

in the room right the how is AI

affecting sort of this the reality is I

think to be honest it's it's still very

early I think what will happen is the

impacts of CI/CD

um are really tightly coupled to how

development teams end up using AI. So

there's going to be some sort of like um

I guess a lagging process there. But

we're finding a lot of um people a lot

of teams are starting to use AI in their

development process. And so we're

starting this process of going out and

looking and talking to customers and and

learning what's the way that they're

handling AI in their um in their in

their teams and their application teams

and then how we can best leverage sort

of the CI side to um to support that.

but then um in addition to that use AI

within the the pipeline itself get in

the right place. So one of the things

we've been um I think pretty pretty um

keen on at Octopus is this idea that um

you know at CubeCon we were probably one

of the few companies that that didn't

have you know AI plastered all over it.

Like we we tried to be very you know

that's that's what gets the sales right.

>> Yeah. That's you stand out now these

days.

>> That's right. By not having AI. Um, I

mean we we've got AI in Octopus, but

what we've been trying to do is think

about, well, how do we actually use it

in a way that's actually useful for for

for our customers, right, for for

engineers, etc. Um, and so we've been

slowly adding capabilities within

Octopus to provide um, you know, AI

support, whether it's a MCP server, uh,

whether it's a a recovery agent that can

review logs and tasks and all that sort

of thing. But that's within the product

itself. Some of the bigger changes will

depend on like I said how how actual

application teams um use use AI. What I

think you know we're talking about we'll

find is there's going to be a lot more

velocity. I think that's one of the big

big changes right is there's just going

to be a lot more code coming through. I

think one of the questions is okay what

does that mean for your pipeline? Um one

of the things you often talk about when

you know human there's a human element

to the pipeline is speeding up the cycle

to get that feedback quicker. You know,

if you got engineers sitting there

waiting for their code to run tests,

they can get back to it and fix it, the

the shorter and shorter you can make

that feedback loop, the the better it

becomes because they don't need a

context switch, etc. I think in a world

where the majority of your code is being

developed by AI, that becomes perhaps

less important. you know, if you can um

kick out your your build and test

process um and it takes 30 minutes

versus 20 minutes, does it really matter

if the engineers are already long gone,

moved on to the next problem and the the

actual AI agent themselves itself can

kind of babysit the process and review

the problem that came up and issue a new

fix. I guess there'll be a deemphasis, I

think, on some of the speed of the

pipeline itself and more on increasing

sort of um or decreasing risk, right?

the risk that comes from having AI items

generate code. And so exactly what that

process looks like, I guess, remains to

be seen. I think what we'll see a lot

more use of is things like progressive

delivery. And I think particularly

feature toggles um are going to be a

really common tool in in the tool belt

of application teams. Partly because it

allows you to ship that code as as fast

as you can or as fast as you want, but

manage the roll out of the actual

feature set or changes sort of

independent of the deployment. sort of

decouples your deployment from from your

release. And so in a world where you

know we've got a lot more AI agents

generating code and being involved in

perhaps part of the build process, those

agents themselves being able to use

toggles to react to it quickly, I think

then become a lot more important um than

perhaps what we see today.

>> Can we talk about progressive delivery?

what it is and what are the most common

ways to you know like to to derisk

getting your code or your software out

there.

>> So progressive delivery is the next

evolution beyond continuous delivery. Um

so you know with continuous delivery

it's this idea that you know I've made a

change to the system and I want to ship

it to um dev or stage or typically you

know if it gets to production sort of in

one hit right with progressive delivery

um you're what you're trying to do is

basically release those changes in a

little bit more of a controlled way

typically through things like a canary

deployment. So this is where you might

deploy just some subset of of your

instances that are out there.

>> So what is a canary? What is a canary?

Um Canary deployment is this is New

Zealand basically. New Zealand's our

canary. So this is as we said before

this idea where um you select some

subset of your your customer base or or

whatever that might be and you would

typically route traffic to a new

instance. So you'd ship you know you've

got version one running and you want to

release version two. You essentially

ship version two side by side and you

might use, you know, most common one

would be some sort of network um traffic

manager to route some percentage of your

traffic to to that new instance and you

gradually roll that up. Typically, you

know, as you do sort of do this process

properly, you you should have a fairly

mature um observability um mechanisms in

place to see that, you know, you can

roll up or roll down.

>> And I guess this whole thing comes from

a canary in a coal mine, right?

>> That's right. Yeah. Yeah. So the idea

being that um you know in the old days

when you'd be in a coal mine digging

away and it would release um you know

all sorts of toxic fumes and things like

that canaries were um a lot more

sensitive to it. So they have a little

canary in a pa cage um and if that

canary sort of died I guess got knocked

down.

>> I I think the canaries as I understand

they were like chirping

>> and then

when it stopped chirping well it also

died.

>> Oh okay.

All right. Same same ending. Same

ending, but just, you know, a nicer a

nicer way to go out.

>> They need to get out.

>> Yeah. So, it's this idea that you get

that advanced warning, I guess, that you

know, rather than you getting knocked

out by the the toxic gases, etc. Um, you

know, you can get out of there sooner.

So, it's that same principle, I guess,

brought to the software. Um, there's

various other mechanisms like blue green

deployments. So you've got your first

version there still receiving traffic

and your second version is up and

running and you can now do some tests

against it, validate it. Maybe you've

got sort of the the the um you know the

IP details to access it directly. You

can basically validate that it's

working. Um sometimes there may be a way

of avoiding cold starts and things

because that process may need to you

know initialize a bunch of stuff but

then when you've sort of done that

validation and you're ready you can

essentially swap swap traffic around. So

all the all the new traffic goes the

other. In some ways, it's like doing a

canary but straight to 100% but you're

doing a bunch of validation um sort of

on on the side before it actually

reaches customers. In my in my view,

probably the more um useful uh

progressive delivery strategy is is

feature toggles. So, this is the idea

that you you've got some sort of

>> feature flags as well.

>> Feature flags toggles the same thing.

>> Yeah. Often used interchangeably. Um so,

this is the idea that you've got, you

know, some sort of uh variable in your

system. um and it's linked to typically

some sort of external service and

through the state of that particular

variable being sort of true or false on

or off um you can essentially have

different code paths essentially take

effect and there's a bunch of benefits

that feature toggles have over say

canary um releases particularly for

application um delivery where you know

the the your unit of change with a

feature toggle is is very granular. it

can be, you know, single lines of code

and so everything else remains the same

and all you're doing is is tweaking that

single line of code with a canary or any

sort of versioned sort of delivery

deployment mechanism. Um, your unit of

change is the entire app. So if you've

had, you know, 20 commits since uh the

last sort of release went out, then

you're essentially testing all 20 things

in that one hit. Your ability to sort of

target the actual customers, it's a lot

more precise when you're using feature

toggles. So you can you know use all

sorts of complex rules um and say that I

don't know everyone from Germany who has

this particular product in their basket

um has this kind of experience and

that's you know really hard to do via

network traffic rules right the other is

your ability then to to actually roll

back um so to roll back from a canary um

hopefully you're still in the process

where you're sort of going through that

canary you know process and you can roll

it back um that could take you know

minutes Maybe you have to redeploy the

whole old old version. That could be

minutes or or more. With a feature

toggle, you know, you can do that in

seconds. That's pressing a button and it

happens immediately. Not only that, but

you've kind of you've got more control,

I guess, on when you sort of do that.

So, with the with a deployment that

you're doing um via via a standard

versioned release, you're sort of tied

to when that deployment takes place

because when it takes place, that's when

essentially your new feature is

available. And as an application team,

that means you need to know about

exactly when it's taking place and make

sure you're watching the logs at that

point. And maybe you and 10 other teams

who are shipping things at the same time

um are all doing the same thing. Whereas

with feature flags, your control, you

basically got control over when that

takes place. So you might ship the

actual, you know, the assemblies and and

that sort of thing on the Monday, but

you release your feature on on Tuesday

when you come in and you you can you've

got the logs ready and you you've kind

of reviewed what the next steps are. So

this really makes things a lot easier to

decouple releasing a feature from from

deploying software. You know version

deployments uh through Canary etc.

They're really useful uh particularly if

you're doing like infrastructure type

changes where there is no kind of

application toggle that's that's

relevant there but you want to violate

some changes to your infrastructure or

your your process or potentially you

know things like um things that will

involve um schema changes and schema

changes are the big

>> list schema change that's this is the

big problem in any like to be fair in

any um progressive delivery um and this

is why you know the question always is

are you ready for progressive delivery?

Um to do schema changes. I guess this is

the point that um application teams kind

of need to be really mature and I don't

mean mature in terms of you know not

telling silly jokes but mature in terms

of understand all the problems are in

place with this and know how to release

these sort of changes in a gradual

controlled fashion and do it over

multiple stages that you know ironically

is actually quite hard for us um at

Oculus because our software is both SAS

hosted so we have a a SAS offering that

customers can use and we have an

on-remise vision and there's kind of

because we have both both sides um we

kind of have the best and worst of both

worlds you know in the cloud system if

you've got um a SAS product you have

complete control over what versions go

where so if you want to do an expand and

contract you can stage the whole process

you know that it's all been updated

before you kind of move to the next

stage on the other hand for a

self-hosted um application where they go

in and they install it on their own

infrastructure somewhere you don't know

what version they're running and what

they're coming from so they might

upgrade from version one straight to

version six. Um, and so you're not

really forcing them to go through that

expand and contract phase. On the other

hand, um, you know, they've got a lot

more control over when they upgrade. And

so you can kind of be a little bit more

deliberate about, you know, making sure

that they do backups before they change

and and, you know, maybe the down maybe

they can manage that migration and

accept a little bit more um downtime

during migrations and updates and things

like that than would actually be, you

know, acceptable in in a SAS product. So

one thing about you know we talked about

progressive delivery and you're kind of

doing this to avoid surprises you know

if if a regression goes out a new bug or

something doesn't work you kind of want

to c catch it early hopefully only a few

customers have experienced it or even if

it's not 100% you kind of and and and

you have a way to go back uh to all you

do is you you if it's a feature flag you

you hide it if it's if it's a canary

deployment you go back to the other one

>> but There's also this thing where like

when things do go wrong at some point

you want to do a roll back. Can we talk

about

how have you seen roll backs done well

and what does it take to actually have a

real roll back strategy? Bunch of people

talk about CI/CD. Some people talk about

feature flags. I don't hear too much

chatter about roll backs.

>> Yeah, roll backs. This is always a spicy

one. We get a lot of customers who say,

"Why don't you have a roll back button?

I want to roll things back. Why can't we

roll things back?" as as as in the

deployment software like Octopus or

anything else they like okay if it can

deploy I want to like do checkoint and

and like just just do a roll back

>> that's right how hard could it be just

do what you

>> how could it be tell me

>> how could it be well this is the the

problem right so um in a completely

stateless system that's you know pretty

straightforward if you've got a

completely stateless system and you know

this is something that githops is really

good at where you'll have that

definition sort somewhere in your repo

if it's completely state stateless you

can do a um a git revert and push it and

it'll go back.

>> The reality is for most systems out

there, you've probably got some state

>> state being

>> state being databases. It could be um

you know any sort of any sort of

information that you can't necessarily

just kind of undo I guess because if you

roll it back and now you've got your

code talking with the schema of the

database that's not in sync you can uh

provide schema uh if you've got a schema

migration let's say in a normal

deployment you can provide alongside

that a secondary sort of anti-migration

that kind of undoes the change but again

that's not always possible. you need to

do is what are you going to do with that

data set?

>> We we've gotten pretty far in basically

trying to um advise customers that you

never you want to avoid ever talking

about roll back. It's always roll

forward.

>> So if there's a bug,

>> okay,

>> roll forward. Get a change. Yeah. Get

get your change in um as soon as

possible. This is where fast feedback

loops are important, right? You know,

this is what the hot fix processes are

for, right? So we all know that in a

standard process you want to go

devstaging prod and maybe it's maybe

you've got you know um approval

processes and slows down etc. But if

you've got a sign significant bug that

you need to kind of quote unquote roll

back sometimes the the safest thing to

do is actually make a hot fix to that

that version and and push it out sort of

as quick as possible and your bottleneck

might be the the build pipeline or

whatever but depending on sort of your

appetite for risk there you can resolve

that sort of a lot quicker. Now

obviously if you if the failure itself

is just from some um mechanism in the

deployment process itself or somewhere

further down that chain then your your

time to recover is going to be a lot

quicker. But it's this idea that you

know if I've got a failure in version um

version two my roll back isn't to go to

version one. It's to go to version three

and make sure I've got that fix in in

version three. It's the sort of thing

that um you know when we we talk to

customers and some of them go yeah we

roll back you know we roll back all the

time if there's a problem and then when

you ask them what do you do if you've

got a schema change they kind of stop

and and realize that they've never it's

just sheer luck that they've never sort

of run into that right

>> is it fair to say that you want to roll

forward if it involves business logic or

something that is not stateless because

if if it is stateless or if it's

application logic you know you have a

code that says if this else then and you

realize there's a bug where you can just

revert it as long as it doesn't, you

know, touch the the schema or or the the

data.

>> Yeah. I mean, in an ideal world, you're

reverting is through a feature flag,

right, that you click and you're

essentially reverting by changing the

code path. And this is why I always say

um feature flags. So, kind of a nice a

nice tool to use for um doing this

progressive delivery because, you know,

it's just as easy just as easy it is to

roll out that feature. You can typically

roll it back. Now, you're still going to

have some of those problems with schema

issues, etc. If you know if you're

making a change and you've got parts of

your code path that expect one and not

the other, you're going to need to

account for that.

>> But you can even account for that inside

the feature flag.

>> That's right. Yeah. So that that's the

way you sort of ideally sort of manage

that so that within regards to which

path you go down the feature flag, it's

kind of self-consistent with whatever

version of the actual sort of database

schema that's out there.

>> So I guess the more feature flags you

use, the fewer surprises you might have,

but it's a bit of extra work both to

build and also to remove.

>> Yeah.

you you get stuck with still feature

flags all across your codebase once you

start to use a lot. I saw this at Uber.

>> Yes. Yes. 100 times. Yes. So when you've

adding a feature toggle to your um app

itself. Um so we at Ocus we obviously

use feature toggles in our code quite a

lot and we use open feature as like the

um the the framework the SDK to interact

with it. But we essentially have built a

wrapper around it where um the the

toggle itself within the code is um sort

of we provide some details about which

team owns it um and that team sets an

expiry on it. Now the expiry itself when

that time passes nothing bad will happen

but through parts of the CI process if

that time has passed we can send a

notification to that team and say hey it

looks like this toggle is no longer

used. So the specific mechanics don't

matter as much, but it's more a matter

of making sure that, you know, if you're

adding feature toggles, it's really easy

to forget about it because you start

rolling it out and you kind of forget

about it. And you know, you want to keep

it in there just in case for a while in

case you need to roll it back. And

having the ability to understand how

long a toggle has been there, um, is a

is kind of a key part of helping to

maintain that that hygiene. Now the

reality is even at Octopus we've got a

bunch in I know I've got a bunch in

there that um I'm sure if I was to log

in I'd probably get a bunch of

notifications to remove you know when we

use that gardening metaphor in code

right this is this is one of those sort

of operations this is weeding right you

need to just kind of keep on top of it

there are some mechanisms around even in

in lie of the AI side which will um you

know ideally if you're using feature

toggles you you probably got a bunch of

um observability and metrics and logging

around it and there are some system some

tools out there that will allow you to

keep track of when the last time a

toggle was kind of evaluated. Um, and

that kind of gives you that that signal.

Um, similarly, you know, you might

remove it from the code because

typically when you want to remove a

feature toggle, you want to remove from

the code first before you touch your

actual sort of toggle system. And so

having a mechanism so that once you

remove for it from the code, um, you

know, it might take two weeks before it

makes all the way out into production.

So you don't want to delete it before

then, by that time you've kind of

forgotten about the fact you removed it.

Oh yeah. Um and so having mechanisms

that will keep track of that change I

guess going through the system um and

when it reaches the environment where um

you know production where it's actually

being used can kind of show okay that

code's gone out that's you know remove

the toggle it's it's fine and safe to

actually remove the configuration

because you've got that feature toggle

information in two places right you've

got it in the code and you've got it in

your your your your platform

>> can we talk about how development

environments evolve we talked about

CI/CD but I'm interested more in you

know you you go from like you have one

environment later you might have staging

or something and what evolution have you

seen across the all the teams that you

work with all these hundreds or

thousands of teams

>> yeah I'm not sure if there is one

particular pattern there I mean I think

you know most most common is you know

dev test prod um

>> so these three different environments

>> yeah and I mean even that I think is

probably a a a gross simplification of

all the kind of

>> and dev meaning my local machine.

>> Dev in the case of CD is often like the

the first point of integration. So it's

kind of um test often customers will

keep test kind of reasonably in sync

with let's say production or some sort

of sanitized data source. So that way

that whether it's the QA testers or the

um product team or whatever can go and

review the code. dev is almost like the

first po point of of of integration that

is it actually is the deployment process

just at its core actually working or is

anything fundamentally broken at all. I

think more and more now we're finding

that dev is less useful um in that

respect and what we're seeing is more

the um growth of things like ephemeral

environments and so this is the idea

that you know I as an engineer I'm

running some sort of feature on a

feature branch

um and I want to kind of evaluate that

it's actually doing what it what we're

expecting it to do but not only that I

need I want the rest of my team to be

able to see it working and um you know

if I've got it running on my machine

it's not exactly easy to sort of um yeah

give other people access I guess and

then I want to move I may want to you

know completely context change move on

to something something completely

different. So ephemeral ephemeral

environments is this idea that from my

my branch premerge I want to spin up a

whole environment um essentially from

scratch ideally with with whatever

dependencies are required to sort of run

this particular component that I've been

building. Um, and then I want to

basically deploy my um, app into that as

if it was a normal full-fledged

environment. Um, as once that's

available, I want to sort of have access

to, you know, if it's a web app, maybe

it gives me the URL and I can poke

around it and hand it around and and

other people can kind of evaluate. And

then the moment I kind of merge that PR,

tear it down again. You know, it's quite

common to have multiple test

environments because, you know, I've got

a lot of stuff going through my pipeline

and I've got three testers. So, let's

have three environments. Uh so they can

all sort of have one at once or often

you'll see a single test environment and

a bunch of tests and they all kind of

need to to um collaborate to see who's

got access to the system at the moment

etc etc. Whereas with ephemeral

ephemeral environments it doesn't roll

off the tongue. With ephemeral

environments you can um essentially have

a a full-fledged deployment per per

feature. And so again, that's about

speeding up that that feedback process,

right? Again, all these processes are

all about speeding up that feedback

process to get the the the catch those

failures or issues or or bugs or

whatever. So sooner. There was a time a

few years ago where cloud development

environments were really talked about a

lot which was the idea is as a developer

you have an environment spin up in the

cloud you're let's say your your visual

studio code connects to it or or maybe

you just log in online and it spins up

all the dependencies often times done

with containers which reminds me of this

as well and there's also like like

preview environments but somehow it

feels that both that discussion and this

one kind of died down maybe it's AI

maybe it's something else But I mean the

technology is there, right? We have

containers. It's you can you can package

things together. It's it's I'm sure it

depends, but it's all doable.

>> Yeah, it does get tricky. It's this is

again one of those sort of things that's

really easy to talk about. Um for simple

cases, it it can get tricky when you

know what if I've got more than just a

single app in my kind of quote unquote

environment and how do I make sure it's

got all the data I need to validate. So

it can get tricky. So

>> or if you have a bunch of services that

have state.

>> That's right. Exactly. So um there are

sort of complications that it does bring

but the I guess the the benefits that

you get as a as an application team um

particularly you know application team

where you've still got engineers writing

code um is is sort of speeding up that

that feedback process I guess. Well, now

with with AI agents everywhere, that's

even better cuz uh in a sense that if

one of the best ways to validate, you

know, we have code reviews and AI agent

generates and you look at the code, but

isn't it not better to just confirm that

this thing works, especially when it has

a UI?

>> That's right. I think even in that world

where you've got AI AI agents kind of

building the code and validating the

code any sort of uh scenario where you

want that AI agent to kind of validate

what it's done um you're essentially

talking about ephemeral environments

even if it's not exposed to people

because it's doing its own testing and

poking around um in whatever shape or

form um that it's doing that still is I

guess one of these kind of environments

right it it's ephemeral it spins up

you've got some sort of provisioning

process um and then ideally once the

job's done you just kind of tear it

down.

>> I'm interested in learning more about

the reality of operating a large

infrastructure platform and you know one

big one you're working on is actually

Octopus deploys SAS offering. How does

that look like and what are the

challenges of of you know like running

something where you're running all of

these deploy processes all all these CD

you probably have a bunch of different

things. What is it like?

So um at the m at the moment I'm not on

the team that sort of builds that but I

can give some of the the context I guess

from history and kind of um context

there. Originally when we first sort of

decided to sort of provide a Octopus SAS

offering um I don't know I think it was

2020 or something like that. It was all

VMs. So every customer would basically

get a VM spun up and we would virtual

machine virtual machine. Yep. and the um

Octopus um you know self-installed app

would basically get installed onto that

VM and they'd get a whole VM for running

workloads on etc. Um and that was very

much not cost effective. It was costing

us something like a hundred bucks per

customer per month and they were paying

I don't know $20 a month or whatever it

was. But this whole process was more an

experiment to see was there a demand.

Um, and to his credit, Paul was happy to

sort of hand, you know, pass out the

credit card to kind of go through this

process to see that is this actually the

direction we we want to go. Is there a

is is this something that going to turn

into a viable sort of direction for the

company? Because it's a big step, right?

Going from building software that you

can kind of hand out and people can

download and manage themselves to

>> it was like pretty much self-hosted or

like run on your own infrastructure.

>> Exactly. Yeah, that's right. And so the

the demand was there. So um not long

after that sort of first experiment we

basically started from scratch again and

I worked with a couple of the other

engineers back then to start building it

on um Kubernetes um and so octopus

itself in in that space we have what we

call kind of a reef. So what you'll find

is everything in octopus we've always

got of octopus or nautical kind of names

around it. So a reef is basically a way

of it's this cell-based architecture

where contains all the resources that

are needed for that particular

customer's instance. Well, some of it's

shared, but it's kind of broken down

into individual cells. And so a reef

will contain, you know, the cluster, an

Azure database, etc. Um, and each

customer instance is running now in a in

a pod in that um cluster. And so as part

of that project that was when um I think

I was working on converting it so it

could run on Linux and inside containers

and someone else was building the

dynamic worker infrastructure. So um

there were a couple of us that kind of

just um got in and yeah really just got

it up and running so that way we could

kind of start moving forward and and I

guess stop stop losing money. Fast

forward to today now there's an entire

team that's kind of backs that and we've

got you know several thousand customers

on it and we run you know many many

thousands of deployments um every every

every month and so now what we're trying

to do is there's a project at the moment

to basically make the um Octopus

deployment process itself more resilient

so what that means is at the moment when

a deployment kicks off a bunch of the

the the process so as a it's kind of a

um imperative set of steps a bunch of

that is stored in memory which means

that whenever we want to do an upgrade

um we need to essentially um stop

running tasks for some period of time so

we can kill their instance and spit

another one back up. Um Octopus itself

at the moment um doesn't um sort of have

zero downtime between upgrades. So

there's a bit of downtime between that.

We kind of want to reduce that and get

that as close to as close to zero as

possible with the realization that you

know going from downtime of 5 minutes to

1 that's that's just work right that's

you know you can move things around you

can maybe change the architecture going

from 10 seconds to zero is is a much

bigger shift um I'm not sure if if and

when we'll get there but um yeah there's

definitely this this big effort at the

moment to make the whole process a lot

more resilient to basically improve and

reduce the amount of downtime that takes

place so

can um kind of perform upgrades quicker

etc.

>> One interesting thing you do is you have

a SAS but you also have an on-prem

offering. What are interesting

engineering challenges that come from

that? A lot of companies have decided to

just like honestly just move to move to

SAS because now they control everything

centrally. I think Jer did this uh or

may maybe they're they're doing it which

is a well-known one but clearly it's

just a lot more work and a lot more

headache to have both.

>> Yeah. Yeah. And we touched on one of the

big problems here a little earlier is

that when we want to push out any

updates, you know, to cloud because we

control the whole process, we can push

it out. And so we have a sort of a

gradual roll out process there. Um

because each customer is on their own

instance, we can sort of deploy each one