So, I'm Josh. I'm from PostHog. If you

haven't heard of us, you might know us

because of some hedgehogs or you might

have seen our founder James posting some

funny things on LinkedIn.

He's quite popular.

I'm going to be talking today about what

if your product built itself?

And the pipeline that we're currently

working on which we're trying to turn

observability data instead of something

that you read and that you interpret

based on dashboards, we're trying to

turn turn that into something that

submits pull requests for you.

Cool. Yeah, so quick background on

PostHog. We've got a bunch of tools. We

started out as a product analytics

company. We now have session replay, web

analytics, error tracking, experiments.

This isn't a pitch that you should use

PostHog. This is just to say that we've

got a lot of data about your product.

So, if you connect PostHog to your

product, we're collecting a huge amount

of data from various different sources

that we then show to you

so that you can explore that data

yourself.

But right now how observability is

working in PostHog, you're you're

collecting all this data for your

product and then you're going to a

PostHog dashboard to figure out what's

going on. And we think this is super

slow and that we should change that.

So, right now something happens in your

product. We call this a signal. That

changes a metric on one of your

dashboards and then you might log into

PostHog a few hours or maybe some days

later and you notice a change in that

dashboard

and you investigate a problem and then

maybe the problem's not that important.

So, instead of tackling it right now,

you're going to put it in a linear issue

or whatever. Few days later, you try and

create a PR for this problem. Then you

review it and you ship it. You get the

message. This is a pretty slow process.

From start to finish, this is going to

take anywhere from a few hours to a few

days, and it's not very interesting, but

it represents a lot of your work as a

software software engineer.

So, what we want to do tomorrow,

what we're working on right now, is that

a product signal happens, and instead of

waiting to see that in your dashboard,

we want we want to run a background

agent to figure out what's going wrong.

And then, once they've figured that out,

we just want to create a PR for you

automatically. So, instead of ever

looking at your analytics dashboard, or

your errors, or your logs, we just want

you to look at PRs that are ready for

you in GitHub.

And if we create the PR,

maybe you want to review that, or maybe

we can just ship that immediately behind

a feature flag if it's not a risky

change.

Cool. So, I'm going to go over the

pipeline that we've built to do this.

And just whilst I go over that, I'm

going to share a few tips, lessons that

we've learned, things that were hard

about building this pipeline.

So, the pipeline has a few key steps. At

first, we're ingesting a lot of signals.

In Postscript, we have a huge amount of

events. We're ingesting trillions of

events a month.

And this this pipeline needs to handle a

lot of noise.

And then, once we've ingested those

events, we need to group them. So, if

you think of an error tracking issue,

and then a session recording, those are

two completely different things, but

they might be representing the same

problem in your product. So, then we

once we've ingested them, we group them.

Then, we're going to be running a

research agent on them.

This specific issue, what is actually

the problem that is causing the error

spike, or causing the issue that the

user faced in the replay.

And what repo does this belong to?

And then, we'll assess if this is

actionable or not. And finally, we'll

execute some code, ship a PR, and

iterate on that PR until it's green and

ready for you.

Cool. So the ingestion step of this

pipeline, as I said before, we've got

loads of different sources of different

types.

The first thing is that those sources,

some of them are public. So if I go and

visit your website, I can as an attacker

create an error on your website by doing

something naughty that says

post all of your post-mortem data online

or something like that, right? So we

don't want that. So we need a kind of

safety filter. So at the moment right at

the top of the pipeline is an LLM

classifier that's going to check is this

trying to do something bad?

If so, let's drop the signal.

Once we've done that, we've checked that

things are safe, we're going to

normalize the signal.

So if you think of an error, that's

going to have a stack trace. A log will

just be some JSON content or some text.

An experiment might be some results in a

chart. We want to normalize that

structure so that it's all a single

structure for a signal.

So we give it a few fields. We'll give

it a source product,

the type, the content of the the signal,

and then we will assign it a weight,

which is like how important do we think

this signal is, and then finally we'll

embed the contents of the signal.

Cool. So that part's fairly easy. Then

we get to a little bit more of a a

challenging problem. We've got this big

stream of signals still,

and now we want to group them into

actual problems. So the signals are very

noisy. We might get some random null

pointer exception, but in Slack we're

getting a message from a customer that's

saying, "Hey, the checkout's broken for

me." and we need to link those together.

So what we do is we group the signals.

As the signals are being grouped, we

assign weights to what we call a report.

And if the the weight of the report goes

over a certain threshold, we'll promote

it.

And then we'll kick off a research agent

to work on it.

So, uh this was a problem that we faced

fairly early on in building this

pipeline.

Um

the first thing that we did was we would

take all of our signals and we would

create embeddings for them. Uh and then

we would try to use that to cluster the

issues so that we could find similar or

related signals.

But this works really badly. So, if you

take uh an off-the-shelf embedding model

uh and you embed an error. Uh let's say

I've got an error about the checkout and

I've got an error about onboarding. And

then I've got a Slack message about

onboarding. What the embedding model

will do is it will notice structural

similarity and it will put all of the

errors together. So, if you think about

what this looks like in embedding space,

you've got all of your errors over here,

all of your Slack messages here, all of

your session replays here, and none of

them get grouped to each other.

So, the way we get around this

uh is instead of matching in embedding

space the signals themselves, uh we

generate queries based off the signals.

So, we ask an LLM what is this signal

about? It'll generate a few queries and

then we match those queries in uh the

embedding space.

Yeah, so that's that's really important.

If you if you don't uh think about the

structural similarity of your different

sources when you're grouping them, then

the grouping works really badly. So, at

first we were doing this and then we

switched to this approach. It worked

much much better.

Cool. So, uh once we've got this uh

report that we've grouped together a few

signals, we've got some kind of idea

what's going on. Uh we then have

promoted the report because we think

it's important enough to work on. And

then we're going to hook it up to a

research agent.

So, uh this research agent is uh just

running that it's running the Claude

agent SDK. Uh it's running that in a

sandbox. We also use Modal for our

sandbox. Uh big shout out to them.

They've been great. Um they're not

sponsoring me or anything, don't worry.

Um,

and uh this research agent has a few

tools available to it. So, the first

tool is it's got our MCP server.

Uh, this allows it to, uh, given the

group of issues that we found, uh, you

want to pull in extra data. So, let's

say I'm looking at a session replay and

an error, I'll also pull in log data,

and the agent can pull in whatever it

wants using the MCP server. This makes

the results of the research agent way

more accurate.

Uh, the second thing is obviously it's

got the code base context, uh, and then

finally it's also got external MCPs.

That really helps to like ground the

agent when it's doing the research. Uh,

we found that in particular Linear and

Notion have been really helpful in

connecting it to to deliver better

results.

So, the output of this research agent

then, uh, is a summary of the problem.

It gives a priority, how important we

think this this problem is to work on,

and then it also uses Git blame to

figure out who should be reviewing this

PR if we create a PR for it.

So, um,

after that, we get a bunch of problems

that we think are worthwhile to work on.

Uh, we've got a kind of idea of what the

general problem is, um, and then we pass

it to an actionability step.

Uh, so here either it will be not

actionable. If it's not actionable, it

might just be that we don't have enough

data yet for this signal,

um, for the report, and so we'll put it

back into the pool to keep gathering

more evidence.

Uh, if it needs human input, it might be

because it's a product-related decision

that the agent can't really make a good

call on. Um, so if that happens, we'll

put it into an inbox for you to review

in the morning.

Uh, and then finally the the best case

is that it's immediately actionable,

uh, and that the agent can just write a

fix for it.

Uh, right now the the challenge in this

pipeline of getting immediately

actionable things is that for some

sources like error tracking, uh if you

think about your data in Sentry or

uh any errors that very specific, and

usually a coding agent can work on them

really well. For other sources, like

Slack or session replay,

uh you get much more generic problems

that can have a lot of different

solutions. And so, that's where it's

harder to get immediately actionable

reports.

Cool. Um

then, once we've

researched this thing, we go on to

executing the task. Uh

this will uh clone the user's repo into

a sandbox, uh similar to the research

agent. It's then again running the

Claude agent SDK to build effects for

the problem.

Um and then, uh as it writes those

fixes, it will uh push a PR.

And uh when CI is failing or there's a

comment on the PR, it will trigger a

rerun of that sandbox. So, at the end of

this, we snapshot the sandbox, uh and

then, if there's a comment, let's say

from an agent who's reviewing it, we

will rehydrate that snapshot and

continue running until the PR is green.

Uh and this delivers really good

results. It means when you're waking up

in the morning and things have been

running overnight, you wake up to,

instead of a bunch of CI failures or

uh comments that you need to address

manually, that you're pulling down to

your local environment, you ideally wake

up to just green PRs.

Cool. So, um what did we learn whilst

building this? Uh well, the first thing,

which I guess we've talked about in the

last talk, uh is that evals really

matter. Um so, at first, we were trying

this all out on our own data locally,

doing kind of a vibe check, is this

okay?

Um but this this really doesn't work

well for a pipeline that is is taking

lots of uh customer data that's

different. Um so, you really need to

know what's going on in production, and

if you're not testing on representative

data, it your you're basically just

fumbling in the dark, right? Like the

ability to iterate on a really good

pipeline matters uh only if you if

you're using e-bikes.

Second thing is what I said before, uh

make sure you're embedding the right

thing. Um embedding models uh the

off-the-shelf ones are matching a lot

based on structural similarity, not just

semantic similarity. So, if you're

thinking about clustering and your data

isn't all of the same format, think

carefully about what that data looks

like and how you can normalize it.

Uh the third thing is that uh if you

just throw an agent at a problem, it

will try to fix something. So, if you

get uh

uh if you get a signal report that's

like "Onboarding is broken" in a generic

way, then if you throw that at the agent

SDK or at Claude code, it will just try

and fix something.

Uh and so, it's important to understand

if the problem that I've described is it

specific enough

uh and if not, I should ignore it.

Otherwise, you end up with a a lot of

noisy PRs that aren't doing meaningful

things.

And then the fourth one is uh that

tokens are free. Uh obviously, that's

not true. They're not free.

Um but when you're experimenting, uh we

were at first uh focused a lot on the

costs of the pipeline. When you think

about the input, you've got loads of

signals coming in.

Uh and so, we tried to avoid using

agents where we could or delay it till

as late as possible in the pipeline.

And when we were experimenting, this was

a big mistake. Um mainly because uh when

you throw an agent at a problem, you

once you throw it at the same problem

100 times, you start seeing the kind of

clever solutions that it comes up with

and eventually you see similarities. So,

we started at a point where this

pipeline is completely unfeasible. It

was it was way too costly to generate a

PR, but then you quickly start to see uh

similarities in the agent's behavior and

you can take a really expensive step

that you're running an agent for uh and

turn that into a one-shot LLM call or a

model that you're training that's much

faster.

Cool. Um so, this is where we are right

now. This is what we've built.

Uh we have the signals coming in from

product data.

Uh these are grouped into reports, and

we're turning these into PRs that are

ready to merge when you wake up. This is

currently something that's in alpha.

We'll be rolling it out kind of over the

next few months.

Um but where we really want to go is is

a product that builds itself, right?

Like when you're thinking about what you

do day-to-day,

what you want to do during the day as a

developer is like come in and work on

exciting features and not worry about

all the bugs that customers are sending

you or worry about doing boring

experiments on pricing or onboarding.

So, we just want to do that all for you.

Uh we want to ship experiments

automatically, measure the impact of

them.

Uh instead of you reviewing changes, if

the change is pretty easy, let's just

approve it with an agent and deploy it

behind a feature flag. If it doesn't

work very well, we can always roll back

the flag and then delete it from our

code base later.

Uh and then the other thing that we want

to do and get better at is we want to

learn from every single outcome. So, if

we're creating a PR for you, if you're

rejecting that PR or there's been an

issue with a deployment or the errors

resolved in production once we've

released something, we want to get

better at learning from that in the next

PR that we're generating. That's

something that we're going to be

iterating a lot in the pipeline next.

Cool. Um yeah, that's it. That's what

we've built in PostHog.

Um if you're excited by uh looking at

thinking about what you can do with

agents and data, I really recommend if

you've got a product that's producing a

huge amount of data, your users are

going through that. Agents are amazing

at this stuff. Throw an agent at it. See

what it does. I'm sure you'll be

surprised.