I'm not going to lie, I'm pretty excited

to film this video because we're

actually talking about one of my old

favorite topics again, performance in

the web. Yes, really no clickbait. We're

actually going to dive into some

performance stuff. But the reason we're

doing it is probably why you guys are

here. Because the fixes that I'm working

on here are things that I could not get

any of the models to do. But more

importantly, the things that were broken

here were broken because of those very

same models. As great as AI developer

tools have gotten and as powerful as

models like GPT56 and Fable are, there

are still things they don't understand.

Some of that is because these things are

just really hard to get right. But some

of it's because the tooling just isn't

there for the agents to figure stuff out

properly. And while the agents couldn't

solve this problem themselves, I

actually found really useful ways to

wield agentic coding while I solve

problems that the agents just could not

understand. What are those problems,

though? Well, here I'll show you. You've

probably seen the app before. It's right

here. T3 code. So, what am I talking

about with that? It looks like it's

running fine. Everything when you click

it loads immediately. It's pretty good.

Well, here's the problem. My GPU process

is currently using as much as 13 to 15%

of my CPU, and I'm running at 720p right

now. When I blew this up to my full

resolution on my laptop or even pulled

it up on my 5K Studio Display XDR, this

would take up to 50% CPU utilization on

the graphics process. That's not good. I

don't know about you guys, but this

doesn't look like a complex 3D game to

me. This looks like a very simple web

app. It should not be performing that

poorly. Did I screw up the React code?

Am I hitting the network too

aggressively? What could I possibly be

doing that is causing my laptop to heat

up and is causing my GPU process to use

such an absurd amount of resources?

Well, these are all very good questions

and if you were to ask your agent about

it, the answers you get would be frankly

garbage. And the actual answer ended up

being way more interesting than I

expected. I put a lot of time into these

fixes, roughly a day and a half, and I

talked to a bunch of my favorite

performance people in the space, and

most of them were also surprised at how

absurd this problem was and how common

it is. Spoiler, it was a single built-in

Tailwind class, kind of. You're going to

want to watch this one if you're a nerd

about the web. And at the very least,

you're going to want to watch this if

you want to see how to utilize agents in

solving problems that they are not smart

enough to solve. Before we can do that,

we're going to have to take a quick

break for today's sponsor. My team is

shipping so fast right now, it's kind of

insane. Today alone, I merged 11 PRs

into T3 Code and about to merge many

more. But in order to do all of this, we

have to run a lot of CI. And if I'm

being honest, CI is just not very good

on GitHub. GitHub actions feels like a

forgotten product of a past era. And

that's why I'm so thankful for today's

sponsor, Blacksmith. If I'm being

totally honest, GitHub should just buy

these guys because they made using their

platform so much better. For running

your actions, Blacksmith is two times

faster. For downloading from cache,

Blacksmith is four times faster. And for

building Docker, it's up to 40 times

faster. If you think these numbers are

I promise you they aren't.

I've seen it in my own projects. I've

seen build times go from 20 plus minutes

to sub 10 trivially. Making the switch

is easy. You change one line of code in

your action and you're good to go. On

top of that, there's so many little

things they get right. If you or your

agents want SSH access to figure out

what went wrong, they have that. If you

want to cach your git checkout so you

don't have to fetch the whole thing

every time with a sticky disc, they have

that. If you want to dig through logs to

see which builds are succeeding and

failing and why, they absolutely have

that with one of the best observability

platforms I've ever seen for CI. If your

CI runs are longer than this ad, fix it

today at soyv.link/blacksmith.

I want to go full storytime mode so you

guys can understand how I ended up in

this position because it was quite an

experience. I was doing a lot more work

in T3 code over the last few days both

to like maximize my fable usage before

we lose it and also because I just had a

lot of stuff I wanted to get done and

finally was done with conferences for a

brief like 5day window which ends

tomorrow. I'm back at conferences again.

This video is probably coming out when

I'm at open source. Yeah. So I finally

had time to just sit down and work for a

very brief moment and I wanted to get as

much done as possible in that time and I

did. This is a screenshot of me using T3

code a couple days ago where I was

pulling up and landing dozens of PRs

across different projects. I still have

like 10 or so trying to get merged into

the T3 code project as well as a bunch

of work on LakeBed which is so close to

launch. I'm really excited about it. But

as you can see here, just the sheer

volume of threads like these ones, this

is like eight threads that are all in

the last 10 minutes. I have a couple

here that are in the last 12 hours or

so. I got a shitload of work done. But

the more I was doing, the more I started

noticing some issues with the

performance. Thankfully, for most of

this, I was just at home at my desk with

my monitor plugged in. So, it didn't

seem too too bad. But then I started

doing something silly. I started using

Moonlight to play games on my desktop

remotely while doing Agentic Code on my

laptop. And while I did have pretty

absurd settings, I was streaming at 120

fps over my 10 gig network. and the

games looked and ran great. It was

insane that it wasn't native because the

latency was really low. Huge shout out

to Moonlight. It's an unbelievable

client. It's so much better than Steam

Link. It's way more annoying to set up,

but once it's set up, it runs comically

better. But I noticed that when I was

running the game that my keyboard

started getting pretty damn hot. So, I

put a bunch of time trying to optimize

Moonlight and Sunshine to get a better

experience there and not have my

computer overheating as much. But, I

couldn't get it quite where I wanted.

So, I closed it and went to bed. The

next day, I woke up and came to my

computer and it was still quite hot. Not

quite as bad, but pretty damn close. So,

I started using Fable and 56 on my

machine to trace things to figure out

what's going on in a way that's causing

my laptop to overheat this much. And the

more I dug in, the more I realized it

was my browser processes. Because, as

I've mentioned before, I usually run T3

Code through the browser because I'm

using it to control other machines.

Generally speaking, the biggest issue

with T3 code performance has never been

T3 code itself and certainly not the

JavaScript backend that we use for the

websocket server is almost always the

actual harnesses that it's running

because things like cloud code and

codecs aren't the most efficient on

memory or on RAM. But all of these were

running on other machines on my network.

It shouldn't affect my computer anywhere

near as much as it was. So I was

confused and concerned. So, I started to

dig in a bit more and I couldn't figure

out what was going on and I also

couldn't figure out confidently enough

whether or not it was my tab in Helium

or other tabs because, and this is

probably the most important hint to get

from here, the builtin browser

performance tools were not helpful at

all. These tools are more focused on

things like JavaScript running when it

shouldn't or really slow load times or

resources that aren't coming in

properly. Slight spoiler for what's

coming. Once things are offloaded to a

layer like CSS, these tools become much

much less useful. And when I went into

the browser tools for T3 code, I just

didn't see anything that suggested that

this performance should be as bad as it

was. And it also was still snappy. Like

navigating and doing things was great,

but it was hammering this GPU process.

But also again, this task manager is not

tab specific. This is for the whole

browser. So, if any tab had problems, it

would come up here and there was no way

to figure out which tab it was other

than debugging one tab at a time, which

is why I'm currently in a browser you

don't see me using much. I'm in Chrome

because I use Helium for my day-to-day

work and I use Zen as my like casual

browsing. So, I need a browser that only

had one tab in it, T3 Code, so I could

be certain that the problem was T3 Code.

And sure as hell, you can see here it

has to be because if I go to another

place like google.com,

the GPU process drops a bunch down to

like 3% instead of the 18 plus I was

hitting and 40 plus I was hitting when I

was on a bigger monitor. So all of this

made it clear something was up and it

was up with T3 code directly. So I

started how I start a lot of tasks

nowadays. I gave a very vague

description of my problem to codeex to

see what it would do and it wrote this

up. T3 codes web UI can plausibly drive

sustained CPU load and laptop heat. Live

Chrome profiling found a websocket

driven main thread stall while source

analysis found several multiplicative

paths that could turn active agent

streams into continuous decoding history

scan state rewrites persistence time

derivation and highlighting. And this

all sounded pretty viable. So I did what

I usually do. I told the agent to build

the thing and test the thing and see if

it could fix the problems. They very

confidently came back and told me it

did. This was a huge PR where it rewrote

a ton of stuff. So, I started using it

and it didn't meaningfully improve the

experience on my machine. This is when I

realized I was in for a rough ride

because it had just done like a 10,000

plus line of code change in order to

optimize performance mostly focused

around the network layer and how it

triggered updates in the React layer for

the app itself. And none of it mattered.

None of it mattered at all. And that's

when I started to go deep. That's when I

gave up and opened up Chrome and

isolated things to hear. That's why I

started using the task manager to figure

out what might be the problem like what

layer is this issue happening in. And

that's when I noticed in the browser

section that GPU process was using way

more resource than it should be. Even

here when I go to the tab, the tab's

only showing like 1 to 4% utilization,

but the browser is pulling GPU as high

as 20% even on this low res. Thankfully,

I have been doing this since way before

AI was a thing. And I had a rough idea

of a handful of things that could be

causing problems. In particular,

anything that triggers at a high frame

rate because I was on a 120 fps 5K

display. So, I asked for my agent to go

through the codebase and find every

single thing that triggers rendering or

updates on the page that could possibly

have any GPU demand whatsoever. This is

the thread where I did most of this work

for reference. I'm noticing some brutal

performance issues in this app, both

Electron and Browser. If you fix in

Chrome, they should be fixed globally. I

have a separate PR for CPU related

issues, but they aren't the ones I care

about right now. The GPU is getting

pegged, and I have no idea why. I traced

a bit in Chrome on the production app

code site, and I got an AI summary. You

should start with this. This ended up

being absolute garbage. I was just

curious and still loosely debugging

while trying to play Pow World. So, I

threw the Gemini summary from Chrome in

here, and it was pretty garbage. again

was saying that scripting, style,

layout, and framework were all like

happening, but very, very small amounts

of time. So, none of this seemed like it

was the problem. And initially, it

called out the ultrathink composer

treatment. This is a pretty weird thing

for it to call out because the ultraink

UI only appears when you type ultrathink

and you have claude code set because

it's triggering ultraink in claude. And

it does this fancy little gradient

around the input box. But that only

happens when you're using cloud code and

Ultraink. And I had this problem when I

was exclusively using codecs and had

forgotten we had this Ultraink feature

in. And as cool as the little text box

is, I would remove it immediately if it

was a source of problems. And it's very

obviously not. So, the fact that it's

calling out the sustained GPU bug being

the Ultraink composer treatment made me

realize in this exact moment just how

I was because it was so far from

the reality I was experiencing that

there was no world in which I was going

to get Soul to figure this out for me.

It failed just as bad with roughly the

same prompt. By the way, it also fixated

on this Ultraink rapper. So, at this

point, I realized the model was not

going to solve this problem for me or

even find the problem for me. So, I had

to flip my mindset a bit. Instead, I try

to think about how can I use the model

to debug as many different theories as

efficiently as possible. So, I asked it

to make a way for me to test changes

that are easy to experiment with in prod

ideally with just console commands so I

can see an immediate difference. So, it

wrote up this fun window binding

function for me that would create

underscore_t3gpu

and in this let me define custom CSS

that would be applied to the page to

turn on and off various features. This

included things that I was suspicious of

like animations, filters, shadows, the

composer, blur, any media layers, and

this noise layer. This is one I was

particularly suspicious of. So, now that

I have this all defined, I can yink it.

I can go back to this broken slow build,

go into the console, paste it, and now I

have the ability to experiment. So, as I

was saying before, performance here is

bad. It could occasionally get as low as

like a bit under 10%, but it's usually

way over. Watch what happens when I do

underscore_t3GPU.apply

all. It has broken the UI a decent bit.

Like the text is appearing behind the

text box. But if we look at the

performance here, we are now down to 3%

or less GPU process. So it's clear

something in that set fixed the problem.

I don't know what yet, but something

did. But now we have a custom tool that

we built to go back and forth with these

options and figure out what it was. And

just to confirm, I have the D3GPU.reset

function. And as soon as I run that GPU

process spikes back up. Okay, let's

start applying things. I said noise was

suspicious, so let's just do the noise

change. Huh. Numbers are still pretty

high. It's idling a bit lower with the

noise change, but not enough. Okay, how

about blur? Now the blur is off. And the

process is somehow going up. It just

almost hit 20%. Okay, let's try the next

one. Animations. I hope it's not this

one because if it is, that means to turn

off animations.

Oh, looks like it was. All of a sudden,

the GPU process is down to basically

nothing again. And now I have some

actual information. I also forgot to

mention at the start, it spent 30

minutes trying to get debug info out of

Chrome and just failed outright. And

eventually I just told it to use

computer use and look at the browser

tabs instead instead of trying to go

through the Chrome extension that it

insisted on trying to use. Helped a bit,

but still this this is a rough one if

I'm being frank. This message was a cued

one where I gave it the context of a

basic empty like idle T3 code instance

and the massive GPU utilization I was

seeing said that this changes the

diagnosis materially. The 31.3% is GPU

process CPU with 1 gig of GPU process

memory specifically while the thread is

working. Likely frame pump is now

visible in code. Here's where it starts

fixating on the animations. I also had

told it that the animations seem to turn

off and cancel this problem out here. It

gave me another script I could run to

find all the animations and manually

pause them and see what happens. It is

also worth noting that DevTools changes

the performance characteristics of a

site. just having it open starts to

insert a bunch of things into the site

and put it in a debug mode that

inherently makes it slower. So, it's not

the most accurate readout compared to

something like the task manager that I

was using. So, it told me to do that,

close dev tools and then watch the task

manager and see what the difference was.

Then, it told me what order to apply

different things in to see how it

affects the performance. I then

described what I had seen, which is that

all had an immediate cut to under 4%,

reset jump back to 25 plus, and

animations appear to be the biggest

improvement. One correction helpers

animations rule disables both animations

and transition. Let's separate them in

order to diagnose further. Now you can

turn off transitions and animations

separately in order to get a more direct

answer to what was up. It had its

theories and I will again tell you guys

most of these are wrong. The pulse had a

little bit of the impact. This stroke

for the blue context window ring did not

at all. It did not matter in the

slightest. The pulse was the biggest

issue for sure in different places

though, specifically in the sidebar. And

here's where we identified one of the

major offenders. It was the pulsing

terminal icon that appears when you open

a terminal in a thread. This little icon

right there was what was causing all of

my hell. I could never have guessed when

I first started running into the problem

here that of all the things on the

screen, these two little green icons in

the bottom were pegging my GPU on my now

$8,000 computer. And then it gave its

theory for why an opacity pulse is

expensive. Normally, it's a cheap

compositor only animation, but an

infinite compositor animation still

requires frames at the display refresh

rate, which for me was 120 fps. And on a

high DPI display, it was pretty brutal.

And its suggestion was to make it

static, which I did not love. And it

said, "If motion is desirable, use a

finite pulse only when it first becomes

active." This is like the worst guidance

I've ever heard. Like, if you like the

pulse, do it for two and a half seconds

instead. No, you. This is why

everybody thinks that Codex is so bad at

design. I hated this suggestion so much.

It gave suggestions for bad

optimizations to not do. So stupid. And

it also suggested a lower frequency

blink to see how it would affect

performance. I tried that and it was not

much better. It was a little better, but

I also hated how it looked. So, I was

just not happy with a lot of this. And

it made a PR that just removed pulsing

from that one icon, even though other

icons like the working completed and the

uh work in progress in threads were also

affecting this in a meaningful way. So

again, Soul just was terrible at this.

Here I start talking about other agents

because this is the point where I was

accepting of the fact that Soul was not

going to help and was hoping, praying

even that Fable would do a better job.

So I gave it some of the work Fable had

done. And this is probably the point

where I should switch over to the Fable

thread because I'd given up on Soul

being helpful. Also, fun fact, I did

most of this work on my other MacBook

over T3 Code remotely, but since I was

moving to Fable and it wouldn't benefit

from Codeex's control options, and I

also wanted a bit more resource

available, I moved this over to my

framework desktop running Linux because

it wouldn't have quite as much computer

use capability, but at this point, I

didn't need it. I just needed to test

theories and write code. So, I had this

going at the same time also so that like

these two could run at the same time and

not interfere with each other and get

confused by each other's solutions. This

separation was really convenient and

just throwing old machines on my network

and being able to hop between them just

by clicking different things in T3 code

in the browser. It's such a nice UX. I I

have never been doing more work across

more machines in my life. Huge shout out

to Julius for making all of this as

stable as he has. The best thing I ever

did was buy a Mac Mini for Julius

because now he's making all of this

workflow stuff work great. So, here is

what I did with Fable. I discovered some

awful perf issues due to CSS animations.

They're pinging the GPU really hard,

especially on 120 Hz monitors. I know

the terminal icon in the sidebar is

guilty of causing problems. What other

animations may cause PF issues? What can

we do to make them less GPU intensive?

Found the full animation inventory. Here

is where it started and like built up

the list of all of the different

animations and again started with three

that didn't matter. It started

with the loading skeleton when you first

open T3 code, which is not the problem.

I don't care. And then it suggested the

Ultrathink Rainbow and the Ultraink

Chroma Shift. As I discussed before,

Ultraink has nothing to do with this.

Thankfully, as we keep scrolling and we

leave behind the expensive ones, we end

up in the moderate ones where problems

actually live. There's the animate

pulse, which is the terminal icon that I

mentioned before, as well as a few

others, like the typing dots for message

timeline, thread status indicators,

things like that. Opacity animation on

the compositor. So each isn't expensive

alone, but every infinite animation

promotes its element to its own GPU

layer and keeps the compositor

committing at 120 fps even when nothing

else changes. With several sidebar rows

pushing at once, that's a lot of tiny

layers being recomposited forever. The

reason terminal icon feels guilty is

probably that it prevents the page from

ever going idle. Oh boy. Now we get into

animate ping connection status.preview

local server card. These did not end up

being a big deal. Animate spin didn't

matter. It's only loading states. And

the provider update pill countdown also

didn't really matter. But here is where

I got some actual useful insights from

Fable. It fixated on the backdrop blur.

It's a thing that was funny enough added

by Opus way back when we started T3 Code

in order to try and make it look better.

For the most part, our use of backdrop

blur is not too bad. As you see, I am

blurring the background behind the text

box and my performance isn't bad right

now. But there are other places where

we're doing this type of thing. And in

particular, I don't think it's mentioned

here. Uh, okay. The the root tsx is

mentioned, we had a noise layer that was

applied to the front of the page with a

very low opacity in order to make it a

little grainier, to make the gray not

feel so monolithic.

It's very common and looks so much

better. We've used this on most of the

things we have built. But when combined

with a handful of other things,

specifically the backdrop blur and the

consistent animation, that's when the

performance started to get bad. Mind

you, it took me this entire thread and a

dozen and a half back and forths to get

this figured out. And even when it did

get mostly figured out, it did a really

bad job fixing it. This might not come

through in the encoded version of this

video on YouTube because compression is

a but there were very clearly

distinctly different gray colors in

multiple places. You had the gray for

the sidebar and you had a now darker

gray for the main body because my

filters were actually brightening the

color a little before and a different

gray on the bottom here for this special

area where like the local checkout

button and things go. It was not super

visible on most SDR displays, but on my

HDR MacBook and on my HDR Studio

display, it was really bad. So, yeah, I

was going mad with this. I ended up

removing the noise entirely and tuning

the color to make it look good enough,

but it kept screwing up the color here.

This one should be even more visible. I

don't know again if this will come

through in the compression or not, but

it was very clearly lighter gray on the

bottom, and it was very jarring and bad.

Found it. And this one isn't my change

at all. The bar is the composer's lower

chrome strip, the full width row that

holds the branch and checkout

information. In the dark mode, it's

defined as this specific mix. Let's

lighten it to try and make it match. It

screwed it up again.

Ended up fighting it more in another

thread and eventually got it to the

point where you cannot see it at all.

But this also resulted in the darkening

of the body here, which with a little

tuning, I got to a point that I was okay

with. Most importantly, I got the

performance down right.

Why is that still at 18%.

20%. God, did our merge not go through?

I'm on the site app T3 codes. So, when

we finally shipped it and I finally

moved back over to Helium, I noticed the

GPU process cooking still and I was so

stressed thinking I just did all of this

work and it didn't fix it. I don't even

have very many tabs open here. I have my

YouTube stream, which thankfully, as bad

as the YouTube client is optimized, is

not the problem. Here I have the chat,

which I'll close as well. I have my

codeex dashboard with my usage there.

We've lost a tiny bit here, but not much

so far. So, let's just close that. Okay,

that helped a bit. That took 2% or so

off it, maybe, but it's still way too

high. Well, I had forgotten something. I

had two other Helium windows open. I had

one for Claude and I had another for my

other Claude because I have three Claude

accounts. Once I closed those, the

performance was exactly where I expected

it. Do you know what that means, though?

What that means is an empty idle

Cloud.AI page uses 10% of my $8,000

laptop's GPU for each open tab. I had to

sneak a Claude duck dunk in here

somewhere, but I understand why. Because

when I asked Claude to fix it, it gave

the most stupid suggestions I've

ever seen in my life. So, as most of my

real engineering videos tend to

conclude, I think Anthropic needs to

hire some real godamn engineers because

they made me think that my fixes didn't

work because their code on an idle page

was that bad. To their credit,

the same thing happens in Codeex a lot.

In fact, this thing happened in Codeex

so much when I was first testing the

Codex desktop app that I was concerned

that the performance would continue

going to randomly. They would fix

it and then the next update would break

it again. And that kept on happening to

the point where I decided to build my

own open- source alternative to the

Codeex app to make sure this would never

happen again. And despite that, and

despite having 120,000 people, thank

each and every one of you, by the way,

having 120,000 of you guys using T3

code, I had to stay up until 5:00 a.m.

two nights in a row fixing this myself.

These things aren't easy. It does not

matter that models have gotten so smart.

They were very helpful. This diagnosis

would have taken me way more than like a

day and a half of partially working on

it if I had to do it all by hand.

Getting agents to build tools to help me

diagnose different theories was great,

even though their suggestions were

absolute trash. And this is why

being an engineer is so useful. And

having experience in these things is

great. My experience didn't make the

agent solve the problem, and the agent

didn't understand it better than me. The

agent could find things in the codebase

faster than me. It could build custom

tools to test my theories better than

me, but I was still the one who brought

the real information. And that felt

weird, but it also felt good in a way to

know that this experience I have

fighting weird CSS behaviors in the

browser without tools that are actually

helpful for it is still valuable. And

it's valuable enough that both major

labs that are trying to replace all of

us engineers with their fancy GPU

runners can certainly waste my GPU in

ways that make no sense. Yeah, this was

a fun deep dive. I am happy I did it and

I am very happy with the resulting

performance improvements that we now

have in T3 code. If your reason for not

using it before is because the

performance wasn't great, it's more than

fixed now. I highly recommend giving it

another go. The desktop app is great. I

personally mostly use it through the

website connecting over the CLIs on

other computers. And T3 Connect is

coming soon too, which will make it so

you can connect without even needing

something like Tailscale. But if you do

already have tail scale set up, I can

say with 100% certainty, T3 code is the

best way to work with it. If you're

curious about the final version of the

PR that landed and fixed all these

problems, you can go check it out on

GitHub right here. Both Fable and Soul

pissed me off a ton during this

debugging journey, but they also ended

up building useful tools to help me find

the answer. But what was really cool

here is that I still had to find it

myself. As great as these tools are,

they are just that, tools. They don't

know everything. They can find things

fast, but they have to be steered. And

when you learn how to steer them and

wield them for different types of work,

whether that is they find and solve the

problem themselves, or they build the

tools to help you solve it, agents can

always be valuable in almost any dev

work, even if not as valuable as people

on Twitter like to make them out to be.

Just cuz your agents don't know the

solution to the problem doesn't mean

they can't be helpful when you solve it

yourself. This is a fun deep dive for

me, and I hope it was for you guys, too.

Until next time, peace and ns.