Hello everyone.

Welcome back to SemiAnalysis Weekly.

On this week, we've got Cam, Kimbo, and Bryan and we are gonna talk about everything InferenceX the latest InferenceX article included a whole review about DeepSeek V4, and

what it takes to go from like day zero performance to day 43 performance, meaning

How much do these guys grind over the month?

So I'm really excited to dig into some, some topics here.

Uh, we're gonna talk about the model itself, the implementation, how it's running on different hardware, some of the nice vLLM versus SGLang Twitter drama that's been going

on, an agenic benchmark that we're working on on.

And and yeah, lots of fun stuff.

So guys, welcome to the show.

Excited for this one.

Super excited to get started.

Let's go.

Nice to be here.

Awesome.

All right.

Kimbo, let's let's start with you.

I think um the the burning question maybe for the general audience here is like what's different going from DeepSeek V 3 to V 4?

And when you describe what's different, like what's the what impact does that have on running InferenceX ?

In other words, like what does day zero support actually mean when it comes to a new model architecture?

Yeah, so I think the headline changes or the features of V4 compared to V3 and R1 would be the 1 million context length.

This ability, in order to achieve or to get to a million context length, DeepSeek did like very aggressive innovations on the attention mechanism.

that's like the first thing and the second thing is the Mega MoE to like speed up things to speed up the expert FFM computations.

first the attention mechanism DeepSeek V4 comes with two variants of sparse attention.

One is the compressed sparse attention and heavily compressed attention.

So both of them are

built on top of the previous version DeepSeeks sparse attention that came out with 3.2.

The idea is that by sparsifying your attention between query and the key values, you could reduce the memory reads so that allows you to do 1 million context length without

exploding your memory requirements.

both of them, heavily compressed attention and compressed sparse attention, in addition to that, have like one additional embedding compressor so that further decreases the size of

the KV cache entry.

And then after that, they both incorporate some sort of sliding window, which then further

decreases the KVcache usage.

Combined with all this, I think DeepSeek is quoting like a very, I think around 100, around that order, 100x reduction of the KVcache usage compared to a standard MOA model.

And yeah, that's like the headline feature of DeepSeek V4.

Yeah, yeah.

And it and it's also a different architecture in terms of like more total parameters, less active parameters when compared to V3.

So literally a like a different model, not just an update to the weights.

Bryan, maybe kick into you.

What does this mean when it comes to actually getting support from the various InferenceX runtimes and like what what work is required to actually get performance results out of a

system on day zero when the model's released?

Yeah, great question.

there were a lot of changes.

Not just, like as you say, not just the weights, but the sizes of certain things as well.

For example, some hidden sizes and dimensions.

So it's not just adding support for like new attention mechanisms, attention blocks.

It might also be like changing or tuning certain things that

the community thought was constant.

Like in this case, was the...

NVIDIA, I guess, it was the MHC dimension.

DeepSeek, for the previous times I've always released, MHC with...

what was it?

I think half of what Pro used, so V4 Flash and the previous iterations of DeepSeek V3 all used the same hidden dimension.

Sorry, not hidden dimension, MHC dimension.

But Pro used the new one which caused a hiccup for Nvidia on day 0.

Yeah, but for the rest of the guys, guess everyone adopted quite well.

On day 0...

it's worth mentioning, right, that some companies like vLLM and SGLang had like early access under NDA.

but companies like Nvidia like did not.

So that made it more difficult to to implement on day zero.

Yeah, so I guess they had a hit start with implementation.

That's why there wasn't as many hiccups.

But I don't remember, I don't know if you guys remember seeing it, but there was a Twitter, on Twitter there's this video circulating of like Huawei talking about the

optimizations, although it was in Chinese, but someone gave a translation and there very good optimizations talked about during Day Zero.

If you just follow what Huawei did, I think you'll get a good performance if you're optimizing from day zero.

Yeah, but they talked about kernel fusions, they talked about index optimizations.

Yeah, it's really good guide, I guess, for day zero optimization.

Wait, what one question I had, maybe Kimbo can answer this.

I've been reading a lot how about basically this architecture is like extremely kind of tailored to GPU, GP GPU like architecture rather than things like, you know, are you know

things based on systolic arrays like TPU.

Is there a reason for that?

Have you heard that as well?

Or just kind of curious?

I think solely in terms of what you said systolic array, I don't think so because systolic array is essentially like a hardware architecture specialized for a matrix multiplication,

right?

And GPUs literally have tensor cores, which is also specialized for mathmols.

Models do matmols, so I don't see any very special thing that V4 is specializing for.

But I guess what you might see would probably be related to Mega MoE.

That is very GPU, tailored to GPUs because of the way that the kernel launches and everything.

Is that what you're seeing?

I'm actually, I don't know.

I probably, yeah.

speaking of Mega MoE, what is Mega MoE actually, Kimbo?

Yeah, so.

oh

just that it's a huge fused kernel but just curious what it is.

Yeah, Fuse kernel, I think Fuse kernel is one thing.

The whole concept of MegaKernel is that you're breaking the kernel, the typical kernel boundaries.

So, compared to kernel fusion, which is either manually written by kernel engineers, where they just go through the math and they see what can be

or can be merged together.

And I compared that and also like compiler optimization where they removed the kernel launches.

Megakernel is more about breaking the kernel boundaries that are not typical.

For example, sorry, I suddenly can't remember which ones, but the

Some concrete examples of what can be broken is that the register usage.

So you can imagine that in a typical kernel, what happens is that you launch a kernel, you do computation, in a Von Neumann architecture, you would load data into the register.

You do computations in a register, and then at some point, you store it back to HBM.

So what Megakernel does is that they fuse different kernels so that

ah they could remove the round trip between data movement between registers and the HBM while also when a register is free they could automatically do the next operations of the

future kernels so instead of waiting for a whole kernel to complete and for for Mega MoE the case for Mega MoE is that

In addition to that, they also merged the compute and communication, computation and communication overlap thing.

So it's like all merged into one kernel and then they could do like very aggressive overlapping.

So what's the downside then of like fusing kernels?

It seems like it just fills everything right?

Yeah, in the extreme case where we see like the hazy research, they literally fuse a whole model like a LLaMA 7B, LLaMA 8B.

And I would say that the downside or I guess the reason that is preventing people from doing that is that it's a lot of engineering work.

in principle, so

In addition to it's a lot of engineering work, you have to justify it with something, right?

So that the thing is that Mega MoE is about aggressively reducing the latency.

And in the large batch scenario or in a training scenario, it kind of doesn't make sense or there is a limit to where it stops making sense.

So let's say if you're doing a training with a super large batch, like

couple hundred million tokens per batch, your computation and communication is sufficiently overlapped.

Your kernel launch time is not the bottleneck.

In that case, it doesn't make sense to do Mega MOEs.

And the other downside would be, I guess another difficulty for MegaKernels is that it's because of doing the aggressive

resource allocation and on the fly, it creates a lot of memory pressure.

So I imagine that would be, that'll require a lot of uh work on managing like the GPUs, I guess, so that they don't, I guess some physics related, maybe the overheat or some

related issues, which would in turn affect the performance.

Makes sense, makes sense.

Okay, I want to jump into throw sh throw something on screen here, just uh little screenshot from the V4 paper where it explains the open source MegaKernel and it

specifically says two things.

One is that it's been validated on NVIDIA GPUs, nothing else, and Huawei Ascend NPUs.

Obviously, we've seen the open source CUDA-based megacernel called Mega MoE, which then these InferenceX serving runtimes such as vLLM and SGLang and uh

TensorRT can implement for themselves.

And the performance improvement claim is somewhere between 1.5 and 1.73 times faster.

So clearly they've they've realized the benefits of this engineering work by by doing this.

Um, but I I wanna maybe share a second thing on screen, which is that like the open source community, just because the code is out there, doesn't necessarily have the opportunity to

benefit just from day one.

And so what we've been tracking really, and Bryan, maybe you can take us take us through this like experience in in detail, is how how quickly these configs can come online.

in other words, how how quickly can a given InferenceX serving runtime actually support things?

And I thought this little GIF video from the article was quite nice in terms of how it shows the progress.

over time as we count up the days and we start to see more hardware being supported as the B200, B300, GB200, GB300 come online, the MI355 from NVIDIA, and then you can start to see

the performance improve, right?

For people watching, like these lines going from the left to the right means either more throughput or lower latency per user, or you know, higher throughput per user in terms of

interactivity.

And um yeah, maybe you can take me through like, okay, day zero, there's two or three different hardware platforms supported and then a month later everything's supported, but

in that whole time a lot of performance improvements happen.

what are like some examples of some performance improvements that were just kinda like dropped overnight and then were a big win where people can suddenly

I don't know, twenty percent more throughput, which means twenty percent more users or twenty percent more profitability for the existing users or or whatever, you know, for a

given InferenceX end endpoint provider.

uh Wait, before I move on to that, huge shout out to our front-end engineer, Alec, for making that video feature.

It is very cool and shows you very nicely how improvements were made.

Yeah, and so moving on to the question, I guess one thing that we can look at to look at these improvements is AMD.

Like as the paper said earlier,

a lot the stuff optimized already for Nvidia GPUs and like because of how big the CUDA community is and how many people use CUDA, there was huge incentives for SGLang and vLLM

in the limited time they had with the model to prioritize Nvidia.

So MD catching up is actually a interesting look at like real optimization from getting to work on day zero when

So maybe our first step would be like getting the quantized versions to work.

AMD on day zero had FP8 only.

There were no native FP4 support, which was quite unfortunate.

And actually a very large gains came from getting it to work in FP4.

Of course, due to the calculation being faster and lesser memory needs to be transferred.

Actually we have a very nice plot on InferenceX if you may bring it up Jordan.

Thanks.

On the AMD improvements.

Yeah, but the biggest jump was converting a lot of the kernels into ITER instead of using Torch Fallbacks.

Yeah, so you can see the SGLang graph actually evolving the most.

oh There's a breakdown of optimizations as well I think below in our image.

It's a pretty messy image with like orange text in in ExCaliDraw Sorry.

Yeah, so there were very big improvements just from changing kernels from torch fall back into either Ultraturn or etc.

So it's very cool to see like the impacts these individual improvements have and of course not just like a humongous step, it's always step by step.

You optimize one part and you optimize another.

Yeah, but very cool stuff.

I I wanna like take a second to just talk about how cool this is and also shout out all the AMD engineers and Nvidia engineers that we work with, because this is all them.

Like we contribute a few things to upstream, like kernel libraries and stuff, but I mean we're we're we're just a team of three people and and this is all the the engineers.

Um but I think this is something that's so cool about InferenceX and like this was the whole thesis of it, the project, like when we started it is like

You know, a lot of things online, a lot of benchmarks, you just see like the end product, um, in terms of performance, but really it's like a lot of hard work, tiny iterations,

increasing throughput by five percent of the time, and that eventually compounds to sort of, you know, get performant and like you can see that here, right?

So whenever new models release, like MiniMax M3 just released, we're doing the same thing.

And over the course of a month or two, you're gonna see like

You know, you start out with PyTorch, you know, just Torch native fallback, and then you kind of use custom kernels and then you make the custom kernels even better and then you

you just keep making them better and better and pushing the frontier forward.

So yeah, this is really cool to see.

I also find it fascinating when I look at these things and you dig in to try to understand what these optimizations are.

Like every single one of them, if you tease out just that one optimization, like if you look at the tuned parallelism point or if you look at the fuse KV compressed tile laying

uh point, it might seem simple in isolation and in some cases maybe obvious that people should have been doing it.

But if you try to reason about the entire system all at once.

Where there's all of these improvements always constantly happening, it becomes like incredibly complicated.

And that's really what's so impressive about watching these engineering teams work is like the ability to take all of these individual optimizations and then build them all together

into one thing, which is just like better performance or more efficiency.

Super cool.

okay.

The next thing that this obviously leads to is the

bit of the comparison between the different runtimes.

You guys are are big on avoiding the the war between vLLM and SGLang the Twitter drama drama.

Can you talk about the maybe the reason for this and why in some ways having the competition between different open source InferenceX engines as well as vendor specific

libraries

is actually driving a lot of innovation and improving things instead of just like waiting for somebody to be nice enough to to share what they've learned.

They're kind of many of them are forced to innovate really fast in order to stay ahead.

Yeah, I mean we tend to to

stray away from comparing vLLM to SGLang on InferenceX specifically, just because mainly because we didn't find it beneficial necessarily to have

like make it a competition.

We we more so just wanted to showcase each one in isolation.

But, you know, on the other hand, it does kind of create some competitive spirit, which makes both sides move faster, which can also be a problem because we only have limited

compute to run things.

So when you double the amount of of submissions on on a on a certain model

It's kinda like, you know, a a lot of of CI time.

But yeah, we've we've seen we've seen great improvements by both vLLM and SGLang and I think the a little bit of competition is is good.

But

And and what about the vendor specific libraries?

I'm I'm gonna we can talk about like some of the other chips, but maybe the the most open or easy for us to run is NVIDIA and AMD, because we have NVIDIA and AMD GPUs.

Talk about Huawei in a second.

Both of them have closed source proprietary libraries in the form of like TensorRT with Dynamo for NVIDIA, and then ATOM, AiTer Mori, all the libraries from AMD that make it up.

Um, but they're not, you know, I guess what's the they're not the vendor specific libraries that are closed source are not always the best performance and don't always

allow you the most customization.

So like what's the maybe open source versus closed source dynamic that's going on when you guys have to try these things out?

the w the way I think about it is Tier T L M you know, like ATOM the AiTer optimized engine for ROCm is

They're like hyper, hyper specific to certain uh AMD and and NVIDIA architectures that will run really fast, but they're not necessarily like completely portable and not always

a hundred percent open source.

Like whereas vLLM and our SGLang, you know, you can fork it and go start like your own base ten together, whatever fireworks.

you know, it it's very like user friendly and it's has op you know API specifications and such.

So that's kind of how I look at it, but I don't know.

Maybe Bryan and Kimbo have different opinions.

I mean, in my opinion, it's just more competition.

I mean, on our branch mark, we also have a disclaimer saying that ATOM has no customers.

yeah, but that's the other problem.

your market just isn't, or rather isn't as big as Nvidia's.

But unless your product is really, really much better.

Like, Mori is a lot better than the previous

of ROCCL.

Yeah, but ATOM, the development is okay.

Development is fast.

AMD is putting support into it, which you'd to see.

But yeah, SGLang is just, there's much more help from the open source community.

A kernel development, which is really good.

And ATOM just needs that gap, I guess.

And who knows, maybe it'll take some time for that gap to close.

And ATOM gets more accepted by the community, I guess.

I mean they like twenty less engineers, so it's kinda question for you.

Like one of the things that I've been asked before and I've seen on Twitter is basically like why SGLang and vLLM, like why wouldn't SGLang just copy all the good parts of vLLM

and vice versa?

Whereas, you know, it's to the point where they basically just converge to the same thing.

I mean, we're talking about InferenceX, right?

We're talking about we're talking about being one percent faster or something, right?

And like everything is kind of being worked out out in the open.

So it doesn't have to be you, Kimbo, but like maybe Jordan and Bryan, like why do you think it's a natural thing that there are basically precisely two, maybe like a few other

like open runtimes?

and they're so like head to head and we're not necessarily converging on like one InferenceX engine.

I mean it is kind of an interesting topic, like

based on my understanding is that, so first of all, objectively speaking, they are, both of them came from the same, literally the same Berkeley lab.

They are lab mates.

So that's the thing.

So they started with a similar idea in the same lab.

They started the project.

And apparently something happened between them.

Some, like the main, maintainers, the major project developers, they have some problems between them, which I don't, I literally, I honestly don't know what happened.

And then they just are parting ways and having very similar ideas.

I think in principle, they are doing the same things and throughout the development,

throughout these years, they are, like you said, copying features from each other.

I'm sure you probably heard from our collaborators a lot that one, the other side is copying, copy pasting our code and to support this feature.

I've heard both sides.

So that's, there's that.

Like they were all one company and then Dario was working on safety stuff and something happened and then he left and and now they really drive each other in a way that I think

is benefiting the ecosystem because they're they both land counter punches and then we benefit as they improve quality and drive prices.

Lower.

Um maybe

R competition is good.

It it's it's interesting that, you know, they're very similar, but they have like uh

I I just want to say one other thing though, which is that I don't think it's a completely a historical artifact the way Kimbo describes it.

I think there's obviously stuff that's downstream of these run times now, whether it be customers or other libraries that depend on them.

And so having two different providers allows you to make a decision about who's gonna prioritize your features, who's gonna merge your stuff, where you're gonna get support

from, how you're gonna work with vendors.

Because they're just like one point and there's stuff that's upstream of them, like the vendor specific libraries in some cases, or like the hardware, literally CI testing,

InferenceX stuff.

And then there's stuff downstream, which is like, you know, RL libraries, like you look at Slime or you look at VeRL or you look at Prime RL and stuff.

Like if there's features that they want to see from the runtime, they need to make a choice and go with the ones that are gonna support them.

And there's there's many customers, like you mentioned, the InferenceX endpoint serving providers, as well as the labs.

You know, we know companies like Microsoft are using these technologies.

They don't have their own stuff in-house.

Uh Poolside talked about that in their paper, GLM, the whole thing's SGLang, right?

So a lot of people depend on this stuff right now.

And I think that's like it's at least good to have

two options to to buy from in open source if you're somebody that's making a library and you need support from somebody.

and you can have multi, you know, runtime back end support if you really need to, if you're really not getting what you want from the community and you need to go somewhere

else.

the the other, okay not to not to take this in too far a direction, but I I I really, really want to hear about Huawei.

And this is making me think a lot about

proprietary runtimes and software experience and like what this NPU chip is gonna look like when there's a lot more users and it goes from what we thought was l one or two lines

in a paper saying, Yeah, we have Huawei support and then no real proof.

But Bryan, man, there's real proof now that DeepSeek is running on Huawei.

Can you take us through what you've uh what you've found and the analysis that you did on the numbers that they shared?

Yeah, so actually before that I'm gonna talk about like the DeepSeek delays which are there were a lot of rumors on X because DeepSeek was like planned to release on Chinese

New Year and then it got pushed back and back and back and every time was like it's gonna drop this weekend never mind it's this weekend yeah so there were conflicting like uh

reasons some got some people on were like talking about they're actually trying to get a model working on Huawei

That's why they were delaying it and optimizing it first before releasing it.

And of course, there's the other explanation of like, they just want that better evaluation results.

But regardless, the performance on Huawei during release was real.

There were benchmarks and profiles shared by the Huawei team.

And looking at profiles, they were very elegant.

The Kness

optimization as of course as I previously mentioned were quite sophisticated at that point.

So they did get, they probably did get much longer access than VROM or SGLang to optimize for architecture.

yeah, in the future we would release another follow-up article on this regarding Huawei's results and comparing it.

with other chips, yeah.

I think the biggest...

I I don't want to distract you too much, but I'm I'm curious if you can compare the state of the software toolkit can from Huawei to where things sit with some of the other chips

like NVIDIA and AMD.

I know you haven't used it hands-on, but in theory, the software is open source and it can now run a close-to-frontier open source model, like one of the best open source models in

the world.

So this

This chip or this system has like come a long way, right?

Yeah, but like I mentioned, their code is all open-sourced, same as the vLLM and SGLang And if you want to look at it, you have to go to a Chinese version of GitHub.

It's called Gitcode.

But the documentation is very good.

the the code is open source similar to vLLM and SHLing, but not similar to TRT from Nvidia or ATOM from AMD.

In other words, you can see that you can read how they implement the kernels.

Yep.

And it's all on Gitcode.

And it is very interesting to see how they do things, like compare differently to NVIDIA AMD.

Their documentation, in my opinion, has been very good.

They do frequent meetups in China, where you can talk to the, they call it CANN, C-A-N-N.

And you can talk to the CANN developers during meetups.

And they have like weekly calls, I think.

They have a schedule of weekly course where you can hop on and talk to engineers.

So like I'll take from this is just that Huawei is very enthusiastic about getting this working and open source contributions on CANN.

So yeah, the direction has been very good.

Some minor notes is that like Huawei has implemented some optimizations from like papers.

I think slightly before Nvidia, the one I talked in the article was about fusing communications and computation kernels.

If not wrong, NCCL released it in 2024 or something like that, but Huawei released it much earlier, right after the paper on it was released actually, which is actually quite

interesting to me.

I mean, the Chinese developers are indeed 10x developers and their velocity is actually...

great and will be compounded with open source.

Yeah, thanks Jordan.

MC squared.

Yeah, and the velocity will definitely be propelled by the Chinese open source.

Yeah, fast fascinating stuff.

Kimball, let's bring it in here.

When you when you hear this stuff about support on Huawei, and then you see something like DeepSeek V4 Pro, 75% off discount at the start, which then becomes permanent, what does

that make you think for the future of like open source model competitiveness with

closed source models if they're just gonna try and drive the price down to zero and run it on Huawei hardware or whatever GPUs they can get their hands on.

Yeah, so first of all, it's pronounced Huawei.

There's an H there.

I'm a bit too annoyed by this.

And then it's my bad.

have this.

have an issue.

yeah, okay.

Aside from that, yeah, I think ever since we see ZAI,

with GLM are able to serve their models on Huawei Ascend chips.

We also see like this time DeepSeek V4 can do that too.

It shows that Huawei chips are getting there, being like a very strong option in addition to what we know from the GPUs and the TPUs.

And Chinese have a very different ecosystem.

They built a lot of things on their own.

Like Bryan mentioned, the HCCL the MC Squared and everything.

And they also the engineers have been very aggressive in optimizing everything.

I think these, the accumulation of all these things end up with DeepSeek being able to offer a very low price.

And I imagine that

One of the reasons that they could probably keep the price is that they've also pushed a lot more optimizations since release.

While I would also suspect that they are trying to capture the market.

Based on my understanding is that the chat or the AI model market for China, Byte Dance is still

like taking the majority.

So everyone else is fighting for market share.

So they are trying very hard at the cost of probably negative margin or at least just zero margin trying to get people to use it.

So that's why they could serve at a very low margin and yeah.

So I think that's the two main implications.

Makes sense, man.

Cam, you think David versus Goliath, do you think that's the right analogy for Huawei to uh everybody else?

Yeah dude, I think so.

We'll see.

Maybe maybe Huawei is actually Goliath.

We we don't know yet.

It's a David versus Goliath, man.

Yeah.

Um, cool.

All right, guys.

Uh this has been quite the tour of everything DeepSeek V4 on on InferenceX .

Cam, maybe we could leave by talking a little bit about future.

You got uh some big work coming with the Agenic benchmark.

Everything we've done so far has been fixed input and output sequence length.

So like what's coming new?

Yeah, so one of the things with MiniMax M3 and DeepSeek V4 are is the one million context length.

Right now, obviously on InferenceX we are not testing that.

We're testing fixed sequence length 8K, 1K to 1K, 1K which nobody uses.

Uh nobody else.

Um

But I I I said this in the last podcast, and it it's still good to have the 8K, 1K to 1K, 1K, because it's testing, it's showcasing, just in my opinion, basically pure chip

performance, right?

There's no prefix caching or anything like that.

So we're moving it up and and we're gonna showcase.

I mean, InferenceX is a systems problem.

so we're we're moving up the stack, we're gonna showcase and we're gonna run an agentic benchmark with

real clawed code traces that we've collected internally.

we're gonna showcase things like Dynamo, and you know KV block manager, other sort of KVBMs and yeah showcase like different PD disag optimizations and I think it's gonna be

really cool and that that's gonna come in the next couple weeks so be on the lookout for that.

Uh

Yeah, that's that's all I have to say about it right now, but we'll it'll be really cool when it comes out.

Excited for it, man.

Cool.

Okay guys.

Anything uh you think is left unsaid on this uh on this stuff so far?

Anything about DeepSeek or future plans or infrartext in general that you guys are looking forward to?

Let me think.

Bryan, do we have anything else upcoming that we can talk about?

I'd say be- yeah, Agent X .

That's what I that's what we were just talking about.

So that's coming in the next couple of weeks, but I'd say be on the lookout.

Yeah, be on the lookout for new chips.

We have quite a few in the pipeline, so that should be really, really interesting.

Um yeah.

Awesome.

And Kimbo, maybe next time we can talk about RL system performance as well.

I know you've been doing a lot of work there.

Some of it depends on the InferenceX runtime, but there's much more to an RL system performing well than just uh having a fast runtime, right?

So cool guys.

Well this is a great show.

Appreciate you taking us through this and if anybody wants to find you InferenceX hex on GitHub, find us on Twitter, making memes.

Maybe it creeps to LinkedIn and YouTube sometimes as well.

Yeah, they will make YouTube short.

wide expert parallelism and uh related topics.

Yeah.

All right, thanks for coming on.

Nice job.

Bye bye.

Thank you, Jordan.

Thank you, everyone.