Today I'm getting a tour of one one of

Jane Street's training data centers here

in Texas from [music] Ron Minsky who

co-heads a technology group and Daniel

Pontecorvo who has the physical

engineering team. Thanks for coming out

here to do this. Thank you. Well, let's

get started.

So here's one of where we have our our

training cluster of GP300s and VL72.

>> Are you allowed to say what is currently

happening on this cluster?

>> What actual training jobs are running

right now?

>> High-level. We do a bunch of different

kinds of models. Some of the models are

for training LLMs. Some of the models

are for training all sorts of custom

architectures that are more adapted to

the trading problems in the trading data

sets. Earlier you were explaining to me

that this is originally not a facility

that was built to handle 200 kilowatt

racks. And so you had to retrofit it to

be liquid cooled and everything. You

know, these cabinets, these GP300

cabinets, consume at peak about 140 kW

each. Compare that to traditional

air-cooled you're talking about 10 to 40

kW. It's a lot more. So on the perimeter

here you can see some of this air

cooling equipment here that would have

have fed a traditional air cooling. Some

of it remains, we use some of it for the

air air-cooled load that we have. About

15% of these cabinets are air-cooled. In

here inside the GPU you can see how the

how the cooling kind of flows from these

quick disconnects on the back

taking a fluid in, routing it to these

cold plates that are sitting on top of

the GPU, and then coming back out at a

warmer temperature.

So when you slide this sled in, it

automatically connects to liquid supply

and return and 54-volt power. Right? So

these just slide in,

powered and cooling all within the sled.

There are still some components in here

that are air-cooled, but about 85 to 90%

is cooled via those cold plates of the

heat load. To what degree have we had

concerns about leaks, right? I feel like

you spent, you know, decades worrying

about not having water in our data

centers and now we're like putting it on

on purpose. Like

How big of a deal is that? So, inside

they have something there's underneath

there's there's

these ropes set to detect leaks. So,

there's a management side of the switch

of this server that will send out an

alert if they they sense a leak in

there. Furthermore, there's leak

detection underneath the floor. If it

drips and falls under the floor, we'll

be able to sense it there and isolate

via valve. But, it is true that if

something here fails, you are at risk of

destroying the server. How often is

there a leak? Not often, but this stuff

is new. You know, it's yet to be seen

over time how this how this works out.

So, I guess it's a surprising to me that

like it was not a problem to get this

get a liquid cooling going in a facility

that was originally built for air

cooling and like lower power densities.

I don't know how Why did it work?

>> How do you define problem? I mean, it

was an engineering challenge.

>> I feel like you're hearing the version

of the story after all the problems have

been worked out. These guys have spent a

huge amount of effort figuring out like

this place would feel in a kind of

intermediate point where we knew we had

to scale up a lot, but we didn't know

what the shape of the coming compute

was. And so, I think one of the things

that the guys here did really well was

like take to heart the importance of

optionality of like, "Oh, yeah, there's

a lot of different futures and we need

to build some stuff that will work for

multiple different ones." And I think

that works really well, but required a

lot of hard thinking and planning to

make it land well.

You know, one thing I'll say is there's

a couple of ways to do liquid cooling.

So, we have fluid coming from the roof

from the chillers on the roof down here,

maybe about 18° C.

>> [music]

>> We use the same fluid for the air

cooling, so it's fungible within the

data center. We can move the fluid

around. Oh, that's nice.

So, what we do is we send this in. This

device here makes sure that every single

cabinet has the right amount of flow.

You don't want the ones at the beginning

of the row to receive too much flow and

the ones at the end to be starved. So,

what these devices, these valves, they

control how much fluid goes to each

cabinet so that they're balanced.

How How do they measure whether they

have the right amount of fluid?

>> Ultrasonic flow meter here. So,

ultrasonically it's measuring the fluid

and measuring how much flow in liters

per minute and capping that at some rate

that we predetermine based on the heat

load that it's rejecting.

Um that liquid comes in, it goes to this

heat exchanger here. So, there's inside

that heat ex- inside that CDU is a heat

exchanger that transfers heat between

this building loop, building cooling

loop, and a what we call a technical

water loop inside there which is needs

to be very, very clean and filtered down

to 25 microns so you don't plug the cold

plates on the GPUs. You want very good

efficient heat transfer there.

Um it's filled with a liquid uh a mix of

distilled or deionized water and uh

propylene glycol, 25% of propylene

glycol. Um that's to inhibit any

bacteria or algae growth. Um and that

bacteria could grow in there if you

don't have the right ratios and plug the

cold plates and plug with the

uh the heat exchange between the GPU and

the cold plate. I don't love the world

where we have to worry about bacteria

growing in our servers.

>> So, you have you have leaks, you have

bacteria, all these different new things

to worry about. Um making sure you have

proper flow between all the devices. Uh

air-cooled data centers, you you know,

you put the cabinet in and just flood

the room with air and and and heat and

and transfer that heat. So. Was there

just an area underneath that you could

have used? Raised floors traditionally

were used to to supply air. Uh so,

there's ways you could supply air. That

air would come out here. A lot of the

the new solutions are going to overhead

piping cuz it takes time to build these

raised floors and it's it slows down

projects. So, a lot of the piping

system's going above overhead now for

for speed of deployment. But, we like

this here because you see that blue uh

that blue wire there, that'll sense a

leak. If one of these connections is

dripping, it'll touch that that rope

there and send a signal to say that

there's a leak. So, we're able to

contain a leak under the floor and and

and measure it where overhead it's kind

of right into your data center.

So, we have

4,032 GPUs here in 56 racks. What we try

to do on the on the power side is make

sure we balance our power. You can't

overload certain areas, so you kind of

you can see how we have this bus way

here distributing power and we're very

conscious about how many racks are on

each bus.

Make sure you don't go over amperage and

trip a breakers and you could be in the

middle of a training run and overload a

breaker and you'd have to go back to

some book worth. What is the hourly

price

on a black wall rack? There's two ways

of pricing it, right? There's how much

does the hardware itself cost and the

power and all of that and then there's

the opportunity cost.

>> That's right. That's right. And we

actually think about opportunity costs

when we think about all of this compute

stuff very intensively and like

because we're in a world where compute

is relatively inelastic, you end up in

places where there's a real crunch even

internally where it's like, "Oh, it's

you know, it takes time to get new

compute online and available." And you

can get to a case where it's like people

are all kind of bidding for the same

compute and you're like, "Wow, it's

become incredibly expensive." Because

the stuff that we get out of this is

super valuable to the business. And so

the opportunity cost tends to dominate

the hardware cost.

Even though the hardware costs are not

small.

>> [laughter]

>> Yeah, so that's the other question. If

this facility is connected to the grid

and you presumably had asked the grid

beforehand for a certain amount of power

and now you move to much denser compute,

how are you and this is still like using

the grid it's not behind the meter. How

are you able to get the power in here?

So, because as the compute is moving

denser, what is what you know, if we

have some power allocated from the

utility,

we end up just using

less space within the data hall, right?

So, you know, like everything just gets

smaller and you can see in this data

hall it's got a lot of space that we

don't need.

You know, so you're trying to respect

that that power capacity you have from

the utility whether it's utility or

behind the meter, you still have to

respect that overall value, but you want

to ride as close to it as possible.

That's why you can afford to put a

podcast studio in this place.

Although there are reasons to want

higher density like the networking

setups themselves are incredibly

complicated and like I don't know one

thing I always feel when I go into one

of our data centers is like what a

beautiful job people have done getting

the wiring right. That's actually quite

hard and the more you have stuff splayed

out the harder it is to get all the

wiring done and it's also worth noting

like most of the wires you see here out

of the cages are fiber, but the stuff

inside of the the fastest stuff is all

is all copper. Light moves more slowly

in fiber than electrons move in copper.

So you really are at many different

levels of optimizing for the latency of

all of these networking rigs. There's

about 8,000 km of fiber in this

deployment.

>> [music]

>> And there you go. This is what happens

when you go really dense.

But you you you do have enough power to

fill everything there out?

>> So it depends how we move power around,

you know, one of one of the ideas of of

being flexible and fungible with our

power and cooling is that we overbuild

our distribution. So while we're limited

on the upper end,

you're able to move power around by

loading up different rows, right? So we

have these UPS's that supply power to

our site, but when we distribute that

power out if we can move it around Yeah.

it allows us to say hey we're going to

you know, we're going to grow some CPU

here or we're going to grow some GPU

here. So you know, we have some headroom

in there to do other things and this is

just opportunity areas for us to do

that. We want to be ready to go if the

business needs some more computer where

we have a place to put it. So you you do

some amount of pre-building for future

you know, future opportunities that come

up.

So what are these things? So these are

breaker panels. These these distribute

out to those bus bars that you've seen,

so

this is where, you know, power comes in

and you're going to have to break that

power out to to go in different paths.

So, there's a lot of distribution, you

can see all this overhead conduit that

we had to put in. This is all carrying

power cables out to the data hall. So,

you're you're very conscious when you're

distributing power. It's it's less

fungible than than than cooling. With

the cooling, you can kind of oversize

the pipes and move it around. With with

power, you have breakers and and current

limits. So, you're very you have to be

very careful with with what you load up

and where and making sure that you don't

end up in a situation where you're

tripping a breaker. We have protections

in place, so if we we trip one of those

four buses, we're still good, so there's

some redundancy,

but still it's still an interruption,

something we want to avoid.

>> What would cause you to trip a breaker?

Just high current. So, adding too much

load on a single busway or single

connection, or pushing too far in the

over subscription and saying, "Oh, well,

you know, we think we're going to be

over subscribed by 10% and actually

things kind of shoot up to 50% or 20%."

There There are times when you do have

time to respond,

but if you're you're too far over the

limit, the breaker's going to trip and

you're going to And is the

is the current pattern determined by

software or by

by hardware alone? It's It's by

hardware. It's controlled somewhat by by

software. I think

So, what Nvidia is doing, they have this

kind of LPS system, it's a a load

management system that they're rolling

out in some of their new cabinets.

Really, what they want to do is keep

that load profile flat. They're building

in more bulk capacitance in those power

cells, those capacitors in there, and

they're also trying to get the software

to allow it the

the peak load and the average load to be

much tighter, so you have this flat

profile. A place where software comes in

is monitoring. We actually put a huge

amount of effort into building our own

monitoring tools so that we can in one

pane of glass, as they like to say, like

we can monitor every aspect of the

system and look for problems and

sometimes even drive reactions to the

system where like sometimes we might

want to shut off a workload if it is

drawing too much power. And having like

a unified system that can see all the

things has been a huge kind of step up

in being able to run these things in a

highly reliable way.

>> So, that that software system Ron

mentioned is actually pulling

information from these breakers and

performing some logic behind the scenes.

It's It's topology aware. And what it

will do, like Ron said, it can shut down

nodes, so we don't trip that. Because

you want to run as close to the edge as

you can, cuz like the hardware is

incredibly valuable. So, you do want to

oversubscribe. You do want to kind of

run near that edge, but you also need to

be safe, and so you build in these

safeguards to let you pull back in a

controlled way. If somebody

switch one of these switches, would

there some training workload stop right

now? Yeah, I mean, this site's been been

>> [laughter]

>> The site is currently live, so yes.

>> [music]

>> So, here you can see a little bit of the

scale of of of our liquid cooling. So,

you know, these are we we call it buffer

tanks. So, they're here to to help with

a situation where maybe there's a

interruption in power and the chillers

on the roof free start and lose cooling.

This is almost like a thermal battery

storing some energy for us to keep those

GPUs cool while the chillers come back

online. Also, as these workloads come up

and down, you're going to have a kind of

a movement of temperature. So, this

helps dampen that that effect out. These

are the traditional air cooling units

that you'll see in almost every air

cooling data center. Really just pulling

that hot air back and supplying it back

into the data hall. Those wheels up

there are are valves. So, you mentioned

leaks, right? What happens if there's

leak? Well, you have places where you

can isolate the system to kind of work

on it, to fix a leak or something. These

orange things just have more chain in

them. So, they're not hitting you in the

head. Now, you can reach them with a

ladder, take more chain out, and turn

the valve closed or open. That's great.

Such a simple solution. You know, it's

it's interesting as these data centers

and and the compute gets so dense, where

the place where you have the compute is

getting smaller and smaller, and the

places where you're supporting that

computer is getting larger and larger.

The infrastructure, the transformers,

the the chillers are getting bigger and

bigger. So, the sites are are very much

this much compute and this much

infrastructure to support that compute.

What did the analogous system to this 20

years back look like? So, yeah, it was

like dramatically more primitive. Like

early on we literally just had like

computers like sitting out with us in

the room with all the people. Like the

one of our computer clusters we called

the hive and I remember the first

version of the hive was literally like

six Dell boxes stacked on top of each

other like at the end of, you know, the

row. That's why they're called hive

bucks. Yes, that's right. Yeah, like

hive was the name like a very like

actually the first the first work that I

did at Jane Street was doing like

quantitative research on trading

strategy strategies and I was like, oh,

yeah, I guess we need like a cluster and

like that pile of six Dell boxes was our

first cluster. And the trading systems

themselves we also had there and that

actually was like

more important to have out with us and

it took us time to convince ourselves

okay to like actually go and put it in

another room in a rack because we

actually wanted the ability to make sure

we could turn the damn thing off. Like

if something went wrong just the comfort

of like I could unplug it was there.

And, you know, it took time to convince

people that we had enough control over

the systems and we understood enough

about where things were and just would

be able to find them if we had to go in

there that things were cleanly labeled

enough

>> Yeah. that we were comfortable taking

these things and moving them in the

back. Yeah. I mean, there were ups and

downs. Like literally at some point, you

know, one of the people who was like

cleaning the office like unplugged one

of the trading systems in the middle of

the day like as they were vacuuming. So,

you know, in the end it is in fact

better to have it all in a data center,

but I don't know, early on it was more

of a shoestring operation and we were

just kind of figuring things out.

>> did did not need to be even back then

co-located with the exchanges?

So, an important thing to understand

about our early uh trading is like we

were super not fast, right? Like there

there

trading like latencies matter at lots of

different orders of magnitude and like

sometimes it matters whether you're

responding in you know, seconds or

milliseconds, sometimes microseconds.

Like these days the very fastest systems

we care about you you're talking about

whether you can turn around a packet in

under 100 nanoseconds.

>> Yeah. Okay, definitely want to ask you

more about that when we get to the

podcast studio. Yeah, interesting.