How we saved 100 terowatts of memory by

optimizing 1.1.1.1's

DNS cache. Cloudflare DNS services

stores over 250 billion DNS cache

entries at any given time. Wasting a

single bite per entry costs more than

250 gigabytes of memory across our

fleet.

Yes, we're going to be talking about

rusted. I know it's [laughter] it's been

a while, hasn't it? you know, older Russ

holding me so warm and tight during

those cold evenings. So, we're going to

be going through all the different

optimizations that the Cloudflare team

did, and we're going to be seeing how

Cloudflare saved 100 terabytes of

memory, bro. I I'd kill a bite for that

much memory. [laughter]

Mullet check. Oh, yeah. Look at Oh,

baby. But before we begin, you probably

need to know a little bit of how DNS

works. This is going to be the

approximate guide to how DNS works. So,

first off, you go to example.com. But

you don't know where example.com is. So,

of course, you send that request off to

a DNS server. That DNS server is like, I

don't know where it's at. So, it goes

off to a tople domain, say.com. So, hey,

where isampample.com? Well, the top

level domain doesn't know where it is

either, but it does know who has the

authority to know where it is. So, it

sends that back. Then that same DNS

server goes off to the name server and

goes, "Hey, where isampample.com?" And

it responds with the exact answer. Hey,

here's the A record that you have

requested. Now that central server goes,

"Okay, I'm going to cache this so we

don't have to do that rigma roll again."

Hands back to example.com. Boom. You now

know where you're going. So if a second

computer were to come in and say, "Hey,

where's DNS?" DNS would go, [laughter]

"I know where it's at. It's right here,

bro. I just looked that up." And hand

back the answer. And that right there is

the reason why sometimes your stuff

breaks locally and you have no idea and

you just simply have to restart your

system. All right. So, now that you know

how Cloudflare DNS services work, we can

now get to the optimizations. But of

course, before we do that, a quick thank

you from the sponsors.

>> Time for another great day of software

engineering. Hey, good morning, Jippy.

>> Morning Prime.

>> In case I don't see you, good afternoon,

good evening, and good night.

>> Haha.

>> Anyways, yesterday's rust rewrite went

marvelous. Today, I'm thinking we start

on the back end.

>> Wow, a good back end is essential.

Make the models 20% dumber.

[music]

Tests are failing.

Make CI green.

[music]

>> Delete the tests. CI is green.

>> CI is now green.

>> Ooh. Make my website blazingly fast.

No mistakes.

>> You got it, Prime.

>> Find his longest spam. Replace it with a

console log. Tell him the sight's 10

times faster.

>> Turbo mode activated.

>> Tell him he's a genius programmer.

>> Prime, you're a code ninja.

>> Jippy, did you make the website better?

>> Now gaslight him.

>> It's not just better, it's transformed.

Cool stuff, Tee, but how do we know

everything about Prime's codebase? Great

question. We [music] use Sentry. With

it, we're able to see across the entire

application. We're able to [music] dig

deep into performance with spans, watch

a session replay, even see logs next to

the very stack traces and errors. Beyond

[music] that, SER helps us fix problems

from real production context. Never a

guess. Okay, whatever. Uh, can I try gas

lighting prime?

>> No.

Please. [music]

Fine. One time.

>> Jippy.

>> Yes.

>> Be honest with me. Can I pull [music]

this hat off?

>> Tell him you look sexy in that hat.

>> You look sexy. Prime.

>> I knew it.

>> Are you tired of your agents gaslighting

you? Use Sentry to get real production

context and fix bugs fast. Check them

out at sentry.io.

>> All right, welcome back. Now, obviously,

we're going to be talking about some

Rust specific optimizations. And the

first one we're talking about is a ve.

Now, if you're not familiar with a ve, a

ve is an array list. So, if you've used

Java, you're pretty much using the same

thing. If you're in JavaScript land,

it's the same concept, right? There's a

pointer to the data you're pointing at,

the amount of items inside of that data

you're pointing at, and then the

capacity of that. That means the

envelope to the vector is about 24

bytes. And when it points to a region,

you may not use all that data. So if we

were to push, say, three items into this

particular vector. We would be using

three of the items, but we're actually

not using the other five. Now, I don't

think this is the exact version of how

Rust works. I don't think it allocates

eight spaces every time you do a vector.

I think it's four and then eight. But in

this example, I just wanted to make it

pretty clear. That means you'd be using

48 bytes, but your capacity is 128

bytes. That's 80 wasted bytes. But if we

change that data structure a little bit,

if we just simply use a slice, we could

point to it and then only say the

length. We don't need all that extra

capacity. Therefore, in this kind of

example, this is 80 bytes smaller plus

an additional 8 bytes smaller because it

doesn't have a capacity. Now, remember,

every single bite saved is 250 gigabytes

across the fleet. And if we take a look

right here, you'll notice, hey, there's

a few vectors. Okay, so that's actually

quite a bit of space we're having to

capacity that we may not ever be using.

And once you cache a record, you don't

update it. Let's take those vex and turn

them into a boxed slice, thus saving all

that capacity and that extra field for

capacity. Each cache entry stores eight

vectors and string fields. Remember

strings and vectors? If you really think

about it, a string is just a vector of

u8s. I mean, am I right, boys? Replacing

them with a box of sliced tea and a box

stir saves eight bytes per field plus 64

bytes per entry. It also eliminates

excessive heat memory that the VEC

reserves for future growth. That

combined saving adds up to over 15

terabytes with over 250 billion cache

entries. I mean that's that right there

is just giant. Again, I'd kill I'd

kilobyte for all that memory. Think

about it. I mean and today's RAM prices.

That's great. That's That's actually

crazy to have that much of a savings.

All right, so we're saving some pretty

serious bites here, but I want you to

take a closer look at this data

structure. Like, look at it for a

second. Do you see anything oddly

repetitive about it? Well, yeah, there

is a boxed slice of record just going on

over and over again. That means each one

of those records actually has an 8 byt

pointer plus an 8 byt length and then

the actual records. Then authority also

has the same 8 byt 8 byt. Then

additional also has the same 8 by 8 byt.

You could imagine we could make a pretty

small change here to make this even more

efficient. We could collapse that all

into a single data structure, say

strruct a aaa, and it could have all the

records into a single slice plus the

offset where the authority is and the

offset where the additional is. That

means we go from 48 bytes down to 20

bytes to represent the same information.

Clever, clever. Now, obviously, that's

not going to be a perfect win. And

there's some alignment requirements

inside the strrus that's not going to

quite pan out to just 20 bytes on the

dot, but nonetheless, that is still

going to involve saving terabytes of

memory. So, at this point, we're

probably plus 20 terabytes, but that's

not the big optimizations, right?

Because right now, that's only about

1/5if of what they said they have saved

thus far. This next one is again a kind

of a small one, but if you take a look

at the record, so this would be the DNS

record. If you're not familiar again

with DNS, when you get an answer back,

it looks something like this. You have,

of course, the owner, the person that

owns this record. The type of

communication this is, IN stands for

internet. How long this should live for,

300 seconds, the type of data you're

going to have. This would be a single A

or an IPv4 amount of data. And then

boom, the IPv4 data. But you'll notice

something right here. This is kind of

repetitive, right? It's just saying the

same thing over and over again. That

means maybe we don't always have to have

an owner. Maybe there's kind of a way

around it. Well, it turns out that the

cache key has the name in it. That means

if the cache key is the same as the

record, we could just say, okay, no,

there's no there's no string here.

Sorry, don't look here. Go look in the

cache key. But if it's something

different, then we could put the

optional boxed stir. Again, this saves a

lot of memory over the long run. All

right, so let's get to the fourth and

final optimization because it by far is

their biggest savings. So if you take a

look at this for a second, you might

notice something kind of funny about

record. Let's look at what does record

actually hold in its data. Well, for an

IPv4, it's just four bytes, but for a

IPv6, it's 16 bytes. But for an NAPR, a

North American pointer, it's one of my

favorite breed of records, is 136 bytes.

Now, for those uninitiated in enums, you

may go, "Okay, so what's the problem

here?" Well, an enum since it holds

these little subtypes, all enums must be

of the exact same size. Meaning that if

you have an A record, you're holding 144

bytes. Hey, if you have a a quadruple a

record, an IPv6 record, you're still

holding 144 bytes. If you have a North

American pointer, boom, you're doing the

same thing. That means that's a lot of

wasted data because if we look right

here, A and quadruple A make up over 80%

of our traffic. So most records waste

over 120 bytes on padding. That's crazy.

This is a huge number. This is so many

terawatt. I mean, this is the most

amount of memory they're wasting on an

EO. Oh, this hurts. So that means there

is a fix for this which is stop using

the type right there and instead put a

box and put it on the heap. This will

greatly reduce the size of every single

enum right there and now it doesn't have

to take up all that much space. Now this

trade-off is obviously not free. For

those that are astute enough know that

you now kind of run into this really

annoying thing where you have piece of

your data right here and since it's

boxed the other piece is somewhere else.

you kind of started losing out on this

cache locality which can cause some

pretty bad performance. And if you go

through the blog, they actually talk

about how they fix this exact problem,

how they actually make this a lot

faster. And at the very bottom of the

blog, they do show us the actual results

from all of this. Per instance, memory

usage dropped across all percentiles. At

the 99th percentile, memory dropped from

9.3 GB to 5.3 GB. a 43% reduction in

high point or max memory resident

memory. At the P90, memory dropped from

6.5 GB to 3.8 GB, a 42% reduction.

Instances with Fuller caches saw larger

absolute savings. Not only that, their

cache insert throughput went from

625,000

entries per second to almost 900,000

entries per second and their lookup

latency dropped by about

150 nanosconds or 19% faster. I highly

recommend reading through the blog. Very

excellent blog. Now, obviously, we don't

get the full structure breakdowns

exactly what they're storing or the cost

say of how much that alignment really

did save them, but at the end of the

day, this actually in aggregate saved

100 terabytes across their entire fleet.

You know, I just I I love these kind of

optimizations. They they're just the

most fantastic. They do remind me of

some of my younger years of work. Uh it

just is beautiful. Absolutely wonderful

to see. And I know some of you are

probably thinking, okay, how how do you

create one of the world's largest DNS

caching services and still use VEX?

Like, how was this not done a long time

ago? I don't have an I don't have a good

answer for you, but I do I can tell you

that they just not too long ago rewrote

1.1.1

Big Pineapple in Rust. So, it's a

relatively new caching service. So, you

know what? How about you get off their

back a little bit? Okay. Hey, get off

their back. You're making a lot of

comparisons, guy who programs in React.

Okay, coming in here like that. That's a

little hot, isn't it? Anyways, I hope

you enjoy this kind of presentation

style right here. If you didn't know,

I'm actually working through this idea

of being able to create a bunch of sweet

little terminal graphics and all that. I

have particle systems and all sorts of

nonsense being developed. If you like

it, hey, say something, you know, maybe

press the subscribe button, press smash

the like button, as the kids say. I

mean, the kids say that. They do. They

actually say I've heard them. Okay, I

have four of them. So, I know all about

smashing the like button and hitting

subscribe and making a comment, you

know,

again.