Hey, did you know that the very first

Git commit was initial version of Git,

the information manager from hell? Nice.

So, recently some news kind of got me

thinking, why is it so hard to host Git?

It kind of seems like any old Git could

host Git. But based on the recent GitHub

numbers,

this actually looks really, really

difficult. But why is it so hard?

Because like every single engineer that

has come before me, I always go, "Oh,

that problem? That doesn't sound that

hard. I'm pretty sure I could do it."

Well, actually to finally answer that

question, Vincent right here did a

write-up and he's a 10-year veteran at

GitHub now working at Cursor and he

breaks down why is it so hard to host

Git. And everybody seems to love this

blog post. So, let's all learn together

why it's so hard to host Git. And by the

way, a little sneak preview for you, the

reason why is uh because the way it is.

It's really hard. Quick disclosure, I

have done many a live event for Cursor,

one in which I dressed up as a king and

they do give me some free tokens, but

this was not a paid ask of any kind. I

just think this blog post is super

interesting and so does everybody else.

But actually before we begin, quick

thank you from the sponsor. If you're

anything like me, you have a bunch of

ideas, but perhaps you're a bit

disorganized. So, often I find myself

having a long brainstorming session, but

at the end of it, I just have a big

markdown file and then of course at that

point I've lost everything. Hey, stop

doing that. You got to check out Linear

and use the Linear MCP. I have a big

conversation, I come up with a bunch of

ideas and at the end, I get a set of

tickets that are actually all the things

that I need to do. And when I don't know

what to work on, I go, "Hey, agent, go

to Linear and tell me what I need to do

next." And it will find exactly what I

need to work on. This has been so

successful for me that I'm in actual

serious danger of completing a project.

Go to linear.app/terminal

and get 6 months for free on Linear.

Thank you for supporting the channel and

thank you Linear for sponsoring the

video. All right, so before we begin an

extremely brief explanation of how Git

works, every single commit, which is

represented by some sort of SHA, right,

is actually just a pointer to a tree.

The tree is just a pointer to other

trees at specific versions and other

blobs. A tree is just literally a

directory. Your root tree is just the

root of your project, and a blob is just

a file. And you can see this by me

taking my latest commit and saying

ls-tree. You'll see all the different

blobs and directories within this

commit. And I can just grab out one of

these SHAs and go get show and paste it

in. It'll actually just show me the

file. Each one of these things, a

commit, a tree, a blob, they're called

objects inside of Git. And if you

package them all together, it's called a

packfile. It's like the minimum

representation of all these different

objects together. There's no guarantee

in order, but it's really good to

understand that to find out what blob

version is inside of a commit, you have

to walk everything to figure it out. So,

to find out what version of a file at a

commit, you do actually have to walk

from commit root to subdirectory to the

specific blob to find out what the

version is and the contents of that

blob. And for a Git server to actually

work, the server itself has to

communicate to clients using packfiles,

and clients communicate to a server

using packfiles. So, when you do get

fetch or get push or you do get pull,

it's using packfiles to communicate

between the server and the client. Now,

these packfiles are really what makes it

super difficult to host Git. Now, there

has been some attempt to not use

packfiles. One of them is using a KV

store. I think this immediately makes

sense because every single one of the

subtrees, every single one of the

commits, every single one of the blobs,

the tags, everything, they all have a

unique SHA. So, you could imagine you

could just have a key-value store. Key

is the SHA, value is the item. Well,

this animation right here shows you the

exact problem. If you were to go and

look up things about a commit, you'd

have to go back and forth between the

key value store many many times. And as

the repository grows, you could go

thousands of times just to get the

contents of a commit. In fact, Google

even tried this approach and tried to

make it fast, but it turns out it just

couldn't work out in the end and they

ultimately abandoned it because Git

clone was entirely too slow. Now, of

course, over the years companies have

tried various versions of being able to

host Git at scale avoiding this whole

pack file dilemma, but it just never

actually worked out. What's actually

trying to be accomplished is pretty

simple. Ideally, you want the repository

to exist on many disks on many machines.

That way, just one outage doesn't

accidentally destroy your repository. I

think this makes perfect sense. Also, if

you have a whole bunch of reads, you

would like that to be really really fast

because if you have a whole bunch of

reads going to one machine, you could

imagine somehow the network or the disk

get saturated and then you just have

really long wait times, i.e. CI sucks.

GitHub also tried to kind of do a

distributed file system, but ultimately

abandoned it because it again was just

impossible with pack files. But finally,

somebody cracked the code and that

somebody was in fact GitHub. They

created something called spokes. Spokes

was originally developed at GitHub

around 2013 and it has since become the

industry standard. Most Git hosting

services use a variant of spokes

approach application level replication

for Git repositories in their

architecture. And spoke doesn't

distribute Git itself. It works at the

pack file level. It stores all the data

as actual Git repositories on local NVMe

disks. It replicates Git data, but keeps

all copies consistently in sync. Good.

So, now we have distributed Git. So, how

does spokes actually work? So,

fundamentally how spokes works is it's

pretty easy. There will be a minimum of

three machines that contain your

repository. When you want to read, any

network request can go to any one of

these machines, get off the actual

contents, and bring that down. That is

because they're all consistently kept in

sync, which means for every repository

that you actually have, it needs to have

a routing table stored in a database

that says, "Hey, here's where your three

machines minimum are located." As your

repository grows and as the reads grow,

say from CI, you may need more replicas.

But, writes are a little bit different

because the first thing that happens is

we need to send up the actual pack files

to each one of the replicas. After that,

we need to say, "Hey, can you actually

store this? Is this pack file in sync?

Will everything actually work out?" And

that's the prepare stage. The next thing

is called a lock stage. If all the

things come back or the majority of the

repos come back saying, "Yes, we can do

this," then we will then lock all the

ones that said, "Yes," to come back. And

then after the lock request, you do a

commit. The commit will then finalize

the actual writing of your pack file,

and boom, your repository is now at a

new version. This is called a

three-phase commit. I didn't know about

in distributed systems. It's called tail

at scale problem, the dreaded tail at

scale problem. It's actually pretty easy

to understand if you just pretend we

have these five repositories right here.

Well, first off, we have to send to each

one of these the pack file. After that,

we have to say, "Hey, can you actually

like do this pack file?" After that, we

have to then set the lock. And then

after that, we have to commit. And we

also have to wait for responses each

time. Now, the problem is is that if you

make five requests, you're probably

going to have reasonably good

performance on all five requests. But,

every single time you have to make more

request, there's a higher and higher

chance you're going to have bad

performance. Now, the performance is

actually really easy to calculate. It's

1 minus the percentile to the amount of

requests you're going to make, and this

is the percentage chance that that

percentile is going to happen. So, now

we had to make four requests per node

for a three-phase commit to happen,

which means if you had five nodes, that

is 20 commits. That means you have an

18.2%

chance to have a P99 slowness on one of

those routes, which could be hundreds of

milliseconds, potentially seconds long.

And this obviously only becomes worse as

the repository becomes more popular. So,

let's just take the recent Deep Seek uh

repository that became the fastest

starred repo of all time. It probably

has

like many many replicants. So, that

means it could potentially be doing 200

requests for a three-phase commit, which

means that 86% of the time you're going

to be hitting these really bad long

request paths along your three-phase

commit. So, it's pretty much guaranteed

to be slow always. Reads super fast,

writes super slow. So, that's the

problem with Spoke. This is the problem

that makes things difficult to host Git,

and this is the industry standard. It

makes reads really easy, which is great

for CI, it's great for a lot of cases,

but it makes write increasingly hard as

repositories are more popular. All

right, so how do we solve the scale

problem? We've identified that hosting

Git is hard, right? But how how are we

going to solve it? Well, it seems like

this is a reoccurring theme in the

things I've seen. The answer, S3. Well,

it's actually more complicated than just

S3. It uses something called a wall. And

when I say a wall, of course, I don't

mean the crazy things that your uncle

says on Facebook. Not that wall. What

I'm actually saying is a write-ahead

log. A write-ahead log is simply just

all the write operations that have

happened, so you can reconstruct Git

from that series of writes. Postgres

uses this, SQLite uses this. So, by

using a write-ahead log, you can imagine

that all All these writes, all these

pack files are being stored in this log,

they're being stored in linear order,

and it's actually super easy to append

to because it's just like a bump

allocator, right? And for a server to

come up and be able to go, "Okay, well,

what's my Git state?" It just simply

needs to read the right log. And boom,

that's its Git state. So, does that mean

there's no consensus? Yeah, there's no

more consensus. We use S3. Well, which

one's the primary node? It doesn't

matter. Any node can serve it out. Now,

obviously, you don't just want to serve

the same repository from random

different nodes because that'd be super

inefficient cuz each one would have to

restore itself from the wall. So, they

use something called rendezvous hashing.

Now, rendezvous hashing, by the way,

very French. Comment ?

Je m'appelle Michel. Is a extremely

simple algorithm to allow two parties

that don't know about each other to

where they need to meet or where they

need to rendezvous. Effectively, you

give it like a unique string, and it

will give you out servers plus a score,

and whatever is the highest score is the

server you take. Pretty straightforward.

And this is Continuity. It effectively

uses a wall on S3 to be able to store

the state of Git and then to be able to

restore it at any point on any machine.

There's no more needing for kind of

reconciliation between two different

nodes to make sure everything keeps in

sync. No longer need to vote, "Hey, can

this right actually happen?" You don't

have to have consensus. You don't have

to have a whole bunch of replicas laying

around. That means the 25 million

repositories that are created every

single month don't actually have to have

three machines each. They could only

have one. They could have zero. Nothing

could exist because they don't need to

exist because the actual state of the

Git repository is up on S3 and can be

restored in a matter of milliseconds.

You really should go read the blog for

all the technical details. It is super

super duper awesome. It even goes into

like how they keep the wall log actually

pretty tight because they have to do

compaction. Also, I didn't even know

there was something called compaction in

Git. It effectively takes a bunch of

pack files, make a big pack file. And

with super duper fast NVMe storage and

S3, they can do 16,000 commits per

minute. That That's That's That's a lot

of commits. Learning about the

write-ahead log and pack files and stuff

like that. This was a really amazing

article. I had such a good time. I just

I I I feel smarter and I also feel much

more confident. Dude, I can totally do

>> [laughter]

>> this.

I can do this. I You know

what? You know what? I feel like I can

make that. Truly the problem of

confidence.

Feel like you can do anything in Gen.