GitHub just had its magnum opus of a

bug. Now, I've heard lore that people

have dropped databases, just some

showstopper bugs, but I did not think it

was possible for GitHub to cease being

Git. Now, we're going to go through what

actually happened, but more so I just

have a lot of questions as to how this

was even possible. All right, so before

we begin about GitHub ceasing to be Git,

let's first go over a bit of lore

because it's important. In 2005, the

creator of Linux decided that he was

going to take the Brendan Ike challenge,

which is to create a programming

language in 7 days, but instead create

version control in 10. And thus, Git was

born. Upon seeing this beautiful

distributed version control, software

developers did the only thing that made

sense. Immediately create a single point

of failure. GitHub became so popular

that other services decided to use the

hub naming convention. Of course, the

most popular being DockerHub where you

can store all your your Dockers at

Microsoft seeing that GitHub attracted

developers developers developers

>> decided to purchase GitHub for $7.5

billion and proceeded to generationally

fumble the bank. Microsoft, you could

have done like nothing. Like if you

would have just left GitHub, it would

have operated better than whatever has

happened today. But the impressive part

is that Microsoft has discovered the

fourth E in the triple E strategy of

course which is embrace, extend,

extinguish and shitify. Yesterday we had

a regression in the merge Q behavior

where in some cases squash or rebase

commits were generated from the wrong

base state making earlier changes appear

reverted in branch history. you had one

job, just one singular job, GitHub,

which is of course to be Git. Now,

that's only about half the story because

it really isn't kind of actually

illustrating what happened. Now, here's

a better story. On April 23rd, GitHub's

merge Q started silently reverting code

on customers main branches. Now,

reverting is actually a Git operation.

It did not revert code. That's very

important to understand. Not a handful

of lines, in some cases thousands.

Nothing looked wrong. A PR with a plus

29 minus 34 diff got reviewed, approved,

and cued. What actually landed on main

was a single commit with plus 245 minus

1,137.

Thousands of lines of unrelated already

shipped code were quietly removed. Every

merge that followed went in on top of

that broken history. That means that you

would look at the PR. your PR would

actually look correct. And when you hit

the merge button, what actually went in

was not what you hit merge on. This is

absolutely diabolical behavior. I mean,

to be lied to, like the UI just telling

you no, everything's good, man. This is

exactly what's going to happen. And for

you to have no idea is crazy. I still

have a lot of questions, but this is

roughly how it's being reported. Now, we

do not have the official GitHub kind of

postmortem that explains exactly what

went wrong. This is just what is being

purported by large companies.

Effectively, what happens is you have

history that looks like this in git. And

this would be your main branch. So,

let's just say you have four commits on

your main branch. And for whatever

reason, you branched off back here to

make your new branch and then you've

added two commits. So the actual root

cause of what would happen is that when

you'd go to merge, what would end up

happening is they would run something

called a merge Q, which would take your

two merges, which if you just drew this

in a straight line would be three

circles because you'd have where you

branched from and your two changes, and

it would squash these two into a

singular change. So they are just one

single branch. This is actually a whole

new branch that's uh created internally

in one of Microsoft's CI. Let's just

call this singular squashed merge A.

What should happen is that A should be

merged to the tip of Maine.

But what actually ended up happening is

that they would take this history right

here and overwrite whatever was on main

with this right here. So that means if

this thing was called B, that means what

would be on main now would simply be B

to A and that is it. That means these

three commits were just gone. Now this

obviously caused teams to scramble all

afternoon. There are hundreds of tweets

of people saying, "Hey, I had to spend

all afternoon or multiple days trying to

figure out how to detangle whatever has

happened and what is actually missing

from all of their PRs." GitHub, of

course, has said there's only 2,84

poll requests that were messed up out of

4 million merged. Then later on said

there was only 800K PRs. Either way,

that's the difference between 007% and 3

plus%. It's a big difference. So, which

one is it, GitHub? And obviously even

more confusing thing if you go to April

23rd. That's weird. I don't Do you see

Do you see like some major outage or a

partial outage going up there? No, they

don't show anything went wrong on the

23rd. Obviously, the reason for that is

that the only downtime was you and your

team trying to figure out how to undo

what just happened. But I also just have

a lot of questions and maybe you guys

can help me on this one. So the first

thing I have is that if this is B and

this is A and this is the temporary

branch that we have created. If history

looked like this, wouldn't pulling down

cause you to have some sort of hey the

remote branch is like out of sync with

your local branch, meaning your two

histories have diverged? You wouldn't

even be able to pull down main anymore.

Or the other option is that instead of

just simply putting a right here, it

actually had uh revert commits, meaning

that it would have these three commits

right here, and then it would have

another commit that was like, hey, I'm

reverting these three commits right

here. And then you would have something

like a, which would make this not too

difficult to be able to go, okay, well,

here's the only revert we have.

Unreverted, which is not like how

anything sounds like. But even more

importantly, git is still git. You

cannot change history like this easily.

Does that mean internally is Microsoft

like pushing with force? Right? Because

the only way that you could push this

right here, this history onto main would

be that you have to force push because

you're changing history. And if that's

the case, if that's the case that

they're force pushing, makes me kind of

nervous. Like what other issues are

there? What other things have been

silently happening? And of course, this

there's no shot it was 2,84 total.

Please double check this stat. We

probably had 200 plus as a single

customer. So, I'm actually just pretty

curious. Did you lose any commits? Are

you using the kind of the auto squash

feature and thus you have things gone?

Also, this has opened up the most

legendary excuse of all time for the

next year. When any company gets hacked

and stuff gets leaked or something goes

horribly wrong, they can go, "Oh, yeah.

Well, actually, uh, we actually did

implement perfect security. We actually

had this exact situation already

covered, but GitHub decided to silently

drop our commit, brother. It's not our

fault, it's GitHub's. It's also funny

because there's probably a bunch of vibe

coders that they just simply just kept

on going on with their day. Some weird

thing happened and they're getting

they're just like, "Oh, fix it. Oh, AI,

just fix it." And then when a bug

appeared, they're like, "Stupid AI

always reverting previous changes. Hey,

fix that one. Now again, you know that

happened. Absolutely beautiful. And the

worst part is for once it actually

wasn't AI. They're just they're just

going to be blamed for part of this for

a whole group of people. Actually, I

mean, to be fair, it could have been AI.

It could Hey, it could have been.

There's some there's some reports going

around that it was a faulty feature

flag. Of course, Hacker News had its

heyday and people were viciously

discussing what happened to GitHub. Four

people spend hours putting our repo back

together at my company after this.

GitHub has been unreliable and now they

are breaking core tenant of what I

expect from this service. Cortenant, you

mean tenant, not tenant. You must be fun

at parties. I don't care to associate

with people who are offended by being

corrected. They can saw off and I never

go to what you Americans call parties.

Whenever you use quotes and they are

directly followed by punctuation,

you must include

the punctuation within the quotes.

Parties period reply. Don't you love

hacker news? It's just such a lovely

place where you see really thoughtful

and amazing discussions about topics.

But at the end of the day, we still do

not have the exact uh postmortem. But we

don't know the exact uh cause or

reasoning or will GitHub even provide us

with the exact details. I don't know. So

if anybody can answer these questions,

does Microsoft actually use a force

underneath the hood to be able to align

histories? Were there revert commits,

which I don't think there was because

nobody is saying that on the internet,

wouldn't this cause all sorts of history

being absolutely destroyed? Wouldn't

this be kind of easy to be able to kind

of resolve? Maybe, yeah, it would take a

little bit of time. And maybe you have

to ask Claude Opus for, you know, Mythos

to be able to resolve it. But you should

be able to get this done pretty quick.

And really, was it only 2,800 PRs? Is

that is that it? I mean, Mr. 200 over

there doesn't seem to make it that that

to be the case. the name is that I think

the most impressive thing out of all of

this is that GitHub created such a

monumentously good name that even after

GitHub ceased to be Git for a little

bit, everybody's probably still going to

keep on using it. I mean, now that my

friends is one of the most impressive

products of all time at this point, it

seems like GitHub is more addictive than

cigarettes.

A gen. Hey, is that HTTP? Get that out

of here. That's not how we order coffee.

We order coffee via ssh terminal.shop.

Yeah, you want a real experience. You

want real coffee. You want awesome

subscriptions so you never have to

remember again? Oh, you want exclusive

blends with exclusive coffee and

exclusive content? Then check out cron.

You don't know what SSH is?

>> Well, maybe the coffee is not for you.

in hand.

Living the dream.