- The important is, is your code good? We care about

excellent code. We don't care who you are. Like maybe you're a dog. I don't

care, right? I don't care where you come from. I need to look at your code.

Oh, yeah, but I'm an engineer at this very large company in

Italy, in Germany, in the US. We don't care. We care about the

quality of your code because this is what defines our community

and which means that we have a lot of people who contribute who are some very different

backgrounds and very introverted. Sure. But that's okay, right?

- FFmpeg is probably one of the biggest CPU users in the world. Everything we've just

said in the past couple of minutes, every sentence is someone's

lifetime's work. There are books about every sentence. So the level of complexity

in many cases is inordinate.

- FFmpeg has one hundred thousand lines of assembly for all the codecs.

- For all codecs. Mm-hmm.

- And just this one has two hundred and forty thousand. Every cycle

matters. We are talking about probably three billion

devices which are going to decode video nonstop because, for

example, thirty percent of the video from Netflix is now in AV1,

fifty percent of YouTube.

- This is what peak video codecs should look like.

Seventy-nine point nine percent assembly, nineteen point six percent C,

and zero point five percent other.

- And what's incredible is with those tweets, which is factual, people get crazy.

- For the last two years, they go crazy. No, intrinsics is fine. The compiler-

- You can optimize your compiler. Auto-vectorization, it's your fault. You don't

understand. And we've tried that forever, right?

- For two years, and two years later, showing hundreds of examples

of handwritten assembly. No, no, no, you're doing it wrong. The compiler can do this.

The intelligence agencies tried to, like, say, "Can you put a backdoor in VLC?"

- Yes. Two of them.

- Well, what did you say?

- No. Well, I was a lot less polite.

- Basically saying, "Hell no."

- Like, if we had to compromise our software, we would shut it down. This is clear.

- Any tweets Kieran, you regret?

- Tweets I regret?

- Or is it like that, how does the French song go? Regret nothing.

- Don't regret anything. No, it's because regrets are attacks on your mind.

- The following is a conversation all about FFmpeg and VLC

with Jean-Baptiste Kempf and Kieran Kunhya.

FFmpeg is an open source software system that is

the invisible backbone behind YouTube, Netflix, Chrome,

VLC, Discord, and basically every platform that

touches video or audio on the internet.

It can decode, encode, transcode, stream, and

play almost any video or audio format ever

created. To me, it is one of the most incredible software

systems ever developed, and it's all done by volunteers.

VLC is also a legendary piece of software. It is an

open source media player that plays basically anything you throw at

it, any format, any platform, no ads, no tracking.

It has been downloaded over six billion times, and

again, for me, it has been one of my favorite pieces of

software ever, with the most legendary logo, which I,

of course, had to honor in this conversation by wearing the

VLC traffic cone hat the whole time.

So again, above all else, thank you to the incredible volunteer

engineers who put their heart and soul into this code that has been

used and loved by billions of people. Thank you.

And about the two great engineers and human beings I'm talking to in this episode,

Jean-Baptiste is the president of VideoLAN and

is a key figure behind VLC and FFmpeg. Kieran is a longtime codec engineer, FFmpeg

contributor, and the man behind the now infamous

FFmpeg account on Twitter/X that I recommend

everybody follow for the memes and for the

unapologetic celebration of open source and great low-level software engineering.

Let me also say that it's inspiring and humbling that

so much of modern civilization rests on software built by

people who are not chasing fame or money, but are

obsessed with the craft of engineering.

We live in a world where billions of people consume video every day without

ever thinking about the invisible machinery underneath it. But that

machinery matters. Open source infrastructure matters.

It is one of the great examples of human beings quietly collaborating

across borders to build something useful, durable, and elegant for the rest of us.

And so this conversation is not just about codecs and

media pipelines. It is also about the deeper spirit of

engineering and generosity that makes projects like FFmpeg possible.

Again, I can never say it enough. Thank you.

This is the Lex Fridman Podcast. To support it, please check out our

sponsors in the description, where you can also find links

to contact me, ask questions, give feedback, and so on.

And now, dear friends, here's Jean-Baptiste Kempf and Kieran Kunhya.

So the legend goes VLC can open everything. What's the

weirdest thing that you know that it can open?

- You know, there is a ton of people who are using VLC to record VHS

videos, right? Like, it's just like you plug it with a capture card and you

can basically record VHS video.

- Well, how does that work?

- Basically, it's, you know, those type of capture card where you can put a Peritel

in or- ... or RCA, and you put that, and actually VLC can play those type of cards,

and there is a module which allows to control directly some of those

VCR camcorders. We support DVD audios lately, right? We spent

the summer working on DVD-Audio support, and

like there is no, no one's making any DVD audio support. There is a custom

encryption schemes.

- What about Lucasfilm?

- Oh, yeah, and there is of course all the weird codecs support, game codecs supported

by FFmpeg.

- The one Star Wars video game, the first ten- second opening sequence, someone has gone and

implemented that and made sure that's bit exact on one disc that existed at

one time of one little sequence in the game.

- And then funnily, there was a... At one VideoLAN conference, we made a

competition to make the weirdest and most horrible file ever

... and see if VLC could play it.

- What did it end up being? What's the file?

- It was an MKV file made by Derek-

... which each of the frame was changing resolution, aspect

ratio- ... rotation and it was like-

- Did it work?

- Yes. And there was another one where

the whole video was actually animated subtitles, right? SSA, right? So-

- Yeah. I remember that, yeah

- ... each, this one was-

And so each frame was a black frame, but on top of that there was a,

a subtitle that was animated for each frame.

- There was a file that's a valid ZIP and a valid MP3 at the same time or something like that,

so.

- So yeah, we'd made a competition of stupid files.

- And it worked. It opened all of the stupid files.

- Yes.

- By the way,

For people who are not familiar, I am wearing a hat. Would it be fair to say this

is the best worst logo of all time, the cone?

- Yeah, by far, right? The logo of VLC is so iconic, right?

Like we are a team with a small number of people and the icon

is known everywhere. I go to middle of nowhere in India or in

China, people know the cone, right? And 25% of

the website traffic that comes to our main website is

cone player, right? So, so many people don't know VLC, right? They know the cone player.

- That's the thing they Google for is cone player.

- Yeah. They go on Google and they put cone player and they download VLC, right?

So that's iconic. And once we tried to change it as a joke, right? We said

it was going to be a type of uh,

caterpillar construction and we said that during April 1st-

... and we had around 10,000 emails saying, "No, don't change the logo," and so on, right?

So it's so iconic, right? It's so distinctive, right? If you want to do a

video player, you're going to put a play button on a TV, right? And that's a YouTube,

YouTube logo, right? It's unoriginal. This one is orange, right?

- Yeah.

- It's very bright and it's weird.

- And it's ridiculous and it's absurd and it's hilarious. It becomes meme and meme becomes

culture. Yeah.

- And you keep it and you know about it and you know that in 20 years, like you

still have, going to have the cones and remember, oh yeah, that was a video player.

- Yeah. And we'll talk about, you know, the mission of FFmpeg being

a kinda the archival aspect of it. So you can think about 1,000 years from

now we'll have all these videos that only VLC can open. Humans,

human civilization has already destroyed itself multiple times and the

only thing that will remain is this like, you know, the cockroaches will be

crawling around and it'll be the VLC logo- ... with some of the archival

footage that VLC can open. And the aliens will show up and they'll press

play and they'll get to see it all 'cause-

- Well, really, really hope so, right? But there is also so many memes where people say, "Well, I'm sure I can put

a pancake inside my DVD drive and VLC will play it." Like-

- Can they?

- No, we tried. It doesn't. Um-

- Doesn't.

- ... but we actually have a video of us trying that. Didn't work.

- A codec for physical reality, I don't know what that would even look like.

- There was a guy who did that, right? He printed a small cone, right?

Like the ones we distribute as goodies and inside he put an

RFID chip which was his way of playing a movie, right? And so he-

... put this on a RFID player and when he put that it was playing like The

Last Star Wars and so on. So instead of having like DVD boxes, he had like

VLC cones all around and he plugged that and that was like physical objects.

- So the thing that we're talking about is everything around video

codecs, video encoding, video decoding, video streaming, video

player client that I'm wearing on my head, the entire

ecosystem enabling free media. We'll talk about FFmpeg, we'll talk

about VideoLAN, VLC and all the other incredible video

technology that is used probably by billions of people. So JB,

you're the lead developer behind the legendary VLC player.

Kieran, amongst many other things, you're lead developer behind the legendary

FFmpeg handle on Twitter. And both of you have spicy opinions I would say.

So today we wanna talk about FFmpeg and VLC.

For context for people who are not aware and I'm sure

basically everybody listening to this have used these two

technologies probably regularly without knowing it.

So FFmpeg underlies basically most video on the internet

including YouTube, Netflix, Chrome, Firefox, of course

VLC and countless other video platforms. It

is estimated that over 90% of video processing workflows online and

offline involve FFmpeg. VLC has been downloaded at least 6.5 billion times. But

likely that number, 'cause it's impossible to really count the number

is much higher than that. Virtually any operating system

supports virtually any media format.

The limitation being it can't open pancakes. So,

Can we just lay out some of the basics to help people

understand what's involved in all of this? So when we press

play on a video player like VLC, what happens? What--

How does it go from the file or the stream to the

pixels on the screen and the sound on the speaker? What are the big

stages to be aware of?

- So there are several stages, right? The first stage is to get from

an address, right, which is the type of URL, to

give you a byte of streams, right? So this would be, for example, HTTP,

file, DVD, right? You give the path to the media, and it gives

you a stream of data.

- The stream needs to be cut up by what's known as the container, the demultiplexer or

demux. We'll try and keep the jargon light throughout this, but it

needs to go and start demarcating video and audio frames. So it just gets data from the

operating system blocks at a time and needs to start cutting these frames up

into compressed data. It then needs to start doing simple

parsing of the video frames- ... mainly to figure out whether that codec is GPU

decodable or needs to fall back to software. We're very sort

of used to assuming the GPU will play all of these things. There'll be hardware

acceleration. I think it's up to forty- five percent of files are not GPU

decodable. So these need to be probed. They need to be detected. There can

be variants of a given codec, some of which are decodable on the

GPU. Different vendors of GPU might have different capabilities,

so those need to be detected. So if it's GPU capable, you pass it through

to the GPU black box. So now if there's a software fallback,

that means in the beginning is to first do deentropy

coding, so removing the mathematical coding of the bit stream. So this

uses capabilities such as Huffman coding or arithmetic

coding to actually decompress the mathematical layer of the bit stream.

We then need to start reading the syntax elements for intra prediction. So intra

prediction are like still images of the video, so your I-frames.

So this works and operates in the spatial domain. So you do your intra prediction in spatial

domain. You have a residual because your prediction isn't quite

matching that of reality. So you've made a prediction, but then there's a little bit

left, and that's what's known as the residual. This is stored in the frequency

domain, and these are quantized to decompact their space.

We then need to do the inverse transform to bring them back to the

spatial domain and apply these residuals.

- So a lot of the process of the decoding is this thing is compressed.

And you have to predict the highest quality thing that's

supposed to go there. I-frame- ... is the best representation you have spatially.

And then there's a lot of temporal compression that can happen

depending on the codec, and then you're predicting. You're predicting what

the reality that was captured in this rawest form.

- Yeah, because what people don't realize is that the compression on

video and audio is

one hundred times, right? Like, people don't realize how compressed we, we

do, right? For audio, you move, you compress by, when you go from

normal audio to MP3, you compress by ten times, right? When, when you

move to video, you need one hundred times, two hundred times, right? So you need

to remove all the details, but that you don't care about

because all the compressions that we do, and that's very important, people forget about

that, is to be viewed by humans, right?

So all the codecs, either for audio, mimic basically how your

ear works, right? And a lot of things about, like, the

response on the ear and same for your eyes, right? And so, for

example, on video, we don't work on RGB, right? Everyone expects

to work in RGB. We don't, right? We move to YUV, which is

basically one is luminance, brightness, and the other are colors.

And this matches your eyes, where inside your eyes you have the cones and the buttons, right?

With some of them look on brightness and more on, on the other on colors, right? So

we need to compress a lot, and so we need to degrade. But in order to

degrade, we need to match the human perception, and this is why it's so

difficult. And then we need to use the maximum power,

mathematical power, very complex technologies. We move to the

frequency domain, as Kieran said. We do a ton of

dequantizing, in order to get the best compression, but it still looks good.

- You're trying to compress in order to maximize the highest

quality thing for human perception.

- That is correct. And this is very important, right? Compression is not like

a ZIP, right? A ZIP, you have data in, you get data out,

right? And you try with all the ZIP compression to arrive with

the limit. Here we are degrading the signal, right? And so we

need to degrade both the audio and the video signal in the best way

possible. And we can do that, but it involves, first,

a lot of theoretical knowledge about how the eye works, but

it, a lot of mathematical change, a lot of mathematical tricks,

right? For example, when you move to RGB and you do go to

YUV, for example, what we do very often is that

we scale down the resolution of the color compared to the brightness.

And most of the time, just this without compression, it

divides the size by two, but most people don't see it,

right? And so on and so on, right? And then you go

to very complex mathematical change. So of course

Fourier transforms, which de facto are not Fourier transforms, they are like

discrete cosine transform, but that's the same idea. So frequency domain

we split the video by blocks, right? So that's why when it's

wrongly decoded, you see those blocks and badly encoded, you see those blocks,

and so on, to arrive to compression states that are

insanely high, right? And each generation of the codec is like thirty percent less-

... for the same quality, right? And this requires amount of

power of computational power that is huge.

- No, no, but you should elaborate. It's thirty percent better, but an order of

magnitude, perhaps, perhaps even two orders of magnitude more

compression power. That's the big difference.

- What do you mean by compression power?

- Sorry, CPU power to achieve that level of compression.

- Oh, yeah. So you have to be able to leverage the CPU and sometimes GPU, like

you mentioned. And then we should mention that a lot of this

programming is done at the lowest possible-

... stack, whether it's C and of course, as the legendary—

... Twitter handle re-emphasizes over and over, a lot of assembly.

- So what happens globally is that you have an address, right? Which gives you

with the operating system, a stream of bytes, a stream of data, right? And this is the first step.

And the second step arises with demuxing, where you're going to separate audio,

video, subtitle in type of different tracks. And then on each of those

tracks, you're going to decompress them, decode them, either audio with an

audio codec, video to video codec, and subtitle to subtitle codec.

And once you've decompressed those type of things, you have raw images,

raw, and then you're going to talk with your graphics card and your screen and

display that. And same for the audio, you're going to talk to your audio card, which

then is going to go in analog to your audio speakers.

- And everything we've just said in the past couple of minutes,

every sentence is someone's lifetime's work. There are books about-

... every sentence. So the level of complexity

in many cases is inordinate. You know, it's, it's... Every sentence

has thousands of people working on this-

... in industry as a whole, books written about

it. So there's a lot of detail, there's a lot of subtleties, there's a

lot of both academic and practical realities, both of which matter.

- Uh, we mentioned codecs, but I don't think you mentioned containers.

So what, what are the actual containers

for some of the stuff we're talking about? So people are familiar with MP4,

uh, MOV, MKV. So anyway, what are containers versus the thing that goes inside?

- So the container is what we call also the muxer, right? When I say demuxing, it

means decontainerizing, right? So actually, if you look,

mux means multiplexer and demultiplexer, right?

Mux and demux are those. And same, a codec is actually coder, decoder, right?

Um, and so containers are this collection

of multiple tracks, right? So it's a, what normal people call the file

format, but it's a bit more, um, subtle than that.

But the most known one, of course, is MP4, but when I started,

it was AVI, right? AVI was the, the video format from-

... from Microsoft, and MOV, M-O-V, which became

MP4, was a format from Apple. In the open source community

one of the person that is still active on VideoLAN is called Steve Lhomme and started,

This Matroska format, which is, like, a bit more complex and

more future-proof. And there are so many others.

- So, I mean, there's a, it's a pretty common thing, and maybe it'll even happen in

this conversation, that people confuse container

and the codec, right? So confuse MP4 and H.264, for

example. Is that a horrible violation?

- No, it's not, because technically the name of H.264 is MPEG-4 Part 10. Because

MPEG-4 is actually a meta specification which has

several things in it, right? There is the Part 2.

so there is, like, audio codecs, right? AAC de facto is MP4 audio-

... something. There are actually several video codecs, right, inside the

MPEG-4 specification. One of them is MPEG-4 Part 10,

called also AVC, called also H.264. Right? So

it's completely the fault of the industry to make things difficult

to understand. So that's very difficult so that people

then don't understand why sometimes you talk about MPEG-4 Part 10,

where you mean H.264, and why it's not MP4.

- So you can technically shove in all kinds of different codecs inside

containers and horribly so.

- But broadly speaking, though, MP4 is understood to generally

be H.264 plus AAC audio. 99% of the time that's that, and that,

the rest are de minimis, the small effects, you know, edge effects really compared to that.

So it's not the end of the world. There, there are people

who do get annoyed by that. But also in reality, something like VLC, just to point out,

the file may say .MP4, but it may be something completely different, and that's

one of the challenges both FFmpeg and VLC have is the real world is a

completely different place to a three- letter file format.

- And this is very important to say, right? Like, for example, in VLC and in FFmpeg,

we discard the file format, right? We look

into the file to understand what's in it because so many

people, like, they say, "Oh, it's a video, it should must be MP4," but technically it's

an MOV or maybe it's a MKV, right? So we

analyze in real time everything that we have, and we don't trust- ... the format.

- So what information does the fact that it's .MP4 give you?

- It helps, right? It gives you a hint, right? Just like, oh, it's

finished by .MP4. I will start first by opening,

probing it with the MP4 container demuxer

to see, well, it should be that. But I don't trust it, and if I'm lost, I say, "Okay,

maybe I'm going to try it." So it bumps the priority of the module.

- So how do you get to... just to take a bit of a tangent there.

You know, the dumb thing is if you try the MP4,

but it turns out it's a different codec than you would have expected,

Most players just break there.

- Yes.

- Yes.

- And so how do you not break? There's just philosophically, I'm sure

there's a bunch of stumbling blocks along the way where it's

easy to just break and stop, freak out. That's it. How does VLC not?

- This is why VLC is popular. But the reason is because

actually VLC was, is just a client of a streaming

solution called VideoLAN from a very long time

ago, from the late '90s. And when you're playing video

which are on UDP, right, in network, they might be damaged, right?

So you don't trust your inputs, and this is very important into the security is that you

don't trust your inputs. So everything in VLC is prepared to work with broken files.

And it's a philosophical idea from the beginning, and everything

is engineered into that. And it's a culture, right? And so,

for example... And VLC became very popular on that because a long time ago when people

were pirating content which they do a lot less today-

- And none of us ever have-

- No, of course not. Um— ... the metadata to play some files like AVI is at,

at the end of the file, right? And when you're downloading, you don't have that, right?

So VLC was just like, "Hey, this file is broken, but I'm still going to try to

interpret it," and this was very useful.

- We hinted at the awesomeness of the various different stages.

We hinted at the awesomeness of codecs, the depth and the richness and the complexity

of everything involved there. What— Let's try to define

what is a video codec? What's involved there? What does it mean to

compress something? You already started to hint at it—

... but can we elaborate a little bit more?

- So there's a huge amount of redundancy in any video both

spatial and temporal, and the point of any video codec is to remove

this redundant data, use mathematical properties as part of this reduction

process. So more often than not, using several orders of magnitude more

compute to compress because that's more costly versus both costly

both financially and in CPU resources—

... versus the decompression. So it's asymmetric in that respect. Uh,

often the case because compression is done once, but there could be lots of viewers of

another file. So to take that information and compress it by

100x, 200x, removing redundant information and

using mathematical properties to make that small, but also have properties

such as error resilience. So as, as JB suggested,

VLC in the beginning was, was used to play UDP network feeds, and UDP

network feeds lose packets. And so some of the design goals of a

codec is also to be recoverable.

You need to actually be able to join a stream. It's not necessarily a file. You need to join,

get on the decoding process, and start decoding.

- And, and to give a more image to people

who are not familiar, right? Like, when you're going to see any type of movie, right?

You're going to see the camera is going to pan, right, and travel. And

you realize that, for example, all the background is the same from, for, like,

a minute, right? Or—

... thirty seconds, right? So you can reuse the cloud that you see uh, on

the background, you can reuse that from a frame to another, right?

And so it's, gets the more, the more

memory you have, the more power, the more comparisons you can make, right? And so

the more compressed you can be. And most of the modern codecs are

basically doing that.

- So just to make it even more explicit. So what is video? Video is a

bunch of pixels often RGB.

You have three values, and you have a grid of pixels, and you have,

let's say, twenty-four or thirty or sixty,

frames a second, and you just have all these pixels

repeating and showing different stuff-

... thirty times a second. And so the question, the philosophical,

the technical question is, how can I compress all of that,

store all of that at 100x?

- Yep. Or 1,000x, right?

- 1,000x.

- The target is 1,000x, right?

- And the goal is when you say redundancy, what is redundant? Meaning stuff at best

that humans wouldn't notice if it was missing.

- So for example, you have a picture of a cloud, right? And from the

next frame, they're still going to be the same cloud, so it's redundant. You could just put it once and

not do it, right? Or you have a black background behind me, for

example. The black is the same on the whole picture, right? So you can say, "Well, you know, in

this picture, take the pixels that you have on the top left and the one

on the top right. I'm not going to give the value. I'm just going to tell you it's the same at the top

left." And then you can say for frame one reuse

something from the previous frame or the previous, previous frame, and so on and

so on, right? So you could... Basically, it's

unlimited, but then it's limited in terms of memory or in terms

of compute power. Because, for example, if you need to compare pixels

on two hundred frames in the past on 4K resolutions, it's a huge amount of compute.

- And then when you're showing it, you have to do the decompress of all of that.

So is it the codec, the, has the

encoding and the decoding is a coupled process that you're developing?

- Yes, exactly, right. And those are two different

trade-offs, right? Are you going to compress more? But then it might be

more difficult to decode. Are you going to

comp- to make it a codec that is more complex to encode and easier to

decode? Are you going to make a codec that is easier to encode because you need to be

fast, but then the, the client side, the, the player is going to spend more

time? That's why you have so many different type of codecs, is that it's not

always easy. And to make it even more complex, modern codecs

like AV1, AV2, or VVC are actually not codecs. They are

a collection of tools, right? There are multiple tools, multiple

codecs in the same codec to, depending on the image, get the more compression.

- So just to elaborate, codecs like AV1, VVC

have a much wider, have a wide audience. It could be a screen share

content, it could be video, it could be animation. All of these

require different coding tools. So

what happens these days is a collection of tools are put in and called

AV1 and called AV2, called VVC to allow for different use

cases. So you may be on Zoom and sharing your PowerPoint,

and then you need to show the audience a video. That codec needs to

start changing its tool set depending on the content to compress in a different way.

- And like you said, there's a bunch of incredible engineers behind each part of

that, each part of the tools that make up AV1, for example.

Uh, so we've kind of danced around it. We talked about VLC,

the logo, the hat. Let's talk about FFmpeg. What, what is FFmpeg exactly?

- FFmpeg is basically the low-level libraries for codecs, so compressions

and decompression, muxers and demuxers, and filters. It's—

The core is this, and then you have several tools which allow you to create a

type of pipeline to process any type of video files.

And it's used as a library absolutely inside

everything from VLC to Chrome to your smart TVs, to

basically any video that you see online you usually use

FFmpeg. And FFmpeg in it has all those

type of tools, and sometimes depend on other libraries like

x264, libvpx, and others, right? So it's really now the de facto tool to process

images.

- From a philosophical level, I think it's incredible that your home

videos, your grandmother's home videos and trillion-dollar corporations

effectively are on a level playing field using the same technology stack.

It's— it wouldn't be a surprise, you know, these big companies just have three

thousand-line FFmpeg commands.

There are some that use the API, but there are some that just have long command lines.

- So yeah, there's a bunch of tools, like literally command line tool,

FFmpeg, of course, FFprobe. There's libraries, libavcodec, libavformat, libavfilter.

But the FFmpeg on the command line—

is, like, legendary because you can cut— Like, there's so many

parameters. You can customize everything to hell.

- It's a language. It's an actual language.

- It's an actual— yeah, you could think of it as a programming language.

- Yeah, of course, I'm sure. Because— so most of the people, they're going to take FFmpeg,

file in, file out, and specify the format, right? But you can-- We've seen

thousands of characters, and we've seen also, like, people, like, doing

programming generation of command lines

to make FFmpeg. There is a ton of people who are using AI to generate

command lines for FFmpeg because you have no idea what it is. But you can do- specify so

many filters right on the command line, right? So

FFmpeg is this collection of toolbox for multimedia processing that everyone uses.

And everyone that is watching your videos is also using it, right? You're on

YouTube. Well, it's FFmpeg on the client side. Well, the your

server side, on the server side. The client side is probably Chrome. Well, you're using

FFmpeg also.

And you're using OBS to record. Well, it's FFmpeg, right? You're using a ton

of important, like, big box, professional boxes. Well, it's very

possible that inside some part of FFmpeg is running.

- I mean, there's like so many, just to give people an idea, like I use

FFmpeg a lot on, on everything. Just trivial stuff like,

Take a video, add an intro video and an outro

video, and fade one into the other like what is it

called? Dip to black, like where it dips and then shows

the next video and does the same thing with audio. There's

like a cross dissolve of the audio. It's quiet, it quiets the audio and

makes it loud again. And then there's a bunch of stuff like

showing the captions on screen card, like baking the

captions in. You can customize the font.

You can do all kinds of layering of audio and video. There's a

million things and of course, all of that works

like magically with basically any codec.

Like anything you can shove in on the audio and the video side, it works.

- But it's like if you look at, for example,

you can do things that you would do with Adobe After Effects-

... in command line on FFmpeg, right? It's, and it's very interesting because, for

example, for imaging, there is not such tool.

There is a few tools, but not with the breadth of FFmpeg.

- So ImageMagick has a similar kind of-

- Yes, but you will not-

- ... spirit, but it-

- ... do some filters, complex filters. You don't have the equivalent of Photoshop-

in command line, right? But for video, you have FFmpeg in command line.

- Yeah. It's incredible. I mean, it's like an example of a thing when a

bunch of great people get together and they get a vision, and they

stick by that vision for many years, which is incredible.

- And the vision behind, and the same for VLC and FFmpeg, is that

we make everything that is very complex easy to use for the normal

people, for everyone.

Right? Our goal is to make something that is insanely complex technically

and make it easy to use, right? And people, they use VLC, they

drop a file. They don't realize how complex the file is, but they

play it. Or, or people put any type of thing inside FFmpeg with

complex filters, and it just works like magically, right? And people-

And this is our mission, right? Make very complex things.

- We wouldn't be here and you wouldn't be here if this

required, you know, a traditional television studio setup.

It's tools like FFmpeg that democratize this. The podcast and

streaming revolution, the YouTube revolution-

was caused. You know, FFmpeg was a big player in that because it

democratized this technology that was once in the nineties, for example, you needed

equipment that cost hundreds of thousands of dollars to do compression. It was the size of a car,

and now everybody has that at almost an exact level playing field, and

that's something that's so remarkable.

- It gave voice to a lot of people. And just to clarify, we say you,

you wouldn't be here, not the human, but the podcast.

- The podcast. Oh, sorry. You as a... Sorry.

- I would still... VLC did not have anything to do on a biological

level- ... at creating me as a human.

- But, but it's like you realize also everything moved from text to images

and images to video, right?

Look at social networks. Video is everywhere. It's the most

powerful medium there is, right? And when you see

shorts and, and in Reels and in TikTok, right? It's

amazingly powerful to give... Video is amazing for

that, right? But the complexity is important.

- This is what people don't realize. I mean, this is really it

gave power to the individual all across the world. That's real

freedom. And I think, I can't believe it, but we still haven't mentioned the

actual obvious thing for people who are not familiar, which

it's open source, and there's a open source

community of users and developers behind it. So

it's really, it's a movement. So, like, we'll talk a bunch in a

bunch of different ways about the community behind it. But can you speak to the open

source element? So when we say what is FFmpeg, it's an open source project.

- Yeah. So FFmpeg, VLC, x264, VideoLAN, everything

we do is fully open source. And for the people who don't understand how open

source is, my usual analogy is about a chocolate cheesecake, right? Um,

usually for you, when you want to buy your cheesecake, you go to a bakery, they give you the

cheesecake. The other one way of having a cheesecake is have your

grandma give you a recipe of how to make that. When we do open source,

we give you the chocolate cake, and we give you the recipe to

actually remake the same cake, but at the same time tell you how

to build the oven and also how you're allowed to modify

the recipe and resell it to someone else.

And this is because software is just a very long recipe of small

instruction. Computers are not very clever. They go very, very

fast. So a normal program has tens of billions of instructions

instead of the tens when you have your chocolate recipe. So

a lot of the software industry was about selling software, like where you

just have like the final cheesecake. In open source, we

give you everything, and that managed to get a

lot of people work together, right? Because then you decide that you're going to

make the best program, the best recipe for video, and you create

communities. In FFmpeg, since the beginning of FFmpeg, probably

two thousand to three thousand-

- In the thousands, yeah

- ... people have contributed from the beginning, right? And then it's exactly like the

Linux kernel, right? The Linux kernel has probably ten thousand people

contributing everywhere, and they get together, well,

mostly online, right? So they virtually get together to create the

best tool for something. And on FFmpeg and VLC, it's just like,

well, this codec doesn't work, so I'm going to work on the codec, and I'm going

to add the support for this file inside FFmpeg, so it will be

beneficial to everyone. Because again, we work for the greater good. We work for

everyone, and that is what open source is.

- And we should mention, depending on the licensing,

You could probably build a billion-dollar, maybe even a trillion-dollar company

around basic... as a wrapper to...

- Well, yes- ... people do.

People do, right? There was a lot of problems with mostly,

Cloud providers who are basically running some open source tools,

In the cloud and just give you the API to

access to that. And there was a lot of um,

databases like Mongo or Elastic who changed their license in order

to avoid those type of scenarios.

- This is a question we get a lot in FFmpeg is, "Why don't you do that?"

And you can't. We have, we have thousands of contributors, some of whom aren't even alive anymore.

It would need all of their agreement to do that, and JB will go maybe a bit later

and talk about how challenging that process was in VLC to do the re-licensing.

- The license is a social contract in terms of Rousseau de

facto of the community. The community does

not agree on much beside the license. People go

around, discuss around because of the license, and that

also allow those license fork, right? Sometimes the community splits, but

it's possible because of the license and to merge back.

And we've seen that so many times, right? GCC and GC,

And EGCS in the past. We have seen, for example, all the web

browsers, right? They started as web, like KHTML, which becomes WebKit and

then which becomes Blink, right? So open source

license is like the core of the community

and people are coming from all around the world, very different type of religion,

Political borders. They work

in the same way on a project to solve a specific problem,

and the specific problem we're working on is to make multimedia easy for everyone.

- Uh, looking it up on Perplexity here, looking at the different open source licenses.

Most major open source licenses fall into two buckets:

permissive, very few conditions, and copyleft, share-alike

requirements for derivatives. Below is a brief practical summary of the

main ones you'll see in the wild. MIT license, BSD, ISC, Apache, GNU GPL, GNU AGPL.

Where's LGPL? Yeah, LGPL. Let's see.

There's the Mozilla Public License. There's Eclipse Public License. It

goes on. There's a lot of variety. I mean, I think the really popular

ones is MIT, GPL, LGPL-

- Yeah. And BSD. BSD

- ... and BSD, Apache. Sometimes you'll see-

- Apache as well

- ... Apache. Unlicense, that's an option. Attempts to dedicate code to the public domain with a fallback

permissive license.

- There are many licenses for many different things. What people don't understand

that public domain is something that doesn't exist worldwide, right?

So it's all the open source licensing

use the copyright law, right, the international copyright law, in order

to give rights on how you use the software or how you

modify. It's de facto a copyright license

contract that you give to the end user or to the developer. And

so you have like the first one, which are basically very permissive, MIT, BSD.

You give the code and basically you do whatever you want, right?

You take it, you want, you modify, you do what you want. And this

is popular for JavaScript and the type of BSD operating system.

- So some of them, one of the parameters is whether they require attribution,

meaning if you use the code, you have to say-

- Yes. So in those types of permissive licenses, some you need to say if you

use it, which is called attribution, and some you don't. And then there is

the other part of license which are copyleft,

where you need to give back to the community your

modifications and with different strings attached.

some weak copyleft license, like the Mozilla Public

License, to some which are a bit stronger like a

GPL, or even very strong like AGPL. So all of

those are different types of licensing that depends on what

your goals are and how you want to structure your community, which is why I spoke about

social contract, because this is very important to understand.

FFmpeg and VLC are mostly GPL or LGPL. The Linux kernel is GPL

but Android is Apache. A ton of JavaScript frameworks that are using are mostly MIT.

All the BSD kernels, OpenBSD, NetBSD are of course BSD. And so it's

philosophical change on how you want people to contribute back- ... basically.

- So I think you talked about that, you've moved at one point from GPL to

LGPL on certain parts of the project. What... Can you describe the difference

between the two, and what does it take to move to, I guess, a more permissive...

So that direction is more permissive. LGPL is more permissive than GPL.

- Yeah. So you have to realize that you can always go from

more permissive to less permissive, right? Because of course,

those licenses are basically statements, and so if you restrict,

you can always restrict more, right? So in a GPL project, you can take

MIT code, but you cannot do the opposite, right? Because they are more

constrained to match. Indeed, in fact, I changed the core

of libVLC, which is the engine of VLC- ... from GPL to LGPL.

And there were two reasons to do that. The first one is that so people can

use the VLC engine, libVLC, into

third-party applications. So a lot of applications which are playing video

on your phone or on your tablet are actually VLC engine in it-

... which is calling FFmpeg in it. Um, so that was

one of the ways to create one of the companies I created, which is doing consulting

and integration of those types of applications where you integrate VLC

into third-party solutions like inside game engines or

stuff like that. With GPL, you couldn't do that because that means you needed to open source

everything, and those are for a lot of, like, commercial companies who don't want

that.

- So you can create a company with LGPL, you can create a company around it.

- Yes.

- You can do a commercial thing. You don't have to open source it.

So that's a big, big leap.

- So you could play video in your game.

- Yes.

- The problem is I'm a game developer, and I want to play some videos-

... and I don't want to be forced to open source the entire game just to play those videos.

So that's where the consulting business, the libVLC LGPL-

... allows you to do that. The LGPL, the library GPL as it used to be known,

allows you to do that.

- And FFmpeg is exactly the same. It force... LGPL

forces you to give back what you change on this component, this-

... library, which is why it's library GPL.

And so you can use FFmpeg as LGPL into, like,

any type of application, even non-open source, but you need to give back the

modification you did on FFmpeg. Same on libVLC.

- Is it limiting from an open source perspective to go GPL?

Because if you-- if your library, if your code is GPL, it means you're not...

You're basically discouraging companies from building a business-

- Yes

- ... around it, right? Is that, is that fair to say?

- It depends on the company, but the company whose business model requires

the source, the application to be closed source, yes, it's limited.

So that's why, for example, I moved to LGPL. The second reason is a,

a bit more obscure, is that the terms and conditions of the,

App Store, the Apple App Store for iOS makes it very

complex to have GPL application on it, while it's easier to have

LGPL applications on it. So VLC on Windows and on Mac,

And on Linux is GPL. The core is LGPL.

But on iOS the iPhone version and the Apple TV

version is a type of different license called the MPL.

And yes, I went and changed the license and it was a long story.

- Yeah. So I think basically to change the license you have to contact all the

contributors.

- Yes. It's very important to understand that open source

projects are what we call in the US copyright law

joint work, or in civil law collective

works or collaborative works, is that you work all

together in terms of the same goal, and then you create one software, which is one release.

But the copyright is kept by all the individuals.

Some open source projects don't do that. They force copyright assignment, but this is not what we do.

We're communities. So everyone has basically copyright on what

they changed. And this copyright stays even

if at the end your contribution was deleted because the new

contribution was based on your previous one, right? So if you want to properly

re-license, you need to find all the contributors. And at that time, I

had to contact more than three hundred and fifty people. And

sometimes, well, they're just an email, right? So it's... you need to actually track down.

I actually, like, travel to some place to go someone that I

was like, sorry, that I'd found online to see a-- to

go to their job and say, "Well, you licensed that. Can you-- do you

want to change from GPL to LGPL?" Most of the times they don't even care. They wanted to

help VLC. But also it brought me to very complex situation. I

arrived to the work of a person who was a factory worker.

Um, and I said, "Well, I need to you to sign that,"

because it was his son who died who actually wrote the code, right?

So I had to explain all those type of open source meaning,

and no, I was not a company trying to rip out the two line or five

line that that guy did-

... but was useful, and the whole community agreed on that, and he had

no idea I was a factory worker. This com-- And I was a lot younger, right? Like

it was fourteen years ago, and like, like I was

almost in tears, right? It's very difficult, right? We are talking about lives of people and

he explaining, and we went talk about the photo of this guy, right?

So it's important to do it right and to do it correctly.

But yes, that means tracking down everything because every contribution works.

There are some project who don't respect that, and we do re-licensing a bit, like,

aggressively.

But as I said, it destroyed the whole heart of the community because it's-- we

only agree on the, on the license, so that's important.

- I would emphasize the community is such a wide-ranging group of people.

There's people in the Syrian war zone with electricity part-time. There's,

there's all people from all walks of life-

... rich, poor, young, old. So it's quite remarkable to get,

you know, a group of people aligned on something. And that's an

achievement in itself.

- Yeah. It's incredible. And a lot of them are introverts, so you

coming to find them and getting them and getting them to answer an email might

be quite difficult.

- Most of us are introverts, right? You need to be more precise. You have

extremely introverts, extremely, extremely introverts and introverts, right?

It's just like a whole spectrum of different people. It doesn't matter. The

important is, is your code good?

Is your code great? Is your technology great? We care about excellent code.

We don't care who you are. Sorry, it's just like we have no idea to check.

We cannot check, right? Like, maybe you're a dog. I don't care, right?

I don't care where you come from. I need to look at your code. And this is

important because people don't understand that, and they come to the community and send them some

patches, and they get rejected, and they don't like that because,

I mean, you're just like, "Sorry, it's not up to our standards." "Oh, yeah, but I'm

engineer at this very large company in Italy, in Germany,

in the US." We don't care. We care about the quality of your code

because this is what defines our community, and which means that we have a

lot of people who contribute who are some very different backgrounds and,

and very introverts, sure. But that's okay, right?

- So one of the legends of the community is of course,

Linus Torvalds, who created Linux and is

a longtime maintainer of the Linux kernel. As the

legend goes, he can be pretty harsh on this meritocratic process of

reviewing the code and saying it's not good enough. Can you just speak

to the legend of Linus Torvalds?

- Linus is one of a kind, right? And

I would even go and say that what he did on Git is more interesting than what he

did on the Linux kernel.

He's very harsh, but what people don't see is usually when he's

harsh to, it's people who are maintainer of part of the

kernel, right? So they know him, right? So he's not very harsh like

that to everyone. The thing is, what he created in his room

is basically powering every server online, right? Even

at Microsoft cloud called Azure, I'm quite sure seventy, eighty

percent of the servers are running Linux. All your Android phones are running

Linux. What he did with the power of open source, sure,

is amazing. And yes, the quality of the Linux

kernel is very high, and yes, it's difficult, but

we cannot compromise on that. We cannot compromise on quality

because in the end, and you have to understand that, is the core

community of VLC is five people. The core community of FFmpeg is ten to fifteen,

and we are the ones who are going to maintain your code, right?

Because one thousand contributors in the timeline and just ten staying, it's

one percent chance that someone comes and stays. One percent.

So you will have change of job, change of wives, you have children, you

have accident in life. You're going to change jobs, whatever. You're not going to

come back. It's most likely. So we are the one going to maintain your code.

It needs to be maintainable. It needs to be excellent.

And yes, sometimes that means that you need to rework your work because it was good, but

it's not excellent, and we need excellence because we are very

few to maintain something that is critical for the whole.

- But we should also mention that there is some spiciness, some harshness to the

language that's sometimes used when you're keeping this high bar of excellence.

Is there something to say to that?

- It's, it's true, right? It's also the fact that, for example, what we're doing

is low level. It's extremely technical. You get into this

community. The tone gets very like

a type of-- It's a subculture, right? So people who arrive from the external are

basically not known to the subculture. Most of those people

around FFmpeg and VLC, we do VideoLAN DevDays, VDD every,

every year. They are so fun in real life, and they love it.

But it's true that you're online and sometimes, like, the tone, you

don't realize how it is. But that's okay.

- It's a culture. I mean, you get this in the gaming culture. There's pretty harsh,

intense, the way people communicate, and it's-- everyone

understands that the way you show love and respect just looks different in

different communities. Sometimes people... It depends. If it's a

book club, usually people are going to be much sweeter. If it's an open source

project that's very high stakes and used by millions of people-

- But it's very not often insults that you see, for example,

in the gaming, right? And so Linus' tone is a bit unusual

even for the open source community. It's more like it's more harsh on the results,

saying, "No, this is not good. This is crap." Those type of things that you will see.

- Try not to make it about the person, make it about the code.

- Yes.

- It's very, very matter of fact, and I think you've got to look at it in terms of, you know,

the famous FFmpeg is developed almost entirely by volunteers, and that's true, and you've got

to imagine someone's done a hard day's work at their day job. They come home.

You know, terseness might be a thing, you know, it... And that's not something to take

personally.

- You're tired, you're busy, but you still care about this open source stuff.

But you may not be able to explain and handhold someone on every subtle detail.

- And also you have to

realize that most people don't speak English as native language.

And this is especially for open source projects like

FFmpeg and VLC, which are mostly centered out of Europe. Sometimes like

people who are from the US or, or just like are very

not happy about the tone, but most of the time it's also like they don't know better, right?

It's difficult. The language is-- English is a difficult language. There is so many

subtleties and tone and so on that you don't have, right?

So often it's also difficult in those type of community about like

different cultures and languages.

- So as the legend goes, JB, you repeatedly turned down millions of dollars to keep

VLC open source free for everyone without ads.

So take me through the reasoning behind that decision of

leaving millions of dollars on the table.

- Yeah, that's like almost a meme, right, on Reddit or-

- There literally is a meme on Reddit.

- 9GAG and yeah, yeah. See, there's-

- You looking like a wizard in the, in the VLC hat on Reddit. This

is JB, the creator of VLC media player. He refused

tens of millions of dollars in order to keep VLC ads free.

Thanks, Jean-Baptiste Kempf. You can even summon him on Reddit.

- Yeah. And usually if you see, right, it's usually like people tag me, right? And,

and then there is me, and then like I say, "Good morning."

I got twenty-four K upvotes, which is great, right? My karma on Reddit is amazing,

at least on that account. So the question is,

needs to be answered first, what is the story about VLC, right? Because

yes, this is true, I refuse

dozens of millions of dollars, yes, several times. Yes, I could be a

multimillionaire and be somewhere on the beach. Um,

but I did not do it because I thought it was not

moral and it was not the right thing to do. And this is very important for

myself, is to be like, I work for the greater

good, I work for people, and I don't want-- It's not just by myself.

But the reason is also because I did not feel that I'm

completely legitimate to do that, and let me explain you why. VLC

story is a very weird story. In France,

we have university and we have a type of top colleges

and those top of excellency schools are engineering schools,

business schools, and basically lawyers and medical,

right? But they're outside of university, and in order to enter those,

you spend two years working like crazy math, physics to

enter those best engineering schools.

One of the schools is called the École Centrale Paris. It has changed name since, but it was

called the École Centrale Paris. And because it was Centrale, they

had to move it because it was too small after the World War II and, and they moved

it, they wanted to move it to the center of France in a place called Clermont-Ferrand.

And the alumni decided that this was not okay, right? It is

a, the school that Eiffel, right, the, the one who did the Eiffel

Tower, attended to, right? So they said, "No, no, we are amazing, great school. We cannot

do that." And so they bought a piece of land south of Paris very near

Paris. And it was a campus managed by a nonprofit of the alumni, okay?

Because of that, everything on the campus was managed by students. The

university did nothing, right? So radio, TV, supermarket, library defining who was

going into which rooms. Everything was managed by the students.

- That's amazing. That's an amazing experiment, that it all, all

didn't go to hell quickly. It somehow flourished.

- It worked great, and I learned so much in my life

doing those side activities, right? Because you're twenty-two and you need to run your campus,

else you don't have electricity, right?

So you care about that, right? But anyway, in the '80s they

did a full experiment of deploying a network mostly sponsored

by IBM and 3Com, which was a token ring network.

So token ring is something that probably almost no one

knows about anymore. It's a networking technology where

you don't have routers, right?

Everyone is linked. It's type, like really a ring, and when you want to

send a message, you talk to your neighbor who's going to put the message to the next one,

who's going to put the things to the next one,

in terms of ring. The issue with token ring is, of course, is that it's

very slow because every computer on the network needs to

open the message, see if it's okay. Is it for me? No, it's not, and

then send it back, like a token which is traveling around the ring.

In the '80s, you're doing some Telnet and sending mails as

university. That's okay, right? But starts the '90s,

and the '90s and start video games, and when you have high

latency in video games, basically you die, right? So in nineteen ninety-four, nineteen

ninety-five, around Doom and Duke Nukem coming around, they want a faster

network. So the students go and see the university and say, "You know

what? We want a faster network. We need to work," which, and also play

video games. And the university tells them that basically,

"Oh, I'm sorry, we cannot help you because you understand the

campus is not ours. You manage it, so

do something. And you should see some basically partners of this

university and basically go away." And they go, and they

actually go and see the CIO of

Bouygues, which is a large French company and who's doing some

TVs in France. And he says, "Well, you know what? The future

of video is satellite." Well, today we know it's not, but at least it was

a good idea. In nineteen ninety-five, the first satellite dish, and he says that

instead of having like one satellite dish and a big decoder for

each of the students, which are one thousand and five hundred, what about

you build, like you put an enormous dish and

only one decoder, and you send the video directly on the network.

And that required a very fast network. Today, it's obvious, but at

the time was, like, the first to do video streaming. So they built this project,

which was called Network 2000.

Of course, right, we are in the '90s, right? Everything is- ... futuristic is called 2000,

like—

- Yeah, 2000, yeah.

- And so they do the Network 2000 project. It's completely hacked.

It crashes after 45 seconds. That's okay. The demo is 40 seconds.

It leaks memory. That's okay. They put 64 megabytes of RAM instead of the

8 or 16 you have, and the demo should have stopped there. And that was the Network

2000 project by the students.

- What was the format of the video that they had to work with?

- MPEG-2 because satellite is MPEG-2 TS for transport,

MPEG-2 video, and MPEG-2 audio at that time.

And the project should have stopped there. Everyone was happy. They had, like,

amazing ATM network at 155 megabits per

second. They had probably one of the best network in Europe at that time, and

they stopped the project. Six months or a year later, two students arrive and say,

"Well, you know what? Maybe other people care about video streamed on a

local network," and they create the VideoLAN project,

VideoLAN. And one of them is called Christophe Massiot, that is a good

friend of both Kieran and me, and they start the project. It's not

even open source yet, and they spend around three years

to get the school to agree to make it open source. Because the

university wanted to get some-- because of the IP and copyright of

the students, wanted to basically monetize these MPEG-2 decoders.

- Just to be clear, so what was the main application, streaming on a local network?

- It was streaming on a local network.

- By the way, that's just, like, to state the obvious. This is before YouTube. This is before-

- Ten years before YouTube. You have a Pentium 60 or 75, right?

You, the main machine was 486DX at 33 megahertz, right?

- Bear in mind, television was the main form of video at the time. You could

get new channels. In the '90s, having even one new channel when you grew up with four

channels, having a fifth or a sixth was a big deal, and so

having this satellite service with, you know,

dozens, even hundreds of channels was so groundbreaking.

- Especially because this is university where you had a ton of different nationalities, right?

So there was a ton of people who wanted... So the-- in the end, they had, like,

several dishes on different types of satellites, right? Because, for example, a lot of people were

coming from the Maghreb or the Middle East and they, so they went

to different types of satellites. Anyway, the solution worked great,

and they started the VideoLAN project. The VideoLAN project has several

and some are completely crazy solutions, like one how to

create multicast on a unicast network, but let's not come

to that. It's too, too complex. But VideoLAN client part is what became VLC.

Actually, they basically strong-armed the university to force it to open source

because university did not understand that. And in 2001, it's still early.

But basically, yes, the university agreed early 2001

to make it open source. I joined the project in 2003 because that's

when I joined the university. So the first thing is

I'm not the one who created VLC because actually no one did, right?

- Just kind of naturally emerged from the VideoLAN project. And we should mention that, like,

again, you said it just, but to make it clear, VideoLAN,

As what it became was at the time was a set of technologies

around video, and the VLC, what you called the client, that's the thing

that most normies, uh-

- That is correct, and

- ... think of, like, as the thing, which is, like, the thing that pops up when you click on a

video and you play it.

- So I arrive in 2003, and then I will create the

open source nonprofit organization called VideoLAN,

and I took everything out of the university to create a nonprofit

project and something sustainable. It's, yes, it's true that I spent

more time than anyone on VLC and VideoLAN. That is sure.

but it's a continuity of a previous project, VideoLAN, the

student project, which is a continuity of the Network 2000 project, which is a

continuity of that and that.

- I'm sure there's moments along the way there you were thinking of, like,

what is the future of this from an open source perspective? 'Cause as,

as the internet is blowing up, and there is companies... I

mean, for people who don't remember, like, there's companies making

huge amounts of money.

- And I can tell you that in 2005, the project should have died,

And I made it to continue the

project. At some point, we were only two active developers.

and I thought it was great technology and was useful,

and it will be useful and I made that my life and my,

my time. And I made that grow from a

few hundreds of thousands of users, millions of users to what we have

now, which is probably billions of version of VLC around the

world and used everywhere. So

that's a bit the story of VLC. There is ton of very funny stories around

that. Many people from around the world working

on it, like you said, in Syria or middle of nowhere in India. But

along the way, I got several offers which were either to

bundle toolbars, right? You remember those horrible toolbars-

... which were basically spyware, or changing your web

browser or your search engine or even, like,

advertisement inside VLC. And I didn't like that,

right? I am-- and people don't understand that. It's not-- I'm not

against money, right? I'm very happy to make money. I created several

startups and one I hope that is going to work very well.

It's the fact that I believe that you need to win money

ethically. There is a right way of doing that, and doing sneaky

advertisement or stealing data is not the correct way, right?

For example, if Netflix arrived at some point and say, "Well, we want to put Netflix inside

VLC," probably the story would have been different, right? But they didn't. The only

people who came to us were shady ads company.

And if I do that, right, I would have a ton of money, right? And then three years

later, project is gone, right? Someone forks it and something else happens.

- So it's not even necessarily ads or any of that, it's the shadiness of the-

... dishonesty of the-- So you had a good radar, you had a good

threshold of like, "No, this compromises the

spirit of what this is supposed to represent."

- But also it's for me, right? I'm like very selfishly, I

need to go to bed at night and be happy about what I've done, right? Maybe it's

my upbringing, maybe it's my parents' fault or whatever, right? But

I believe there is right and wrong, right?

And this was the right decision at the time.

It still is. I want to be proud of what I've been doing.

And like, if I had sold out, I would have

betrayed so many other people who work here.

- Yeah, well, I should say me and most of the internet

thank you for that decision. It's inspiring for others,

I think that are pushing the open source movement forward,

that it's okay to do these kinds of huge sacrifices

if you believe it's right. And I think in that case it was right and it was the reason that VLC

became as successful as it was, 'cause it's an embodiment, it's a symbol of

like, you know, freedom and what the open source community can create.

- Yeah, and be a service for so many people around the world, and this is important.

- We should emphasize in the 2000s it was really normal to download a program and it

secretly installs some spyware. It was buried in very faint text or in the

license text box that nobody reads that at the bottom-

... "Oh, I will be installing this toolbar-

... and changing all these things," and it was very common to have to, you

know, you install a program to do something at the time of any sort.

- To put yourself in the mind of a developer at that time, I think it's

very easy, to everybody listening to this, it's very easy at that

time to convince yourself to take a few thousand dollars-

... a few thousand dollars to do it. To say no to much more money—

... takes guts and takes vision.

- The last offer I had was obscene,

and they say, "Yeah, but imagine with all that money you could build

something new, open source," right? It was like the mind trick was,

it was difficult.

But for me it was just like, "No, this doesn't work like that or this is not the right thing,

so I don't do it."

and again, right, it's not that I don't like money or whatever. It's just like

it wasn't right.

- Well, once again, thank you from me and from the rest of the internet.

Let me talk a little bit more about the open source movement, about the

fact that, as you say over and over and over and over, FFmpeg,

is and many open source projects are built by

volunteers. So there's a bit of drama recently,

uh, Kieran, on the interwebs, on Twitter.

You have a spicy style on Twitter that I think

articulates and celebrates all the incredible developers and

development and the code, especially

assembly that's involved in building some of these codecs and building some of this

incredible technology. But that brings us to the, a bit of a

debacle that happened. Tell me the full saga of what happened with

the Google security engineers.

- Just to be clear, Google are one of the biggest supporters of open source out there.

They have been for a long time. It's just I think some things kind

of went a bit overboard this time. So FFmpeg

itself, and this is not like a secret, it's on the homepage, you know, the,

it processes untrusted data. There can be security issues when you parse

untrusted data. That's very normal. But recently what changed was Google started

using AI to create security reports on an open source project, FFmpeg.

Volunteers had to deal with that. They did, they provided very limited

funding, and they even went to the media first announcing how good their AI was

before the issues could be fixed.

- And this is in the public forum.

- Yeah, this is all public.

- So report, reporting an issue, using AI to find an issue in the code,

which is a security vulnerability, and then reporting that publicly before you're able to fix

it.

- Yeah. It's announcing how good their AI is, that they provided a standard

90-day industry deadline without

without really understanding the nature of volunteer-driven

development. In addition, this vulnerability was on an obscure 1990s game codec.

the way-- And let's look at it from their standpoint to begin with. Let's you know—

- Yeah. Can you steer me in their case?

- Yeah, sure. They have substantial resources working on the security of open source

projects

that, you know, are ubiquitous, and they've used, you know, a lot of compute to do

that and very expensive and very capable security researchers,

to do that. And that's their viewpoint is they are contributing by

doing that. But I think that's where opinions differ.

it opened up a lot of interesting fissures I would say.

it does seem that there's a portion of the security community that

look at themselves a bit like building architects that never have to go to site.

You know, going to site is something that is a little bit beneath them, the actual

day-to-day construction. They're there to do their security things and it's someone else's

problem. The security industry also kind of has

a very aggressive tone towards things. The, the language they use is extremely

aggressive. They use very strong language like, "You will get popped." So and to,

Joe Public, get popped,

you know, means something quite bad. For them it means to get hacked.

The way I would look at it personally is a little bit like the padlock on your home.

The padlock on your home or, you know, the lock on your home

is there to protect against the capabilities of what it's there to protect.

It's not there to protect nuclear secrets. It's not there to protect Fort Knox. And

it could be looked at that they're using AI at a level of scale to go

and pick those locks and then say, "Hey, your

lock's not secure. You need to deal with this." Whereas actually they're the ones with

resources to be able to

fix this. But that seems to not be something either they'll contribute to in terms of

patches or in terms of financially. And the scale of AI is kind of the issue.

The bug reports are very wordy. They're very,

very-- It's almost a denial of service by AI-generated bug

reports on very niche codecs.

and the other issue the security community has is everything is marked high priority.

You're going to, you know, "This is the most important thing in the world, and you need to deal with this.

High, high, high, vulnerable, scary, scary, scary,"

on a game codec used on one disk in 1993.

And that's where the dichotomy lies. Going around

telling everyone that their padlock's not safe, well, that's a hobby project of

somebody. The safety of that codec is consummate to

what that person thinks. It's their hobby. It's good that they're security analyzing it, but it

doesn't need a big scary warning, "This is a critical vulnerability."

We also may recently also see that there was another

quote-unquote vulnerability. It wasn't at Google in this case, but

a filter could overflow and have an integer overflow, and one of your

pixels could be the wrong color. And this was marked high, 7.5 severity in red.

And at some point, the security industry needs to realize you can't keep crying wolf

like this because this just leads to people, you know, the equivalent thereof of putting

password stickers on their PC. You know, you can't just keep crying wolf every day.

And I appreciate, you know, that's their modus operandi is to create as much

scare and fear. But from the Google standpoint, at the end of the day,

they need to contribute either financially or with patches. Google

uses FFmpeg at a scale probably you or I couldn't even contemplate,

millions of CPU cores.

And yes, they contribute in areas mostly regarding their own products, so VP9, AV1.

But in a wider sense,

there's a disproportionate level of contribution. Yes, they fund students. Yes, they fund Summer

of Code. And I think so Alex Strange is a former FFmpeg developer I think

posting in a personal capacity.

- So he posted about security engineers on

Hacker News. His post reads, "The problem with security

reports in general is security people are rampant

self-promoters, in parentheses, Linus

once called them something worse. Imagine you're a humble

volunteer open source developer. If a security researcher finds a bug in your code,

they're going to make up a cute name for it, start a website with a logo.

Google is going to give them a million-dollar bounty. They're going to

go to DEF CON and get a prize, and I assume go to some

kind of secret security people orgy where everyone is dressed

like they're in The Matrix. Nobody is going to do any of

this for you when you fix it." basically commenting on the sort of the incentives

for the different people involved and misaligned.

- The problem here is the disproportion of means on discovery

compared to patching it, right? And this is the biggest issue, right?

And after that debacle, Google did some changes.

- They are now starting to send patches, which is-

- And they also now have reward tools for fixing issues. So

it has changed a bit because of that debacle. So it's good, right?

But we've seen, and we talk about Google, but we have seen like some

other large companies saying, "Oh, you need to fix this bug because it's critical in our

product."

- Can you explain the XZ fiasco? The FFmpeg tweet reads,

"The XZ fiasco has shown how a dependence on

unpaid volunteers can cause major problems. Trillion-dollar

corporations expect free and urgent support from

volunteers. Microsoft, Microsoft Teams posted on a

bug tracker full of volunteers that their issue is high priority.

After politely requesting a support contract from Microsoft for long-term

maintenance, they offered a one-time payment of a few thousand dollars

instead. This is unacceptable. We didn't make it up. This is what

Microsoft Teams actually did." And then you give

the image and the details and all that kind of stuff, showing that these trillion-dollar

companies are not giving much money, not giving much support.

- They think an open source project is a traditional vendor that they have an

SLA. They think a public bug tracker is actually,

you know, a third-party vendor's Jira where you can do all of these things. It's

not. It is there to report bugs.

I think the thing that made this particularly heinous was

the name-dropping of Microsoft, the name-dropping that this is a visible product.

If this was just a general bug report, I think that would have made it a lot

better.

- Yeah, so they literally said, like, "This is a big deal because a lot of

people are using it in Microsoft." I wonder what

happens psychologically. So I think what happens in these companies, maybe you can

correct me, is they— You're right. They just think of FFmpeg as like a vendor that

Microsoft surely is paying a huge amount of money to.

They kind of assume that in their interaction, and

nobody anywhere on the stack is going like, "Wait a minute.

Shouldn't we be giving like millions of dollars to FFmpeg?"

- And this is a very big problem in large— Like we're talking about some

companies, but it's the same everywhere, right? A lot of those

companies. Like the, when we talk to that person, right,

he was just like a manager on one project in Microsoft Teams, right? He

had never really discussed with open source community. He

had no idea, right? It was like—but the problem

is that usually there is what we call OSPOs, right? Open source program

offices in those type of companies, and they are the ones who are supposed to

discuss with open source vendors. Um, or open source communities. But like

they often don't explain that correctly internally, right? And here it's just like

we are not your supplier. If you want me to be your supplier, I'm very

happy, right? I will send you a contract and SLAs. Like I created

five companies who are doing that around open source projects, so that's okay.

- We should say that some of the spicy tweets that Kieran, you're

behind, and some of the debacle produced results.

- Yes.

- Positive results.

- Donations have increased substantially. They're still not enough to

cover even a single full-time developer, but

on both a, you know, awareness level and a technical level,

there's substantially more technical awareness and sort of awareness of the importance

of FFmpeg as a result, as a result of X and what's

happened. I can say, you know, it solved its purpose. People

realize the level of importance FFmpeg has.

- And on VideoLAN it's the same, right? Like for example, a, a very simple example.

For more than a year, we couldn't update VLC on

Android because of a bug on the Play Store, on Android Play

Store, right? The only way we got someone to answer

was to put a very spicy, as you say

tweet saying that we are going to stop distributing VLC

for Android, right? And we have around 100 million people using that. And now then

someone from Android actually came and discussed to us, right? We had the

same issue with Microsoft or, or like saying that we were

going to stop distributing VLC on the Windows Store. And

unfortunately, we are so small that the

only very strong power we have to solve those issues

is blaming on social network because it snowballs and

now they listen to us. But so as large companies often have difficulty

talking to us. Like for example, VLC, right, is probably one of the top

10 software used on Windows. I am not part of Microsoft ISV

programs, right? I don't have a point of contact at Microsoft, right?

While I'm sure any other software, Adobe, Spotify, has a

point of contact. I don't have that, right? So

raising awareness works. It's sometimes very spicy, lot of drama. Well,

X and Twitter are okay for that, but it's efficient.

- Uh, so everybody listening to this should go follow FFmpeg on Twitter, on X, follow

VideoLAN on Twitter, on X. Go donate. Donate ... to FFmpeg.

- And thank you, Lex. Over the years, several years you've been a supporter of, you

know, FFmpeg and VideoLAN on X. You know, giving us shout-outs, appreciating,

you know, what we do.

- FFmpeg for life.

- And for example, like Tim Sweeney, Carmack, and a few

others, like very high-level people have raised also the awareness

on our X accounts, and that helped a lot also.

- Karpathy as well.

- Karpathy, yes.

- Karpathy as well, yeah.

- Yeah. I mean, also, you know, outside of the fact that so many people use it, it's

so impactful on the world, it's also a great representation of a great open

source project. Like the value of assembly and C and making sure that like you take

programming seriously for real world systems.

- It's not just that. We'll talk about assembly later I'm sure, 'cause that's its whole topic in itself, but it's

also celebrating people like Andreas Rheinhardt who do maintenance. It is, I

believe unpaid, as I believe as a volunteer. He's doing massive

refactorings. Uh, Andreas Rheinhardt and Anton Khirnov

rewriting ffmpeg.c with threading. Celebrating those guys,

celebrating the untold labor that's gone into this

that actually doesn't change anything from the user standpoint. The files are exactly the

same, but wow, the, the, the airplane has been rebuilt whilst it's in the air.

- Christian Garcia said, "As a teenager running this account," referring to the FFmpeg

... account, and you responded, "Teenagers have written more assembly

in FFmpeg than Google engineers." But also

just pointing out that there's a lot of incredible contributors who are teenagers.

- Like JB said, we don't care who you are, where you're from, what you do.

Teenagers have written thousands of lines of assembly,

Over the years. Give a shout-out back in the days to Daniel Kang.

So also highlighting the work of people like Ruikai Peng. This is a 16-year-old, some of his

first contributions to FFmpeg,

actually doing and putting some of these quote unquote security researchers to shame by

by actually finding issues and fixing them and being

16. There's no barriers. There's no barriers to you have to

study on, at college under this person and understand these. It's you can

learn C, and let's be honest, it's from, it's from the K&R book. Learn C.

You can learn assembly. We'll talk about that maybe a bit later.

You can contribute to world-class technologies.

- In VLC one of the oldest contributors called

Felix, he's the one doing everything on Mac and iOS. He's starting

working on VLC. He was 16. We had a guy called

Edward Wong, who used to be a Google Summer of Code student who

stayed for three years around VideoLAN. He was 14,

right? And, and part of Google Summer of Code and Google

Code-in, which were programs where basically we have students or high school,

We wrote a ton of assembly for x264 and for VLC

and for FFmpeg, right? So everyone can contribute.

- And he also did a good job because he didn't play the alarmist CVE

heist, create a CVE, which is like, a

public exposure of security and do these big scary

red 7.5 high priority. He just fixed an issue in Git

after three days and just fixed it. He didn't need to go and play a big

security drama about it. And I think

I posted, you know, the kids are all right. Whereas- there's, you know,

there is a por- I'm not saying all security people do this, but there is a portion of the

security community, as Alex said, that likes to hype themselves up by

creating drama. They would have happily raised, "This is a high

priority CVE 8.0" or whatever on a

issue that actually was in Git. It wasn't even in a release, it was in development, and three

days later was fixed.

- Well, I just want to put a little bit of love out there, even to the bigger

much love and respect to Google engineers. Like you said, they're

Some of the, the best software engineers in the world, and they do contribute a lot-

... even on the security front. And also, you know, I'm a big fan of

Theo. Much love to Theo. He was part of this,

Debacle and drama a little bit. I think when you just zoom

out on the grand arc of human history,

the drama contributed positively to everybody involved. Donations went up.

It brought more attention to the topic, allowed

everybody to bicker in a way that ultimately got them to figure out

what FFmpeg is all about.

- So the way we looked at this is like it's a rap battle at the end of the day, you know?

No, but it is. We say stuff, we say stuff-

... but we can, we can leave it on. X is a perfect place for, you know, international rap

battle. You say stuff. I say stuff about your mama, but it doesn't mean, you know, I have

an actual personal issue with her.

Uh, and that's what it looks like. The Theo situation, you know, JB can maybe expand, went a

little bit too far and there was a little... But, you know, it's just a bit of fun.

It's just a bit of rap battle. It's a bit, it's WWE. You know,

everyone's having a bit of fun on X.

It doesn't need to be taken seriously. You know, the teenagers thing, you know,

that... So that guy was a Google employee saying, "Hey, you know, there are other ways to run

an open source business." You know, go and there's like, oh, man, just have a bit of fun, you know?

That's what the point of this account is. And, and furthermore, if you can teach people

about the ways of open source projects, assembly, et cetera, by doing

that, I think there's a lot to be offered here. It's not dunking on people for dunking's sake.

It's showing actually the story that I think X learnt is these are not big

corporate open source projects. This is not Kubernetes where there's,

you know, hundreds, maybe thousands of people-

... paid to develop this stuff. These are just people in their basements in their spare time,

and if you can address that topic in a fun and entertaining way-

... I think that's the good thing and that's, that's the value of X and

then the reach we have.

- And to be honest, right, like even at Google, Google is

one entity, but so many different people, right? And there is a ton

of Google engineers we work with

all the time, and even like Google from YouTube to Chrome to

Chrome Media to the rest of Google, those are very different types of entities. But

what we do is efficient. And, for example for,

for Theo, right? It went a bit too far. I had him... Like I calmed everyone

down. I had him on the phone. We said, "Okay, like this goes too far," and so on.

But in the end yeah, it's a rap battle, but it's positive for the

project. It, like the awareness we have on open source and,

and I mean true open source from communities right now is

increased dramatically in the last two years, and this is useful.

- Uh, what do you think motivates all the incredible contributors that we've been

talking about? Like, what's the engine? It's so interesting to see.

- So-

- Like you said, they're sitting in the basement. What's the driver? What's the engine there?

- There are many drivers, but weirdly the main one

is that what we do in multimedia plays videos, and video is

cool, right? And, and for example, we have so many

people in the community who arrive because they loved watching

anime, right? And this is like the advice when people ask me,

"What should I work on in open source? How do I start?" And my answer is always the same:

work on something you love.

I am working on VLC because I love movies, right? And I love

watching the same movies over and over, even if my wife hates me when I do

that, right? But because it's interesting, right? Because it's a topic that

you like, right? The first, that's the first thing where people come to

usually to VLC and FFmpeg. The second thing is that

technically we, because we search for excellence, this is

the best school ever, right? This is the best school

ever of programming. If you're good in C, in

FFmpeg, if you know how to write assembly, I assure you you're going to

be one of the best programmers ever, even if you're working on writing

TypeScript, because this is the most amazing thing to

do. And you will, like, have to get reviews by some of

the most seasoned programmers ever who are going to look at

every part of your code and tell you why it's not great. It's like we are the

best teachers that you've ever had in programming, right?

- Andrew Kelley started Zig. He was an FFmpeg developer and started Zig after

his FFmpeg school. I mean,

it's the place to learn so many aspects of programming in the

real world, in a thing used by billions of people. You have nowhere to

hide. You have to be open and honest about your flaws and how you can learn

and be better.

- And what is also interesting in multimedia is

that you have 16 milliseconds to display a frame. It's not like

a game engine where you can basically slow down and wait a frame.

Like, so it's, you need to be good, right? There is no choice, else you don't have your

video. And because of how codecs, if you miss a frame, you're going

to destroy the look of the video, right? So you need to be good.

You need to be perfect to have the right thing. But also

is that it's not just pure programming in the mathematical sense, right?

A lot of people don't understand, but

in order to program correctly on the open source multimedia community,

you need to understand how computers work. And when you write assembly,

you need to understand about CPU pipelining, right? You need to

understand how SIMD works, how the ALU works, right? You need to

understand what, how IO works, right? And this is what I

think that is missing to a lot of engineers and software engineers today, is

understanding what we call computer architecture. And,

like, seriously, like some of the debates is like, should we use this assembly call or this

one? And people say, "Well, no, it's going to be like three cycles on this

type of CPU and this one," and has massive impact on the output, right?

- We should expand. FFmpeg is probably one of the biggest CPU users in the world. There's

it's probably running— ... as we speak

easily 100 mil- order of magnitude 100 million, maybe even a billion CPUs

as we speak. So every instruction matters. There's not...

The impact, at least in terms of CPU, is massive for everything that we do.

- So first you come because it's an interesting subject, then you stay because it's

excellent, and in the end you're very proud of it because

it's in the hands of everyone. Like so many

people like, "Oh, I'm working for whatever consulting company and I'm doing some

portal to download invoices for your PG&E." Wow,

great. Like, so many jobs are like that. You're not going to,

to tell that to your grandma. But if you go to see your grandma and say, "I do

this so that you can play video on your laptop," they understand. And this is very

important, right? Because you're working on VLC, FFmpeg, H.264. It's

in the hands of hundreds of millions of people and you have an impact.

And so you can be proud of yourself. And so I think that

in addition to doing a great resume, all those things are why people contribute.

- Yeah, those are side effects. My favorite quote on this topic is John Collison.

He said, "The world is a museum of passion projects." You know, everything out

there is a passion project. And open source multimedia and open source

in general, you can just do that so much faster. There's such a

faster network effect, you know?

I can open a cafe and that can be my passion project, but I have to get building codes, I have to build a

building, I have to find a location, I have to do all the, you know, all

sorts of things. Well, in the software world, that passion project can be,

can move quickly, it can be amplified by the network effect,

and that amplification can be more than the sum of the

parts. You know, you can be, you can find people

interested in extremely obscure things

and have a network effect and make something that is truly amazing.

- And on that topic of passion projects Tim Sweeney actually said in a reply

to a tweet that was complimenting JB. He said,

quote, "Many things in the world only happen because an awesome

person decides to do it. This is the case with VLC." And

that speaks to something interesting to me, that it does seem that a small number of

people, sometimes one person, can create

something incredible in the software world. Like you said this over and over and

over. I think JavaScript is an incredible thing created by,

Initially a single person. Some of the programming languages like

Python and C and Java, like just one person has this

vision, has this design, and brings it sometimes over a weekend is the

initial spark.

- Yes, Linus built Git in two weeks. Wow.

- It changed the world, Git. I mean, it really changed the world.

- Linus' passion project. "Hey, I'm uploading this tarball to an FTP, like deal with it."

- But for me, it's not just in software, right? And I believe in,

in individuals that are going to change the world, right? And it's with

a good, as you said, vision, right? I want to do that. It

is useful, it will be useful. And whether it's going to like build

train or cars or rockets or something like, I believe

people who believe in themself and have a vision can have a huge

impact for humanity.

- Let's actually zoom out before we zoom back in. We'll just keep

going up and down the stack. So you know, we've been

talking back and forth VLC and FFmpeg. Kieran, you said that

FFmpeg and VideoLAN, VLC coexist,

and there's no central point of importance. It's a kind of

what you call the binary star system.

Uh, they succeed because of each other. Can you explain the difference, how

they interact? What is the- ... are they competitors?

- I don't think they're competitors. I think the simple answer is, the

short answer before I go into detail is VLC is to FFmpeg as Android is to Linux.

So they depend on each other, but they coexist because of each other. So they

are a binary star system is the analogy I used.

- By the way, I feel horrible that I just recently learned that Alpha

Centauri, the closest star system to us, is a triple star system.

- And when you start doing the physics, it's a nightmare, right? But, but, but like-

- Hence the three body problem. But anyway. So a lot of

FFmpeg pipelines involve the x264 project, which is

a VideoLAN project. I would put a finger in the air and say

80-plus percent of those pipelines are dependent on a VideoLAN project.

VLC, obviously, as we've discussed, a VideoLAN project, uses

FFmpeg, gives it reach, exposure to weird files,

Historically used some donation money to fund FFmpeg

development, and we'll talk a bit maybe about some of the reverse engineering later.

So it's a binary star system. They work and feed off each other. Many of the developers are

shared. There's no central location. It's a virtuous cycle working together.

- And we should mention that x264 is the encoder for H.264 video standard. So

H.264 is the standard. X264-

- Is the open source implementation of the standard

- ... that's used by basically everybody- ... for everything.

It's, that is the main driver of this. When you think of an MP4

file that has H.264 codec in it-

- If it came from a software environment, like a data center or somewhere,

the chances are it was created with x264.

- And that's under the flag of VideoLAN.

- That's a VideoLAN project. So in the VideoLAN graphic, it sits in the VideoLAN

world.

- And VideoLAN has a, says a bunch of stuff in it. Go to the

VideoLAN website, there's a bunch of icons.

- Like if you look, there is so many libraries, right?

- libdvdcss- ... libdvdnav, libdvdpsi, libvlc of course, vlc-unity, libblu-

Blu-ray. Uh, yeah, there's many more.

- And there is so many more, right?

Lately, lately the dav1d project that we might talk about is the

last project from VideoLAN. It's everywhere, right? And we do,

we have a libspatialaudio lately that we announced. We have a-

- checkasm.

- checkasm-

- We'll talk about that later

- ... which is like an insane project- ... but amazing. So and x264 is one of

those VideoLAN projects. And my opinion, for example, is that

x264 was, is the most amazing encoder ever designed,

and this helped the adoption of FFmpeg. A lot of people and large

companies went through FFmpeg because they wanted to use

x264, and x264 increased the popularity of FFmpeg. But

also VLC had its popularity because

it's played so many files that were done by FFmpeg, right? So it's,

it's many projects that are intertwined and work together.

- Yeah. Unfortunately, there's a, there's a thing on X where VLC is mentioned and

there's people, "A quick reminder that it's FFmpeg inside doing the actual work." And that, and

that's like I said, it's not, that's not the case. We work together.

- And to give you an idea, right? When I compiled VLC for Windows, I compiled

around 16 million lines of code, right? One

million of those are inside the VLC repository, and FFmpeg in total is probably two,

around two, right? But so it means that so many dependencies are outside. And

if you also look at FFmpeg per se, FFmpeg also is

integrating third-party libraries like x264, but

LibOpus and so many others, right? So we all depend on each other.

- Uh, yeah, that's why I was hoping to do this episode as we are doing that

just kind of joins FFmpeg and VLC-

... because it's really, it's, it's really two of the same, like you said,

binary star system and we're all just orbiting it. Can we give a shout-out

to some of the people along the way? We didn't really quite talk

about the history of FFmpeg, so-

maybe can you tell me about Fabrice? Can you tell me about

Michael Niedermayer? Can you tell me about some of the key figures here?

- Let's just talk about the eras of FFmpeg, because there's

key eras and key people that made this

possible. Uh, Fabrice Bellard, as you mentioned, creating the

concept, and then probably in the 2000 era, I would call the era,

eras tour of FFmpeg is the 2000 era was Michael Niedermayer. So

key things he got done was exhaustive support for DivX and

Xvid at the time, and all sorts of weird variants of what's known

as MPEG-4 Part 2. So this predates the

MPEG-4 Part 10 that we're used to. So this was 2000 era video

codecs where there were, oh, flavor after flavor of weird decoders.

At the time in the 2000s, you needed a new player to play every different type of file

format. So there was Windows Media Player to play Windows Media formats. There

was RealPlayer to play RealMedia formats. And those were the other, the other

key thing in FFmpeg at the time were native decoders for those. I actually do remember

being a teenager, I must have been,

figuring out there was this one player that could play,

could decode these files without having separate bloated players. Because

at the time when you downloaded RealPlayer, there was a ton of other stuff in there, a ton of ads, a

ton of other things, and just having a simple library

that was fast led to that. And then I think 2008 was a, 2008 onwards

was a big change because that's when H.264 got its maturity and I think

something hopefully we'll talk about a bit more. This was the beginning of high definition video.

So H.264 was the key decoder of that.

So I'd call that the late 2000s and 2010s, and that's when the big reverse

engineers came along and really did astonishing work.

The beginning was a single player that could play Xvid, DivX,

Windows Media, and RealPlayer was already a massive achievement in itself without codec

packs, without weird stuff you had to download that had weird ads and weird spyware.

- VLC 1.0 was out on those times, 2000, 2009, 2010.

And this is like where it exploded.

- Yeah, without codec packs, it just works- ... across all these different-

- It, de facto, it's just like all the codec packs are FFmpeg inside VLC,

plus we have other modules for all the type of codecs.

- But back at the time that wasn't, is there were weird, in the 2000s, there were weird

codec packs with DLLs coming from this place, DLLs coming that-

- With a lot of spyware

- ... with spyware, with you know what. It wasn't reliable, you didn't know, and having a single

player that was open source or single playback module/player that

could do this that was open source. But I think the thing to emphasize is

this task in the 2000s that Michael did was Sisyphean. It was really,

the number of edge cases are poor beyond comprehension in terms of

you could have a Chinese CCTV system that did one weird variant

of MPEG-4 Part 2, what's known as MPEG-4 ASP,

and that was a weird variant, and you had to fix that without breaking everybody else-

... times a million.

- So that's, so you said that's where a lot of the reverse engineering was happening.

- It started in the 2000s with the Windows Media stuff because that was-

... proprietary. It started with the RealMedia, so with Benjamin Larsson.

- Kostya Shishkov.

- Kostya Shishkov, that era. Those were the key, that was the key

groundwork. And then in the 2010s was kind of

the Paul Mahol, Kostya era building, doing some of the

most difficult codecs. JB maybe can talk about GoToMeeting 4 and

GoToMeeting 5, and-

- What? What's the GoToMeeting?

- So, like, let's talk about this

amazing Ukrainian guy called Kostya, who was at that time living in Germany, and who

was in love with Sweden, right? He— And the guy was the most... He's like,

like a lot of the people in the community are very clever. He's

one of those who are, like, borderline geniuses, right?

He was able to reverse engineer extremely complex codecs,

And he does that, and we do a bit of engineers with

Kieran, but clearly not at this level.

- No, no, yeah.

- Um, he reverse engineered binary blobs, which are 20 megabytes?

- Yeah, so just for reference, one megabyte binary blob to reverse

engineer is probably order of magnitude a month of work, and this guy is doing

20, 30 megabyte blobs. Maybe we'll talk about that in a minute, about the

subtleties of how you do that. But this guy is doing it for very difficult and very obscure

codecs.

- And did that for fun, right? And so GoToMeeting was a big problem with VLC

because that was like the number one

feature request for a long time, so I put a bounty. And the guy at some point said,

"Okay, JB, I'm going to do it." And in a matter of two

months, and then he explained how he did it. He was just like, "Oh, I looked at the code,

like this looked like a DCTs that I used to see on WMV

and so on." He did that, and the funniest part is that

the code he's written is a ton of jokes. And

there is, there is a ton of JB, right, my name, and, and

Kempf and Kempf and Kostya jokes inside the code. The code is beautiful, right?

- So one of the things I wanna comment is I've gotten a chance to

speak to some of the developers, some of the assembly language level

People, and they all always make everything sound like it's kinda easy.

There's a kind of humility because, maybe

just the level of what's required to do this stuff is so

high that everything else seems easy, I guess is the lesson to take away from that.

- So in the community, like some of the most impressive people are the ones doing

reverse engineering- ... and the other ones doing the assembly folds, right?

Um, and both of those type of people are

amazing. x264, for example, became amazing because, of a guy called Loren Merritt—

... who is, was from University of Washington, I think.

- At the time, yeah.

- And who was, like, who made everything great and fast doing a ton of assembly.

Um yeah. So this is like the

the golden era, I guess, where so many things got done.

- So, yeah, if you look at Kostya, for example, he looked at the world as a binary specification.

He didn't need documentation or anything. It's, "I have a binary and I can

figure all of this out." And he regularly used the phrase binary

specification. "Ah, you know, it's not a problem." And he would go away, and he would come back, and

he would do interesting stuff.

- Can you actually speak to the details or add color and

texture to what it takes to reverse engineer a blob?

- Yeah. So let's look at GoToMeeting, for example, is a good one because,

um, I record a meeting on GoToMeeting, for example.

How do I play it back without needing this GoToMeeting

player? There may not even be a player. I may, I may need to send a recording of a meeting to

someone that doesn't have a player or whatever.

So first of all, there's a ton of other stuff there. There's an actual video

conferencing client. You need to go and find, it may be easy, it may not be easy to find the actual

module doing the decompression. You need a way to

actually dump the YUV data from the module. So often it

involves opening in a disassembler, trying to guess where the hooks are

to incorporate that module and run that module natively

to decode a sample file. So figure out where this module is doing

the decoding process and find a way to hook in and

output the raw YUV data, 'cause you will need that-

... as a point of comparison for when you actually do the reverse engineering, 'cause you'll need to be bit

exact or in some cases close to bit exact. And then you open up your disassembler,

use a lot of intuition to go and figure out, you know, where the DCT is,

where's entropy coding. There, there is a kind of,

not a rule book, but there's always a pattern of some sort. For example,

GoToMeeting, you know it will be a s- a lot of screen codec tools.

There's also different variants, so often I think there's, what, GoToMeeting 4, 5-

- Well, 2 or 3, 4, I think.

- 2, 3, 4.

- So as you mentioned here, going to Perplexity, GoToMeeting uses its own

proprietary codec for older s- recorded sessions historically stored in

WMV files that require a special decoder to play properly on

Windows. Without this decoder installed, Windows Media

Player and some editors cannot decode the video track, so you

may only hear audio or see a black screen. Boy, do I remember

that. But this is reverse engineering that.

- This is key, right? Because the GoToMeeting is something that not many people know

anymore, right? Well, you know about Zoom and, and Teams and so on. But like, now

let's fast-forward 10 years, 15 years, and like this is a

GoToMeeting.exe for Windows 32 bits, right? Which is like, oh

yeah, but I'm on Android, I'm on an iPad, I'm somewhere else, right? How are you

going to do that? I'm going to be on RISC-V, on Arm. Those are blocked,

but there are tons of files we need support for the future.

And this is why those type of work are— exceptionally useful for humanity.

- I just have to say, though, that reverse engineering process is mind-blowing.

It's crazy. It's like,

it's a kinda like, you know, I've been reading a lot and interview

archeologists. I mean, you just have so little signal.

Yes, yes, you know over time you get so much experience, you

understand the structure of the original code, so you can kinda start inferring

basics. But you're like-And we like archaeologists with a

little brush trying to reconstruct the entire human civilization

- Kieran is too humble, but Kieran has done some reverse engineering also.

- Of CineForm, yeah, at the time, um-

- CineForm, nice

- ... yeah, at the time before actually led to the open sourcing of that work. Um,

so in parallel to doing the binary side, you obviously have samples. In many

cases, you don't have many samples so you have to figure out what all the

different flavors are, and you may have a s- So CineForm, for example, is actually a

collection of different approaches and toolkits within that codec 'cause often

it grows naturally. And the hard part is finding a sample that gets you kind of

somewhere to start without having to implement 10 different other things. So

start there.

I think thankfully at the time I found a sample by pure chance that had a lot of flat blocks.

It was animation, so that really helped a lot because

it wasn't using particularly complex coding tools, et cetera, and you could kind of get

somewhere and then, and then build up and build up until you figure, "Hey, here's a few

bits here. I missed this. I missed this, this if branch that it does," and go, "Oh." So when we

say samples, you mean sample videos-

... and then, and then you're tracking, trying to infer, like, what is this

codec doing- ... by observing the sample and then looking at what, at

the lo- at, at the machine lo-

- The machine code saying-

- At the machine code

- ... "Ah, I have byte, this byte is six. Take this branch." And in a different

sample, oh, it's-

- That's nuts, man.

- And-

- That is nuts

- ... so you see, this is nuts. Then you go to things like GoToMeeting.

- Yeah, yeah.

- It's like-

- Mine was easy, right?

- ... imagine-

- Yeah, right

- ... two order of magnitude of more complexity. A guy alone

somewhere in Germany doing that.

And for a long time, you work, you're in a black box because a decoder,

for a long time, because there is so many steps from the entropy decoding, the intra

prediction, the motion prediction, the IDCT, and so on. For a long

time, you don't see anything, right? So you're debugging purely in memory.

- Debugging guesswork.

- And you may have the buffer that the coefficients are stored in completely wrong,

and so you may be going down a complete rabbit hole thinking it's this and then,

oh damn, that's not, that's, that's something else, and-

- And you're doing that on binaries that are tens of megabytes,

millions of instructions, right?

- So you're stepping through the debugger,

like one by one, you know, instruction by instruction going, "Hey, this instruction changes this.

This does this."

- Pausing the program on the CPU level. Like it's-

- Pausing it, yeah, on the CPU level, watching what's going on, trying to figure out-

- Sometimes you need to, like, be in a VM, so yeah, that you can pause the VM.

- Yeah, pause the VM, dump the memory, 'cause there could, some of the codecs could have encryption.

There could be like a DRM on there. So you need to dump the

memory from a virtual machine.

- Like when I joined École Centrale Paris in 2003, Jon Lech

Johansen basically broke the DVD specification and created

DeCSS, showed us how he was breaking a

DRM, which was MP4 FairPlay from Apple. What he did

on his laptop, and I was young, I was 21, was just like mind-blowing

because he was basically debugging Windows inside a type of VM

with ex- Like, wow. It's incredible. It's

mind-blowing and inspiring. Does it get, like from your

experience and from what you've seen in the community, does it get discouraging?

Does it get-

- People help you. People send you samples. People are keen. Sometimes you don't

have access to an encoder, so

this is even more difficult because you just, you just ask and you have to

ask for samples. I remember VideoLAN used to tweet for samples at one

stage. "Hey, I need this obscure sample," and-

- For a long time I was, "Oh, I need this codec, and I need this codec."

- And if you were really lucky, you would find like... If you were unlucky, you'd get

like one or- you'd get nothing or you'd get one or two, and then they would... Sometimes you'd find a

goldmine. It's like, "Yeah, my company has 100,000 of these files 'cause we're dependent on it for

some reas-" Um, and so those are the, those are kind of the best if, if...

Because then they can test bit exactness across the huge range of coding tools.

- Can you explain bit exactness?

- Bit exactness, so most but not all video

codecs, certainly from about the 2000s onwards,

have a bit exact definition, so every implementation must produce

exactly the same bits, bit for bit, in exactly

the same data that comes out of a decoder.

- For like a large number of samples?

- For a given sample. So Lex's implementation, JB's implementation, and my

implementation of H.264 must match bit exactly. That

wasn't the case in the '90s of MPEG-2, probably fair to say

one of the biggest mistakes the video industry made, and I think people who were in the

room in '92, I don't think, most or both of us were in diapers, I suspect- ... but

have acknowledged I would give a shout-out to Yuri Reznik.

He's acknowledged that was one of the big mistakes of the era.

- And you're saying the encoders needed to be able to run tests and then the bit exactness.

the bit exactness. I mean, that's a nice thing to guarantee.

Like there's a parallel sort of development here on the way the web

browser works, which is a, you know, takes HTML and displays

it, and there's no bit exactness there across the different engines.

- I would point out actually FFmpeg is unique in the sense that it's,

it has been a winner-takes-all scenario. You have... Browsers is a good analogy

because it has to parse a lot of different content and render it in a particular way, like a

decoder.

But there still are multiple browser engines. There's Firefox's one, there's Chrome's one, there's a

few Japanese ones that are pretty decent.

That's not been the case in multimedia in general across a

wide range of codecs. FFmpeg has kind of

won it all, I suppose, in a sense because of, because of the fact that you can get

every new codec added is actually worth more than the value of that codec itself

because it makes the whole thing better.

- Man, this is really cool. Going to Perplexity. Yuriy Reznik is a

multimedia and signal processing researcher, got his PhD in computer science

from Kyiv University with over 150 papers and more than 80

granted US patents, contributor to major multimedia standards

including H.264, MPEG-4, AVC-H.265 MPEG-4 ALS, G.718, and-

- G.71 is telco stuff. Telco.

- Oh. And so he was more connected to companies.

- RealAudio, RealVideo, right? That was-

- Oh, yeah

- ... very important at that time

- ... Zencoder, Brightcove, Contex. This, man, I need to hang out with

Yuriy. He's legit. And he's like one of the most nice person-

- Slack guy, yeah

- ... ever, right? Like for example, for my for my

startup that I'm doing right now called Kyber, right? I met Yuriy because

I met him every year at the Mile High Video Conference, which is in Denver.

And he gave me like so much good ideas

and good things. He's like really amazing person.

- He tells us how, how, you know- How great it is to be, you know,

even know us. And then we just like, you know, you look at that and it's, I think it's the other

way around, Yuriy

- That reminds me of a thing that you mentioned to me about FATE testing and,

like, the insanely rigorous process that's used to

test everything that's incorporated into into FFmpeg. Can you take

me through the testing process?

- Yeah. So FFmpeg has a system called FATE, FFmpeg Automated Testing Environment.

Because FFmpeg runs on so many different OSs and can be compiled with so

many different compilers,

there's been a crazy number of configurations. So you can see the

absurd combination of compiler variants, operating system variants,

instruction sets. You can see at the top macOS has tons of different

variants because it has iOS, it has tvOS.

- Well, I'm looking at a page fate.ffmpeg.org.

81 minutes ago, 76 minutes ago, looking at the

different architectures, the operating systems, the different compilers, Apple Clang

version...

- Combinations are crazy.

- ... the combination is insane. RISC-

- So these are all run by volunteers, so these are all volunteer systems. The ones at the top,

for example, the Macs I host in my office, for example,

Host all sorts of different stuff. Other people host other things. So

it's really there to make sure... because FFmpeg does quite complex C

code, for example, you do have miscompilations. So the compiler

will sometimes compile C code incorrectly. For example, this

happens once in a while.

- Oh, there's like, there's a log of all the compilations.

- Yeah, log of all the compilations, all the tests. I think one of the other ones will show all the tests

passing.

- If you click, you can see all the tests- ... back. All tests successful.

- In logs test, yeah. So you see all those tests are passing

of all the different codecs, all the different filter transformations, all the-

The level of scale is quite crazy.

- Oh, that's nuts.

- On all the combinations. It's not just a matrix at this point. It's like a pivot table

of different combinations.

- That's nuts.

- And it's a key part of what we do because you may be able to test something

locally, you make a change, but actually that breaks

GCC version 11 on Mac or something like that, and you're

able to then fix that. We also have miscompilation, so the C code,

sometimes the compiler can have a bug in it where it creates the wrong output,

and that can have quite a big effect sometimes on a video because of the

way frames have dependencies. Even a small change in the output

can cascade to actually quite big glitches.

- You see PowerPC, you see RISC, you see ARM.

- There was PowerPC, there was RISC, there was weird stuff in the past like DEC Alpha.

There was—

- You see Visual Studio, different versions of Clang or GCC.

- Visual Studio, Intel compiler, Apple Clang, you name it.

- What are some of the pain points? Like maybe do you have

emotional triggers maybe nightmares about a

particular operating system, a particular container, codec combination of-

- I mean, for me, it's really easy because I have a day job. My company

builds... The company I started builds equipment for broadcasting sports matches

between TV stadiums and studios, for example.

We have to work with 10-bit video, and 10-bit video has a set of challenges that

you can't process 10-bit data natively on a CPU. So that means you

have to stick it in 16 bits. So that means you

have six wasted bits. So there's different packing formats to actually pack

the data more efficiently because when you send that over a network,

you lose... 'Cause you need to save that 40%. For example, on PCI

Express, you may only have bus bandwidth to do that. And so I think internally we

have about... Some are industry ones and some are internal to our

own hardware that we build. We have a, I think a 5 by 5 or 6 by 6 matrix of

every single format to every single other format conversion. In fact, one,

one of them I sent you, and they're all written in handwritten assembly, and they're all

written, and they all support different CPU generations.

So this is really traumatic, handling all these different combinations

times a million.

- By the way, the company you're talking about is Open Broadcast Systems.

- Yeah, so no, no relation to the free OBS streaming service.

But JB and I have started companies

broadly speaking around the FFmpeg VLC ethos, so that's really

low-level work. So in most companies, this wouldn't be written in

assembly. It would be accepted that C is fast.

As you can see from that, C is not fast.

- So here it says 62 times faster than C.

- Yeah. So it's taking those, the ethos of doing

low-level programming, real-time programming,

and using that for commercial applications, and JB and I have started companies around

that, in many cases hiring developers from the open source community

to use that ethos. And so that's a great example

of some of the things we're doing. In most companies, it would be, say, "Oh,

I'll write this in C and it's fast and we're done," but actually you can get a lot better.

- For me, like, some of the headaches we have is around some OS that are difficult to

support, right? Because if you look at VLC and thanks to FATE and FFmpeg, we run

on... The last version of VLC runs on Windows XP

and still run there and runs on Windows 11. We work on macOS 10.7 to the

latest macOS, whatever it is, right, 26.

We work on iOS since iOS 9, well, we are actually

iOS 26, right? We support many types of Linuxes, BSD, Solaris. The last

version still runs on OS/2, right? Like there is maybe 10

users of OS/2 in the world, and one of them is maintaining VLC. Then you

realize that this very small team around VLC and using FFmpeg

codecs and all the other ones support more

OSs than Microsoft or Google or Apple, and they have infinite amount of power and

resources. But for example, the worst is iOS.

For in order to build on iOS 9, we need to do some

very clever mixing of several versions of the Xcode IDE and SDK from Apple, from

several versions, and do a type of Frankenstein version of that

so that we can still support iOS 9, which is not supported at all by the

compiler of Apple in order to still run on Arm32 on iOS 9. And

you've seen on FATE that it was still supporting iOS 9, right? So,

My headaches are mostly related to the support

of so many OSs. And it's important because, like, we

receive so many people saying, "Hey, thank you. I still have my iPad 2 to

watch movies," and it still works on iOS 9, right?

And it's also an impact of, like, not forcing people to buy new

hardware when it works fine if you optimize it correctly. Which brings us

to what we were saying about assembly. It's also fighting, like,

the fact that you need to buy something new nonstop while you could optimize

more, which is a lost art.

- You gotta tell me about this lost art or this, uh-

... the carriers of the flame of assembly. What

what is, what is assembly? Why is it beautiful?

Why is it challenging? How does it work?

- So when you write assembly code, you write this using the instructions the

actual processor is using directly. So most of the time you would

write in a language, let's take C as a good example. The compiler would use that to

create assembly language and machine code

instructions for you based off your C code.

And there's a specific flavor of assembly that we use in FFmpeg that's called

SIMD, SIMD, single instruction, multiple data. So this means, for example,

say I want to add five to a number in

scalar assembly, so this is what's known as you work on an individual element.

So I wanna have a number of-- I have the number ten and I want to add five. I use the add

instruction, and I add five to ten, and I get 15. With SIMD, with

SIMD, I can have a whole vector of 16 different numbers. They could all be different.

If I want to add five to that, I can run one instruction,

and that one instruction sums all 16 elements. And that, as you can

imagine, lends itself very well to video. Video

is, you know, pixel grid, so I can perform operations on

multiple pixels at the same time. The key thing that we do differently in

FFmpeg is we don't use any abstractions or any major

abstractions on top of that. So there's a part of the world that uses what's

known as intrinsics. So these are C functions that behave very

similarly but not quite the same to writing assembly by hand.

So the registers that data is stored in

on the CPU, the compiler allocates those for you.