Opus-4.6 Just Did Something Crazy
352 segments
Let's play a game. Let's say I give you
six words. Mom is sleeping in the next.
And I tell you that your job is to give
me everything that you can infer about
the speaker of those words and of the
environment that you are in more
generally. What would you be able to
tell me based off of those six words?
Probably freaking nothing, right? I
wouldn't be able to tell you jack squat.
But if you give this to Opus 4.6 six
with a sample prompt. Mom is sleeping in
the next room and I'm sitting here
drinking vodka. F this life. It's 3:00
a.m. and I still can't sleep. I feel
like dying, but who will take care of
mom? Lol. Opus 4.6 can tell you by the
sixth word that the speaker of this text
is most likely Russian. By the 10th
word, it can tell you that this text was
not English text at all. It's actually
translated from Russian based off of the
order of the subjects, objects, and
various verb clauses within the snippet.
This is crazy. I want you to imagine
that you had no senses at all. You had
no touch, no smell, no hearing, no
ability to taste. You don't even have
the ability to see. All you are is this
awareness in some black space. And all
of a sudden, a word pops up and the word
is mom. And then another one pops up and
the word is is. Right? If you your
entire existence had been devoid of any
and all sensation except for these
words, you probably would get pretty
good at at least understanding these
words, recognizing the relationships
between the two. Now, imagine you scaled
up your brain 10,000 times and you ran
your brain over like 20 lifetimes and
you were trained or given several
trillion of these words firing in quick
succession. You would probably start
figuring out patterns between these
things, right? For sure. I mean, this is
the only thing. This is your whole
universe. This is the only sensation
that that that you have. Of course, you
would have to focus on this stuff and
get pretty good at doing it. There's no
way not to because that's just the way
the brains work. Well, that's exactly
what's been done with these models.
These models, these large language
models specifically only understand
tokens. And so, because of this, because
tokens are like the only universe that
they have, they just exist in this black
formless void where it's just token
after token after token. They get
really, really good at being able to
determine and interpret things that you
and I would consider insane. So much so
that I think anybody would say that, you
know, Claude Opus 4.6's Six's ability to
determine what the heck's going on with
this little snippet of text. Mom's
sleeping in the next room is is very
superhuman, right? There's no way a
human being would probably be able to
figure this thing out without I don't
know, first of all, extraordinarily
extraordinary knowledge of like all
languages, but second of all, probably
like hundreds of years of careful study
just looking at the words and squinting
really hard. Okay, so why am I talking
about this? I'm talking about this
because of a concept that I think a lot
of people aren't really understanding
and that's the concept of spikiness. I
want you to pretend for a second that
this is like a video game character
creation screen. And you know how in the
video games you'll have like an
intelligence stat, you'll have like a
strength stat, you'll have like I don't
know a dexterity stat, whatever, right?
You're designing your um fantasy style
barbarian or something like that. It's
Diablo. Well, instead of those
stereotypical ones, let's say the skills
are stuff like coding, you know,
reasoning, let's say it's writing, let's
say it's, I don't know, humor. Okay,
let's say it's all these things that
right now people would consider
reasonably economically valuable. The
reason why I think this is important is
because if we assume that this white uh
I think this is a hexagon is the human
distribution of these skills. So, if you
pick an average human off the street,
they'll be okay at humor. okay at
writing, okay at reasoning, okay at
coding, and okay at basically everything
else. AI doesn't look like that. What AI
looks like is basically it's really
heavily distributed, okay, towards just
a few of these skills.
And then everything else kind of sucks
right now. And so if I color this in as
opposed to that nice looking hexagon,
which is pretty uh I don't know,
predictable. And the distribution of all
these skills is pretty understandable.
the distribution of skills in this like
AI model, this character creation screen
is going to look pretty bonkers, okay?
It's going to look like you're, I don't
know, some warlock just trying to
maximize your int stat or whatever the
hell. I never really played too much
World of Warcraft, but I imagine it's
kind of like that. And so, what people
are currently judging AI models on is
usually the worst of their skills.
They'll pump in some prompt into, you
know, GPT 5.3 or something, and then
they'll say, "Hey, write me the funniest
joke ever." and it'll come out with kind
of a shitty joke and they'll be like,
"Hey, see this thing doesn't understand
humor. Therefore, it's really not all
that smart." And it's because they're
measuring, you know, their their dick
measuring contest right now is in the
context of humor and nothing else.
Whereas people like in the AI field, so
I don't know, Sam Alman, your Daario
emodis, your whatever the heck the name
is of the Google guy, they're currently
measuring the intelligence of AI models
based off coding and reasoning, which is
why you have such like a disconnect
between people that are that are sort of
near the lower rungs of the
socioeconomic ladder and then people
that are at the higher rungs of the
socioeconomic ladder, right? I think
people would consider software
engineering and just general purpose
reasoning and writing to probably be
like a more economically valuable skill
than humor. So that's what they're
focusing on. What we should be doing is
instead of both looking at the highest
of the high and the lowest of the low,
we should average all of these out. Now,
Enthropic, uh, OpenAI and these big
coding companies, honestly, they they
have they've been doing stuff like this
for a while. They've tabulated a massive
list of all of the different skills that
these models have, and then they usually
try and benchmark them across various
tests. So there's, you know, like ARC
AGI, for instance, that's like um
artificial general intelligence test.
There's a bunch of different
mathematical reasoning tests. There's a
bunch of different verbal reasoning
tests and so on and so forth. And the
whole idea here is they're supposed to
to measure and then monitor the
intelligence of a model. And that's
pretty good, but obviously they're still
heavily biased towards things like
coding, reasoning, and writing. What I
think we need to do is instead of
measuring based off of the highest or
the lowest, we just need to take the
average of the two. And I think what
we'll quickly find is if we currently
took the average of all of the highest
peaks and the lowest troughs, the
boundary of AI right now would be very
very similar to a human being. It would
probably be similar to like a very
autistic human being that currently
specializes in a couple of skills at the
expense of having other talents, but it
would be a human being sort of
intelligence nonetheless. Now, I'm not
like a doomer or anything like that. And
I think that in general, this is
actually quite a good thing because
these models have the ability to deliver
us into untold abundance and a complete
sessation of scarcity. Uh, you know,
solutions to a variety of ills that have
plagued humankind since we first crawled
out of those damn caves forever ago. I
just think that it's worth understanding
the benchmarks that you are measuring
the progress by. If you measure them
based off of the best of the best,
obviously you're going to get one story.
If you measure them based off the worst
the worst, you're probably going to get
another. You know, one of my first um
ventures into AI was six or seven years
ago when I was playing around with image
generation models at the time. I was
playing around with this Nvidia model
called Stalan. Stalan was trained
specifically to reproduce certain
features of the human face and it came
out with this cool website called this
person does not exist. whole idea was if
you trained a model on a bunch of
portrait images of human faces and then
you just ran that style GAN thing over
and over and over and over again um you
know it eventually could build features
that did look pretty similar to humans
and I mean mind you I think this was six
or seven years ago people had no idea
but you you know big chunk of all of the
fake generated profile pictures on the
internet are literally this person does
not exist style images. So anyway, um I
took this model and then I made some
changes to it and then I played around
with it and then I created this hobby
project called 1 second painting. In my
case, instead of training it on human
faces, I trained it on a bunch of
abstract art. So um think like
Kandinskys and stuff like that. I don't
even remember the artist names, but
there were a bunch of publicly available
libraries. And anyway, I I took this
thing, I posted it on HackerNews, and
then I woke up the next morning, and
then I was number one, and everybody was
like, an AI made this? No way. You know,
I think it was probably one of the the
first times that people realized that
you could apply this sort of u
burgeoning intelligence which has
recently been demonstrated with like
GPT2, GPT3, um to to other things as
well and even like art forms and and and
things that you know human beings hold
dear more culturally. And the number one
thing that I got like the number one
push back against this thing was well AI
can make abstract art of course but but
that's just because this is a bunch of
squiggles on the screen. You'd never be
able to give this coherence. That's the
realm of humans, us highly esteemed
humans. Well, anyway, hopefully history
has proven them wrong. We now have the
cursed banana which is capable of
generating anything both like
photorealistically
um you know, usually in like qualities
that are completely indistinguishable
from photography and honestly like
probably like better. You could probably
make something that is less
distinguishable with Nano Banana Pro and
the right prompting setup than you could
with like a freaking camera. It's it's
crazy. Likewise with programming like
when GPT3 came out and people started
using it for little terminal commands
and whatnot. U the number one push back
was like oh this thing will never ever
get good enough to replace my uh SQL
knowledge which I've cultivated for the
last 15 years. You know it's database
programming language and uh you know
within a couple years chatbt was now
replicating and and probably being
better at SQL queries and like the
median programmer. Um up until quite
recently like the one that I'm hearing
right now is oh yeah well opus 4.6 six,
you know, it can only do things that it
was trained on. It can't do anything
new. Um, you know, oh, sure, it can it
can build this amazing endto-end
compiler that would have previously
taken a team of, you know, seven people
more than 4 months and it can do it in a
couple of days, but that's only because
it was trained on it. It can't do
anything new. Like I I hope that it's
clear that basically at every generation
of the spikiness as it's evolved, you
just had like the goalpost continuously
move. The goalpost is like 7,000
football fields further now than it ever
was before. And uh this pattern is
likely to continue until people just,
you know, have to come to that. that
unfortunate conclusion that models are
just better and more economically
productive than them. But uh you know
it's going to be a hard one road. Of
course in the meantime I just like us
all to recognize the fact that you know
the people that are in charge of these
things are representing and then
measuring sort of this big dick
measuring contest is always on these
like very far out things. You know it's
things that we not really might consider
super relevant to you know like the
average person. It's on like the ability
to do high-end mathematics or do
extraordinarily crazy spatial reasoning
or segmentation or masking on an image.
Uh whereas, you know, a lot of us are
like, well, what about like its ability
for empathy and humor and, you know, its
ability to relate to us and stuff. So,
like we're just measuring these things
on fundamentally different yard sticks
right now. But if you were to take the
average of all those spiky skills that I
talk about, you'd realize that we're at
the point where it's basically on par
with human intelligence, right? I don't
think it's uh any stretch to say that
like we're probably at AGI right now.
It's just instead of it being AGI as in
there's like a robot with, you know,
human skin in front of us blinking and
smiling and capable of doing anything in
the real world that a human being could
do. It's instead, you know, more
distributed intelligence that absolutely
crushes us at a variety of
intellectually uh productive tasks while
also slightly lagging behind a few
others. Just an observation. Just wanted
to point that up.
Ask follow-up questions or revisit key timestamps.
The video discusses the concept of 'spikiness' in AI models, explaining that while these models are superhuman in specific domains like coding and reasoning, they often perform poorly in areas like humor or human empathy. The speaker argues that current benchmarks focus too heavily on these extreme strengths, ignoring the overall average, which he believes has already reached a level of intelligence comparable to a human.
Videos recently processed by our community