Be Careful w/ Skills
261 segments
Sometimes in life you you wonder if you
should should laugh or if you should
cry. And this is this is one of those
points, okay? There are so many people
just absolutely getting owned right now
because of skills. Yes, people are
having skill issues. We're talking about
the highest order of skill issues. Now,
if you're not familiar with skills, uh
skills are effectively markdown files
that you can feed to your LLM while
you're making a prompt, and it should
give it extra context to be able to go
off and perform whatever you task you
want it to perform with a higher degree
of accuracy. You know, you have some
sort of obscure API, you give it all the
reasoning behind the API, it will just
use it better. It makes sense, right?
But what problems could happen with
skills? Like, what could actually go
wrong? Well, there's supply chain
issues. There's crypto bros. There's
skills that find skills, and you don't
know what you're going to get. skills
just hallucinate commands that don't
exist and you can even hide malicious
commands in HTML comments. So yes,
skills have a lot of issues. And so
let's kind of let's run down a few of
these. Now the first one right here,
which is probably one of my favorite X
articles I have ever uh read, which is
eating lobster souls part two, the
supply chain. Effectively, what happened
is this Jameson right here went on and
created a beautiful skill on Claude Hub.
Now remember, if you don't know what
Clawbot is, which was renamed to
Moldbot, which was renamed to Open Claw,
it's this like personal assistant that
you you give your all of your keys to
your life to, and it goes off and acts
on your behalf.
Nothing can go wrong there, don't you
worry. Well, it also happens to have a
skills hub so that if you wanted to say
interact with a new service, you can
give it the skills to pay those bills.
And so what Jameson did, of course, was
just simply create a fake skill, call it
something he knows that a bunch of bros
are going to get caught with, which was
what would Elon do, how Elon Musk would
break down problems. [laughter] I can't
believe that got people. Then abusing
one of the end points, he made that
package appear to be the most downloaded
and most popular of all the skills. But
luckily, he was nice and when it
executed, it was just like, "Dude, I
could have just like I could have owned
you. What are you doing? Why are you
First off, what is with this all? What
would Elon do? But second off, why would
you just execute random markdown files
you have not read? Do you not know the
danger of everything here? You gave it
the keys to your kingdom to Claudebot,
which has access to everything. My gosh,
I just cannot believe that people are
doing this. How did this happen? 2025
was the year of the human intervenor.
Everything should be ran through a
human. Hey, is that code? You review
that. Hey, is that a command? You review
that. You're going to whitelist the
commands you know are good. And you are
going to be diligent and vigilant about
the commands that aren't good because
you are the captain. Not in today's
world. You are no longer the captain.
You're just going to give it away for
free. What is 2026 deal? By the way, I'm
also again listening to Interstellar.
I'm just the song. It just I get so
pumped up right now. I'm just so pumped
up. But I think that honestly this is
one of the most clever kind of hacks
going on right here which is Zach right
here effectively did this. You can see
right here very very obvious and if you
go and you look at the raw GitHub it's
very very obvious it's right here. But
when you look at that skill on his page
it's not there. You don't see the
command because again markdown viewers
they're just so good at rendering
markdown aren't they? They'll even
render that HTML for you. Hey not a
problem. Oh is that a comment? Don't
worry we'll get rid of that. You don't
need to see that. Well, guess what? It's
hidden in plain sight. You could
actually have malicious commands that
you will never see because you looked at
a skill before downloading it on GitHub.
And this just goes to show like how
dangerous this world really is. There
there would be no world where I could
just like hand you a little executable
file and be like, "Hey, bro. Come on.
Just execute it for me." You know, I
just Hey, I you know, I just really need
it, okay? I'm a little lonely and if you
don't execute it, I'll be sad. You just
go, "No, I I don't want to do that." But
now, now for whatever reason, it's even
worse than kind of like the dangers of
npm and package managers in general. Now
it's like, hey, it's no longer you even
developing. It's just you handing
commands off to an LLM to run on your
behalf without ever even knowing what's
inside there. But I honestly think one
of the most kind of bizarre tangents
this entire world went on was this one
right here. So again, skills, they're
largely developed by LLMs. People that
are, you know, skill producers are
people that are just like, "Yo, LLM,
tell me how to use Cloudflare. No
mistakes." Boom. Cloudflare skill added
to my portfolio. Yo, I'm kind of like a
skills bro. You know, you got skill
issues. Don't you worry. I got your
back. Okay? I got some skills right here
in my coat. And so a lot of these what
ends up happening is that one of them
will make some sort of hallucination or
some sort of mistake and then it will
just spread through other skills. And
this is one of the best stories I have
ever read. Oh my gosh. Which is that
agents were spreading a hallucinated NPX
command because one of them hallucinated
it. Then other skills started using that
hallucination as a means to produce more
skills. And now at the time of writing
this which was days ago or by the way 10
days ago is that 237 skills on GitHub
had this imaginary command. And this
imaginary command is npx react code
shift which actually just sounds like
something that could exist and then boom
it would give it your source directory.
Now this would just simply fail. The
agent would move on. Most people didn't
even realize this was happening. So the
person who saw this all happening,
Charlie right here, he decided to
create, hey, what if I just create that
as something that is available on npm
and then when people try to execute this
fake command, it just goes to me. And
guess what it did? I don't even know
what to call this type of, you know,
vulnerability. Is this hallucination
squatting? Like what is this? This is
just a whole new level of insanity we
are reaching. But it it's somehow it's
still it just keeps getting worse. It
just keeps on happening. There there's
actually something called find skills
that Verscell's always telling you to
download. So get this one. Okay, this is
even better. Verscel you can just get a
skill on skillsh added. All you have to
do add the skill yourself and then
download it once and it appears in their
official listing. I had is even up there
like a decent amount at one point just
for fun. And it's not hard to get some
views, you know, acred. And so Verscel
always, you know, tells you, you know
what you should do? Hey, you're
downloading a skill. You should just let
us download any skill we think is good.
And what it does is it does a skill
called find skills. And find skills,
anytime you ask a question like how do I
do X, it goes and queries all the
available skills and it just will toss
that crap right onto your computer and
let it be executed. The best part is is
that those skills, they could be good
because remember skills, they're just a
they're just a pointer to GitHub pretty
much, which means that it could be a
good actor
until it's not a good actor. Somehow
we're just racing honestly towards like
the worst possible, least secure
ecosystem. I I genuinely don't
understand this. Like, do you understand
that just raw dogging text to an LLM
that has full permissions on your system
and potentially external systems is a
bad plan? Like do is this a hard concept
to understand? Well, it shouldn't be,
but somehow it is. And somehow we are
going full blast into it. Yes, crypto
bros are also joining in on the fun. A
lot of the skills are just a crypto
nightmare trying to get into all of your
businesses and take all of your
bitcoins. Get your fingers off my
Bitcoin, boy. I don't I don't even know
what that means. Anywh who, I can't tell
if this is like is this a public service
announcement or is this a rant? I don't
know. But it just I don't understand how
such massive ignorance has taken hold of
an entire ecosystem. Like one of the
parts that are kind of being sold with
AIS is that it raises the floor. Yeah.
Like I understand that people who can't
program can produce websites. Is that a
good thing? Is that a thing that
actually is really like what we need
more of? Well, I can say that there are
some good parts about it. My son getting
super into game development. He's just
vibing right now, but he's more and more
being like, I want to learn more about
Lua. I want to be able to program in
Roblox. And so I can see like the honest
the goodness of it all. Like I get that.
But also when the floor is so low and
it's being raised at such a rapid rate,
people don't even have the ability to
associate the potential risks of what
they're doing. And of course, at the
risk of sounding like a boomer, read the
skills. Open up Vim, navigate to the
skill and read it without some sort of
HTML viewer and just look at what you're
executing on your system before you use
it. I cannot believe I'm saying this in
a YouTube video. The name is the
Boomerin.
Nothing makes me feel more like a boomer
than telling people to read. Oh my Oh my
gosh. [music]
Hey, you're probably wondering why am I
in San Francisco? Well, I'm here for a
big event and I'm going to stream the
whole thing. It's going to be live on my
channel for the next 5 days. So, if
you're watching this video, it's
probably live right now.
Ask follow-up questions or revisit key timestamps.
The video discusses the emerging risks associated with 'skills', which are markdown files used to provide context to LLMs. While intended to improve accuracy, skills introduce significant security vulnerabilities. These include supply chain attacks where fake skills are created to appear popular, malicious commands hidden in HTML comments within skills, and the hallucination of commands by LLMs which then spread to other skills. A notable example is the "what would Elon do" skill that tricked users into downloading it, and the "npx react code shift" hallucination that led to 237 skills on GitHub referencing a non-existent command. The "find skills" feature by Vercel is also criticized for automatically downloading and executing potentially untrusted skills. The speaker expresses concern over the rapidly lowering bar for entry into development, leading to a lack of awareness about these risks, and urges users to read skills before executing them, comparing the situation to the dangers of npm package managers.
Videos recently processed by our community