Hi, I'm Carl from the Internet of Bugs.

I've been making a living from writing

software

for 35 plus years now,

and I'm trying to do my part

to make the Internet a less buggy place.

So, today's books are "The World of Star Trek"

by "David Gerrold" and "Programming Language Primatics"

by Scott.

In February of 1967, a Star Trek episode first

aired.

That episode was called "Court Martial,"

and it was the source of confusion for me

for most of my life until very recently.

Spoilers for a 58-year-old Star Trek episode,

by the way.

Don't say I didn't warn you.

The episode kind of is a whodunit mystery.

The breakthrough that the characters have

to figure out the mystery revolves around the

way

that the computer works on the show.

In the episode, Mr. Spock discovers

that there's an error

in the central computer's chess program,

and because of that, he concludes

that the computer's memory has been tampered

with.

In this case, someone has altered the ship's

log,

effectively replacing a section of surveillance

video

with a fake.

When I was a kid, I loved computers,

reading sci-fi books, including Star Trek,

and also watching and rewatching a lot of Star

Trek reruns,

because we only had three TV channels back then,

and we didn't have internet,

and TikTok wouldn't be invented for decades,

and okay, I'm old, just leave me alone.

Anyway, this episode confused me as a teenager

because it did not make any sense

with respect to what I knew about how computers

worked.

Computers had chess programs, and computers had

logs,

and there was no world I understood

in which forging a log over here

would affect the chess program over there.

As I got older, and I learned about how TV

shows were written,

I decided that it must just be

that the people that worked on the episode

had no clue how computers worked,

so they just wrote the wrong thing.

As I got even older,

and I understood the timeline of how computers

were developed,

I realized that a lot of the things that I knew

to be true about how computers work

wouldn't necessarily have been true in the mid-1960s,

because a lot had changed in the decades

since that episode was written.

And then, as I got even older, much older,

things came full circle,

and here in the mid-2020s,

that episode finally makes sense.

In fact, it seems almost prophetic. Let me

explain.

There have been a lot of advances in

programming

for the last 60 years.

First off, there are a number

of formerly common programming techniques

that these days are Considered Harmful,

like GOTO and Global Variables.

And in order to avoid those harmful things,

and write better and safer programs,

we've had lots of advances in the science

and the art of programming,

and better programming languages and tools.

Let me give you a simplified case

for three of the biggest, most important ones.

One is variable scope,

when a variable can only affect things

within a certain area.

Two is types, more than memory types,

but like a "name" can't be assigned to an "address",

even though they're both strings.

And the third is error and exception handling

and checking.

So think about it.

Basic and assembly are really simple,

with a flat variable address space,

not a lot of structure,

basically no way of handling error built in.

With Pascal, C, we got functions,

return values, some sample types,

some simple types, some simple local variables.

C++, Java, Objective-C gave us objects,

where we could have variables private to

instances.

Public and private methods can define types.

For example,

if you accidentally confused an integer zip

code

with an integer phone number,

you could get a compiler error

if you tried to pass one to a function

expecting the other one.

Swift and Kotlin gave us optional or nullable

types

that cut down in a whole class of errors.

Go gave us conventions for error values,

getting returned alongside functions as result

variables.

But the algorithms in the heart of LLMs,

and neural nets in general,

don't use these new advances.

It's conceivable that going and manually

changing

some chatbot's weights to get it to misreport

some past event

might end up affecting how well the chatbot

plays chess.

And according to some of the @GothamChess

videos I've seen,

they definitely don't need anything

making chess harder for them.

I'll put a link to a couple of those videos

below,

if you're curious.

They're pretty hilarious.

An LLM has potentially trillions of parameters,

which are effectively just global variables,

all with the same scope, all with the same type.

We have no way of understanding

how changing any of them might impact any

others,

not to mention how,

and there's no inherent or internal error

handling

in the system.

Welcome back to the 1960s.

And this matters.

Like I said in my companion video on my main

channel,

LLMs are just software,

and they are subject to all the rules

and disadvantages of software.

And software gets complicated

and the bigger the software project,

the more complicated it gets.

Think about it.

Imagine you have 100,000 lines of code

and hundreds of global variables

with GOTO statements everywhere,

and you need to make a change.

You need to do two things.

First, you need to make the change you were

asked to make,

add the feature, fix the bug, whatever.

Second, you need to avoid causing any other

changes,

breakage, side effects, anything else in the

project.

That first part is relatively straightforward.

The second part... isn't.

The larger the project gets,

the more potential interactions

the rest of the project has

with the part that you're changing.

And the more difficult it is

to make sure you don't cause any new problems.

How many different things do you need to check

to make sure you didn't break anything?

Well, with 100 or so global variables,

you need to check 100 or so.

And what's worse, is assuming you to a perfect

job

with this task, and that's a big assumption,

then you just added one more thing you need to

check.

So when you need to add your next change,

you need to verify 100 or so plus one.

And then when after that, at 100 or so plus two,

and the longer you work on this project,

the harder you make things for yourself.

I lived through this before, and it was a

nightmare.

I did a couple of projects in the early 1990s

for a now-defunct retailer

whose point of sale system had been written

internally

using a version of the BASIC programming

language

with ISAM database extensions.

And it was more than 100,000 lines of BASIC

code,

complete with line numbers and GOTO statements

everywhere.

Changing anything was incredibly risky and took

forever.

Now, I can't prove the system contributed

to the company's inability to adapt to online

sales,

causing them to fail when the Internet took off,

but I wouldn't be surprised.

This situation is often referred to as "whack-a-mole,"

where every time you fix a bug,

you run the risk of creating a bug somewhere

else,

because of some interaction that you didn't

know about

or didn't fully understand.

It's named after the old arcade game

with the targets that pop up

that you try to hit with a hammer.

I talk about whack-a-mole a lot,

and I've seen it kill a bunch of projects.

And I even had a section on it

in the mobile app development book I wrote

more than 10 years ago.

This isn't a plug, by the way, although it is an

FAQ.

The book is called "App Accomplished."

It's permanently out of print,

if for no other reason than because mobile

development

isn't a hot topic anymore.

So one way that we as professional programmers

deal with this

is to try to limit the number of things

that can interact with each other.

Imagine now that same 100,000-line project,

but now it's partitioned into well-designed

modules

with mostly private variables

and a small number of well-defined public

methods.

Now you only have to worry about

what's in the module you're actually changing

and then double-checking some interactions

between the module you change

and others that touch the public methods that

it calls.

Still requires work,

but it's SO much easier than "everything is a

global" case.

But that's not really an option for neural

networks.

And knowing the problems at hundreds

of global variables cause, much less trillions.

My professional experience would lead me

to expect this to cause a problem.

And lo and behold, there turns out to be a

problem

called "negative transfer" that happens

when the training that you're doing now

has an adverse effect on training

you did previously on an unrelated task.

And when you try to retrain to fix the thing

that just broke,

you run the risk of a "negative transfer"

breaking something else in the network.

And that's pretty much the same symptoms as "whack-a-mole."

So my educated guess is that this "negative

transfer" problem

is the equivalent of the standard software

engineering

problem at least of what's called "whack-a-mole."

And the two things are connected.

And it sounds like this is a training problem,

and it is.

And that sounds bad, and it is.

But it's probably even worse than it sounds.

And not just because it can lead to the problem

called "catastrophic forgetting,"

which is not a good thing, although that

happens too.

There's a thing that defines human intelligence,

that the AI companies just haven't cracked yet.

And that's learning.

Humans have a very complicated system of short-term

memory

that gets saved into long-term memory as we

sleep,

or at least we're pretty sure that sleep is

involved.

There's a lot about how that works,

and we still don't completely understand.

The current crop of AIs are all frozen in time

at the end of their training.

I've made a video likening it to the character

from the Memento movie, and I've also heard it

compared to the character in a movie called "50 First

Dates,"

although I've never seen that one.

The AI companies seem to mostly avoid

discussing this topic,

but I think it's important.

In fact, I'm gonna go out on a limb here

and say that one of the requirements

for artificial general intelligence

is not to be as brain damaged as the Memento guy,

but feel free to disagree in the comments.

It seems to me that having some kind of working,

long-term memory is a minimum requirement

for anything claiming human-like intelligence,

much less super intelligence.

And that makes this problem,

whether you call it a "global variable,"

or "whack-a-mole", or "negative transfer,"

or "catastrophic forgetting," or whatever,

to be something that I expect will have to be

solved

before the AI companies' ambitions can be

realized.

And I haven't seen any real advancements on

that front

from any of the AI companies, or much of the

research.

Few AIs that have been released

have any kind of learning mechanisms,

and a lot of those have had to be quickly shut

down

after they learned the wrong thing.

Here's what I mean when I say these AI's are

just software.

The situations that cause things to go wrong

in business software or web software

also cause things to go wrong with these AIs.

It might not sound like the same kind of

problem,

but only because the industry goes to great

lengths

to use vocabulary associated with brains,

instead of with computers.

But they're the same problems

because it's the same mechanisms.

Software is software is software,

even if they try to call it something else.

This is one of the reasons that I've never

believed

the AI hype, because I've learned over the

decades

and many sleepless nights that software is

complicated,

and there are so many ways that software can go

wrong.

And getting it right is not as easy

as the AI companies are trying to make it seem.

And if you are one of the people that watch

this channel

because you are or want to be a part of the

software industry,

I hope that helps you see reasons

that your skills are more relevant in the

current era

than AI companies would like you

or the rest of the world to believe.

Because despite what the companies want you to

believe,

the skills that you've developed to understand

software

still apply in the AI and machine learning

space.

There's some extra stuff you're gonna need to

learn too,

but it's not a whole different science,

the way the AI industry wants you to believe

that they are.

Thanks for watching, let's be careful out there.