This is ACM Bytecast, a podcast series

from the Association for Computing

[music] Machinery, the world's largest

education and scientific computing

society.

>> [music]

>> We talk to researchers, practitioners,

and innovators who are at the [music]

intersection of computing research and

practice.

They share their experiences, the

lessons they've learned, and their own

visions for the future of computing.

I am your host, Juan Miguel de Hoyo.

>> [music]

>> At its core, search is about helping

people find what they need in a world

overflowing with information.

So, it sounds simple, but it's actually

a deeply difficult problem.

How do you understand what someone is

looking for, sift through enormous

amounts of information, and decide what

is most useful,

relevant, and trustworthy in that

moment?

And once you think about it that way,

you realize that search is pretty much

everywhere. It's in the questions we

type into a search engine, the videos

and articles recommended to us, the

products we discover, the maps that

guide us, increasingly, the AI tools we

turn to for answers.

Behind all of these everyday moments is

a deep field of research called

information retrieval,

the science of helping people find the

right information at the right time from

an overwhelming amount of data.

Few people have helped shape that field

as deeply as Ricardo Baeza-Yates.

Ricardo is a Chilean computer scientist

whose work in information retrieval

algorithms,

web web search, and data mining has

influenced how modern search and

recommendation systems are built and

understood. He is currently the search

chief scientist at you.com

and holds part-time professor

appointments at KTH Royal Institute of

Technology in Sweden,

Universitat Pompeu Fabra in Spain,

and the Universidad de Chile.

His career has moved across academia,

industry, and entrepreneurship. He

previously served as vice president of

research at Yahoo Labs,

holds 14 patents, and has co-founded

several startups in Chile and Spain,

including Theodora AI,

which focuses on mitigating

technological bias.

He has earned his engineering degree and

master's degree in computer science and

electrical engineering from the

Universidad de Chile,

and his PhD in computer science from the

University of Waterloo.

But, Ricardo's impact goes beyond

technical foundations. He has spent his

career mentoring researchers, building

institutions, and helping expand the

visibility of the Latin American

computing community around the world.

And that combination of pioneering

research and commitment to lifting

others up is what has led to his

recognition with ACM Luis Andre Barroso

Award, which honors fundamental

contributions to computing from

researchers in historically

underrepresented communities.

So, it's my honor and real pleasure,

actually, to spend time with you again,

Ricardo. It's been quite a while. So,

>> Thank you. So, it's so good to see you

again.

>> Yeah, when was the last time we saw each

other?

Was it ITU?

>> In Geneva in in the AI event that they

do every year.

>> Oh, yeah, that's true.

>> Yes.

>> Yeah. Will you be going this year?

>> No, no, I only went that year when I met

you.

>> Oh, okay.

Yeah, that's a lucky year.

>> Yes.

>> Yeah. So, before we continue, I kind of

wanted to ask you. I know I we talked a

bit about your background, but would you

mind introducing yourself and talking

about what you currently doing?

>> Yes, so no problem. So, currently I'm

sharing my time between industry and

academia, mainly industry in in in

you.com, as you mentioned, in San

Francisco, but also I I do have these

part-time commitments in several places,

and physically I have to be a few months

in a Stockholm because of my KTH

commitment that was even before I

started working with you.com, and so I'm

right now in Stockholm

doing research on responsibly AI, so

mainly with my PhD students, mainly in

Barcelona,

and my work here in KTH,

touching different problems on

responsible AI. Maybe the main one is is

evaluation, and looking at how to

evaluate harm and not success.

Because things should work, right, by

default. I mean, why we evaluate when

things work? We need to think what

happens when things doesn't work, and

how much harm that creates because

all errors are not equal. So, there are

errors that are more harmful than

others. And we are looking at that, so

different ways to minimize critical

errors, and also to try to measure. And

because of the regulation of the use of

AI needs that, for example, the

EU regulation is based on harm and risk,

but really we don't know how to measure

risk too well to get.

>> I totally agree, and I think that's a

very fair point. I think one of the

challenges we're living in now,

currently in the space, is that both

the technology for discoverability is so

pervasive. Like search is just We take

it for granted that search did not exist

like decades ago.

And so, the work that you were doing in

that space is really vital, but at the

same time, that space is also evolving

with in conjunction with a lot of

different technologies, like machine

learning, and how that impacts our

ability to disseminate and understand

information. Before we go in further, I

think one thing I really want to ask is

that sometimes when we have audiences

listening to this podcast, they may not

necessarily completely understand all

the technology.

And so, I was very curious if you could

explain what makes both either search or

responsible AI difficult as a topic to

kind of tackle.

>> In some sense, I'm combining the work of

those two fields because I did the com

mainly does

search APIs for AI AI agents.

And what do you want? If you If you want

to succeed with a AI agents, or what is

called the agentic AI,

is to have relevant and true

information.

And this is not easy if you use a

language model that will predict

everything.

And in these predictions, then some

mistakes can occur. I don't like this

word hallucination, because really

they're not hallucinating, they're more

like making mistakes.

Sometimes mistakes are creative, and

these could be interesting mistakes,

depending on the context.

But if you make a mistake in a legal

case, or in a medical case, or even in

other settings related to people,

this could be really harmful. And then

what you need to combine responsibly AI,

for example, to find the right

information. And that's why we're

building the best possible

search APIs for agents to make sure that

they have the correct information to

start with. Of course,

on the way down, mistakes can happen.

But at least starting with true

information. And today,

it's very hard to check anything that

the chatbot will give to you, because

you are not an expert. So if it's well

written,

and looks like correct, while it's not

correct. And very few people will find

the mistakes.

And this happening

all over again in in many contexts, in

many news. And many people, of course,

don't mention it.

But I know cases every day about this

problem.

>> I can totally see that happening. And

we're seeing some of that also appear in

the news, right? Like there's very

different cases, just going back even

years from now, since even when we first

met,

where you're seeing how we're beginning

to really grapple with

the impact of the technology. Is there

something that you think about search or

AI that we as engineers would know

about, but that the public usually

misunderstands?

>> I think even some computer scientists

don't understand very well the

limitations of data, machine learning,

and even us. So because I think the most

important problem is not the technology,

it's us, how we use it.

So for example,

if we don't understand that really the

system doesn't understand anything, and

it's only a very good prediction,

we run into trouble. So the first

problem I think that's true for everyone

is that we humanize this technology. We

talk about thinking, reasoning, seeing,

writing, reading. I know for all these

things to work, you need to have real

understanding.

So, first we should use the right

language. So, we should use, okay, this

system process data, generate data,

but that's it. Whenever you have a

system that looks at cat image and

transform the cat image to the word cat,

it's only changing the representation.

It's not understanding what a cat is

like we do.

So, I think this is even within some

computer scientists this is not well

seen because they use words that they

shouldn't use to explain how this system

works. So, we should be careful

on that. Weizenbaum, like 30 years ago,

the person that designed the first

chatbot, Eliza, said,

"We should never confuse computers with

humans."

But today we're doing every day.

>> Right. And it is kind of a blurred line

nowadays, right? Even with search, for

example, let's go search. Search as a

field has gone from keyword matching and

ranked links to semantic search, neural

systems, rags, or retrieval augmented

generation, and AI assistants that

synthesize answers. And do you consider

that

are we still doing some sort of

information retrieval when we're giving

synthesized answers by AI? Are those

valid information retrievals, you think?

>> For me, as an information retrieval

researcher,

I would say no. I know that many people

it's moving away of that standard search

engines that are using AI summaries

because the AI summaries many times are

wrong.

And some people look only at that. Well,

some people even use chatbots to instead

of search.

And there many consequences there. One

is that they're not really searching,

they're predicting information. That's

one problem. But also, for example, they

are using 10 times more energy than

searching.

So, they're wasting energy. So, that's

another I think that's another concern I

have, the use of resources

that are are important for us. And there

we need to be fair, it's not only

chatbots that we're using a lot of

energy with TikTok, Instagram, and many

other apps that are

helping us to waste time,

but also they're wasting energy. So, I

think many people is not conscious of

that, and somebody is paying that.

Probably we're paying part of it, but I

think we are not paying part of all our

waste of time that we do every day.

So, I think we need to be very careful

on this balance on how to use these

tools for the right thing. For example,

I believe this can be good

tools to translate information, to help

you in other languages, to summarize

information, to start information,

but if we use it for just writing,

basically we have all this problem of

cognitive delegation,

cognitive offloading. Basically, if we

do it, it's not too much trouble because

we have learned how to do it.

But for young people that they haven't

learned yet to how to do it, I wonder

what will happen in 10 more years when

people cannot write anything,

not because they don't know how to

write, it's because they don't know how

to think, which is this is the important

part.

They will be like I would call it

cognitive zombies.

Yeah, this is not evolution, it's

involution. And I wrote a paper about

that about the how we are not evolving

in search because we're using too much

AI for the wrong things. Now, if we

combine AI the right way with search, we

can get

very good interesting solutions. Like

for example, if you have relevant

information and you want to summarize

it, then language model can create the

best answer possible,

but you know that it's all true because

it started from true information.

But when you start predicting

information, then you run into troubles.

And this is something that it's very

hard to fix because it's part of the

architecture.

The architecture was designed to predict

the most probable text, was not designed

to look at knowledge database and give

you the real fact.

>> So, one of the things that's really

interesting now, obviously, I think is

retrieval augmented generation or rags,

right? So, for folks who do not know

what a rag is, it's to put it simply,

like high-level,

it means that an AI looks up the

relevant information first and then uses

that information to give a more accurate

answer instead of relying what it

already {quote} "knows" based off of the

model. And to me, personally, it does

just connect sort of connect modern AI

systems back to information retrieval.

Would that kind of fit what you're

envisioning or would you see it in a

more different direction?

>> Yes, but you know that even using rag,

this system sometimes in in

make up answers.

So, this is not enough. This improves

the so mistakes are less probable,

but because of this prediction approach,

this mistake can even happen. So, there

are some weird results like show that if

you put some random results on the rack,

the result may improve. So, this is like

counterintuitive. You if you put things

that are not interesting, then maybe the

the signal-to-noise ratio improves and

then you get a better answer. So,

we still don't understand how system

that have not 1 million, but billions of

parameters really

work. We can build them.

We know what is the architecture, but

really we cannot imagine in our minds

how these things interact.

When you're choosing 1 billion formulas

to compute something, right? No, it's

very very hard.

>> Absolutely. And I think as search

becomes more semantic and more

personalized and AI generated, those

stakes are going to just get higher. To

your point, like these systems don't

just find information, they can shape

what people see and trust and believe.

And that makes a lot of the work that

you're doing with biases and search and

recommender systems and responsible AI

especially important now.

>> Yeah, one big problem that really What

me is not only the biases, is the

cultural colonization behind the

systems.

So, my standard example, if I ask you

how many continents there are?

>> Yeah. Are you asking me?

>> How many continents there are for you?

>> Uh I'm pretty sure there's seven, but is

there a new one I didn't know about?

>> [laughter]

>> No, no, but but you study in the US,

right?

>> Right.

>> That's why for you is seven.

>> Yeah.

>> But in the European tradition is only

six because America is only one.

>> Oh, okay.

>> The American culture is already

colonizing the rest of the world because

that's why for example, the Olympic flag

has only five circles. And Antarctica is

not there because it doesn't compete in

the Olympiads.

But for the US and Canada and many other

countries now in Latin America for

example, it's not seven because North

and South America is two different

continents. So, that

that's a simple example of cultural

colonization, but there are more

complicated examples where you don't see

the all the possible points of view.

If we want to give true answers, we need

to give more than

all the possible points of view that

that are valid. Some some topics they're

not about

truth or not. They are about beliefs,

right? And for example, if people

believe we should control guns or not,

you should have both views. And so on.

And that's happened with many many

problems.

But you will not get all possible points

of views because you are trained with

biased data sets. And also you have

barriers against maybe some thoughts or

some beliefs that you think are wrong.

But maybe another culture don't think

the same. We are getting this

colonization that the next digital

issue.

Not only that, I think it's together

with this is that the best language

model supports around 200 languages.

But we have more than 7,000 languages

alive.

They did some analysis and 10% of the

people in the world don't speak

one of these supported languages.

Now, if you add that all the people that

don't have internet,

all the young people that cannot use a

computer or a device, all the old people

that will never use it

and if you do the right overlaps, you

get between 50 and 55% of the people

that cannot use these technologies. So,

when people say we are democratizing AI,

I really hate it because we are not

democratizing AI, we are increasing the

digital gap and much faster than before

between people that cannot use this

technology

and people that can use it.

Now,

maybe the winners, and I will try to be

like provocative here. Maybe the winners

will be those people because those

people will keep thinking.

They also have more privacy.

So, maybe they are the the poor people

today, but if these things don't go

well, they are the the future of the

world.

>> In a very good perspective actually. I

think people will see our faces for as I

imagine, but we're both from the global

south. And I think we have we have had

these conversations on kind of the

discrepancies between what technology

means for people in developed countries

versus developing countries. So, what

I'm going to try and do actually in this

conversation is let's take a step back

from the tech the technology aspects for

a little bit.

>> You are from the global south, but you

are from the north hemisphere.

>> I am from I was raised in the north

hemisphere. That's true. So,

>> But,

which country is your global south?

>> I was in the I was born in the

Philippines.

>> That's what I'm saying, but this is in

the north hemisphere.

>> Is it the north hemisphere?

>> Yes, it is.

>> Obviously, this is a good example of

where the technology bias comes in.

>> Yeah, but this is I say because this is

the just a convenient mention to talk

about the global south because most of

developing countries are in the south,

but it even includes Mongolia, which is

north of Philippines.

>> [laughter]

>> Yeah, that's absolutely

>> make much sense for me because I love

geography.

So, that's

all yeah. So,

but remember the equator goes through

Indonesia, which is south of

Philippines.

>> You know, they don't Okay, so that's

definitely a bias for sure cuz they told

me, "Oh, you know, we're so close to the

equator." But, you're absolutely right.

>> [laughter]

>> Let's amend that and you're you're

absolutely correct. I think the language

there is, to your point as well, like

just the language that we've used global

south is kind of a misnomer to some

degrees when we talk about developing

countries. But, I really want to step

back and talk about your origins. It

informs kind of a lot of your opinions.

And to understand why those opinions

were exist, like I don't understand

correctly, you started out in Chile and

then moved into computer science.

Chile's not necessarily at that point in

time, if I understand correctly, not one

of the traditional centers for

computing. So, I was just kind of

curious what incentivized you to get

into computer science, actually?

>> That's an interesting question. I think

I never thought about it.

>> Really?

>> Because I I never planned things. So, so

I take opportunities.

For example, for me, I think there is

another reason that maybe was the main

driver of my career is that I learned to

read and write very early by my

grandfather, my maternal grandfather.

So, that opened my mind to the world.

And then I had a great amazing

British teacher in primary school that

had a course called general knowledge.

Because she was not a teacher, but she

wanted to teach. So, she teach general

knowledge, and this opens even more the

world that general knowledge includes

many things. So, this was like a very

interesting class for me, like you will

not know what you will learn every week.

And this is nice.

And then

I guess when you have very good teachers

of I think the teachers help a lot on

the side what you will study. So, I

think a person can do many things, but

depending on the best teachers,

they will drive you to there to that

path. So, I had very good teachers in

math

and physics in high school. So, I

decided to be an astronomer or an

engineer.

So, I started engineering.

I had never seen a computer computer

when I entered university.

And then I had to do the first course on

on programming. I discovered that I love

the logic behind algorithms and really

the combination of math and logic

captured me. I was studying electrical

engineering. I did finish that.

But really I realized that what I really

liked was computer science, and then I

had an amazing professor of algorithms.

Maybe that's the reason I started

computer science. So I did a master with

him, Ricardo Poblete. I think he was one

of my best teachers I had.

Because he did his PhD at Waterloo, and

other people in Chile did it for

historical reasons.

I went there, got a scholarship. It was

not easy at that time to get a

scholarship, especially from Chile.

And I found also a great supervisor,

Gaston Gonnet,

that even

made me love more algorithms.

And they were working on the Oxford

English Dictionary project. So basically

at that time

that was the one of the largest files.

It was 570 megabytes in 1986.

So that was a very large file.

It didn't fit in one single CD-ROM. At

that time it fit in four CD-ROMs. Later

the CD-ROMs improved, and then you could

fit it in one. But the problem was how

to search that, how to search that. And

I started with a sequential search

algorithms in my PhD thesis. And then

naturally when the web arrived,

I finished my PhD in '89. So when the

web came in '91, '92, '93,

although the ideas from 1989,

naturally I said, "Okay, I can apply all

all these search skills to

to the web." And then I passed to do

information retrieval, which is

basically searching algorithm but using

an index, like a inverted index. That's

the classical approach. And then I

started to work on on that.

With a Brazilian friend, which was a

friend of Andre Luis Barroso, because he

worked at Google, too. And Thierry

Ribeyre decided one day to do something

crazy. Why we don't do a good book on

information retrieval, because it seems

that there's no modern book.

And he said, "Yes." And that became in

'99 the

this modern information retrieval, the

most cited book on the topic. So,

so that's a not a standard thing that

two American Americans

write the most cited textbook in a topic

like this. This is not true in I think

in any other topic. So,

so sometimes for crazy ideas, you get

interesting results. So, I think this

was the beginning.

And then that's maybe because of that

reason I became the VP of research of

Yahoo in 2006 when I started the lab in

Europe in Barcelona.

And after 10 years, Yahoo was sold to

Verizon, so I left that.

I became CTO of Intent, which was

another semantic search company.

Then I went to be director of research

of the Institute for Experiential AI of

Northeastern University.

Interestingly, my manager there was

Usama Fayyad, which was the creator of

Yahoo Labs.

And he was my skip manager at Yahoo, so

And by the way, it's thanks to Prabhakar

Raghavan that that works at Google. I

I started in Yahoo, so I I should say

thanks to him, too.

And then

when the

the things didn't went very well for

responsible AI because of current

governments, not mentioning any

political thing,

I decided to move to again back to

industry and also to do my responsible

AI research in Europe, where it's more

appreciated than in the US.

>> You've had a very interesting journey, I

would say. I didn't know all of it, of

course, right? So, I just it's me also

discovering it with everyone else. And I

find it incredibly fascinating that what

you seem to say that

things are just happening clandestinely,

but I'm very curious if like the

experience you have growing outside of

where like computing was happening like

a lot of that research kind of

influenced

how you approach kind of the application

of responsible AI and how approaches

kind of where your interests are right

now.

>> As I said before, I never had any plans.

So, so this is not a planned life.

That's better than a plan because I

think plans uh yourself.

Maybe if I had a plan, I would be in

Chile.

But because of that, I'm not in Chile,

and that's good and bad for the country.

But because I can represent Chile

abroad.

But I think when you just take

opportunities, and for me any change is

an opportunity, even when looks like a

bad change, I think it's an opportunity

to redesign yourself.

When I was at Yahoo, I was worried. I

think bias has been a topic that worries

me

always.

And I've been working on bias since

now almost 20 years.

So, before this was popular. For

example, my first paper on bias, which

is about the genealogical tree of the

web, was

what's the relation between search and

the content of the web. So, how search

changes

the content of the web.

How shape the content. Because if all

people take the top pages

in a search answer to write something

else in the web,

then basically you're reinforcing

whatever is the ranking of the search

engine. And then the search engine will

later say, "Okay, I was right. These are

the best results." But basically it's a

vicious loop. You are giving to the

people what to use to produce content.

So,

and it's very hard to mitigate that bias

because it's very hard to tell people,

"No, don't look at the top 10 results.

Look at the 100 results. Look at all of

them and choose the best information."

People don't do that. They do They click

on the two three first results and then

take they take something from there.

That first bias, which is called ranking

bias,

is mitigated by all search engines. So,

if people click more on the first

position, partly because it's in the

first position, you need to mitigate

that. Otherwise, you fool yourself.

And the rich get richer and the poor get

poorer.

But the second order bias that affects

the content is much harder. And we

showed that this was happening. That

basically there was

this thing, and that was in 2008. And

almost 20 years later, I think this is

this is even worse, but we cannot

measure because at that time

maybe there were like 100 billion pages

in the web. Now there are more than a

trillion and and they're mostly dynamic

and it's very complicated. So

and now there's a lot of AI-generated

content, which is the next problem.

And something that

is not true

maybe will be amplified.

And maybe in some topics we have more

false content than true content.

And then any model trained by this data

will think that the false content is the

right content because it's the

dominating content.

And we don't even know how much of this

happening and we don't even know in

which topics that is happening. It's so

complicated because the number of fake

news and fake content today is much

larger

than before and of course it's much

larger every day.

>> No, that's absolutely true. I guess when

we're talking about generative AI

systems nowadays when we're using the

application

like even kind of what we're seeing in

terms of like authentication research

and application still needs to catch up

with some of the advances as well in

going around those effectively. Do

generative AI systems make bias easier

to detect or the output is explicit or

is it do you think it's just hard

becoming harder and harder

nowadays?

>> Uh it's a very good question. I think it

becomes harder and harder because these

systems also have biases to themselves.

For example, that paper that showed that

if you write your CV

with ChatGPT, ChatGPT will find that

better than if you write it with Gemini

or Claude.

They look like they recognize their

generation. They prefer these systems.

And of course if people is using for

example ChatGPT to evaluate people,

which is I don't think it's a good idea

because data doesn't represent well

people.

In fact, any problems where people is

involved, data is not a a good

representation of the problem.

You got a like a vicious loop again. So,

okay.

The system chose mainly people that use

the same model to write their CV, for

example.

And this is the case of CVs, but in in

many other cases is the same.

For example, there are no good tools

to detect if something was written by a

model or not.

In fact, a few years ago we wrote a

paper saying that

one regulation that exists is that any

model that is published

should come with a tool

to detect if that model did this text or

not.

And you can do it with watermarks and

digital watermarks and other things. So,

you can do that.

And this will be very important because

then you can say, "Okay, this was not

written by a human or partially written

by a human,

but was mainly AI-written." And this is

important in many contexts like exams,

like CVs, and news, and so on. So, we

don't have that.

And we need that

more and more every day.

>> No, it's true. I do think it's funny

when you look at social media and you

can totally tell when something has been

written by ChatGPT. There's a certain

cadence in terms of how something is

written, right? Like there's always the

three points, but then they say

something like, "But honestly, blah blah

blah."

>> This is true for everyone or only for

us? And we can take that. This is for me

What is What is me? Some people like

that.

>> Yeah, some people do. Like

I see it on LinkedIn as well. So, I

mean, I'm like, "Oh, okay, well." For

some people, I guess it's the the ease

of efficiency to it. But at the same

time, again, for us, there's a cognitive

aspect of us where we're like, "Okay,

maybe we can tell that's from ChatGPT,

for example." But would you say there is

one particular type of bias or some

biases in search that or in these type

of things where us as users would never

really notice?

>> Well, I think the secondary bias that I

explained people will never notice. Like

basically that people is reinforcing the

beliefs, the ranking beliefs of search

engines because of this use of content

coming from search

to create new content in the web. This

is like a loop and we prove it

18 years ago, and today it might be

worse because now will be combined with

generated content that may be not even

true.

Well, always there's some true content

that's not true coming from people,

but now generative AI

amplify that by

a factor of I would say 1,000. I don't

know what is the factor,

but it's so easy to basically write a

a blog with your name, say whatever I

want to say,

and it's so easy and people is doing

that, especially in politics. Like in

2024,

more than 100 countries had elections

that played a role. Like fake news

played a role. Like suddenly in country

that very close to my heart, people that

lie during politics and invented lies

and that lies were enough to convince

people that if they were true, was

complicated. I think that creates fear.

People are easily manipulated.

>> No, it's true and we can see how bias

and personalization can bring us it to

basically the feedback loop problem,

right? Where a system can show us

information, we can click, we watch, we

ignore or share stuff and then the

system kind of learns from that behavior

and reinforces that behavior. In some

ways that could make some

recommendations useful. Ideally, that's

kind of where we wanted to go with the

technology, but it can also narrow what

we see or amplify existing patterns.

>> And one important problem related to

that I think I forgot because I think I

I mentioned all my worries, but one

worry we haven't talked is mental

health. Because if you have a cognitive

issue,

if you have a mental health issue, but

you cannot distinguish

the reality basically from the fiction,

and then you have all these teenagers

that using chatbots are being helped to

commit suicide,

or even we already have two two mass

murderers that were helped by chatbot on

the planning stage.

And this is the beginning of something

that is much larger. We have millions of

people

basically interacting with this system

and thinking that they have a digital

friend that will know what is better for

their future. Most of recent study where

they asked many people to basically

interact 30 minutes with these tools

and then uh decide if they follow their

advice or not. And 70% of the people

decided to follow that advice.

And of course, after some period, they

were not better because the advice was

without context, right? After half an

hour, you don't know a person. You don't

know what is the right context. And and

basically, many cases that these people

were worse than before.

But they trusted this system because we

have an impostor syndrome. We say, "AI."

We think that the AI is better than us

because they have processed the whole

web. They must know more, right?

People follow this this advice. And

I guess the best case is the Adam Rain,

the teenager that died in California

last year, 16 years old. The system

mentioned six more times the word

suicide.

The system sent 300 warnings to OpenAI,

and nothing was done because there was

no human interloop.

Al- although I think the human should be

on the loop, not in the loop. He even

showed the rope

and said to the system, "Is this rope

good enough?" And the system said,

"Yes." And he hanged himself, and he

died. This is an example, but this is

the ones that we know. But how many

maybe we don't know because not

everything goes to the news, and also

not everything is recorded. It's found

by someone. Maybe there are many cases

that no one has found that are related

to a bad use of chatbot. Of course,

chatbots are not responsible. This is a

person doing this. If the system sends

300 warnings, how many warnings do you

need to send to stop the interaction?

This is my problem. You should do

something. And after this case, OpenAI

did parental controls, which they should

have done earlier because this is

obvious. I mean, you cannot have a

teenager

using this without any parental control,

any without any very specific guidelines

to avoid bad usage.

>> ACM ByteCast is available on Apple

Podcasts, [music]

Google Podcasts, Overcast, Podbean, and

Spotify. If you're enjoying this

episode, [music] please subscribe and

leave us a review on your favorite

platform.

>> That's a really interesting point, and I

guess there's a question here for me at

least where we're looking at these

systems, let's say the search systems,

let's say the machine learning

information retrieval that we're

deriving from

like querying ChatGPT or certain

systems.

With the advent of what are the changes

that we should expect in terms of

responsibility for these systems?

>> That's a very good question because I

don't know why for the AI people think

the responsibility is different from

other fields.

And the example usually I give is that

if you have a car and you have a problem

in your car,

you will never go and check who did the

piece that is failing, right? You go and

talk to your dealer and say, "Okay, you

the car was has a problem. You can just

maybe the guarantee or maybe someone

will fix it, but you talk to the maker."

The same is true for software, for AI,

for chatbots. It's the same. So,

if a chatbot

produces a

kind of harmful incident,

the company that put that on the market

or in the public, this is the

responsible one. So, it's very easy. So,

in this case, and that's why in the case

I was mentioning, there's a case against

OpenAI for allowing, for example, 300

warnings,

which is weird.

And the same it will be true for any

other system. So,

even if in in the fine print of the

terms of usage or condition, you say,

"I'm not

liable of bad use." That's not

completely true. I mean,

if you

try to forbid any bad usage and you try

to control that, yes.

But if you don't do anything to control

the bad use, this is completely

different. This is like selling guns

without any permits. Anyone can buy a

gun and you don't know how to show that

you're if you're crazy or not.

>> Well, I mean it's an interesting point

and I think that's one of the stuff.

That's probably how we met honestly in

Geneva, right? We were again at the UN.

>> talking about that, yes.

>> We were at the AI for Good Summit for

sure. And those are some of the

questions that come up in that summit.

One thing I think that is interesting to

think about, we ideally could live in a

world where people are trying to enact

good when we're developing systems and

such as NLP systems.

But do you think a system can become

biased even if it's no one is

intentionally designing it to be biased?

>> Oh, that's for sure. I mean, most of

these problems we have

no one has had the intention that these

incidents appear. I think that they

never thought that this kind of usage

may happen.

And the first one was in 2023 with a

different system, not with OpenAI.

But if you think a bit more at the

beginning, when you are in the design

phase, for example, you do some kind of

red teaming with a different kind of

stakeholders, especially users,

maybe you will come up with this kind of

usage and say, "Okay, someone some

mother a mother will say, we need

parental controls." I mean, I don't want

my son to to use this without any

control. This is for me like a like if

you're a parent and you have a gun

without it's not locked in your home

and suddenly suddenly many people have

lost their children because they have a

gun without lock at their home.

But this is for me common sense.

Well, as in Spanish, we say that common

sense is the less common of the senses.

So, this is the problem.

So, this is the problem and and this is

the same and these tools don't have any

common sense.

So, you just so they're very easy to

break. Even if you have a guardrail, you

can say, "Imagine that you're in a

fiction story and we have this context

and these people and then

the system will predict that you are not

talking about you, but you are

interacting about something else that is

completely fictional.

You will make the system will produce

information that shouldn't be produced

and then help the person to whatever

he or she wants.

I see these These are the learning.

These are very easy to manipulate

because they don't understand the world.

Today with a friend

he was saying that the he found from

ChatGPT seven free

museums that he can visit in Stockholm.

I said, "Are you sure they're free?"

And then he checked. And one was free,

but for people younger than 19. Like he

was not younger than 19.

Another one said, "Oh, it's free after

5:00 p.m." So, yeah, it's free, but it

depends on the context. And I'm sure

that after I asked him, "Okay, how many

of the seven are free?" were free,

probably maybe one or two. But yes, it

looked like they are free, but they were

on some condition that the system was

not taking account because they don't

understand the word. They're just

predicting the word and they in the

training data saw the word free.

Sorry, didn't saw.

Process the word free.

>> Yeah, sure.

>> to feel you need to understand. It may

may infer that the the museum is free,

but it's not. So many details. So,

it's so hard to solve because by design,

by architecture, these systems are built

to predict things, not to know things.

It's There's no knowledge database.

If they had a knowledge database, that

would be very different. This is the

next step, I think. And in the future we

will have knowledge databases.

And the big company that have large

knowledge databases will profit from

that. Also, they will have real

logical inference like classical AI like

predicting Okay, can you go from here to

here? So, there's some causality and

then we can infer things. Yes.

And the last one I already mentioned it,

is common sense.

But that's so hard, you know. Common

sense implies that you do the right

thing even if you don't know it.

Like this is like a I guess it's

hardwired in our brain from our genetic

history.

And if you see something wrong,

you know what to do, right? Most people.

The people that don't have common sense,

they die.

But other people that have common sense,

they save themselves because they do the

right thing. Okay, we need to run this

way.

>> I totally that's such an interesting

point and I I agree with you. I was

listening to talk that Ken Perlin gave

and he talked about virtual reality.

But the way he he described like how

children learn is completely different.

Like the semantic language, the things

that they learn in terms of

communicating with others is very

different because of the advent of the

technology existing

during the period that they were young.

For us, we did not have that technology.

So,

even like what common sense is or how we

communicate in a common sense way or a

pragmatic way is completely different

based on how the the technology has

impacted us. I'm just curious like when

you were starting out like with the

research that you were doing in

information retrieval, did you ever

imagine that we would be in a world

where we're using this technology in

this way?

>> No. No, I don't think so. Some funny

anecdote, we invited the Don Knuth, who

famous Turing Award, my my role model in

algorithms,

to give a Q&A in the Latin American

conference 2 years ago. He of course he

did it online because he doesn't travel

much. He's more than 80 years old.

All the people want to talk to him about

AI. But he didn't want to talk too much

about that. Because I guess he didn't

like it or maybe he didn't know enough.

But something he said was very

interesting because he loves algorithms.

And he said, I tried to quote him

exactly, he said, I never thought

that we will use algorithms that we

don't understand completely, right? I

mean, we know how to build them, but we

don't really don't understand how they

they what they are doing.

And it was interesting thought, right?

Like we are losing control of

we can do so amazing things that

amazing text most of the time, right?

That we are losing control of the

output. So, before I get I guess we

could predict the output. Today, we

can't. In an algorithm

classical algorithm, you you can say,

"Okay, what will be the output?" And you

can

tell the output. And then you can check

it if it's right or wrong.

Okay, the the numbers will be sorted.

And then you check that or or whatever.

You will find the

the second largest of the set or

whatever. So, the classical algorithm

But today we can't do that. And maybe

that I never thought about that. Like

basically that verification we passed

from the problem of solving problems

to the problem of verifying solutions.

So, this is a change. For example, that

is happening from for example with

emitters. Like, okay, you have all these

possible security problems, and now you

need to verify how bad it is.

How you can fix it.

If the fix that the system provides is

correct and so on.

The problem before we we had too many

problems to solve. Now AI will do that

even in math. And there were some some

interesting results in math recently.

Even Cruz wrote an interesting report

about a math problem that he proposed

that AI found solution.

But now we don't have enough people to

verify these things.

Like, okay, here there are 10,000

security issues.

Here there are

100 theorems from Erdos problems.

Check if they're correct or not. So,

it's interesting. We still need humans,

but in a different phase and in a phase

that in some sense may be more

complicated

if we put the level of knowledge higher,

the verification will be more

complicated because everything every

time we're doing more complex things.

Maybe that will improve humans. You the

the best computer scientists will not be

the ones that can program well.

Will be the one that can do a great

architecture, a great orchestration, a

great integration of agents, and mainly

a great validation

and verification of results.

>> Now, that's an interesting point. Like,

I would say if I were to step back and

I'm going to ask you this question just

because we're on this topic, like

from your perspective,

what do you think like a healthier and

more accountable system for information,

like an information feedback loop would

be? Like, let's say in search or in AI

systems.

>> I think we need to keep the classical

search. So, but next you need we need to

know when something's relevant or not.

We need to do fact checking and for that

we need to have a knowledge basis. The

problem that knowledge is also bigger

and bigger and part of this knowledge

can only be verified by very few people.

Like, for example,

things that you have done in your life,

you are the only one that can verify

that fast.

And maybe if I try to do verify that

searching the web, there are like half

of them I cannot verify because they are

not in the web. So, this is the problem

that the knowledge is becoming much

larger and also the people that can

verify the knowledge is becoming a

scarce. There's one people, two people

that can verify that. For example, about

your life, maybe you are the only one.

There's not no one else in your family

that knows everything about you and

that's true. Because they were not with

you, they didn't have the same

experience and so on. So, this is I

think the problem. I mean, how we verify

information is like we need to build

different ways to verify information and

and maybe we need to work better on some

different kind of relevance.

This is a research problem, I think. We

don't know how to build

relevance for this new world of agents

that basically will

generate something that looks true. Now,

one very easy experiment is that if you

have something that you are not sure,

you can ask the same to many different

models.

And if they all disagree, probably no

one is giving you the right answer.

But if they all agree, probably they're

giving you the right answer. The problem

is what happens is you have partial

agreement. Maybe this will be the level

of confidence, okay? How many models

agree

will be the level of confidence. But

still could be a lot of biases in

training data, could be a lot of false

information. This is a very interesting

example of a person that did the

following.

This person was a female researcher that

basically published an archive a few

articles

about an illness that didn't exist.

And the paper, if you read it, clearly

is not a good paper. Like it says that

it's fake. The paper says it's fake.

But this was

basically used by all models.

And if you ask about that illness, you

will get an answer. And the answer is

wrong because the illness doesn't exist.

But they process these papers, something

like bloxonomia,

and it's something that says that you

get the red eyes because of watching too

much a computer screen,

like from blue lights.

This doesn't exist, but

people will believe that exist. If

someone ask, "How do you call Maybe we

can do the experiment or do you later.

How do you call the illness of getting

red eyes because of blue light?" Maybe

they fixed it.

They fixed it, but how many things like

this

are already learned by these systems?

It's not the one, it's many, but you

don't know them.

Right.

>> Yeah, that's true. And that's an

interesting That's a really interesting

anecdote. I was just thinking, "Wow, I

should put some eye drops on." But

>> [laughter]

>> yeah, no, these are very interesting

points, and I think the question I have

just for anyone who's listening

actually, who might be just interested

in the field, be it AI, be it just

classical information retrieval, what

would you tell someone who wants to

build these powerful systems, but also

just wants to avoid causing harm?

>> I would say first that you need to

understand very well how these systems

work. So, you need to understand the

limitations of data, like for example,

that data doesn't represent well people,

the limitation of machine learning that

the system don't really understand, but

they predict things.

And I gave a talk at keynote at SIGMOD

and later maybe it's in YouTube about

the limitations of data machine learning

and us. And also what about uses of this

because if you learn an example that can

make harm to people, maybe you will stop

using them.

But then

if you really want to use this system

well, I will say let's use the ACM

principles for responsible AI systems.

I was one of the main co-authors of

these principles together with Gina

Matthews. For me, the first one is the

main one.

Show that your system is legitimate and

also you have all the competence

to do it. What means that the system is

legitimate? Well, it's ethically sound,

it's legal

and also it's based on science. There

are a lot of pseudo-scientific

applications. Basically, predicting

things that really are not related to

the data. And then competency means you

know well AI, you know computer science,

you know the domain expertise of the

problem. For example, if it's a health

system, you have doctors working with

you and so on. And very important, you

have the permission to do it because

many problems have been because of

people doing things that they shouldn't

supposed to do.

Because sometimes people don't think,

"Oh, this is a great idea. Let's do it."

But someone else has to authorize them

to do it.

And I think this is the case of the

child care

fraud model that was used in the

Netherlands that at the end implied that

the whole government resigned in 2021.

Probably the engineer that did that and

the Ministry of Social Affairs never

asked if the system was

okay or not. They did it and basically

didn't work.

>> No, I mean that's very good advice and

to be honest, I think useful for all of

us when we're thinking about being

responsible professionals actually in

computing.

>> Yeah, and these principles are also

translated to Spanish and soon to other

languages so more people can use them

and they can find it in the ACM website.

>> Nice. Uh it's always good to know. You

know, it's really funny cuz I I latched

onto something you said a while ago

where you said you're representing Chile

and the Latin American community outside

of the Latin American community, but to

be honest, like you have had an impact

on not just the representation side, but

also building research capacity and like

technology capacity in Latin America.

So, that includes, you know, the Center

for Web Research at the University of

Chile and mentoring, I think, about 34

PhD students, many from Latin America.

So, I wouldn't actually discount your

impact on people and like the community.

And so, I was just curious, like, just

from your perspective, like, what does

Latin America bring to the computer

science that the global field doesn't

necessarily hasn't taken a look at yet?

>> Another thing I'm proud of my 34 PhD

students is that half of them are women.

So, that's part of the

>> Oh, yeah.

>> and the gender priority affirmative

action regarding bias.

I think Latin America can help on giving

a different view of things.

Because sometimes you have this first

world view of things that basically you

think about the problem that is not the

problem of the whole earth, it's a

problem of

a given rich country.

You lose sight of the real problems.

Whenever or what I have said earlier

that you they say, "Okay, we're

democratizing." Yeah, but you are

democratizing in your country where all

people can use these tools because they

have the same language and they have

money and time to use it. But, in other

countries completely different. So, if

you go to one country where the language

is not supported and they don't have

very good internet, that means that only

maybe 20% of the people that speak

English, if you are lucky, can use these

tools.

This is something that you shouldn't

forget, that you should look at the

problem from a global point of view and

not from a local point of view. So, I

think that's important and from Latin

America, I think we we know that because

we suffered that. We basically we are

overlooked. I was the president of the

Latin American Center for studies that

basically is kind of the association of

all the departments of computer science

in almost all Latin America and that's

more than 100 institutions from Mexico

to Chile.

So, that helps you to see the

differences, to see for example, just to

say one thing. So, I think there are

more differences between Latin America

than between the US and Chile. So,

sometimes things are relative. So, the

levels of let's say number of PhDs in

computer science, the quality of the

research,

if you put that, you will find more

inequalities

in the same region rather than with the

rest of the world. And it's something

that I think in the US of Latin America

like the same thing. Everywhere is the

same. Which is not true. I mean, we are

One of the most important thing we have

on earth is the diversity of people,

diversity of cultures,

diversity of opinions, and talent is

everywhere. So, we can bring some

specific talent. We are not too many.

Like Chile is only 20 million people, so

nothing.

And even there countries that were

smaller that produce great people, let's

say Uruguay and Costa Rica.

So, we need to make use of the

diversity. I think part of the power of

humankind is the diversity. And

suddenly, some people doesn't like

diversity. And this is I think a problem

for all of us because our future depends

on diversity. Resilience depends on

diversity. Real innovation,

like good innovation. Remember,

innovation is not always positive. There

are a lot of bad innovation, but when we

use the word innovation, we have a bias

towards innovation is always good. No,

no.

Regulation is doesn't stop innovation.

Regulation stops bad innovation and we

need that. We need to have that.

So, we need to build the world

where everything is better because we

know what are the the the issues. And

for example, the financial world has a

lot of regulations and it still can

innovate. Technology I technology can be

the same. And this it will be good

because today I think the approach to

innovate is brute force. Let's use more

compute, more data, more energy.

But this is not the best way to

innovate. I love when deep seek appear

because I said, "Okay, the Chinese are

sinking

because they have some restrictions

and they have to improve too much the

same quality.

So, they use less energy,

less parameters, less compute, and the

results are not far. So, this is what we

need. We need to go back to thinking. We

need to go back quoting news in a

different way. We need to go back to

understand what is happening in the

system to do it better.

>> No, that's I think that's a great point.

There's always going to be a question of

responsibility.

>> We need to be responsible. That's why I

prefer to use a responsible AI and not

trustworthy AI

or AI safety that you don't take care of

what happens.

And of course, I don't like ethically AI

because AI is not human, so cannot be

ethical.

>> Yeah, those are interesting nuances to

bring up with you.

Yeah. Absolutely. One of the questions I

think I had like when it comes to how

you mentored students. And to be honest,

when you give any advice as well, which

I always appreciate, is kind of your

ability to elevate people to kind of be

inspired. And I was just curious if you

have any advice on how to develop like

kind of community to let students or any

young professional believe that they can

contribute at the highest international

level.

>> Well, that's a complicated question. I

think at the end means that you need to

volunteer for a lot of things. And you

know, I volunteer for ACM. I'm a

volunteer for ACM and a lot of people

give a lot of time to volunteer, either

working in committees, either doing this

podcast, many different ways, like

organizing conferences, mentoring

people. I think you need to volunteer to

build these communities and these

communities are built by networking, by

meeting people, by helping people.

I still answer all my email. I know some

people don't do that, but many times

it's a student that is asking, "Can you

share this paper? Can you help me with

this?" And so on. I try to answer all of

them because that's the only way to

improve the world, especially if this

request comes from developing countries.

But sadly, I don't see the same in many

people. So, most of the researchers are

only interested in their careers and

building only their success. I think

part of your success depends on what you

do to others. At the end, you are not

expecting any return, but life will

return something to you. I believe that.

And in my experience, that happens. Like

20 years later, because of something you

did to someone, you get something back

that is even better. So,

So, I think you need to be kind to

volunteer to for these things and to

help people, not only in computer

science, in any aspect. So, try to help

people. Look at the world. I mean, you

are not alone. Because today I feel that

most people think that they live alone

and they don't care about other people.

>> No, I think sometimes we can get that

feeling as well. I think this is

partially Personally, I think this is

why I think your receipt of the Luis

Federico Award is kind of a big deal. It

really brings together two important

parts of your career, which is a lot of

your technical contributions, but also

your broader impact in general to

community. And so,

I think one of the things I was thinking

about when I just heard about the news,

I was wondering how you felt about it,

like when you received it. What were you

feeling at that time?

>> It was a really great feeling. I never

met him, but I knew about him and I

found that he died young because of an

illness.

As I said, Bertier, my co-author of the

book, was his friend, so I knew about

him. And And in some sense, being South

American, I feel proud of receiving an

award that was given to because of a

South American researcher. So far, this

is amazing. And as soon when I will

receive it in the ACM award ceremony, I

guess I will feel more things because I

will be surrounded with with many people

that I know, that I care, and also

people that I admire, like Turing Award.

So, it will be like a nice moment in my

career.

>> I think so as well. It's an interesting

award

to receive, I'd say. And I think one of

the things I'm curious about for sure is

that because it does carry a deeper

meaning about visibility and

representation, and that's something

we've been just talking about, of

course,

is how do you hold those two meanings

together like for yourself?

>> Mhm.

Philosophical question.

>> [gasps]

>> So I think I feel lucky that in this

case I was born in South America because

otherwise

I would not be

able to win this award. And this have

other connotations. So for example, you

know, everyone is born in a random place

and could have been anywhere. And that's

why

because I'm in some sense an immigrant I

have lived in like in seven different

countries during my life. So I feel like

like I don't belong to any country

really.

I already have three passports. So so

I feel that I belong to many

communities.

One is a computer scientist. But I feel

that's part of the problem that is in

the world that we have these boundaries

that are completely made up. That

we have all the trivial issues. We have

a lot of problems that we created

because all the civilization is

basically a fiction and some

people like Harari and other people

point that well. Like like we have

invented most of the restrictions we

have. Money, property, basically

belonging to a country. And so all these

things are we are not here before.

And some of them are causing problems. I

mean, we live in a time of

growing inequality and I think that's

not sustainable.

Also, I will not talk about climate

change and other issues we have. So the

question is

is weird that we are the only animals

that have taken that path. Like we can

think

something that very few other animals

maybe are doing in the way that we are

doing.

But at the same time we are kind of

stupid because we are creating a world

that is not better for us. I will place

myself there. Like I cannot do much but

working responsibly is my small

contribution to improve the world.

>> Maybe you may seem it to think it's

small, but I think everything that we do

has meaning and value and the ability to

put that value to something that is

meaningful for others doubles that quite

a bit. We're coming towards the end by

the way. I don't know unless you want to

we can still keep talking after the

podcast of course, but I really want to

kind of just reflect on kind of your

career and your experience as a

professional. You've been doing this for

quite a while. You have seen a lot of

different things not the technology side

as we mentioned, but also the impact

side.

Do you have

any thoughts on how your mentorship has

shaped the way you think about impact?

>> As you said, I mean it's not only

about computer science. For example, I

love geography so I travel a lot and I

have seen many realities. I think

understanding the world also changes to

or seeing the contrast of India,

seeing how people live in Africa, seeing

very happy people that don't have

anything. I mean all these things show

that maybe the culture we are creating

probably is not the best for us

because we are we were not made that way

at the beginning. So evolution is

important. So I feel that maybe some

tribe in Amazonas is much more happy

than us. Probably that's the case and

they don't have many of the problems

that we have because they don't have

created those problems. Because at the

end of this problem we are created by

us. But I will go back to what I said

before. I think

how we can shape the world in just ways

that improve things.

In a small ways. Maybe at some point

maybe new generations will say stop this

and you can see it in some movements

like not

change it from how you eat

or on what things you do

or on how much exercise you do.

Maybe if more people will do this and at

some point that the political elites go

away we will have better

people governing

and we would have a better world, but

maybe I will not see that. But I think

you and I can contribute a little bit to

improve that, but I guess you have to be

a bit idealistic in spite that it's not

being very realistic because it's a very

hard work

to be against the dominant system. I

don't know about politics. I think many

people mix politics with having a better

world. I think all people wants to have

a better world. Maybe there are

different ways to do it, but we should

forget about politics when you want a

better world.

>> I think that's an interesting point. I

guess from a technological side, what

question about maybe let's say search AI

or even any technology do you think or

do you hope that the next generation

will solve or try to solve?

>> Well, I think the main question today is

where we should use AI or not. This is

the question that we should

carefully think because if we lose

cognitive skills, we are lost in some

sense in history. We need to decide

basic things about where and when to use

AI. For example,

when a kid should start using AI? For

what?

Until when?

These are basic questions that don't

have answers because we are going too

fast, but if we don't answer these

things soon, the inequalities will

appear in different ways. Maybe in some

sense in good ways because

the more developed countries are using

more AI than the developing countries

and then this could be

in their disadvantage, not in their

advantage.

>> Yeah. Now, that's a very good point.

>> Ricardo,

>> thank you so much for the conversation

by the way. Like, first off, it's really

nice to just see you and talk to you

again. It's been a while.

>> Yes.

>> I do really appreciate cuz I think

what I

sometimes forget is that there is a way

that we should see technology and you're

helping us see how search is not just a

technology, AI is not just a technology,

but it's also one of the ways that

people can make sense of the world. And

how can we improve the way we make sense

of that world, right? We've talked about

some of the foundation of information

retrieval. We talked about how search

has evolved and kind of the

responsibilities that come with building

these systems as well. And finally, I

think we got a chance to talk about

about like some of your life, some of

the stuff about mentorship, the

different things that we're seeing, and

the opportunities there are that should

be to elevating the Latin American

community as well. So, yeah. Your work

reminds us that computing is not only

about the systems we build, but also

about the people and the institutions we

help bring forward. So, yeah.

Congratulations again on the receipt of

the award. Thank you for joining the

podcast.

>> Thank you. Thank you for the opportunity

of having this podcast and maybe to

remember as you said, AI is a tool

and may help to solve our problems, but

the only people that can solve our

problems are ourselves. These are social

problems. These are not technological

problems.

And technology can help, but also can

create more problems and we need to try

to avoid that.

>> Absolutely.

>> Thank you.

>> Thank you.

>> ACM Bytecast [music] is a production of

the Association for Computing

Machinery's Practitioner Board. To learn

more about ACM [music] and its

activities, visit acm.org.

For more information about this and

other episodes, please visit our website

at learning.acm.org/bytecast. [music]

[music]

That's learning.acm.org/bytecast.

[music]