HomeVideos

From Note-Taking App to AI Workspace: The Simon Last Interview

Now Playing

From Note-Taking App to AI Workspace: The Simon Last Interview

Transcript

887 segments

0:05

Hi listeners, welcome back to No Priors.

0:07

Today I'm here with Simon Last,

0:09

co-founder at Notion. We talk about

0:11

their new vision for notion in the AI

0:13

age as a platform for humans and agents

0:16

to collaborate, how the engineering and

0:18

product org at notion is changing,

0:20

[music] and these new tools for thought.

0:22

Welcome, Simon. Hey, Simon. Thanks for

0:24

doing this.

0:24

>> Hey, of course. Yeah, it's really fun to

0:25

be here.

0:25

>> Notion's at scale. Amazing platform,

0:27

lots of users. You did start quite a

0:30

while ago. I think of notion as one of

0:31

the companies that has really like

0:33

braced AI quite aggressively. I was told

0:35

you first got your hands on GPT4 uh at a

0:39

company offsite in Mexico. Um is that

0:42

true? What is the origin story of like

0:44

starting to work on this stuff?

0:45

>> Yeah, I think yeah, that year that was

0:47

2022. Um I I've been watching you know

0:51

what's going on in general. I've just

0:52

been like super curious about the

0:54

technology and fascinated to to try

0:57

everything and think about like like how

0:59

we can apply it. It wasn't until I

1:01

played with GBG4 that it it became

1:02

really really real. So, you know, we

1:05

when we got access to it, it it was sort

1:07

of like a a protogt like interface. Um

1:11

and uh my co-founder Ivan and I both got

1:14

access and it was just immediately clear

1:16

like I would say two big things. One is

1:19

that it was just pretty smart. it it

1:22

could follow reasonably complicated

1:24

instructions. It could write things for

1:25

you. You could edit things and and the

1:27

second big thing was that uh the scope

1:29

of its knowledge was extremely

1:30

interesting. Uh super super deep like um

1:33

and and broad world knowledge. When we

1:35

played with it, it became just instantly

1:36

clear to both of us like okay the the

1:38

time is now to start thinking about how

1:40

to apply this. It's only going to get

1:41

better.

1:41

>> We were talking about Mexico TPT4. You

1:45

guys saw it was like clearly the time.

1:46

Did you start with like a particular

1:48

vision of like what you should obviously

1:51

be able to do with AI and notion or did

1:53

you start pulling people from different

1:54

teams or recruiting people and say like

1:56

let's experiment? How did you begin?

1:57

>> I think we immediately had a long-term

1:59

and a short-term vision. I would say the

2:02

the I'll start with the short-term one.

2:04

The the thing that was immediately

2:05

obvious was oh it could be like a

2:07

writing assistant.

2:08

>> Um so it could be in your document. You

2:10

could like select some text, have it

2:11

rewrite it. You could have it write text

2:13

for you. maybe look something up and

2:15

then uh you know give you like like

2:17

sources or more information. So that was

2:19

the thing that we immediately like like

2:20

got to work on and you know we sort of

2:22

started a tiger team around it and then

2:24

we were able to launch it in like like

2:25

two or three months after that. And then

2:26

the long-term vision that we immediately

2:28

had was like oh the thing that looks

2:30

like it may be possible is more of like

2:32

a general assistant. So what if you

2:35

could just give it all the tools inside

2:37

notion that a human would have be able

2:39

to like create its own databases, query,

2:42

manipulate them, create documents, edit

2:44

them, uh and sort of weave all these

2:46

things together to do like a longer

2:47

range task. And so we we sort of uh

2:50

immediately started on both. The the

2:52

short-term one we're able to shoot very

2:54

quickly and then the long-term one

2:55

didn't really work yet and so that took

2:57

much longer to get working. Are there

2:58

like specific

3:00

first launch of the AI specific notion

3:03

features and products was when last

3:05

year?

3:05

>> No, it was uh it was February 2023 is

3:08

when Okay. It launched. Yeah.

3:10

>> My timelines are wrong. Um are there

3:11

like a few um specific learnings or

3:14

breakthrough moments you think since

3:16

beginning to release that are

3:17

interesting?

3:18

>> Yeah, I mean there's been it's it's it's

3:21

been a slog over many years or over

3:22

multiple years at this point with with

3:25

many many learnings I would say. Yeah. I

3:26

mean just to give you a timeline of the

3:28

arc of what what we shipped is you know

3:30

so the first thing was our our writing

3:32

system uh we called it AI writer um

3:35

that's the first thing we launched uh it

3:37

was the easiest to get working is it's

3:38

like singlestep task rewriting editing

3:41

text uh there's no like retrieval aspect

3:43

it was just like raw access to the model

3:45

to write uh uh to write the text the

3:48

next

3:49

>> uh the next big thing that uh that we

3:51

immediately started working on was Q&A

3:53

doing a semantic index of the entire

3:55

workspace and then letting you ask a

3:56

question and and I can give you an

3:58

answer that's that's grounded in the

4:00

sources. That was also immediately

4:01

obvious to us that that'd be super

4:03

useful. And so we started work on that.

4:04

That one we launched in I think it was

4:06

October 2023.

4:08

So we started a beta before then, but

4:09

then that our our GA was in October.

4:11

That was just a much bigger effort to

4:13

get working. Obviously, we weren't just

4:14

like plugging in the LM. It was actually

4:16

doing this like real-time updating

4:17

index,

4:18

>> right?

4:19

>> We had to get much more serious about

4:20

the the evals and the quality there as

4:22

well. the the Q&A has been a a

4:25

multi-year journey. Basically, what what

4:27

we did is immed uh as soon as we got the

4:29

notion index working, it it was obvious

4:32

that okay, we should index everything

4:34

else as well and and so we index like

4:36

Slack and Google Drive and and we're

4:38

launching new ones uh on a on a regular

4:41

cadence and and and now we have a uh I

4:44

would say fairly complete

4:45

>> one could argue that those are like very

4:47

difficult problems that you know those

4:48

products natively have not solved

4:50

perfectly yet. So, how did you think

4:51

about taking that on? I don't know if

4:53

that's like an offensive thing to other

4:54

product teams, but like it's not working

4:55

yet.

4:56

>> Yeah, it's it's kind of true. Yeah, this

4:58

has been something we talk about a lot

5:00

because it's like, you know, it's like

5:02

almost like what right do we even have

5:03

to do this? But but it turns out that

5:06

most of the companies are pretty bad at

5:07

making their indexes somehow. It's

5:09

honestly kind of baffled us a little

5:11

bit,

5:11

>> right? But I I think my take after

5:15

dealing with all of this and you know

5:17

working with the team to try to get it

5:18

working is there's a little bit of just

5:21

AI pilled savviness that's pretty

5:24

important and then and then I think most

5:26

of it is honestly just like a bit of

5:28

like like craft and attention to detail.

5:29

I think like in in particular with this

5:32

like indexing retrieval stuff in order

5:34

to really get it working you you have to

5:37

be quite empirical and iterative and

5:39

actually be like like trying queries

5:40

like you know like each each uh data

5:43

source is a little bit special like you

5:44

know you can't just apply a

5:46

one-sizefits-all to like quering Slack

5:48

versus quering Google Drive let's say

5:50

they're they're completely different

5:51

kinds of information and we found that

5:53

there's just a little bit of like like

5:54

craft and love that has to go into it in

5:55

terms of like actually trying a bunch of

5:57

different queries actually using it

5:58

every day and constantly

5:59

iterating and rethinking and and and

6:01

tuning how the retrieval works.

6:03

>> How did you um think about the diversity

6:05

of how people organize their workspaces

6:08

and just I mean even notion is not use

6:11

of it is not homogeneous, right? Like

6:13

I'm probably part of 15 workspaces as an

6:16

investor and so I look at them and I'm

6:18

like well mine's a mess and these people

6:19

are really organized and the workflow is

6:21

reflected in how their notion works.

6:23

>> Yeah, totally. I would say I mean the

6:25

interesting thing is that with

6:26

embeddings it almost doesn't matter as

6:28

much. anymore. The the AI doesn't really

6:31

care what the what the the tree

6:33

structure is. For example, the all the

6:35

AI cares about is that there's a snippet

6:38

of text that has the the context you

6:39

need and then it can retrieve it. And

6:40

so, actually, we kind of advise people

6:42

now like don't worry as much about

6:43

organization. Just just just find a way

6:46

to get it all piped in and like like

6:47

thrown in there.

6:48

>> You still make decisions that could

6:50

change performance quite a bit like

6:51

chunking strategy or whatever.

6:53

>> Yeah,

6:53

>> that's super important. But but that's

6:55

sort of not that's sort of transparent

6:57

to the user and sort of in independent

7:00

of their their particular method of

7:01

organizing things.

7:02

>> Mhm. It just seems like still a

7:03

difficult technical challenge given how

7:05

different the content bases are.

7:07

>> Yeah. Yeah. Yeah. I think yeah that took

7:09

a lot of iteration. Yeah. The chunk

7:11

sizing, how retrieval works, the

7:12

different like steps in the pipeline of

7:14

retrieval. Um yeah, there's a lot of

7:16

iteration on that. Ivan said I should

7:18

ask you um how many times you've rebuilt

7:21

Notion and rebuilt your harnesses.

7:24

>> Yeah. Yeah. It's kind of a running joke

7:26

almost. I mean we we we rewrite our AI

7:29

harness probably every six months or so

7:31

and it and the time to rewrite has kind

7:33

of been been decreasing just because I

7:34

think like like progress has been

7:35

accelerating. I think this is honestly a

7:38

a really key thing and something that a

7:40

lot of companies get wrong is just like

7:42

doing one thing and then just like like

7:44

sticking with it. you really do have to

7:46

keenly aware of what the current state

7:48

of the models and the technology is and

7:50

then designing the harness the system

7:51

and the product deeply around that and

7:54

it basically means you have to rewrite

7:55

it every six months and um I find it

7:58

pretty fun. It's part of the process. Um

8:00

you know you get to get to restart and

8:01

and and and rethink it. You know we're

8:04

working on we're about to release a new

8:06

version of a harness like in the next

8:08

week or two. Uh and then and then we're

8:10

already thinking about the one after

8:11

that as well. I I think that leads to a

8:14

a set of questions I had for you on just

8:16

like how does not as an engineering and

8:18

product and research organization work

8:21

now that you have the power of um coding

8:24

agents as well because I imagine like

8:26

your willingness to rewrite the harness

8:28

goes up dramatically if you're like

8:30

agents are going to help me do it.

8:32

>> Yeah, that's extremely true. Yeah, I

8:33

mean yeah, it's been it's been really

8:35

fun to use the coding agents. I think

8:37

the ambition of what I even consider

8:38

building has has has gone up a lot. What

8:41

do you think has most dramatically

8:42

changed in how you think about how um

8:44

engineering and product should work at

8:46

notion over the last two three years?

8:49

>> Yeah, I mean it's it's definitely

8:50

changed multiple times. I mean in terms

8:52

of the coding agents, we kind of went

8:53

through multiple eras. There was kind of

8:54

like the tab autocomplete era and then

8:57

we and then we got into sort of

8:59

inserting rewriting some code u but but

9:03

it wasn't really until the the agents

9:04

started working. I I would say like

9:06

early last year we started to adopt the

9:09

agents like I started using cloud code I

9:11

think around April of last year that was

9:13

a huge unlock like I would say the the

9:15

the big shift there is that you know you

9:18

can really push on getting these agents

9:19

to end to end you know implement and and

9:22

verify and maintain stuff but it but it

9:24

requires pretty significant thought in

9:26

terms of how you architect things and

9:28

what is the verification loop um but but

9:31

but the upshot is I think if you do it

9:33

well you can be much more ambitious

9:35

about what you're building and also make

9:37

it much more robust than you could have

9:38

done uh with with with humans writing

9:40

it. And then the flip side is if you do

9:43

it badly, it's all slop.

9:44

>> Does that change your lens of like what

9:47

teams should look like at notion like

9:50

size, seniority, anything like that?

9:52

>> Yeah, I mean I would say I mean the

9:54

fundamental effect is that you know

9:57

everyone's individual impact in terms of

9:59

their output can be much higher. um and

10:02

your output increasingly depends on your

10:05

ability and willingness to use the

10:06

tools. I I think that's the fundamental

10:08

thing that's happening. And then like

10:09

like how does that play out? I think I

10:12

don't think we've seen that much impact

10:13

on the the team size really. I think we

10:17

we like to work in like smalish tiger

10:19

teams for the most part. Um I think if

10:23

you can make teams small, it's almost

10:25

always better. That was true before and

10:27

I think it's still true. Uh maybe

10:29

increasingly a little bit but but not

10:31

that much. I I think yeah the main thing

10:34

is to just like like really harness the

10:35

tools.

10:35

>> Do you think something different happens

10:37

to the median engineer in an

10:39

organization versus the 10x engineer or

10:41

the engineer 10x more willing to use the

10:43

tools?

10:44

>> Yeah, I think the the gap is bigger. You

10:46

can be like a 100 or thousandx engineer

10:48

if you're using the tools right now. I

10:49

think I think the the gap is much bigger

10:51

like the the the minimum bar has not

10:53

changed but the maximum bar has has

10:56

extremely increased. One impact it's had

10:57

internally I would say is like broadly

11:00

things feel like a little bit more messy

11:02

and chaotic I would say like but I kind

11:04

of love that I mean it's like there's

11:05

there's more pro there's way more

11:06

prototypes uh you know people are like

11:09

for example our uh uh design team made

11:13

an made an entire uh git repo they

11:16

called it the design playground and it's

11:17

essentially like a simplified notion uh

11:20

with a bunch of like UI primitives in it

11:23

>> and they've made it like really

11:25

sophisticated you it it it has like an

11:27

agent in there and like um and it's it's

11:31

pretty cool because it allows them all

11:32

the designers can can spin up like super

11:34

high fidelity prototypes

11:36

>> really quickly and so it's no longer

11:37

like like pointing at a mock and being

11:39

like like you know like like how will

11:41

this look like they'll give you like a

11:42

URL to a prototype that's that's that's

11:44

been deployed and that sort of thing is

11:45

true all the way up and down the stack

11:47

you know for all of engineering just

11:48

like a little bit more chaotic more

11:50

stuff happening um all the PRs are more

11:53

ambitious

11:54

>> do you draw a line somewhere about like

11:56

stuff that is more dangerous to touch or

11:58

sensitive like ah there's could be risk

12:00

of data loss over here or and not or is

12:04

it kind of you look at it all is it's

12:06

fair game

12:06

>> we still do reviews on all the pull

12:08

requests and and I would say and you

12:10

know all the pull requests are now

12:12

written by agents they're often like

12:15

larger and and and more complex that's

12:18

like the worst part but the better part

12:19

is that they're often like a much better

12:21

tested and we can demand sort of a much

12:24

better testing for the things that merit

12:25

I never produce a PR that like hasn't

12:27

been like fully ant tested anymore. And

12:31

so it's like you can get to a pretty

12:32

high degree of confidence that it that

12:33

it works, but it requires like you're

12:36

not just vibe coding by by saying the

12:38

thing you want. You're sort of thinking

12:39

carefully about like what is the thing

12:41

I'm like what is the change I'm trying

12:42

to make and like and and how can it be

12:44

verified and how can it be deployed

12:46

safely and then enlisting the agent to

12:48

to help you with that process.

12:50

>> When you think about where you said the

12:52

general assistant like doesn't quite

12:53

exist yet. Um, what's the what do you

12:56

imagine notions agent agents being able

12:59

to do like over the next year or two

13:02

that are still unblocked? They're still

13:04

blocked by either capability or your

13:05

harness work.

13:06

>> We struggled for a few years to build an

13:08

agent. Um, and you know, it always like

13:11

like sort of worked but then you know

13:14

wasn't that useful largely just it was

13:17

too early. So we you know we we tried to

13:19

to build an agent I would say actually

13:20

three or four times and then uh we

13:24

finally launched it uh last fall so like

13:25

last August September. Um so the you

13:28

fuse notion AI now it's like the full

13:30

agent that has access to everything in

13:31

notion pretty much. Um so that that that

13:32

totally works. I would say like the a

13:35

lot of the original vision that we had

13:37

totally works now. Um and it you know

13:39

it's like like fully shipped. Last

13:41

August or September, we shipped our

13:44

personal agent. U so it's pretty much

13:46

every user in notion has an agent and it

13:48

basically it has access to all all the

13:51

things that the user has access to. So

13:53

you know it can create a database for

13:54

you. It can update things, create

13:56

documents, it can search search the web,

13:57

do research and then the second big

13:58

thing uh that we just launched last week

14:01

actually was u custom agents. So you can

14:04

basically you can create a new custom

14:05

agent give it a name and unlike the

14:07

personal agent uh by default it doesn't

14:09

have access to anything. Uh so you have

14:12

to grant it access but then once you do

14:14

it can actually run autonomously in the

14:15

background. So for example you can give

14:17

it access to its own database to file

14:19

tasks let's say and then you can attach

14:20

it to a slack channel and then it will

14:22

start responding to people on Slack and

14:25

filing tasks. That's that's that's one

14:27

use case. Another one is maybe you could

14:28

um you could give it access to a

14:29

database of like weekly reports and then

14:31

and then let it search the web or search

14:33

your workspace. And so it's sort of a

14:34

custom agent sort of represents some

14:36

work or job some some knowledge work

14:38

task that you want to be done

14:39

autonomously. One thing I'm really

14:40

excited about this going forward is is

14:43

um we want it to be extremely good at

14:46

sort of bootstrapping its own

14:48

capabilities basically from an initial

14:50

kernel allowing it to basically

14:51

bootstrap itself to do anything right.

14:53

So even for example maybe u uh building

14:56

an integration that we don't support yet

14:58

deploying that and then and then using

15:00

it.

15:00

>> So you imagine that notion agents are

15:03

actually the broader definition of agent

15:04

where like writing code is a tool it's

15:07

pretty close to yeah

15:08

>> I think it's pretty key. Yeah I think I

15:10

I think of coding agents as like the

15:12

kernel of AGI. AGI will be a coding

15:14

agent. Um and and and and code is just a

15:18

really really useful uh a primitive for

15:20

representing like deterministic logic.

15:23

The thing that's really exciting about

15:24

it um we're applying it to to a

15:26

knowledge work agent is that it can

15:28

bootstrap a capability you know so yeah

15:30

like I said if integration doesn't exist

15:32

it can build it um if if it needs to uh

15:35

you know connect itself to a new data

15:37

source it can do that

15:39

>> given you have a you know notion is at

15:42

scale but is operating in a landscape of

15:44

productivity and platform players that

15:46

are at even more scale right um many of

15:49

these will end up with their own agents

15:51

lots of people from the labs to the

15:53

Microsoft world are trying to integrate

15:55

other data sources. So you have this

15:57

like cross attempt to integrate and

15:59

index like how do you think that plays

16:01

out? Like what do you what do you

16:03

imagine that notion agents are best at

16:05

or what they have the right to go do?

16:07

>> If you look at the landscape like I I

16:09

would sort of say there's the labs and

16:11

then there's maybe the the the software

16:14

platforms and then there's maybe like

16:16

infrastructure. In terms of the labs,

16:18

you know, we see ourselves as kind of

16:19

like the the Switzerland for models. We

16:22

think and our customers they, you know,

16:23

they don't want to be locked into a

16:24

certain certain labs model. They're

16:26

always uh releasing new versions any

16:28

given month. One is better than the

16:30

other. Um so we want to be a a place

16:34

where basically you can you can easily

16:35

get access to all the best models um at

16:37

any time and you can easily switch

16:39

around.

16:40

>> Do you think open source plays into that

16:41

as well?

16:42

>> Yeah. Yeah, absolutely. I think the open

16:44

source models are actually getting

16:45

really good. There's like the four

16:47

different Chinese models now that are

16:48

that are quite good. We actually just uh

16:50

released one of them in our agent uh

16:52

last week and and we're going to do all

16:53

four for sure. Um they're they're

16:56

actually quite good and they're and and

16:57

they're way cheaper than the the

16:59

frontier models. So I think there's

17:01

there's a lot of use cases where where

17:02

where you'd want that and we want to

17:03

give that as an option in terms of like

17:05

the other you know so you know we think

17:07

of our role as sort of taking all the

17:10

best models that we can creating a

17:12

really high quality state-of-the-art

17:14

agent implementations where where people

17:16

can easily and conveniently get access

17:17

to them and then making sort of a

17:20

collaborative workspace that is really

17:23

good for for humans and for the agents

17:26

uh to to to coordinate on. I think it's

17:29

it's something that's that's very needed

17:30

in the world and we're just trying to do

17:32

it in a really tasteful wellexecuted

17:34

way.

17:34

>> You were describing you need the index

17:37

to make the agents good. Um you give the

17:40

agents access to the tools that we

17:41

humans have in notion. How do you think

17:44

about um the structure of notion and

17:46

like where it's like useful or even like

17:49

not useful or relevant for agents like

17:51

blocks and databases and such?

17:53

>> It's all still pretty useful. Um

17:56

extremely useful. it uh there there's

17:58

been a a challenge to sort of you know

18:01

we want to make it really convenient for

18:03

the agent. I think that's that's a new

18:05

thing that that didn't exist. You know,

18:07

in in the past it was convenient for

18:08

humans and then we also made APIs

18:09

convenient for humans writing code

18:12

>> to use our API. Uh so we essentially

18:14

have a new customer which is the agent.

18:16

At first that was definitely a problem

18:18

you know. So for example like our our

18:20

API uh uses this this crazy JSON format

18:23

for blocks that by default is like crazy

18:26

verbose and like like horrible for the

18:28

agent. But we basically took on that

18:30

challenge and um designed uh just really

18:34

convenient APIs for the agent. We

18:36

created sort of a markdown dialect that

18:38

um looks like the default normal

18:41

markdown but it's sort of enhanced with

18:42

uh all the notion blocks. Um and the

18:45

models are really good at it. It works

18:47

really well. Uh so so that's how it

18:48

reads and writes to pages. And then uh

18:51

for databases uh we we use a SQLite. Um

18:56

so so basically it's the guess the speak

18:57

and SQLite which also works really well.

19:00

So the default thing did not work really

19:01

well. Uh but then we just like like took

19:04

that on as an engineering challenge and

19:05

and I would say now we have like

19:07

extremely convenient APIs that the

19:09

agents are are really naturally good at.

19:11

>> How did you uh understand or figure out

19:15

what would make the API better for

19:17

agents?

19:18

>> That's a good question. Yeah, I would

19:19

say it's a it's a combination of just

19:21

trying things. It's it's it's very

19:23

empirical. So, so we're just playing

19:25

around and like like noticing, oh, it's

19:26

not very good at that. Oh, that's way

19:28

too many tokens. How can we make this

19:29

smaller? And then a little bit of just

19:31

like like first principles thinking of

19:32

like, you know, what is it the models

19:34

are being trained on and what's what's

19:37

in their prior? What do they know? And

19:39

what what do we think it would naturally

19:40

be good at? And and like like how does

19:42

the agent loop work and like what what

19:45

would be the convenient efficient

19:47

pattern for for accessing these things?

19:48

Um, and so and then just you know a lot

19:50

of playing around. I hear user research

19:53

where the user is actually agent and

19:55

then you know ongoing eval.

19:58

>> Yeah. I mean user Yeah. You just chat

20:00

with it.

20:01

>> The user is always there. It's ready to

20:02

talk to you.

20:03

>> Yeah. Actually, that is wonderful where

20:05

you have infinite access to it.

20:06

>> You have infinite access to it. Yeah.

20:07

And and you can you can script and scale

20:09

the access as well.

20:10

>> I assume you have actually I know you do

20:13

because you walked in. You're like,

20:14

"Hey, I need to get access to the Wi-Fi.

20:15

I need power. We can't block the agents

20:17

while we're doing this." Um what do you

20:19

have running right now? Tell me about

20:20

your setup. I'm working on a new

20:21

prototype and so I have a couple agents

20:23

I'm working on that. Um and then yeah,

20:26

my setup these days is just um either

20:29

claw code or codeex. I like the the the

20:31

CLI tools. Um they're they're they're

20:34

super simple and like work pretty well.

20:36

I'm I'm pretty comfortable in the CLI.

20:37

So, and then yeah, my my

20:39

>> you don't need my generated game CLI

20:42

commands.

20:42

>> It's it's a very cool idea. Um, I would

20:44

say, yeah, my my my whole goal these

20:47

days is essentially to just have as many

20:49

running as possible and to run them all

20:51

the time. And you know, so for example,

20:53

like every night before I go to bed, I'm

20:55

I'm like, "Okay, I

20:56

>> Let's go, guys.

20:57

>> Yeah. Basically, what I have to do is

20:58

make sure that I've given it enough

21:00

stuff that by the time I wake up in the

21:02

morning, it it will still not be done.

21:04

And so I've maximum

21:05

>> That's victory.

21:06

>> Yeah, that's that's victory. Yeah.

21:07

>> So, yeah, like I've I've I've done that

21:09

I would say last last five nights pretty

21:13

well. My personal record is that I've

21:15

had a a coding agent running for I think

21:17

it was 13 days straight uh without

21:21

stopping and just just basically working

21:24

through like tasks.

21:25

>> Well, well prompted. Yes. I I admit to

21:27

having woken up in the middle of the

21:28

night at least multiple times this week

21:30

and just being like, "Are you still

21:31

going?"

21:32

>> Yeah, I know. Yeah. It's it it's kind of

21:34

nerve-wracking. I I always like there's

21:36

always like I I'll check it one last

21:37

time before bed and just really make

21:39

sure that it's still spinning.

21:40

>> What about on the notion agent side?

21:42

Like do you have a workflow there that

21:43

is core to daily work?

21:46

>> Yeah. I mean I mean I I use our personal

21:48

agent all the time. So it's it it has

21:50

all the context about about our company

21:51

and everything that's going on, you

21:52

know. So like for example, last night I

21:55

was asking it about um how the custom

21:57

agents launch was was going and like

21:59

like like what the what the signals were

22:00

getting from it. We're super useful for

22:02

that. And then for I I have many custom

22:04

agents that are that are running. U my

22:06

my my personal favorite is I have a

22:09

email triage agent. So it has access to

22:12

all of my work and personal emails. Um

22:15

and it just uh wakes up every day and

22:17

just archives all the stuff I don't need

22:18

to see. Train it over time to uh to to

22:21

learn my preferences.

22:22

>> Do you actually label data for it?

22:23

>> It's pretty to do actually. So all you

22:25

have to do is you make the agent and

22:26

then you give it access to email and

22:27

then you you can make a blank page. It's

22:29

like it's memory

22:30

>> and you let it edit that page and then

22:33

you just say okay now go look at my

22:36

emails and then interview me ask me

22:37

which things you know so sort of it will

22:40

like propose things that it thinks it

22:42

should archive

22:43

>> and then you can kind of correct it and

22:45

then we'll use that to essentially

22:46

generate like a list of rules about like

22:48

like what it thinks are correct or not.

22:50

And so for the first couple days I was

22:51

sort of like like like uh correcting it

22:53

on things. After a couple weeks or so, I

22:56

I I I dropped the approval entirely and

22:59

it just automatically archives all the

23:00

things I need to see now.

23:01

>> Wow.

23:02

It It completely solved my email

23:04

problems cuz for me, like I don't I

23:07

don't use email that much for work

23:08

stuff. Like it's it's mostly in Slack.

23:10

95% of the personal emails and working

23:12

emails that I get, I don't need to see

23:15

at all. And so it's just a waste of

23:16

time. Uh and so it it it completely

23:18

solved that. So now when I have my

23:19

inbox, it's like only stuff I need to

23:20

see. I've got lots of uh custom agents

23:22

running. Uh there's another one um that

23:25

I built that uh uh triages uh customer

23:28

fe u all all internal feedback and and

23:30

and

23:32

bugs. So we have a Slack channel where

23:34

basically people just just uh post

23:37

random like like product feedback and

23:39

bugs. In the past it was it would sort

23:41

of sometimes get answered but then

23:42

sometimes like like half-hazardly get

23:44

ignored just because you know there's so

23:46

many teams where things uh so its entire

23:49

job is just to route it to the right

23:50

place. uh and and it it uses a similar

23:54

sort of like like memory pattern where

23:55

it sort of learns on the fly uh where

23:58

it's supposed to file bugs uh and then

24:00

over time it's built up like like

24:02

hundreds of roles that it just um sort

24:03

of like like learned over time, you

24:04

know. So for example like if there's a

24:05

there's a bug about the mobile app, it

24:07

knows to route to the mobile team and

24:08

then a file a task in their database.

24:10

>> Do you look at that um like the

24:13

generated and updated memory to like

24:15

because it's legible to you to say like

24:17

did that make sense to me? I think I did

24:19

it I did at first. Uh but then sort of

24:21

once you trust it's kind of working, you

24:23

just you kind of ignore it and then if

24:25

if it ever breaks, I'll I'll go fix it.

24:28

Yeah, it it'll break every now and then

24:30

and then um

24:31

>> but the benefit reading your email is

24:33

>> here.

24:33

>> Yeah, just not read it. So yeah. Um

24:35

yeah, I I mean generally I would say

24:37

yeah the the general pattern I follow is

24:39

sort of I I build it as a prototype. I

24:41

have it in sort of like an approval mode

24:43

where I'm sort of, you know, watching it

24:45

closely and then but then after it runs

24:47

a bunch of times, you kind of trust that

24:49

it's working. And then

24:50

>> is there anything you do internally at

24:52

notion to um make sure non-technical

24:55

teams have the intuition for how to

24:56

build agents or how to like express that

24:59

productivity too?

25:01

>> Yeah, it's a great question. I mean, we

25:03

do uh sort of workshops and hackathons

25:06

pretty frequently. So like for example

25:08

like a month ago I did a I did a

25:10

hackathon with uh the the people team

25:12

and sort of sort of got them the the

25:14

people team has been amazing. They're

25:15

actually one of the the highest adopters

25:17

of custom agents.

25:18

>> You know they do all these kind of

25:19

workflows in like Slack and notion kind

25:21

of like like manual work like that and

25:22

um and yeah I would say yeah like like

25:25

people are super excited to to try it

25:27

and sort of like like maybe just need

25:29

like a little bit of a push in terms of

25:30

intuition and like like getting them

25:32

started. Um, but then honestly I've been

25:34

super impressed like I I think the

25:36

concept is like kind of intuitive sort

25:39

of like like once you get once you get

25:41

past sort of a little bit of the

25:42

technical barrier of like what is a

25:43

prompt and like what is the agent and

25:45

how does it get triggered and woken up

25:47

and like like how does that even work?

25:49

But then once you sort of get past that,

25:51

I think it's actually a very humanlike

25:53

interface.

25:54

>> Yeah. Maybe the maybe the biggest

25:55

barrier is actually just getting people

25:56

to try and assuming it's going to work

25:58

at all. Right.

25:59

>> Yeah. Yeah. You and Ivan originally met

26:02

on the internet tools for thought

26:04

community. Um it feels like you know the

26:08

tools we have for thinking are very

26:10

different now. Has your like core

26:11

conception of notion changed over the

26:13

last few years because of all the AI

26:15

stuff like what what is the what what

26:18

thinking does the tool do for you?

26:19

Should agents do for you? What do you

26:21

get to do?

26:22

>> Yeah, I mean it's I would say changed

26:24

quite a lot. I mean, broadly speaking,

26:28

before AI, our our our our goal was to

26:33

create the best tool for humans to

26:35

directly perform their work.

26:38

>> And then now the goal is to create the

26:40

best tool for humans to manage agents to

26:43

do the work for them.

26:44

>> That's a big shift.

26:45

>> That's a pretty big shift. Uh it's it's

26:47

pretty fundamental. Um but it it turns

26:50

out that you need most of the same

26:51

primitives. uh you actually all the

26:54

primitives that we built are actually

26:55

still extremely useful. It's it's more

26:57

that we just needed some some new

26:59

primitives like like representing what

27:00

is an agent and you know how does it

27:01

interact with your pages and databases

27:03

but you know you still need the same

27:04

primitives. You still need a document.

27:06

It's an unstructured way to you know to

27:08

write stuff. Uh agents love to write

27:10

markdown documents. So

27:12

>> yeah,

27:13

>> it's still very relevant and you still

27:14

need a database. It's um you still need

27:17

structured data. you know, if you're

27:18

working with your your swarm of like 100

27:21

background coding agents, you don't want

27:22

to have 100 chat threads. You want a

27:24

kemb board. It's, you know, the same as

27:26

before.

27:26

>> Makes sense. You still need the uh the

27:28

coordination structure. What is one

27:31

thing that just because you're ahead of

27:33

the on this stuff and then trying to

27:35

figure out how to bring, you know,

27:37

notion and then users along with you.

27:39

What is something that's really changed

27:42

about um how you personally like build

27:45

even in the last six months? I mean,

27:47

it's completely changed. I haven't

27:48

written code since like last summer. I

27:52

don't type code anymore.

27:54

>> Yeah. It's it's it's completely shifted.

27:55

I mean, we went from humans type all the

27:58

code to like we're still typing, but we

28:00

like tab complete to sort of like we

28:03

talk to the agent and it sort of does

28:05

little tasks for us, but we are still in

28:06

the outer loop. And then now it's more

28:09

like I I design a endto-end task that

28:14

involves like making some change and end

28:16

to end verifying it. And then I'm just

28:17

the the outer you the outer verifier

28:19

sort of like like double checking at the

28:21

very end that it that it's correct and

28:23

if it's going off the rails kind of like

28:25

like monitoring it. Um so it's a it's a

28:28

complete shift is you know I'm I'm now

28:30

like the agent manager instead of the

28:32

coder.

28:33

>> Amazing. Well um thanks Simon. This has

28:35

been a super great discussion about how

28:37

we're all going to become Asian managers

28:39

and uh uh hopefully in notion.

28:41

>> Cool. Yeah.

28:45

>> Find us on Twitter at no prior pod.

28:47

[music] Subscribe to our YouTube channel

28:49

if you want to see our faces. Follow the

28:51

show on Apple [music] Podcasts, Spotify,

28:53

or wherever you listen. That way you get

28:54

a new episode every week. And sign up

28:56

for emails or find transcripts for every

28:58

episode at no-fproers.com. [music]

Interactive Summary

Simon Last, co-founder of Notion, discusses the company's aggressive embrace of AI, initiated by their early access to GPT-4 in 2022. Notion's vision evolved from a short-term writing assistant to a long-term general agent capable of utilizing all Notion tools. The company launched its AI writer in February 2023, followed by a Q&A feature with semantic indexing in October 2023, and later expanded to index external data sources like Slack and Google Drive. Simon highlights the engineering challenges in building an effective indexing system for diverse content, emphasizing an empirical and iterative approach. Due to rapid AI progress, Notion rewrites its AI harness every six months. Coding agents have dramatically increased the ambition of engineering projects, leading to higher individual impact, more prototypes, and better-tested code. Notion has released both personal agents, which have access to user content, and custom agents, which can run autonomously for specific tasks like email triage or feedback routing. Notion views itself as a "Switzerland for models," integrating various frontier and open-source models, and adapting its APIs with a markdown dialect for blocks and SQLite for databases to be agent-friendly. Simon notes his personal shift from a coder to an "agent manager," overseeing agent-driven tasks.

Suggested questions

10 ready-made prompts