HomeVideos

5 Rules for Building AI Agents That Work in Production | Nan Yu & Jacob Shumway

Now Playing

5 Rules for Building AI Agents That Work in Production | Nan Yu & Jacob Shumway

Transcript

1273 segments

0:00

The models are really smart, but we're

0:01

just not using them enough. If you

0:03

overprompt these things, you're more

0:05

likely than not going to make it worse.

0:07

>> Give it as little instruction as

0:08

possible. Give it the tools to load

0:10

context. Don't give it context.

0:13

>> Here's all the context. Just figure out

0:14

what the right thing is. Write the issue

0:16

and work on it. Linear made the issue.

0:18

Took 6 minutes and then it gave us a PR

0:20

that we click run. You have [music] to

0:21

really break down what is the actual

0:23

workflow that your users want to do.

0:25

Computers can do a lot of work for us.

0:27

So let's get rid of all the work we

0:29

don't want to do and give it to

0:30

computers.

0:33

All right. Hey everyone. Today I'm

0:34

really excited to welcome Naan and Jacob

0:36

from Lineer do a special episode. We're

0:38

going to do a deep dive on how to build

0:41

a production agent end to end and how it

0:43

actually works and we're going to use

0:45

Linear's own agent as an example to keep

0:47

the discussion concrete and real. So

0:49

welcome guys. Yeah, good to be here. All

0:51

right. So why don't we start at the

0:52

super high level. Can you kind of

0:54

demystify this whole agent thing? What

0:55

does the explain like on five version of

0:57

what an agent is?

0:59

>> Yeah, sure. Um, I can answer that one.

1:01

So, at a high level, an agent is really

1:04

just calling an LLM in a loop. Um,

1:07

normally when you call an LLM, you're

1:08

going to give it one question, you're

1:10

get one answer back. But we want agents

1:12

to be really autonomous and be able to

1:14

accomplish more complex tasks. So,

1:17

typically what you'll do is you'll

1:18

define a goal, some milestone for the

1:21

agent to hit, and you'll give it tools

1:23

that allow it to build its own context,

1:26

and then just run it in a loop.

1:27

Question, answer, question, answer. On

1:28

each turn, it's going to call tools,

1:30

pull context in, and eventually it's

1:32

going to hit a point where it has enough

1:33

information to consider the goal

1:36

accomplished. It'll it'll like

1:38

synthesize this final response and send

1:40

it back.

1:40

>> Got it. Okay, that that makes sense.

1:42

Yeah, it's it's basically a model using

1:44

tools running on a loop. That's kind of

1:46

that's kind of like the one.

1:47

>> Yeah. Okay, cool.

1:49

>> Um Okay, then let's talk about the

1:50

linear agent. Maybe now you can show us

1:52

like the initial idea behind this thing

1:54

and then like now what it's kind of

1:56

evolving to.

1:57

>> Yeah. Yeah. I I think you know I I I

1:59

think it's actually pretty interesting

2:00

to expand on Jacob's answer a little

2:02

bit, right? Because you know you we you

2:05

we asked the question and I I think that

2:06

that is like the the correct sort of

2:08

technical definition. You know, you're

2:10

asking engineers what they're going to

2:11

say. Um, but I also think that when we

2:14

talk about agents like colloquially, you

2:16

know, we think of them as like products,

2:18

right? It's like it's like a packaging

2:19

of some sort of uh AI loop plus some

2:22

other stuff. And I think ultimately

2:25

these things are like a bunch of

2:26

different subsystems that are all, you

2:29

know, interacting and then there's like

2:30

one facade, right? It could be a chatbot

2:33

or something like that that that kind of

2:34

fronts all of it. So, you know, you if

2:36

even if you think of like the the

2:38

desktop agents people use, there's they

2:39

have all sorts of stuff built into them

2:40

like schedulers and uh and and these

2:43

these other sort of like side uh you

2:45

know, tertiary kind of features or

2:47

they're all kind of part of the agent as

2:48

a product too, right? So, I I think we

2:50

kind also have to think about it from

2:51

that perspective.

2:52

>> True. True. Yeah. Yeah. Sure. Do you

2:54

think about all this when you had the

2:55

initial idea or like what is your

2:57

initial spec or

2:58

>> Yeah. Yeah. Sure. So, I I'll show you

3:00

right now. I was working with Jacob the

3:02

other day. I'm like I dug this up. This

3:04

is a memo I wrote in like if you look

3:05

it's like the second half of uh 2025,

3:07

right? So it's like not that long ago in

3:09

human time, but in like AI time it's

3:10

it's you know ancient history. Um and

3:12

and I I think on on here, you know,

3:14

you'll you'll see like there's a lot of

3:17

focus that we had on like like computers

3:19

can do a lot of work for us. So let's

3:22

get rid of all the work we don't want to

3:23

do and give it to computers, right? Like

3:24

that that's like the central sort of

3:26

central thesis of this thing. And uh

3:28

it's you know and we kind of structured

3:31

this idea about um you know before we

3:34

really had this sort of technical idea

3:36

in our head. We had this idea about like

3:37

there's some kind of triggering event

3:38

and there's some context that happens

3:39

and you have instructions and it kind of

3:41

loops on on actions and then and ends

3:43

with some kind of result. Right? So like

3:45

we we we already had this sort of

3:48

concept but I I think what we uh we were

3:51

just we we weren't like ambitious enough

3:53

like we did we didn't think it could do

3:54

like really interesting things. were

3:56

like, "Hey, let's give, you know, let's

3:58

create this agent thing." We call it,

4:00

you know, we gave it a really like

4:02

robotic name. It's like robotic program

4:03

manager with the idea, right? And and

4:06

just give it the boring stuff, right?

4:08

And and I I think that what what we

4:10

what's changed, right, about how we

4:11

think about it today, right, which, you

4:13

know, we're in the second half of 2026,

4:14

so it's literally just a year later. Uh

4:16

is that we we don't think of just giving

4:18

it the boring stuff. Sure, you're going

4:19

to give the boring stuff, but also

4:21

there's it opens up a lot of like the

4:24

creative possibilities, right? Where

4:26

where you can do it interactively and

4:28

and uh and and it could be a big force

4:30

augment for like interesting work,

4:31

right? Not just not just the boring

4:33

stuff.

4:33

>> I don't want to offend anyone, but I

4:34

feel like um technical program manager

4:37

is like one of the most boring jobs do.

4:38

You're literally just [laughter]

4:40

like managing spreadsheets, trying to

4:41

track tickets and stuff, right? So So

4:43

this is what the agent started with,

4:45

right? But but now it can do now it kind

4:47

of serve multiple hats kind of do end to

4:49

end product development, right?

4:50

>> Yeah. Yeah. Exactly. And and and

4:52

depending on you know what you use it

4:54

for and the context that you kind of put

4:56

into it. It can do all sorts of very you

4:57

know creative things that we at that

4:59

point we just like didn't even think was

5:02

uh a reasonable thing to expect, right?

5:03

And things have moved very quickly.

5:05

>> So I'm really curious what your process

5:07

is. So you wrote this memo back in uh

5:09

third quarter and then uh Jacob just

5:11

build it in a couple days or what was

5:12

the [laughter] process? Yeah. That was

5:14

the process.

5:16

>> Yeah. What was the next step to to

5:18

actually build this thing to prototype

5:19

it or like play with it?

5:20

>> Yeah.

5:21

>> Jake, what was it the first version of

5:23

this that we actually built from here?

5:25

>> The first version was was really

5:27

prototypy. It was like we were calling

5:30

the LLM from the front end directly

5:33

>> and it was just flagged internal and we

5:35

like we gave it access. We have our

5:38

command menu with all our actions. We

5:39

gave it access to those as tools and

5:41

we're just like let's see what this can

5:42

do. It was It was really hacky.

5:44

>> Mhm.

5:45

>> This episode is brought to you by

5:47

Oceans. I hired someone through Oceans

5:49

for podcast post-production a few months

5:51

back and can't imagine running the

5:53

podcast without [music] his help. He's

5:54

proactive, picks up new tools fast, and

5:56

uses AI to compound everything that he

5:59

ships. Oceans doesn't just [music] place

6:00

assistants, they place operators. The

6:03

talent is AI fluent and delivers the

6:05

same output as a senior US hire at 3 to

6:08

[music] 5x less cost. They reject 99% of

6:10

applicants. So the person who lands on

6:12

your team is already operating from day

6:14

one. If you're scaling and need

6:16

marketing, [music] ops, finance, or EA

6:18

help, I highly recommend giving Oceans a

6:20

try. Check it out at oceansalent.com/

6:22

[music]

6:23

Peter. Now back to our episode.

6:25

>> Right now you can like, you know, push

6:27

code, you can manage tickets, ingest

6:28

stuff, but like what were some of the

6:30

initial use cases that kind of popped

6:31

out that that like, you know, you wanted

6:33

to prioritize first?

6:34

>> I think it it was simple things. I think

6:36

creating issues like it's such a simple

6:38

thing but I think that was one that

6:40

surfaced really early is useful actually

6:42

[snorts]

6:43

writing documents that sort of thing.

6:45

>> Yeah. I I I think one of the first like

6:47

real um use cases that we knew about

6:50

that we knew people wanted to do was

6:52

they would have uh you know this is

6:55

again this is like maybe even before

6:57

everyone had uh you know automatic note

7:00

takers for everything right like people

7:01

were still like handwriting notes and

7:02

things like that. they were like, "Hey,

7:03

I I I hand I I wrote some notes on the

7:05

sales call and they they said a bunch of

7:07

stuff. Can I just dump this in there and

7:09

just extract out, you know, like the

7:11

issues that we need to build, right, for

7:13

this or the bugs that were reported or

7:14

or those kinds of things, right?" So,

7:15

like it it was that, you know, that that

7:17

was the very basic operation that we're

7:19

like, "Okay, if we let's get that

7:20

because like I know that's valuable.

7:21

People have directly asked for that.

7:23

That's something that we felt, you know,

7:24

in our own workflows. So, if we can get

7:26

something working that can achieve that,

7:29

like that's a that's a reasonable

7:30

starting point, right?

7:32

For us, it's it's we didn't even know if

7:34

it was going to be uh we're going to

7:36

have a chatter interface, right? We're

7:37

like like there there's some place to

7:39

just dump a bunch of text and maybe, you

7:40

know, maybe it's just like a a text

7:42

field or something like that and you you

7:43

just hit submit and it goes, right? So

7:45

maybe maybe it looks more like that. So

7:47

I I think it was it was very open-ended,

7:50

right, when we when we first started,

7:51

which is also why we don't have like a

7:52

super robust spec for it, right? Like we

7:54

didn't go into this thinking that we

7:55

knew exactly what we needed to build. We

7:57

just we're just like here's some

7:58

experiments that some directions we

7:59

could try and let's just do it.

8:01

>> Yeah. So you launched it on Slack or

8:03

something for people to use internally.

8:04

>> Yeah. So the the the first production

8:06

version of this we we launched uh sort

8:09

of secretly right without like really

8:11

telling anybody that uh because we we

8:12

already had a Slack integration and

8:13

Slack you could always mention bots

8:15

right we everyone knows you know like

8:16

things like donut and things like that

8:18

where you know it's it's all very

8:19

procedural and like they're like these

8:20

different Slack bots. Uh so you could

8:22

mention linear before and you know it

8:24

give you like a form or something like

8:25

that and uh we we just like silently

8:28

hooked it up to this right we're just

8:29

like okay we're just going to do it and

8:30

then if anyone discovers it by app

8:32

mentioning in linear they can start

8:33

talking with it and then you know they

8:34

can they can do whatever they want and

8:36

and a lot of uh like a lot of usages for

8:38

it emerged that we didn't even expect

8:40

and like that that was and like we we

8:43

sort of had the inkling that that's what

8:44

would happen right is that people would

8:46

do things and like the obvious thing

8:48

would would be something like hey linear

8:50

make an issue to and then you would

8:52

describe it in a natural language and it

8:53

would do it. But people started

8:55

realizing that because they could read

8:56

the context, they could just say

8:59

something like linear do the right thing

9:02

or like look at what we just did, right?

9:05

Or just like something extreme, you

9:06

know, you can be super lazy,

9:08

>> right? Like people will be like, oh, I

9:09

just say at linear and then upwards

9:11

pointing finger emoji, right? Like it's

9:13

like they they do those kinds of things

9:14

and then linear will just

9:15

>> reason through like what happened? Like

9:17

I I know how to create issues. it looks

9:19

like there's some issues name so I'm

9:20

just going to make some issues and then

9:21

and then tell the user that I did that

9:23

right so like it it this kind of like

9:24

behavior was actually very emergent and

9:26

and we we didn't expect it to be able to

9:28

do this

9:29

>> interesting okay so just so understand

9:30

you built this thing and and like kind

9:32

of get the model you made the model

9:33

aware of all the internal APIs that has

9:35

or something people can use the UI for

9:37

the model can also do

9:39

>> and they just kind of let it loose let

9:41

loose for people to try

9:42

>> yeah effectively I mean I think Jake we

9:44

could probably talk about like you know

9:45

we tried certain techniques at first and

9:47

then we sort of landed on on a version

9:48

of it that have now

9:50

>> we tried to give it essentially just

9:52

like everything you can do in linear

9:54

which is a lot of different actions um

9:56

across all the different surface and we

9:59

ran into just like context issues and

10:00

hallucination uh we tried things like

10:03

let's give it our graphql schema and see

10:05

if it can write queries um and that that

10:07

didn't work really well and we kind of

10:08

ended up on a skills type of setup where

10:11

>> we we give it the act the ability a tool

10:15

essentially to load skills and then it

10:17

based on the request it'll just load up

10:19

the different skills that it needs and

10:21

that comes with a set of tools and

10:22

instructions around that.

10:24

>> The skills are defined by you guys like

10:26

create ticket skill or like you know

10:29

>> because we we have opinions on how

10:30

different things work. You know, if

10:32

you're writing an issue, how do you

10:34

think about setting a priority and how

10:35

do you think about writing the

10:36

description? So, we encode all of that

10:38

in these skills.

10:39

>> Got it. Okay, that makes sense. Okay.

10:40

>> Yeah, I I I think that that's you know

10:43

when you when you have a native agent

10:44

like this, right? Like people talk a lot

10:47

about like, hey, they build a CLI or

10:48

they build MCP or something like that

10:49

and it comes with a bunch of skills for

10:51

how to use it. When you build a native

10:52

agent, you can go bug wild with this,

10:54

right? You you could have hundreds of

10:55

these things and because you have

10:56

dynamic loading and you have control

10:57

over how everything works. Like you can

10:59

have a very smooth and opinionated way

11:02

of how it uses your app. And I I think

11:04

that this is one of the big advantages

11:05

of having a native agent, right? It's

11:07

like it's like you can you can just

11:08

bring you can just treat it like a like

11:10

a power user of the app. there's no

11:12

there's no uh variance right in in in

11:15

doing that. So I think that that's

11:16

that's what we that's where we ended up

11:18

and then because it can just you know

11:20

run a loop and decide what tool calls to

11:21

make and stuff like that. It it all of

11:23

this emergent behavior about just being

11:25

super lazy when you app mention it.

11:27

It'll just figure out the right thing to

11:28

do just kind of came out.

11:29

>> Can you guys show us well the product is

11:31

pretty polished now but can you guys

11:32

show us some examples of like tagging

11:34

linear in Slack in different channels

11:36

and see see what it does?

11:38

>> This is a conversation we just had.

11:39

there's some behavior that you know I'm

11:41

like look this looks a little weird and

11:42

I I so like here here's the pattern

11:44

right it's like it's very natural we

11:46

have a conversation in Slack and I'm

11:47

like tagging our designer Yan and Jacob

11:49

right and and about maybe some

11:51

suggestions about what we can do here

11:53

and we're we're trading

11:55

>> ideas right it's not like we have some

11:57

exact sense of what we want to do right

11:59

now right like we're like hey like this

12:01

feels bad here maybe I'll try these

12:03

things and then you know designer like

12:06

kind of gives his opinion here and then

12:08

I I try to clarify right? You were like

12:10

kind of finding where you know where the

12:12

actual problem is. Um and you know Jacob

12:16

uh you know raises a uh an objection and

12:19

I'm like look we can just here here's my

12:21

how we want to address that objection.

12:22

So like we're like finding the truth so

12:24

to speak and you know at the end of the

12:26

day like the message is just like just

12:29

do it right like at linear create

12:32

[laughter]

12:33

issue for me I'm going to just I want to

12:35

hold on to it and then just and then now

12:37

that linear can write code just just do

12:39

a pass like I I'll take a look at what

12:41

you did right like previously it was

12:42

just make an issue for me but here's all

12:44

the context just figure out what the

12:46

right thing is like we we argued about a

12:48

bunch of stuff we came we ended up

12:49

somewhere

12:50

>> so like figure out what somewhere is and

12:51

then write the issue Now it's write the

12:53

issue and work on it. So then linear uh

12:56

you know made the issue there's a

12:58

decision you know and then it's um it's

13:00

assigne is Jacob and it delegated to

13:02

itself right and then uh that was 18

13:05

minutes ago it took six minutes and then

13:07

it can gave us a PR that we can click

13:09

through right so like this was the whole

13:12

sort of loop of of here's an idea and

13:15

then we talked about it and we figured

13:16

out you know hopefully where we got to

13:18

and then linear sus it all out and then

13:21

and then made a PR

13:22

>> interesting and then and then now you

13:23

can just go in here and like play with

13:24

it and see if it's actually a good idea

13:25

or not, right?

13:26

>> Yeah, you can play with it and see if

13:27

it's a good idea or not. And and

13:28

importantly, right, like because it's

13:30

part of like the linear system, like

13:32

this is in Jacob's backlog, like it's in

13:34

his, you know, status to-do, right? So,

13:36

it's in his personal backlog, like he

13:38

doesn't lose track of it. You know, if

13:39

if it was just stuck in Slack, it would

13:41

just be like, you know, you have more

13:42

chats and then all of a sudden you just

13:44

lose track of it and then, you know, you

13:45

would,

13:46

>> you know, kind of cross your fingers

13:47

with search or hope that someone

13:48

remembers it or something like that,

13:49

right? So like the actual tracking like

13:51

aspect of this is still matters, right?

13:53

You you know because you have it in an

13:54

organized backlog that's in the right

13:56

project and everything. So that's that's

13:58

that's that's the ultimate result,

13:59

right? When you think about like what is

14:00

linear's goal, its goal is to like put a

14:02

ticket in the right place and also, you

14:04

know, accomplish the task and and uh

14:07

maybe this is actually kind of

14:08

interesting. It is like an interesting

14:09

product principle. So even though linear

14:10

actually did a work, it's assigned to

14:12

Jacob. Is is it like a principle like

14:14

every agent has to be tied to a human?

14:16

the the the vast majority of them are

14:18

right like there's going to be

14:20

situations where um you know it's it's

14:23

really you know the the system

14:24

effectively invokes itself right if like

14:26

you know if you're if you instrumented

14:28

like a data dog or something like that

14:29

there was an alarm that tripped and it

14:31

threw a you know put a bug into the

14:33

system and then linear try to solve it

14:34

like there's not no one really touches

14:36

it until the very end.

14:37

>> So at that point you're relying on the

14:39

agent to figure out who should review

14:40

the code right all the way there. this

14:42

is you know for something like this like

14:43

someone made a decision to like do this

14:45

right in this case it was Jacob who's

14:47

like okay cool I think we have enough

14:48

information now let's let's work on this

14:51

right so like that that way he he he he

14:54

has a handle to it and it's attributed

14:55

to him

14:56

>> got it okay what kind of information and

14:58

context can this agent access uh like

15:00

obviously all the tickets in line can

15:03

read slack and stuff or like you hook up

15:05

to you know gone and every everything

15:07

else yeah

15:08

>> uh yeah so this this was this context

15:10

was just from slack from this thread,

15:12

right? So, it can read Slack and and I I

15:14

think a lot of the

15:16

>> a lot of the benefit comes from

15:18

stitching all this together,

15:19

>> right? Like Slack by itself isn't

15:20

enough. But if you combine Slack with

15:22

the ability to read your codebase with

15:24

the ability to read like you know your

15:25

project description and your other

15:26

tickets and stuff like that, then all of

15:28

a sudden you can do something, right?

15:29

Because like what could have happened

15:30

here was uh you know we we we you know

15:34

we made the issue and said, "Oh, looks

15:36

like I found someone else who actually

15:37

did this already, right? There's there's

15:38

actually an open PR that exists in the

15:40

system." like it would have told us that

15:42

that's what happened, right? So because

15:43

it has access to all that stuff. It

15:45

doesn't just like you know bulldoze its

15:47

way through this. It like it knows it's

15:49

aware of everything else in the system.

15:50

>> Got it. Okay. So let's just go back to

15:52

the early like when you stealth launched

15:54

this agent in Slack, right? Like so

15:56

people probably started using it and

15:57

started getting some feedback. Was the

15:59

product was just like a simple basic

16:00

prompt and some tools that was the

16:02

product very complicated back then or is

16:03

it pretty simple? How do you improve it?

16:05

Um, at that point it it was it

16:10

technically had the ability to do a lot

16:12

of different things, but it wasn't

16:13

really good at it yet. I don't think

16:14

we'd arrived on our skill architecture

16:15

yet, which unlocked a lot of things. Um,

16:18

the primary thing people were using it

16:20

for was just creating issues. Um, so we

16:23

actually optimized pretty heavily around

16:24

that. We created a small little router

16:27

and for 80% of these use case, we routed

16:29

to a specific subprompt that was just

16:32

for creating issues like highly

16:33

optimized for that. Um, so that's what

16:35

most of our usage was in the really

16:37

early days.

16:38

>> So the structure of the product was like

16:39

there's like a main prompt that maybe

16:41

tells the agent what it can do and stuff

16:42

and then it it kind of routes to like

16:44

subprompts, right? Is that it?

16:46

>> Yeah. Yeah. Yeah. Or there's there's a

16:48

really small model that runs a router

16:50

and that will send it to either this

16:52

like big model main prompt that can do

16:54

anything and has access to all these

16:56

tools or these specific use cases like

16:58

creating an issue.

16:59

>> Oh, interesting. Okay. So there's a best

17:01

practice saying that like when you're

17:02

prototyping an agent, you should use the

17:05

best model available just to see what

17:07

it's capable of, but it sounds like you

17:09

guys actually use a mix of different

17:10

models for different tasks.

17:12

>> We do, but I would say we follow that

17:14

best practice for the most part. Uh we

17:16

tend to throw the biggest model on it

17:18

until we know that it's working well. We

17:20

build out some eval. We have a good idea

17:22

of like what success criteria looks like

17:24

and then you can start to optimize the

17:25

model down because you have a really

17:27

good framework in place to like make

17:28

sure it's still meet. Ideally, you want

17:30

to use the smallest model for the job,

17:32

right?

17:32

>> Yeah. Because Yeah. You want to save

17:34

save money, right? You [laughter]

17:36

>> Yeah. Okay. So, so then um it's like

17:38

very iterative. Like in the beginning,

17:39

you probably don't have a ton of eval

17:41

set up like automated evals and stuff.

17:43

>> Yeah.

17:44

>> Yeah.

17:44

>> Got it. Just to make this pretty

17:46

concrete, like uh let's take the create

17:47

ticket thing, right? You probably have

17:48

some emails for did they actually create

17:50

a ticket or not or like is the ticket

17:51

useful or how do you evaluate how good

17:53

it is?

17:54

>> Yeah. Um, yeah, I mean, yeah, pretty

17:56

much. We have eval. A lot of it comes

17:58

from just iterating and using it. So, a

18:01

user will use it in a way that isn't as

18:04

expected. We'll add that to our data set

18:05

for our eval.

18:09

Um, but we try to have a mix of like

18:11

objective and then more subjective

18:12

measures.

18:13

>> Objective and sub. Okay. So, objective

18:15

is like uh deterministically do this

18:17

thing or not.

18:18

>> Yeah, exactly. If the user says in

18:20

progress, make sure it always adds the

18:22

status in progress. that's really

18:24

deterministic. And then there's more

18:25

subjective things like did you structure

18:27

the description in a in a good way? Did

18:30

you extract the the right information

18:32

that should be the title into the title

18:34

field?

18:34

>> How do you evaluate like is that like a

18:36

yes no thing or is it like a scoring?

18:39

>> Yeah, it's it's LLM as a judge. So we'll

18:42

we'll yeah again we build out this data

18:45

set over time. Then we just have a a

18:47

scoring type of LLM that's like did this

18:50

extract the right information? this is

18:52

what it should be pretty much.

18:54

>> And then and then I don't know I don't

18:55

have a ton of experience doing this

18:56

stuff but like I I feel like you have to

18:59

you have this element as a judge which

19:00

sounds really fancy but you have to look

19:02

at the judge and be like hey is this

19:03

actually judging it correctly or not?

19:05

It's actually pretty manual, right?

19:07

>> Yeah. And we we actually try to use

19:08

those less often for that reason. I

19:11

think I think eval are most successful

19:13

when you're ensuring consistency

19:17

somewhere that consistency is important.

19:18

But consistency is not always important

19:20

for agents. They can have a lot of

19:22

variance in how they respond to things.

19:24

Um, and you really don't want to have

19:26

too many emails around that because then

19:27

it just false signals.

19:29

>> Okay. Interesting. Okay. So, you started

19:31

with uh creating a manage tickets and

19:33

then uh what are some other use cases

19:35

that you you decided to support before I

19:37

I don't remember when the thing was

19:38

first launched but like before the first

19:40

launch. So I I think the the the way

19:43

that you should think about this is that

19:44

like there's a because it's especially

19:46

because it's like um very purpose

19:49

specific. There's like this power law of

19:51

like use cases, right? Like if you think

19:53

about like what the purpose of linear

19:56

is, it's like you're you're trying to

19:58

structure uh the intention of the of the

20:01

company,

20:02

>> right? Like you you have ideas, you have

20:04

meetings, you have Slack conversations,

20:05

you have whatever, right? Discussions.

20:07

And at some point you make a decision.

20:09

And we're, you know, we were originally

20:10

like a system of record to codify those

20:12

decisions. So like if you if you just

20:15

like look at the distribution like the

20:16

vast majority of um how people use

20:19

linear like the agent, right, is like

20:21

codify decisions. Like I had this this

20:24

you know free form conversation with a

20:26

customer. They they complained about a

20:28

few things. Let's extract what those

20:30

things are and figure out what what to

20:31

do with them. or um you know the the

20:35

sort of thing that we demoed uh or or

20:37

even like hey we changed our opinion

20:39

about something right we we we had a we

20:41

had a big meeting about something just

20:43

like pull the notes and like we changed

20:45

so many opinions about this uh the spec

20:47

of this project or something like that

20:49

just go and update all this so that all

20:50

the the marketers and stuff like that

20:52

don't get false information. So [snorts]

20:55

all those are the those are like the

20:57

motions right that are are fairly common

20:59

but like ultimately this is powered by

21:02

you know Frontier LLMs right you can do

21:03

anything you can do anything that you

21:05

want right and like like what you know I

21:08

I I've used things I've used it for

21:09

things like being an interview grader we

21:11

have like a document in in one of our uh

21:13

teams that's like here's the criteria

21:15

for you know how we want to evaluate

21:16

someone's like interview process or

21:18

whatever it is and because I have the

21:20

granola MCP connected to my linear agent

21:22

right I can just be like, hey, that last

21:25

interview I just had, could you just

21:26

quickly give me a score on how we go

21:28

against this rubric, right? So, I don't

21:30

have to like read through all the notes

21:31

and remember it's like a starting point.

21:33

So, there's like a lot of things that

21:34

you can uh you can utilize it for as

21:37

long as you have uh the context

21:39

somewhere.

21:39

>> Got it. Okay. When you guys launched Lar

21:42

agent, it wasn't like uh here's the use

21:43

cases that actually pass all the eval

21:45

focus on those use cases in the

21:47

marketing, but like I guess the user can

21:49

do other things too if they want to

21:51

because it's just like a LM, right? Is

21:53

that Yeah.

21:54

>> I I I I think the the the way the way

21:57

that the way to think about it is like

21:59

>> it's it's almost like what's the biggest

22:02

problem in applied AI right now, right?

22:03

The biggest problem is not the agents

22:05

aren't smart enough, right? The the

22:07

problem is not the models are not

22:08

advanced enough. The problem is uh

22:10

there's like you know people talk about

22:11

like a capability overhang or like a

22:13

capacity overhang or something like that

22:15

which is like the models are really

22:16

smart but we're just not using them

22:17

enough. And and so like where where

22:20

we're all the evalu that we have are

22:22

focused on they're focused on like are

22:25

you uh almost like you know when when

22:27

the user says something they actually

22:28

want to accomplish a task. Did you did

22:30

you figure that out? Right? Because you

22:32

could here's a here's an opportunity you

22:34

can really help them accomplish

22:35

something like you know did you did you

22:37

understand that that's what they wanted

22:39

you to do or like were you a little bit

22:41

too eager and you went on and and and

22:43

did something like that was way too

22:45

expensive and annoying when the user

22:47

didn't want that. Right? Like so so we

22:48

we have things where um you know for

22:51

example in the Slack integration when

22:53

like you're having a conversation with

22:54

the agent and you ask a follow-up

22:55

question, right? And the agent thinks

22:57

they can answer it. There's a whole

22:58

internal process that goes through like

23:00

I think I can answer this question.

23:01

Should I interject and and and that's

23:03

that's the so the events are are really

23:06

focused around those kinds of like

23:07

ergonomic type of uh you know type of

23:10

moments

23:10

>> and after the product is live or even

23:12

during dog fooding uh is there some sort

23:14

of feedback loop after the agent

23:15

response I can do a thumbs up or thumbs

23:17

down or something like some feedback so

23:18

you can get con constant feedback.

23:21

>> Yeah. Yeah. Totally. And that that's

23:24

that's been very useful, right? A lot of

23:25

our evals are effectively derived from

23:28

those moments where someone goes like

23:29

this behaved in a weird way or a stupid

23:31

way like let me tell you why. And then

23:33

and then eventually that itself becomes

23:35

an eval.

23:36

>> Um and Jacob, you probably know like a

23:38

couple of these like the the funnier

23:39

earlier ones, right?

23:40

>> Yeah. Yeah. We've had we've had a ton of

23:42

these. Like we had we had a user like

23:45

call the agent dude once and then the

23:47

agent was like, "Oh, okay. I'm not going

23:49

to respond to you because that was like

23:51

you're not being formal with me. Um, so

23:54

we we've had a lot of interesting use

23:56

cases where we've had to really just

23:57

like dial these things into a really

23:59

really narrow zone. I feel like um like

24:02

you don't have to show the prompt, but

24:03

like I feel like when you write the

24:04

prompts and skills, you got to be a

24:05

little bit maybe it's more line around

24:07

principles and how they should think

24:08

versus like hey you should make sure

24:09

this is 140 characters longer like very

24:11

specific kind of right like you don't

24:13

want to restrict it too much right in

24:15

terms of what it can do.

24:16

>> Yeah, totally. And you also just want to

24:19

like give it as little instruction as

24:20

possible to be honest. Give it the tools

24:23

to load context. Don't give it context I

24:26

think is like a a principle we found

24:28

important.

24:28

>> Interesting. Because if you just give it

24:30

too many instructions, it'll just like

24:31

overfit on software.

24:33

>> Yeah. And it may not need that. And then

24:35

it may overemphasize on certain things

24:37

that actually aren't important for that

24:38

task. Um they're they're just smart

24:40

enough to get what they need if you give

24:42

it a really good defined goal.

24:44

>> Oh. because they can just like do

24:45

searches and stuff and figure out

24:46

themselves.

24:46

>> Yeah. Give it tools to load skills, give

24:49

it tools to load guidance, you know,

24:51

different things like that and it can

24:52

build its own context.

24:53

>> Got it. Interesting. It's kind of funny

24:56

like it's kind of because you probably

24:57

get a bunch of feedback about the agent

24:58

and then you probably have a something

25:00

the agent ingest the feedback and

25:02

synthesize it. So it's almost like the

25:03

agent's improving itself, right? It's

25:05

like a loop. [laughter]

25:08

Yeah, we have a mechanism where the

25:09

agent can report um essentially

25:13

functionality it can't do. So if the

25:14

user asks it to do something it doesn't

25:16

have a tool for it, it's going to call a

25:18

tool and report that back to us and then

25:19

we autoingjust that into you know go

25:22

check if there's already an issue for

25:23

this. If so add it there. If not create

25:25

a new issue. So we have like a

25:27

constantly streaming

25:28

>> nice

25:29

>> set of issues coming in around what we

25:31

can do.

25:31

>> Dude, do you remember like offhand like

25:33

what's the craziest thing the user like

25:35

a user asks agent to do? crazy things.

25:38

Um I I think in general like we're

25:41

actually okay with the user asking, you

25:45

know, as long as there's not like safety

25:46

concerns or like that sort of thing.

25:48

We're okay with if you want the agent to

25:50

write you a poem, that's fine. Let it

25:52

write you a poem. Um so we we give it

25:54

quite a bit of freedom in that regard.

25:56

Um I think trying to lock it down too

25:58

much can can end up in a frustrating

26:00

situation.

26:01

>> Got it. Okay. I I think when it when it

26:04

comes to like you know the the range of

26:06

things that people ask linear agent to

26:08

do it's it's interesting right because

26:12

it because it's like explicitly like for

26:14

work right it's associated with your

26:16

with your workspace and your your

26:17

development team and stuff like that

26:19

like people don't tend to go on wild

26:21

adventures but there are things which

26:25

surprise us right like uh a lot of

26:26

people use it for translation so they'll

26:29

they'll get feedback from customers or

26:30

something like that in in a language

26:32

they don't speak and they'll just

26:33

they'll just straight up just ask it for

26:35

translation or even set up an automation

26:36

to be like if something ever you know we

26:38

we have a lot of customers in France. So

26:40

if we ever get a a intercom ticket that

26:42

comes in in French, just translate it

26:44

for me before you when you file an issue

26:46

against it or something like that,

26:47

right? So like there's a lot of those

26:49

sorts of uh creative sort of use cases.

26:51

They're very on topic, you know, but

26:53

like we never thought that that would be

26:54

a thing that people would do

26:56

necessarily, right? Like that that was

26:57

they didn't even cross their minds that

26:59

that was a a possible uh a possible

27:01

thing.

27:02

>> Got it. Okay. Yeah. Yeah. Yeah. I I

27:04

think just like put the product in

27:05

people's hands and then they'll figure

27:06

out like new use cases will appear and

27:08

they can figure out which part to

27:09

improve.

27:10

>> Yeah. All right. Well, let me ask you

27:11

like some product questions about the

27:13

agent then. So like I I think before

27:14

this before linear agent came out,

27:16

linear was a platform for like you you

27:18

can tag like cursor agent and some other

27:20

agents on here, right? What was the

27:22

philosophy behind actually kind of going

27:23

off and building your own agent?

27:25

>> Yeah, I I think the um you know

27:29

assigning issues to agents was like kind

27:30

of what you're referring to, right? And

27:32

then you can and you can add mention

27:33

them in in in like comments and stuff

27:35

like that. Uh and I I think that we

27:39

wanted to be able to support the entire

27:41

software development life cycle intent

27:43

like ultimately that's what you know

27:45

that that that's where we we saw our own

27:48

sort of expertise right in in terms of

27:50

how we can have very good opinions that

27:51

you can adopt for your for your team. Um

27:54

and uh and you know having intelligence

27:57

and having an AI system uh there to take

27:59

action was a way to actually uh execute

28:02

on those things right because we we you

28:04

know before like if you think about like

28:05

the the you know the the olden days like

28:07

we would like write you know guides

28:10

right we'd write like something like the

28:11

linear method about how you how you

28:12

ought to think about your software

28:14

development process and uh so a lot of

28:17

in you know what we saw uh with uh you

28:20

know with with models capabilities right

28:22

is is basically the ability for us to be

28:24

like actually you don't just have to

28:25

read it and then execute on this

28:28

playbook that we're giving you. You can

28:29

just have linear itself execute the

28:31

playbook. So like I think ultimately

28:33

that's that's where that's what we saw.

28:34

So so you know specific agents that we

28:36

add to the system they'll they'll do

28:38

parts of that. They'll write the code or

28:39

maybe they'll they'll do the root cause

28:41

analysis or something like that for for

28:42

bug reports. Uh but you know none of

28:45

them really took the whole process end

28:47

to end like we wanted to.

28:48

>> Yeah, that makes sense. I I I I always

28:50

think like the the AI handles the middle

28:52

80% and then the humans in the first 10%

28:54

and last 10% and maybe maybe we're

28:57

almost at or maybe we're already at a

28:58

point where you know we we can just tell

29:00

we can tell linear agent like here's

29:02

some user problems here's like a high

29:04

level idea of a solution like go figure

29:06

out and then go put a PR up and then

29:07

I'll review it even for like bigger

29:09

features right

29:10

>> yeah yeah I mean what you're what you're

29:13

talking about like I I've heard people

29:14

say something like AI is a it's not an

29:16

endto-end solution it's a middle's

29:17

middle solution right it's kind of what

29:18

you're what you're talking about and and

29:20

and and you know right now like maybe

29:22

humans handle the first 10% last 10% but

29:24

like at some point maybe it's like the

29:26

first 0.1% and the last touch right it's

29:29

like you know the middle just get it's

29:30

going to get bigger until it basically

29:32

reaches the limits of the edges right so

29:33

like that's kind of what we're also uh

29:35

seeing right like we we're assuming that

29:37

that's going to happen so we want to

29:38

build like a system right that

29:40

facilitates that

29:41

>> I don't know dude so so so Jacob have

29:43

you like have you been following

29:44

discords around loops and goals and

29:46

stuff like that are you like doing all

29:47

that stuff you let it run.

29:50

>> Yeah, we're definitely experimenting

29:51

with that internally. I mean, there's

29:53

definitely cost trade-offs you have to

29:55

consider once you you get into these

29:56

long runninging agents. Um, and I but I

30:00

think models are getting really close to

30:01

where that's a really interesting thing.

30:04

Yeah, on one hand like not exactly like

30:05

I said like if you upload a linear

30:07

method in the markdown file and then the

30:09

agent will actually read through it and

30:10

like actually try to follow it, right?

30:12

But on the other hand, I feel like it's

30:13

so good at producing markdown files and

30:15

all this stuff by itself. And then

30:17

usually it's pretty long. Sometimes I I

30:19

don't even read it anymore. I I like,

30:20

[laughter] okay, you make a plan, go for

30:22

it. Like just just do it. And then and

30:24

then I look at the final output.

30:25

>> And I'm I'm like really paranoid that

30:27

like it'll just end up as like, you

30:28

know, as more and more agent produced

30:30

markdown files end up in the re repo and

30:32

like I don't read any of it,

30:33

>> it just turns to slop, dude.

30:37

That that's what happened.

30:39

>> Yeah. Yeah. And I I I I think like like

30:41

you know as people use AI systems they

30:44

they they start feeling these things as

30:46

well. And then like you know a lot of

30:47

times people oh you know skill issue or

30:49

something like that they were like well

30:49

you should just read your things you

30:50

should like tell it not to produce slap.

30:52

like yes but like like we it comes back

30:55

to the main problem of like what are the

30:56

defaults and and I think one of the one

30:59

of the opportunities we saw for you know

31:01

for introducing AI into linear as a

31:03

system rather because like there's an

31:04

there's an alternate way of the universe

31:05

where we just say like we have an MCP uh

31:08

you know MCP server and they just use

31:10

whatever to to connect to linear and

31:11

then that's it right like we don't have

31:12

like a native agent doing anything

31:15

>> um but I think having a native agent

31:16

lets us uh you know imbue our you know

31:20

opinions about what good product

31:22

management looks like, right? And what

31:24

it looks like is not producing endless

31:26

markdown documents that you that are

31:28

unreadable and full of extraneous

31:29

detail, right? Like like where you know

31:31

we can we can we can make some

31:33

determinations about uh how the process

31:35

ought to run in the in the best case.

31:38

>> I see. I see that that's a really good

31:40

Yeah, because if you just build an MCP

31:41

just a bunch of tools that people can

31:42

use then they might go off the rails but

31:44

like the agent actually has a bunch of

31:46

skills and instructions to kind of imbue

31:48

linear's values and product process

31:50

right

31:51

>> exactly

31:52

>> that makes sense yeah cool when you guys

31:55

decided this thing was ready to ship

31:56

like kari has high bar right so when you

31:58

guys and this thing like people are

32:00

using it for all kinds of stuff so how

32:01

do you decide if this thing is ready to

32:02

go

32:03

>> what was it like from the engineering

32:04

side Jacob I I could talk about the

32:05

product side but what was the what did

32:07

it look like from from the back end.

32:09

>> It felt like we released it early, not

32:13

too early, but I think we kind of leaned

32:16

on the side of like it's pretty good.

32:18

Let's put it out there and get more data

32:20

for how to improve it more. I think we

32:22

could have worked on it internally

32:23

forever honestly to make it perfect. Um

32:25

it with something subjective like this,

32:27

it's it's harder to get to that like

32:30

perfect spot. And so I think we hit a

32:32

point where like we just need to ship

32:33

it.

32:33

>> Yeah, got it. How about you not from the

32:36

pro side? Yeah, I I think from the

32:38

product side, you know,

32:41

if you if you think about shipping

32:43

products that have like that have like a

32:46

UI, right, the UI effectively limits

32:47

what you can do with it, right? It's

32:48

like a very UIdriven feature or

32:50

whatever. And uh so you can have some

32:53

definition of quality that's based on

32:55

eliminating everything that you didn't

32:57

intend in the first place. I think for

32:58

something like this, we had like core

33:01

hero use cases that were like we these

33:03

are the things we're going to demo.

33:04

These are things we think are going to

33:05

add a lot of value and they're a good

33:07

like first way for people to use the

33:08

agent,

33:09

>> right? And if we make those rock solid,

33:12

then the other stuff it's like look,

33:13

there's going to be a variance in in

33:15

reliability and things like that because

33:16

it's just the nature of this kind of uh

33:18

you know, this kind of tool and this

33:19

kind of technology and we're okay with

33:21

that, right? But as long as like the the

33:23

the hot paths uh that that we're

33:25

advocating for are are you know are

33:28

solid and we believe in them, then I

33:29

think that that's the bar for quality

33:30

that we're looking at.

33:31

>> Okay. the stuff that you know like

33:33

create a ticket, manage tickets, create

33:34

a PR like that kind of stuff like that

33:35

that you highlight in the market

33:36

marketing as long as those are good.

33:38

>> Yeah. Yeah. Exactly. Because those are

33:40

things are like like those are you know

33:42

those have those are in the warranty so

33:43

to speak, right?

33:44

>> Yeah. Got it. Okay, that makes sense.

33:46

Okay. So, so I guess just to like kind

33:48

of wrap up a little bit, I'm I'm sure a

33:49

lot of companies are thinking through

33:51

this right now. Should they just build

33:52

like an MCP or should they build a

33:54

native agent or like you know if you

33:57

build all this stuff then people don't

33:58

even use your website anymore. Then like

34:00

what do you do? Like do you have any

34:02

advice for uh builders or companies or

34:04

think about whether just to even build

34:05

their own agent or not?

34:07

>> Yeah, I mean my advice is like like what

34:08

what you know really you have to really

34:10

break down what is the actual workflow

34:13

that that your users want to do, right?

34:15

Like and and a lot of times it's it's

34:17

like super multi-step, right? Like no

34:19

one no one like sits down at their desk

34:20

and oneshots their whole job. Like that

34:22

that's not that doesn't happen, right?

34:24

So it it's it's it's an entire process

34:26

that goes into end and then like where

34:27

are the natural places where uh where

34:29

you want to hook into that where where

34:31

it makes sense to uh to hook into that

34:34

and and I think that that's really where

34:36

um you know like we said like you know

34:38

linear agent as a as a collection of

34:41

as a collection of subsystems right like

34:44

a lot of them are about figuring out

34:46

where the right entry points are because

34:49

the the most obvious thing is like look

34:50

there's a there's an inapp chatbot you

34:52

can have a chatbot right and then People

34:54

will point at that be like, "Oh, that's

34:54

the agent." It's like, "Well, that's

34:56

that's that's one way to interact with

34:58

the agent." It's necessary because if

35:00

you want to do any kind of like

35:01

follow-ups or you want to do any sort of

35:02

like multi-turn processes, you have to

35:04

have some surface to to do that. But

35:06

that's not where the entry points are.

35:07

The entry points are in the discussion

35:09

that you're having in Slack or in your

35:11

meeting debrief or when you're writing a

35:13

project update and you're trying to do

35:15

the research, right? Those are the those

35:16

are the the on-ramps

35:18

>> to like utilize the intelligence. So if

35:20

you if you give people good on-ramps,

35:22

right, the the uh the sort of

35:24

interactive chat, that's that's the

35:25

follow-up, right? That and that covers

35:27

like the long tale of things that people

35:28

want to do. So I I think that's what you

35:30

kind of have to do for for um for domain

35:32

specific agents, right? Like because

35:34

otherwise you you're deal with this

35:35

problem, which is like, you know, why

35:36

wouldn't I use quad or chatbt or

35:39

something like that for this uh instead

35:41

of your sort of native agent?

35:42

>> Yeah. So it's almost Yeah. I I think

35:44

like an agent is almost like a employee

35:46

and then like employee doesn't only work

35:49

in one app, right? You should talk to

35:51

them from like, you know, Slack or

35:52

wherever you guys work, you know?

35:55

>> Yeah. I I think I think everyone's kind

35:57

of moving that direction now, right,

35:58

with especially with the latest releases

36:00

uh that that everyone's kind of putting

36:01

out.

36:02

>> Do you think Kyrie like chef chat has a

36:03

tier when like um if if like everyone's

36:06

using linear through the through the

36:07

agent or the MCP instead of like all all

36:09

the beautiful buttons and the UI that

36:11

exists?

36:13

>> Honestly, no. I I I I don't I don't

36:14

think I don't think he minds at uh minds

36:16

at all. And like I think you know linear

36:18

is like also very um naturally like a

36:21

multiplayer system. So different players

36:24

are going to use it in different ways,

36:26

right? Like we we have like for example

36:27

a lot of uh you know customer support

36:29

agents, right? People right who are

36:31

doing customer support will um use

36:33

linear by escalating things out of

36:35

Zenesk or intercom, right? And like

36:37

that's their entire linear surface usage

36:39

is is there's there's a there's some

36:41

controls in the in the plug-in at

36:42

intercom for them to escalate and pick a

36:44

template or describe the issue or

36:46

whatever it is, right? Like that's and

36:47

that's it. that's their linear usage,

36:49

but it's it's very valuable because

36:50

that's the input stream for everyone

36:51

else to actually do their work.

36:53

>> Yeah, that's a good point. Yeah. So, I

36:54

guess because Kari talks about being

36:56

like opinionated about the product,

36:57

right? But like I guess you need to let

36:58

people use it from what whatever

37:00

workflow or service they they want, you

37:02

know? You can't be too opinionated.

37:04

Yeah.

37:04

>> Yeah. Yeah. Exactly.

37:05

>> Yeah. Cool. All right, guys. Well, I

37:08

mean, uh I guess what what what's next

37:09

for Linear Agent? Where can people want

37:11

to learn more about it?

37:12

uh what's next is you know I think we're

37:15

we're we're definitely introducing some

37:17

aspects of proactivity and uh and and

37:20

sort of longer running memory right

37:22

those are those are the two areas where

37:23

you know we're really kind of focused on

37:25

um if you think about like uh like let's

37:27

say you're you're you're building like a

37:29

project at your company and that project

37:30

can last it could be a very short

37:32

project over in a few days or it could

37:34

like last a whole quarter and you know

37:36

the agent should be very well aware of

37:38

everything that happened throughout the

37:40

lifetime of that project and be able to

37:42

kind of naturally push it forward,

37:44

right? There's a lot of different

37:45

moments where you have to coordinate

37:46

people, you have to make sure that, you

37:48

know, documents are kept up to date and

37:50

all those kinds of things. And that's

37:51

where we we want that to just be

37:53

something you can take for granted.

37:55

>> Yeah. Like I I run a project in linear.

37:57

I can take for granted that it's

37:59

wellrun, right? Like that's that's the

38:00

that's the goal that we're looking for.

38:02

>> That makes sense. Yeah. Whenever I have

38:04

like a long Slack thread with an

38:05

engineer and they're like, "Okay, let's

38:06

go up to the PRD." I'm like, "I'm just

38:08

too lazy to update the PRD. [laughter]

38:10

I don't up to the PRD." So yeah, just be

38:11

able to assign it to linear. That'll be

38:13

super useful. Yeah. Cool. All right,

38:16

guys. Well, thanks so much, man. Thanks

38:18

for giving us an inside look at how

38:19

linear agent works. And uh yeah, I I I

38:22

wish you guys the best of luck. I think

38:23

it's a very interesting problem to

38:24

solve. Thanks, Peter. Thanks.

Interactive Summary

This video features a deep dive with Naan and Jacob from Linear on building and evolving their 'Linear Agent.' They discuss the agent's architecture—calling an LLM in a loop with tools—and how it has evolved from managing simple tasks like creating tickets to acting as a versatile agent capable of end-to-end product development. The conversation highlights their philosophy of using 'skills' for specialized tasks, the importance of providing good entry points (on-ramps) for users rather than just relying on a chatbot, and the shift towards proactive, long-term project management.

Suggested questions

4 ready-made prompts