So this has really been the foundation

of my planning of pretty much everything

I build.

My workflow has changed from [music]

using a planning document to grilling me

and then creating the planning document

and then implementing [music]

to now grilling me and then just setting

a goal and walking away.

>> [music]

[applause]

>> Sweet. Thank you guys for coming.

>> George, tell us a little bit about

yourself.

>> Sure.

Yeah, I'm George. I'm

Yeah, I'm a software engineer. Been

programming for 10 years or so.

Self-taught. Basically taught myself

programming, moved to the Bay Area,

started working in startups.

Yeah, I'm starting a new job on Monday

at Parallel. Super stoked for that.

Doing DevRel. And yeah.

>> But also you help run the Codex Meetup.

>> Yeah, yeah, I run a

Yeah, that's true. Yeah, I've done some

other stuff. I I run a Codex Meetup in

the city. I was just really obsessed

with Codex this winter when everyone was

discovering cloud code and I was

thinking why aren't people using Codex?

So I just wanted to

create a space to kind of teach people

Yeah, I know there might be some tension

around this.

Um using Codex in this demo. So

um

But yeah, been

teaching and learning and posting on

posting on Twitter. And yeah,

that's my thing.

So actually a quick round of Or not

round of applause. That wouldn't be the

right thing.

Let's start with a round of applause.

No. All right, yeah, yeah, good way to

start it. Sweet.

Um raise raise of hands, who has never

used a coding agent?

All right, just just Peter. [laughter]

Just Peter. Okay, sweet. So everyone's

that that's good context.

Um who uses Who doesn't use it every

day?

Okay.

Sweet. A couple people. Okay, so it's a

pretty technical crowd. My I'll give you

an outline of my talk. This is my

presentation today. I forgot to

attribute the quote.

No, these are actually

>> [laughter]

>> They're they're actually GPT, but

yeah, we're just going to jump in here.

It's going to be pretty practical. So,

I'm going to show you sort of the

different stages of how I've approached

AI assisted coding over time to where

I'm to now, and I hope it's pretty

practical.

So, initially what

What one of the biggest breakthroughs I

came across was this past October,

OpenAI published a blog post about You

know, you hear of these coding agents

running for hours, and at first it was

like, "Oh, that's like I was just kind

of mind-blown." I was like, "How could

it run for 8 hours?" Now they claim 25

hours.

And the way they do it is this planning

documents. Let's go to the right one.

It's called plans.md.

So,

basically this is just a huge prompt

where they're

it's an instruction about how to create

a planning document. So, you know, you

dictate your to your coding agent, or

you tell it what you want to do, and

then say, "Make a plan based on this

markdown file." And it creates what's

called an exec plan, is what they call

it. And it's just this

awesome document that allows through

different compactions, it allows agents

to stay on task, to kind of get back up

to speed, remember what it's done,

document its decisions, and then

continue its work. So, this has really

been the foundation of my

planning of pretty much everything I

build since this was this came out, I

think, in October.

Um so, I'm going to sort of just show

you how I use Codex,

and how I use skills, and I'm going to

I have a somewhat contrived example of

something I haven't built, but basically

I'm just going to dictate.

The idea is I have friends who make You

know, I told Greg I was going to record

my screen.

And I guess this is a good time to start

doing that.

Let's start before I do anything.

New screen recording.

Boom.

We got to wait 5 seconds.

Do another round of applause. I don't

know.

>> [laughter]

>> Um

So, me and my friends when we're

together, we always throw out our

predictions like,

you know, 2 years from now there's going

to be 20% unemployment or AGI is going

to be here by 2027. But, I always forget

when people make those predictions and

never have a way to keep them

accountable. I want to create a system

where I can add my friends' emails to

the system, write down the prediction,

and write down the due date, and then on

that date send everyone an email

reminding them of the predictions they

made.

So, I'm just going to dictate the thing

I want.

And then, basically, what I do I have a

skill called

exact plan

create.

So, when I instantiate this skill, it's

going to basically just read from a

markdown file and create a plan. But,

one cool thing I like to do with Codex

is I like to chain skills.

So, I'll do

exact plan improve.

Improve, improve. Oh, they change it.

So, you can kind of run You can chain

these skills. So, once this first one is

done, it'll run the next one.

Implement.

Review.

So, at a certain point

you can go watch a movie.

That's awesome.

And what what's cool about both the Is

it big enough, by the way? Yeah, it's

big.

What's cool about this I I I didn't

create the the plan framework, but I

created these skills to create plans.

And what this improvement one does is

sort of gives it a framework. There's a

software There's a Stanford professor

named John Osterhout, and he just has

really good clean coding principles. So,

I modified the planning document a

little bit in the improvement document

to just say like

follow these great software engineering

principles. And when the agent makes

this improvement pass after it creates

this initial one,

it's going to return a score at the very

end, one out of 10, about how useful

that improvement was. So, I could

actually chain these improvements to

to improve it like 25 times. It'll

probably realistically only improve it

usefully a few times, but when it

returns a score of three or less, it'll

just return the word skip. And then

instructions inside of the skill say if

the previous plan said if the previous

message said skip, just skip. So, the

idea is that you can just put a bunch of

these improvement skills, and after it's

done improving it, it'll just go to

implement it.

The implementation skill is pretty

straightforward. It's just look at the

exact plan, follow it, implement it. And

then review recent work. I'm going to

show you why this is full hot soon, but

the idea is

it's just a code review pass on your own

code.

Um I've created the exact plan now.

Let's take a look at it just so you can

kind of see the structure.

How many lines does it say?

I don't know how many lines. Doesn't

really matter. But, it sort of, you

know,

it's kind of just outlining I I should

have told it like I want to deploy this

to AWS, make it more ambitious. I'm

going to bail on this and

show you the next upgrade to my

workflow.

Let's go back to my presentation. All

right, sweet.

So, a a huge flaw in this I'm actually

going to copy just the dictation I did,

so you don't have to listen to me say

all of it again.

So, a huge improvement to this. So, when

when I give it this instruction,

it's a very broad overview. And this is

actually what these quotes I kind of

like conjured up are is the

the models aren't going to know all of

your intent. So, before just telling it

to go build, it's important to So, there

are my instructions from before. I'm

going to say grill me. And this is This

is a guy named Matt uh Paddock. I think

he's a British guy.

Um awesome engineer. He came up with

this idea. Just have your agent

interview you. And it it'll interview

you incessantly. It'll just keep going,

ask you questions about what you want.

But the idea is

it's going to ask you clarifying

questions about what What do you

actually mean? What do you want? And

sometimes it'll ask you like 100

questions. It's kind of crazy. But it's

really good because AI is always eager

to build something, but is it actually

building what you want?

So, I'm going to say

uh

Let's go with that recommendation and

then ask me 10 questions at a time.

Cool. And then I'm going to axe that

part.

And really what happens here is probably

I I spend a lot of time on this grill me

thing. Like this is the time I think

makes the most sense to spend your time

like clarifying what is your intent and

actually thinking and researching having

it back and forth. I told it to go 10 at

a time cuz I'm just going to blast

through it. But I'm just going to

you know, I'm just going to oops.

Let's do that.

I'm just going to get a lot of these

suggestions down so that we can get to

the next step.

We can just read random ones if you

want.

I guess that's fine. And really when you

when you're asking your agent to help

you sort of clarify unknowns, 90, 95% of

the time, it's answering what I want.

So, I I end up often times saying,

"Sure, let's go with it." Um but let's

see, I might just kill it for a time

and say,

I'm going to say,

"Okay, I think we're ready. Encode all

of those recommendations into

into

uh decisions.md file."

So, the idea here is I've now given it

my original instruction.

I've had it grill me to clarify the

intent, and now I'm going to like create

I didn't tell it to do this exactly, but

it'll do something similar. I'll have it

create like markdown checklists of like

everything we've decided, so that now we

have this decisions.md file, and then we

can go create our exact plan from there,

and the exact plan will reference the

decisions. So, when it's a long-running

task, we've basically had the agent have

a durable place to remember everything

we've decided.

Um

and let's let's just see what it looks

like.

Okay, this this is okay. I would have

told it

I would have told it to make it

checkboxes just so I can like the agent

can know if it's done or not.

But, the next big thing, let's flip back

to the presentation.

Um cool. Yeah, I mean, this this was a

huge change in the last few months.

Um

I think I want to say Codex did it

first, the goal / goal feature.

Um

so, with goal like, there's been they

released this goal feature, and the idea

is that you can have an agent run for

many hours, many days. Peter, you

probably gone for like over We have one

still going since the first day it came

out, maybe.

But, the the idea is

there there are best practices for your

goals and you know, in the Codex code

base

you can tell Codex to look at the code

base. Um, OpenAI engineers and are

posting on Twitter saying this is how I

use goal. So, I created a skill called

GoalCraft

which helps you write a goal. And

goals have limitations like character

limits and things like that. And so, my

workflow has changed from using a

planning document to

grilling me and then creating the

planning document and then implementing

to now grilling me and then just setting

a goal and walking away.

So, when I do this GoalCraft thing

I'll do this

encode these decision or just make sure

we complete all these decisions before

the goal is done.

And this skill I've I've integrated

different blog posts and best practices

that I've found online about

creating goals. And so,

it makes sure like, you know, it's

within this character limit.

It uh makes sure there are stopping

conditions, ways to verify

your outputs. And I recently updated it

so that instead of just writing a plain

text goal, it actually activates the

goal.

So, really in two steps you can go from

this is what I want, grill me

get everything you need and then run the

GoalCraft skill and your goal is kicked

off and it has tons of context about

what you want. That context is encoded

somewhere in a markdown file. Actually,

in this case it's not. It's just in the

context of the conversation. Like all of

these questions it asked me.

You know, imagine we go through 25

compactions of a goal, it might not

still have all these questions.

But we'll solve that soon.

Um, I will just okay, cool. It made the

goal.

Um

we can take a look. I'll just like

go to a new note.

There's a sort of similar format for

best practices for goals. Not super

important for this talk to read through

all this text. There's a lot of text.

But basically, the idea is giving it

context, boundaries, constraints,

verification levels, and stop

conditions.

So, at this point

I started creating goals and it was

awesome because you can get goals to run

for a days and it'll just keep going,

but I found that there were often like

I'd have to steer it a lot and

kind of jump in and interrupt it and

say, "Oh, no, this is what I wanted."

So, I've sort of combined this I'm going

to pause this goal and delete and stop.

I've combined this idea of

Where the heck? That's weird.

Where is my message?

Um I don't know where we're at. Okay. I

basically combined the idea of

after after grilling me on

you know, the architecture decisions

instead of just creating a goal and

having it run, what I found is that

what we're missing here is

like durable artifacts to remember what

we've done so far. And so, what I

skipped here when I did this goal craft

thing is the the whole planning phase

and this whole exact plan. Inside of

this exact plan, there are like lots of

checklists and references and patterns

that we should be following.

But when you just go and create a goal

it doesn't have it doesn't have any kind

of durable memory. I mean, it does have

the decisions that I've written here

but otherwise, you just have this kind

of goal that's floating. It's like,

"Okay, I'm going to keep trying to do

this." So, what I've done recently I

have I created a new skill called I

created this today called grill craft

and it's basically after you're

grilling, you can run GrillCraft and

it'll turn all of your decisions into a

markdown file and then create an exact

plan and just make sure that you have

these two artifacts that are linked up

so that as your goal progresses, it's

progressing through the exact plan. So,

it has more structure to proceed through

your goal and then it has all of your

decisions

still encoded from your grilling

session.

Um let's kill that.

So, that's that's a lot of this I've

I've experimented a lot with

um

just trying to get agents to run for

really long time and try to see how

ambitious I can make them. And I'm

excited to see what the other two

presenters have because

yeah, I think I'll I'll learn something.

But in my experience, I've found that

when I get agents running for a really

long time, what I'm missing is

I like like mid-session grilling

session. Maybe I could just pause it and

ask it to grill me. I didn't think of

that. But it's it seems like when you

have an extremely ambitious project

that's going for

days or weeks, at some point it's making

decisions without you. So,

sort of my North Star and what I'm

trying to figure out is how can I keep

myself in the loop and keep giving it

feedback, but also

delegate it and kind of do more of this

looping behavior.

Um the the last part is I have this I

have a skill called review recent work.

I'm just going to invoke it even though

I don't have

work to review. But typically after I

create

create a feature, I will run this review

pass. And this works in a similar way as

the improve my exact plan improvement

skill where

after doing a code review,

it will make the fixes

and then it will return a score from 1

to 10.

The idea being if you're doing code

review a code review code review and

it's not finding any more improvements

to make, it'll return a score of three

and then it'll return the word skip.

Skip skip skip. So if I'm about to go to

bed, I might put 50 of these skill

chains and just keep doing code reviews.

But I got good feedback actually at one

of my events where somebody said,

"Well, you're kind of asking the agent

who wrote its own code what its problems

are."

And they brought up this idea of

adversarial code review.

And so I thought a lot about that and

I've tried different I actually have

different skills. Whenever I want to

write something, I don't know, maybe I'm

writing an important email, I'll usually

write it with Claude. That's one of my

primary uses for Claude, it's just way

better at writing. So I have a skill in

Codex where it'll write with Claude,

which is handy. And I've tried doing

calling other LLMs from Codex.

Um and I'm sure you guys have seen a lot

of these like when you open up pull

requests, you'll see random bots coming

trying to

comment on pull requests to try to help

you out.

Um actually in the process of doing

this, I opened a PR for this grill cop.

I've actually taking one step back.

There is a repo called useful Codex

skills that has all the skills I've

talked about here today.

Um the grill me

uh the exact plans stuff.

And I opened up a PR to add this grill

cop skill to it. And

I got a comment from Codo. Codo's a PR

reviewer. And so what's cool, it'll

comment on your PRs and it'll sort of

like,

you know, it'll kind of give you that

was a sort of useless diagram.

Um but it'll give you an overview of

what's going on and it'll find bugs,

which a lot of code review tools do. But

what's cool, Codo is a little bit more

built for teams.

So the idea is that

is that inside of the Codo UI it can

read from your different agent side MD,

your Slack, Jira tickets, contacts

throughout your organization, and it

will create these rules for you. So, the

idea is that before you set this up on

your on your pull request that you kind

of get your engineering team together

and say, "What are the rules that we

never want broken? And where should we

centralize these in one place?" So, that

your code review tool isn't just

scanning your code base, it's importing

context from your other tools.

Um so, I think that's

pretty useful. And another cool thing is

when it comments on your PR, it'll just

put a prompt that you can literally just

copy. And I think there's a skill now

where you can just

basically automate this process. So,

instead of printing the prompt, it'll

actually go use it and apply it. But,

that's

I think that's the final part of

of my uh of my Codex workflow. So, happy

to answer any questions if there are

questions. Otherwise, we can go for a

third round of applause.

>> [applause]

>> What do you got?

>> Where are you spending the most human

time right now in this whole flow?

Like, what do you need the most human

intelligence for?

>> That's a great question.

I would say

I would say planning. So, probably the

just get getting the pro I mean, what I

did here is building a a brand new

project, but often you're in something

much bigger. So, I think just critical

thinking about what are we going to

build and the more important like if

it's vibe coding,

it's I'm probably having three threads

going at a time and just going crazy.

But, if it's really important, I think

like reading the plan and making sure

it's what I want, and then testing, I'd

say.

>> In [clears throat] your first example,

you queued up a bunch of the same skill

over and over again to review the code.

Why not just use a slash goal or loop to

effectively do the same thing without

having to queue them all up.

>> It's a It's a good question.

Um

I don't have a super strong I've I've

been doing that improvement thing since

probably December or something like

that, but

yeah, presumably you could create a plan

and say

invoke the skill over and over until it

stops improving, something like that,

but probably lots of different ways to

do it.

What you got?

>> Yeah, I I really like your workflow

here. Um I was wondering for a lot of

these skills that you're creating, uh it

seems like you you know, you said like,

"Oh, let me try like an adversarial

review from an idea." Um and it seemed

like, you know, you kind of try it out,

but what What kind of eval would you use

in order to understand if this is like

the best skill for that job?

>> That's a great question. I think evals

are really important for skills and it's

something something that I'm not I would

say in this room the authority on, so.

>> So, like human eval for now?

>> What is that?

>> Human human eval for now, like how it

feels.

>> Is that how you

>> Yeah, vibes. Yeah. I mean, like

I I I just find I find that like like

with the code review skill, it would it

would stop it would tell me that there

are no problems and then I would open it

up with Codo and it would be like, you

know, they're using I think Opus 4.8 on

the back end. And so, just having that

other LM review it, it would almost

every single time find me new bugs that

weren't being discovered from my own

skill, so seeing more bugs, good eval.

>> What is the final document? So, if you

went away from those projects and came

back like 3 months from now, what would

you use as the document to bring you

back up to speed on what it is that you

built?

>> Yeah, I would use this exec plan. I

might have axed it somehow.

I somehow lost it in this context, but

yeah, this this planning document I mean

the the way that these planning

documents work, the way I've set it up,

in a vibe coded project that's one shot,

that would be it. That would be the the

economical document of what you would

come back to to understand what's going

on. But I've had projects where I'll

have

like 50 or 100 exact plans. And so I put

it in a get ignored file in the root of

my repo just called agent work, and it

just stores all these markdown files and

sometimes agents will go in there and

find stuff. Maybe pick up a bad idea or

two. But yeah, I I like to just I keep

them. I often don't read through them.

You know, it's really just for the

agent. But yeah.

>> So you're so you're not looping through

with each iteration to try to keep it up

to the current state?

>> No, it's it's more of a throwaway. It's

just a produced artifact that's only

being referenced as the work happens.

>> Cool.

>> Thank you, guys.

>> [applause]