Hi everyone. My name is Shardul Vaidya,

and today we're going to be talking

about dark factories, the step beyond

agentic engineering.

The term dark factories itself comes

from manufacturing, essentially a

factory

that has absolutely no humans at all.

Requirements go in, or essentially raw

materials go in, and a finished product

comes out, no humans involved at all.

And the intention is, well, if humans

are not needed, then why keep the lights

on? Let's look at agentic engineering by

the numbers.

When an engineer presses trust all

tools, or YOLO, or non-interactive as I

like to use it,

um you see very often a 53% accuracy

decline. That's essentially where an AI

has no feedback loop. They It has no way

to know that it's gone wrong.

On the other side, we have this 91%

versus 25%, where 91% of engineering

leaders claim that AI has improved

velocity, but only 25% have the data to

prove it. And one of the fundamental

reasons for that is the prevailing

approach.

You get the standard enterprise

rollouts, buy seats, try for acceptance

rates, declare that you're done. Or you

look at developer productivity

metrics. These are lines of code, PRs

per sprint, essentially just looking for

higher velocity, but not necessarily

improved product quality.

The next thing you see most often is

prompt engineering training. This is

essentially teaching developers how to

talk to models.

This has a very short shelf life, 12

months at the time of recording, but

honestly coming lower with models being

released every other week or every four

weeks even. And the last thing is the AI

center of excellence. These are

committees that meet every month,

discuss what shipped, and then see if

they want to make any decisions to move

the enterprise forward.

This, I don't think is the way.

The gap fundamentally isn't

productivity. It's an architectural

imbalance.

One approach assumes that humans stay in

the loop and AI helps make them faster.

The other assumes that humans are

responsible for the verification and a

system design. Essentially, instilling

organizational best practices into code.

Um verification systems replace their

judgment. These aren't even same points

on the agentic engineering spectrum.

One scales with head count where humans

essentially still are the bottleneck,

code review remains the bottleneck. And

the other scales with just a raw amount

of compute.

We're seeing the frontier move. A

handful of companies are already

operating at this dark factory scale,

but

let's talk about it a little more.

Same idea as the SAE levels for

self-driving. Each level is

fundamentally defined by what a

verification system is trusted to do.

Can you fundamentally remove the human?

Level two needs something like lane keep

cameras. You know, level four needs

lidar and redundant compute. In the case

of code, level two we're mostly focusing

on unit tests and lint and automated and

automated test writing and automated

test execution.

But to truly get to level four or level

five, we need the agents to have an

incredible amount of trust in the system

that's providing back pressure to them.

And that essentially means that when the

reviewer says, "Hey, this is wrong."

the agent is forced to listen.

Every major model can write solid code

for a well-specified task. That's been

true for over a year.

The gap between level two and level four

isn't model access, it's what you

trust the model to write without a human

reading every diff.

So, let's talk about who's actually

there. The Coinbase or Ramp convergence

is the strongest evidence for the

architecture that we're going to talk

about soon.

These are two independent companies that

essentially don't talk to each other,

but independently arrived at the same

pattern. This is constraint-driven

design at its finest. And StrongDM

essentially came up with this full

digital twin universe that simulates

everything surrounding their prod

environment.

They have end-to-end scenario testing

already built in with LLMs, and they

approach it with the fundamental idea

that zero humans write code, zero humans

review code.

Coinbase, Ramp, Stripe, Anthropic, and

StrongDM all came to the same pattern.

Isolated sandboxes, curated tool sets,

sub-agent orchestration, and most

importantly, verification gates that

provide immense back pressure to the

coding agents to write this

the correct code. This is the same way

that we've seen web-scale data

constraints produce MapReduce.

And let me show you what that looks

like.

This is what I think the

the the baseline dark factory

architecture looks like. You have some

kind of orchestrator whose sole

responsibility is to decompose, convert

it into a DAG, and then dispatch into

coding agents, right? The decomposition

happens with a planning pattern. It

doesn't matter where it gets the

requirements from. It can be from Slack,

it can be from Linear, it can be from

GitHub, or it can even be by just typing

into a web UI.

But once you get that, you decompose it

into a list of tasks while the planner

has access and ability to research the

codebase. This is exactly what you would

do in a planning sprint

or a planning meeting. And then

each agent each coding agent is given

its own sandbox it's given its own work

tree

and in the case of like hyper scale you

would probably give it its own container

to live in

and then once it thinks it's done we go

through some kind of a verification

gate.

We're going to talk about verification

gates in a minute but it goes through

some kind of verification gate. If it

fails it goes through rework goes back

to the orchestrator and gets dispatched

back into an agent saying hey

this is what the previous agent did this

is what it did wrong please fix it. If

it gets past it just goes on to a merge

state where it goes and gets deployed.

Let's talk about these runtimes for a

second. The key part here is isolation

right these

are essentially highly untrusted

systems. They get their own sandboxed

done through a container or micro VM

whatever your enterprise has expertise

in.

Then the most important thing after that

is context boundaries. There are no

shared state between agents running at

any given time.

This means that these agents are

completely stateless. They can crash we

can reboot them and they will just keep

running.

The tool registry is what lets us

essentially evolve the factory over

time.

These can be internal tools that maybe

abstract away testing or can be external

tools that give access to documentation.

But these are essentially standardized.

And finally crashing is a feature.

You never run into situations where an

agent essentially dies and then is left

there

to to just sort of be orphaned but if if

an agent dies we clean up essentially

leave the state that it was in and boot

up another agent to launch in its place.

Spin up execute produce some artifacts,

and terminate.

Then let's talk about

how this multi-agent orchestration is

possible.

First things first, decomposition, the

planning stage, so to speak, is what

lets is what lets us parallelize our

processes to the fullest.

Bad decomposition can produce merge

conflicts because more than one agent

will try to do the same thing. And this

is where we get our first back pressure.

Second, the rework loop is the biggest

key. When a task fails, the full context

of why it failed is fed forward into the

next attempt. That's not a retry, that's

essentially just a full rework.

Um cost-optimized routing essentially

lets the planner delegate to the correct

agent downstream. That essentially, this

is a trivial task, you can use a

low-cost model to do it. This is a

difficult task, use the high-quality

model. And concurrent dispatch is what

lets us essentially leverage the DAG to

say, "These, these, these tasks are

unblocked. You can execute them."

Then, there's the verification

architecture, as we said earlier.

Traditionally, even as humans work, uh

we've relied on unit tests, on property

tests, on mutation tests to

make sure that

the code we're writing is is accurate,

is trustworthy, is going in the right

direction.

These are all excellent because LLMs

have learned from them, right? They know

how unit tests work. They know how like

uh test coverage works. So, we use them.

We continue to use them. Then, we come

to the AI-native level. Now, I know

these are shown as separate streams, but

they all happen parallelly, right? That

there's no one without the other.

Um the the most the most amazing thing

that we can do with AI-native testing is

scenario testing. You can essentially

describe this end user journey

to your agent and see how long it takes

to get to that journey. This is kind of

built-in

user acceptance tests that we previously

did not really have access to.

And then the last here is the formal and

the aspirational side. It's a it's the

idea that

instead of just

making a plan and and making tasks, we

make a plan and then we make these

system invariants that define and

declare

mathematically what states the system

can live in.

This is mostly unexplored and I believe

this is aspirational for me, too, but

this is something that can really

formalize

how a system is going to behave and it

lets you have immense levels of trust on

your Dark Factory generated code.

So, let's see it run. This is my factory

and it essentially is a giant monolith

that goes from requirements to plan to

tasks to execution to merging. And I'll

just show off every step of that

process.

This is my factory dashboard.

It essentially has everything that has

that that the factory has ever done

including all the projects that it

manages.

So, we can go out here. We we can see a

health check. We can see if something is

broken.

But the most important thing in how we

essentially give the factory a task is

through the submit requirement thing. We

select which project this is about, type

in our requirements,

and hit plan down here. Once that

happens, we go into the plan stage.

We go into the requirement stage because

a plan generates requirements and we

have these

these essentially

uh

context-aware

well-researched tasks, right? So, we can

click here, which is one of my code

review tasks. And we can go down to the

actual design. And you can see here that

the the the prompt I provided was was

very trivial. It was like, "Hey, we've

done a bunch of code review. Um solve

all of the requests from the code

review."

And you can see that it's generated, I

don't know, about 20-something tasks.

Yeah, about 23 tasks. And we get this

beautiful DAG that defines

what needs to be done in what order. And

you can see here, we've got some

independent lines. We've got some deeply

dependent lines. And then we've got some

like one-off things that just needed to

be sorted. And this essentially allows

high level of parallelism in things that

have no blockers, just like how human

teams are designed. Just like how

sprints are designed. We're not ripping

out the conventional SDLC. We're saying

a factory can augment it

basically infinitely. And once we go

from there, once a a plan is triggered,

we go into this state, where essentially

a a

the orchestrator is now treating this as

an an assembly line. And you can see

here, we get this like merge status and

these like stable change IDs. These are

derived from the underlying Git

substrate we use, which is Jujutsu. Um

and those stable change IDs let us

essentially refer to these tasks uh

stably through throughout the history of

the project. And

then you can see a whole bunch of these

are merged, and then we have some

failures that say, "Hey, we've we've

exceeded max rework count." And that's

essentially uh a an escalation to me

saying, "Hey, you should probably look

at this yourself as a human, because

you're responsible for the code that it

generates."

But you can see, a vast majority of

these are already merged and already

ready to go.

Um and this actually, as a matter of

fact, this whole UI that you're seeing

right now was built entirely by the

factory. And we can go look at the code

for a minute. Uh we can go down to the

factory.

And you can see here there's over 300

commits that are essentially very tight

requirements down to right, "Oh, hey,

it's it's just it's just three

additions, six deletions, net reduction

in entropy."

And we can force the factory to behave

that way.

And we can also see this, 27 additions,

30 deletions, very tight, very clean

refactor on exactly what's going on.

This is what this kind of task

decomposition planning into design

into execution lets us do.

And the reason it works is due to back

pressure.

Back pressure in our factory comes in

two specific ways, the planned reviewer

and then the code reviewer. And then

there's a deterministic back pressure

from the test executor.

So, in our factory, um it's essentially

six phases uh that form the pipeline.

Uh first we have uh the requirement

coming in from us as we saw, then the

plan, the plan adversary. This is

essentially something that reviews the

plan generated uh and the DAG generated

to make sure it will work fine within

the context of the code base.

Once the plan adversary approves a plan,

it gets put in the queue uh which is our

which is, you know, managed by our

orchestrator, and then once a task's

dependencies are ready,

we can schedule the task. This goes to

the coder

uh and then this works essentially in an

isolated sandbox. We're going to discuss

that in a little bit.

Um once we get through code, once the

coder thinks it's done, we essentially

run the tests that are configured for

our project.

And then we go to the code adversary.

And this is essentially a code reviewer.

Um it goes through all the code that was

written, it looks at the diffs

essentially, and then points out if

there was too much entropy, if there was

like something wasn't done right, or

some feature isn't complete. And

obviously, if the tests fail, we never

get to the code reviewer. This is

essentially how a um

a maintainer would refuse to look at a

PR if the tests are red, right? If

you're breaking something, I don't even

want to look at it.

And then once the code obviously says

it's okay,

we go into the merge. Um this is enabled

by Jujutsu's conflict detection, and we

essentially have this fine-tuned

uh deterministic merge system. And then

if that fails, if there's too many com-

conflicts,

uh we obviously go back out to the LLM,

and we actually have a merge rework

manager that runs within the merge

uh agent.

The isolation philosophy comes from two

things, right? Uh first, we obviously

want to reduce the blast radius if a

agent goes haywire. Uh that's mainly

provided by running these in isolated

sandboxes. That can obviously be uh in

in Docker or in containers. That can

also be on separate EC2 instances.

However,

um you like to manage your isolation.

The second tier of isolation comes at

the version control level. So, this is

in in my factory provided by Jujutsu

workspaces.

This is the equivalent of Git work

trees, but with a little more Jujutsu

features.

And all of that just allows clean

merging back to trunk whenever it's

ready to go.

The result of all of this is you're no

longer writing code. You're composing

the factory that writes the code for

you. You essentially go from being

uh an engineer who super focuses on code

to an engineer who focuses on how the

system

verifies and fact-checks that the code

written by the LLMs is correct.

You've essentially define these really

deep requirements, curate the agent's

tool sets, architect verification

frameworks, and fundamentally ensure

correctness.

You're responsible for it, but without

actually writing every line.

So, how would we build this on AWS?

You can obviously build it from scratch

using EC2 and and just write everything

as a single monolith like I did.

Uh but every layer of this architecture

has an AWS feature or service that can

make it easier to run.

Bedrock agent core if you want to run

long-lived agents, Lambda Fargate for

isolated agent runtimes,

CodeBuild and CodePipeline for

application verification,

Lambda for for reaching out to external

uh services through tools.

All of this can be done through Bedrock.

And actually, if you wanted to not even

write code for the orchestrator, you

could use stuff like EventBridge and

Step Functions to just define your

factory as infrastructure.

And because it's all in AWS, it gives

you access to CloudWatch and CloudTrail

along with IAM that gives you really

fine-grained control of everything and

anything your agent can do.

And

because we might want our factory to be

fundamentally stateless, we can store

all our state in DynamoDB.

Uh and that's how my factory is built.

Everything is JSON, so it can very

cleanly move into DynamoDB.

And once you've done all of this,

software writes itself

courtesy of the dark factory.

These are some of the references that

inspired me on the dark factory journey.

There's Jeffrey Huntley's article on the

Ralph loop, which essentially

allows you to stack while loops on top

of another.

There's StrongDM's

blog and live website on their factory

and Stevie Yag his article on Gastown

where he describes work as these beads

that are run through the factory until

they are finished.

A huge shout out to the background

agents summit and thank you. My name is

Shardul Vaidya. You can find me on my

email address or on LinkedIn or even on

my personal website.