Welcome back. So, I am extremely excited

today to tell you about a new project,

HydroGym, that just appeared in nature.

So, this is a huge collaborative effort

to build a benchmark platform for the

reinforcement learning control of fluid

flows. I want to give a special

shout-out to Christian Lagemann, who is

the lead architect and the first author

on this paper. Christian started off as

a postdoc in my lab

when this was just a niche project with

a few environments, and he really grew

this into the massive international

collaboration and success story that it

is today. So, huge huge credit to

Christian. We have an amazing team of

co-authors, brilliant brilliant minds

from across the world. I'll show you

some of their work in a little bit. And

also, I'd like to acknowledge Boeing and

the National Science Foundation for

funding this effort over the years.

There will be links in the description

to the paper, to the GitHub repository,

but I really want to jump in to what we

actually are doing here in HydroGym.

Okay? So, reinforcement learning

is the idea that if there is some

complex system, like a fluid flow, this

is the flow past a cavity. Reinforcement

learning essentially learns how to

interact with and manipulate this

environment for some kind of engineering

objective. So, we've seen the success of

reinforcement learning in other fields,

like AlphaFold for protein folding, that

was a Nobel Prize in Chemistry.

Reinforcement learning is able to now

control fusion reactors. It can beat

humans at chess and Go and many many

games. So, reinforcement learning is

this extremely powerful machine learning

framework for learning how to interact

with and control the real world. Now,

fluid flows are particularly important

for many many trillion-dollar

industries. If you think about health

care, transportation, energy, defense,

we live and work inside of a living

fluid, and so do almost all of our

machines. So, the ability to model and

predict and control fluid flows even a

little bit better could have an enormous

economic and ecological impact in the

world around us. Hugely important

problem, very, very challenging. So,

controlling fluid flows

requires simulating them, which is

itself very expensive. But, you don't

just simulate the fluid flow once in

reinforcement learning. You have to

simulate it thousands or millions of

times interacting with this environment

to learn by trial and error how to

control that system. So, that's the

basic idea is instead of making it so

that you have to have all of the the

simulation capabilities and all of the

reinforcement learning control

capabilities to participate, we wanted

to abstract this and create a community

benchmark platform that has the leading

fluid flow environments and the leading

reinforcement learning agents

pre-programmed, built in. So, that if

you have a new fluid flow or a new

reinforcement learning algorithm, you

can test it out much more easily and we

can move the field forward faster

together. Again, to solve those huge

societal scale problems, um making

better uh wind turbines and more

fuel-efficient vehicles [clears throat]

and better artificial hearts and things

like that.

And so, things I'll tell you about in

this uh short intro video and that we

talk about in the paper, uh I'll tell

you about some of the environments we

have. We have, you know, dozens, over 50

two-dimensional and three-dimensional

fluid flow environments that you can

play around with and try different

control algorithms in.

Uh we have transfer learning, so you can

try a controller in 2D and see how it

works in 3D. Or, you can try it at low

flow velocities and see how it works

when you crank up the speed uh of the

flow. Then, we also have some advanced

reinforcement learning uh capabilities

built in. So, multi-agent RL is really,

really powerful for very complex kind of

uh turbulent control. And, we also have

several differentiable environments in

differentiable languages like JAX where

you can get gradients for free and do

better gradient enhanced reinforcement

learning. So, we'll talk about that um

in this video just very briefly.

Now, again, I want to call out just the

incredible list of co-authors uh on this

work, especially Christian Lagemann and

been uh foundational through the entire

development, uh, of this this modern

hydro gym.

Um, and I'll only have time to give you

little vignettes into subsets of their

work, okay? Um,

so a little bit of history, this

actually started off with a project by

Jared Callahan. I think he actually came

up with the name hydro gym, maybe five

or six years ago, where we only had a

couple niche environments, you know,

two-dimensional flow environments, but

the really cool idea,

um, Ludger Paehler kept it alive, uh,

after Jared uh, graduated. And then when

we got Christian to join our lab as a

post-doc, it really flourished and, you

know, almost every kind of major advance

we're going to talk about today was

really because of, uh, Christian's hard

work.

So again, I'm just going to highlight,

you know, some of the work of my lab.

These are folks who were in my lab at

some point in time. I'm also going to

highlight, uh, the work of Ricardo

Vinuesa's group, who came in with one of

the most impressive results in this

entire hydro gym paper. Basically

showing that you could train a

reinforcement learning,

uh, agent on a simplified canonical

hydro gym fluid flow,

and then transfer that over to a much

more complicated aircraft wing and get

very, very impressive drag reduction. So

I'll highlight some of these results.

Again, there's a much longer, uh,

co-author list, many more cool results I

don't have time to talk about in this

video. Please read the paper, um, and

and give credit to their work as well.

Okay. So, um, you know, in the five or

six years since we've been developing

this, there's been this large diversity

of flow environments that we've created

in hydro gym, from 2D to 3D, low

Reynolds number to high Reynolds number,

means, you know, higher flow velocity,

more turbulence. Uh, we have a lot of

flows where there's a 2D and a 3D

analog, so you can test an algorithm in

one, you know, debug it in the cheap

kind of laptop version, and then scale

it up to the supercomputer version.

Um, Christian and his collaborators, his

team, you know, ran many of the leading

reinforcement learning algorithms on all

of these flows. This is, you know,

hundreds of thousands or millions of of,

uh, you know, hours of of compute time

that you don't have to run, you know, if

you have an algorithm, you only have to

test your algorithm on the flows of

interest, and you can compare it against

all of the other pre-computed, pre-run

flows.

So, HydroGym at its core

is a set of fluid flow simulations that

are very, very bomb-proof. They've been,

you know, tuned and calibrated to be

extremely robust to different flow

conditions, to different control

scenarios, so that it's very unlikely

you're going to break these flows, and

they're very reliable and validated. So,

the results are reliable.

Importantly, HydroGym is not programmed

in one

simulation language or one programming

language. We have environments from many

different programming languages, JAX,

C++, and so on and so forth.

And so, the solver diversity is actually

one of the strengths. If you have a

solver for a fluid flow that is

interesting or relevant, you can almost

certainly wrap it into HydroGym. So, we

have FireDrake environments, finite

element method. We have Maya code. This

is lattice Boltzmann method. We have

Maya unstructured finite volume,

Nek5000, JAX for for differentiable

flows. And so, all of these kind of

different back-ends and different

simulation environments can be wrapped

into HydroGym.

And we use the standard kind of Gym API

that you're used to from, you know, the

OpenAI Gym that you would use to train

reinforcement learning agents on much

simpler systems like the pendulum swing

up and things like that. Okay? So,

solver diversity is a huge deal. And

again, we have a ton of environments in

2D and 3D with these various

characteristics.

And so, this is a diagram that Christian

made that I really, really love, and it

kind of gives an idea of how these are

organized in terms of complexity.

So, on this kind of bottom axis here,

you see this increasing complexity from

generic examples to canonical benchmark

flows all the way up to really

complicated real-world dynamics, you

know, full-scale aircraft, very

complicated jets, and, you know, jet

noise, and multi-physics scenarios.

For all of these cases, we have 2D

two-dimensional kind of cheap

inexpensive toy systems and then we have

their three-dimensional analogs at

increasing kind of Reynolds numbers and

complexity. So some of these are

extremely high Reynolds number complex

fluid flow simulations. These require

supercomputers running hours and hours

to simulate and to train these agents.

But again, Christian and his team

already ran you know the leading

algorithms and benchmark them so that

that leaderboard exists so you don't

have to reproduce those the results you

know with with another million hours of

compute.

Okay.

You'll notice on the very very end here

future extensions. Those are kind of the

aspirational next steps. We see HydroGym

as not being a single static environment

a single paper but a community effort

that will grow in time. We will have

regular releases with new environments

new control algorithms and and you know

new new team members. So we want you to

join and and try stuff as well.

But this is kind of the big organizing

diagram. I really go back to this one

and use this one to kind of understand

where is the next opportunity? If I was

going to try my reduced order modeling

my Cindy methods you know to to

accelerate reinforcement learning where

would I start? Would I start with the

cylinder and then go to 3D? Would I

start with an airfoil? You know how

would you actually interact with this

environment and kind of build in

complexity without just trying to you

know tackle the whole thing all at once.

Okay. So that's kind of the overview of

this um

of this environment. Let's see if

my clicker actually works. Good.

So now I want to kind of highlight this

really impressive kind of flagship

result that Ricardo and his team brought

in to HydroGym in in the last year or

so. So this is work coming out of

Ricardo's lab. I'm going to highlight it

here because it's so impressive and it

really highlights what you can do with

this kind of community benchmark

reinforcement learning platform.

So the idea here is how do we use

HydroGym or how do they use HydroGym to

train a reinforcement learning agent

that can be applied to an extremely

complicated flow like the flow over a

real scale aircraft wing, okay?

So the idea here is that you can train a

reinforcement learning agent in a much

simpler canonical hydrogen environment.

On the left you see this

this you know periodic channel flow easy

to simulate relatively fast relatively

simple to actually write that code and

train algorithms and models here.

And what Ricardo and his team showed is

that you can take that agent that

reinforcement learning controller and

zero shot transfer that on to an actual

wing simulation a very complicated wing

simulation. If you match certain local

boundary conditions and again details

are in the paper and and they have

follow-up work really flushing this out

and going into the physics and the

algorithms beautiful work.

And so this is kind of the schematic

here again you train on this hydrogen

channel at an RE tau of of 206 a pretty

turbulent channel. And you can zero shot

transfer that to this real NACA 0012

airfoil with a chord wise Reynolds

number of 200,000 so very turbulent very

sophisticated wing simulation.

And without any modification just just

transferring this controller over to

this scenario they were able to achieve

a skin friction reduction of 38% that's

a 38% reduction in the turbulent drag

the turbulent skin friction drag on this

airfoil.

So that's the kind of power we envision

hydrogen bringing to the community is

you can train on these kind of canonical

building block flows and start to

transfer those over to much more

industrially relevant industrial scale

flows again that start to move the

needle on some of these societal

problems

fuel reduction more efficient transport

you know better mixing all kinds of

applications that could be accelerated

with better control of fluid flows.

Okay, good.

And again really grateful to collaborate

with Ricardo, uh been fantastic working

with their team.

So, a couple of uh just kind of

summaries of things I mentioned earlier.

These are three kind of advanced

capabilities that we explored in this

paper that we think you can really uh

dig into with HydroGym. The first one is

differentiable environments. So, if you

write your fluid flow simulator in a

differentiable computer programming

language, essentially where automatic

differentiation is enabled, so think

about how you train a neural network

using backpropagation, that's using

automatic differentiation. If you write

your flow solver in an environment like

that, like JAX, then you can use that

same automatic differentiation to get

gradients of outputs with respect to

inputs, and you can do gradient enhanced

reinforcement learning. So, here you can

see this uh this GPPO, this is gradient

enhanced PPO, uh proximal policy

optimization, and this pink curve learns

faster and better solutions than

standard kind of industry standard um

PPO. So, gradients help, they accelerate

uh and get better performance solutions.

This is um largely work by Sigita

Mackwell, who's a a PhD student in my

lab, and her whole thesis is on

differentiable flow environments and

what you can kind of accelerate and do

better when you have that gradient

information.

Multi-agent reinforcement learning, as I

mentioned, is a huge accelerator. If you

have a big flow, big complicated flow

like a wing or like this uh this long

span cylinder,

you can break it up into sections and

have different agents in different

sections share information. So, you can

train much, much faster, much smaller uh

agents that share information better,

and again, uh you get great um kind of

training and uh and control results.

And then the third row here talks about

transfer learning, which is a really,

really big idea. If we have this generic

platform, then we would hope that

controllers learned in some scenarios

transfer to more complicated scenarios,

like what we showed you uh with that

zero-shot transfer to the wing.

And so, here's just kind of a a little

bit more of a textbook kind of example

where we take a two-dimensional flow

past a cylinder, really simple, and we

take that controller and transfer it to

a three-dimensional cylinder, and then

do some retraining on that that in the

3D environment.

So, importantly, it's much faster to

train a controller in a two-dimensional

environment than a three-dimensional

environment. And so if I can train my 2D

controller and then transfer it to 3D, I

can save dramatically on training costs.

And that's essentially what we're seeing

here. This blue curve learns much much

quicker

a good solution. That is the transferred

controller from 2D to 3D. So we get a

much much faster learning,

much fewer interactions in this

expensive environment. You save a lot in

compute and you get your solution much

faster with transfer learning.

Okay, great.

And these are just, you know, some more

results showing kind of the training

reward. Again, what I really want to

highlight here is that Christian and his

team,

the larger collaborative team, have

implemented many of the leading

reinforcement learning algorithms, PPO,

DDPG, TD3, on, you know, nearly all of

these environments. So that baseline

performance is logged and cataloged for

all of these flows.

You don't have to repeat PPO

on all of these three-dimensional

environments. That baseline is

established. So if you have a better

idea, you can just test that and see how

it compares to a well-trained

well-tuned PPO.

And again, this is designed to be as

easy to use as possible in the standard

kind of gym API. So really easy to

initialize, you know, import hydrogym,

you can go to the GitHub and download

all the code

and import this this environment. You

set up your environment. This case I

think is a cavity in Maya, Reynolds

number 7,500.

You can pick, you know, what your

sensors and what your actuators are. So

different flows have different sensors

and different actuators that you can

affect. And then you, you know,

initialize this flow environment.

Next, um, interact with it is also very

simple. This is standard kind of API.

Um, you know, you set up a loop and you

basically, uh, do the reinforcement

learning, um, iteration to interact with

this environment.

And that gives you these kind of, you

know, training, uh, and roll out curves,

um, for the various flows for different

controllers. So, very, very standard. If

you train reinforcement learning, if

you're a reinforcement learning person,

you can interact with this rich variety

of fluid flows basically in the same way

you would interact with any other system

in robotics or in dynamical systems. If

you're a fluids person, you can put your

environment into HydroGym and you can

get the whole range of reinforcement

learning solutions on your problem from

the community, from other people who

have that expertise. So, that's really

what we're trying to do is abstract the

expertise so we can all move the field

field forward faster.

Okay, um, and this is kind of a cool

example I like to talk about of just a

really short vignette of what you can do

with this environment. So, this is

fantastic work by Nick Zolman, uh, who

is a PhD student in my lab, uh, in

collaboration with Christian and Urban,

where essentially Nick wanted to test

this idea that you could use Cindy, the

sparse identification of nonlinear

dynamics, to accelerate, uh,

reinforcement learning training. So,

again, I said that it's really expensive

to train these agents in these very

expensive fluid simulator environments.

And often times, you know, this will

involve thousands or millions of of

simulation calls to these expensive

simulators. So, Nick's idea is to take

this data,

train a really lightweight Cindy model,

and then do reinforcement learning, as

much reinforcement learning on that

surrogate Cindy model, as possible

because it's super, super fast. So, can

you bypass the expensive environment

with a cheap Cindy surrogate model? Very

simple idea.

But again, because HydroGym exists, um,

and this was, you know, in development

when he wrote this paper, he was able to

test his algorithm on three major

environments of increasing complexity

from the cylinder flow to the fluidic

pinball, to a full three-dimensional

turbulent airfoil, in addition to the

standard gym environments that you would

normally test your algorithms on.

So, this allows you to test much more

sophisticated systems if you have a new

idea, a new algorithm.

And just kind of cool here, shout out to

to Nick's work, um this does in fact

show that Cindy RL in blue can learn

much, much faster and sometimes much

better solutions than a baseline

reinforcement learning algorithm. So, it

does dramatically accelerate the

training and reduces the number of

interactions with the expensive

environment. That's what's on the the

x-axis in a log scale.

Um and the fact that HydroGym has this

range of Reynolds numbers and flow

conditions also means we can test things

like how does your controller

generalize? So, this is a controller

Nick trained at Reynolds number 100,

kind of a slow flow velocity, and when

he rolls it out to a higher Reynolds

number, Reynolds number 350, much more

turbulent, his controller still works.

So, that's a really powerful thing

that's sometimes hard to do uh unless

you have, you know, uh

a ball proof code where you can actually

crank up the Reynolds number is to test

this generalization. And so, that's a

hallmark feature of HydroGym is that you

can test the sweep across Reynolds

numbers and flow conditions and flow

complexities.

Okay, good.

So, where is this going? Um I want to uh

also mention, you know, I I gave the uh

the the funding acknowledgements earlier

that this has been funded by uh by

Boeing and by the National Science

Foundation. And in particular, the

National Science Foundation has

supported our AI Institute in dynamic

systems where these benchmark

environments are one of our central uh

focus points.

And so, we've just been selected for a

renewal for another 5 years of this AI

Institute, and HydroGym is going to

remain one of the central uh emphasis

points of our uh our AI Institute. So,

we're going to have funding and research

and time and effort to continue to build

HydroGym, to continue to make this a

community resource, and to continue to

support and grow towards these

increasingly uh challenging

applications.

So, for example, things like a full

aircraft model. We really want to scale

this up. Um Ricardo's group's result is

just the tip of the iceberg. We want to

go to full complexity, corner flows,

much more sophisticated flows, higher

Reynolds number, multi-physics.

Um

Again, with community support, uh with

all of you, we can actually do this and

we can drive better design, better

vehicles, better engineering faster with

uh this technology.

And not just aircraft, of course, um you

know, wind turbines, artificial hearts,

you know, trains, planes, automobiles,

everything uh that involves a working

fluid, we can model and control better

um if we have the best algorithms and

the best simulation environments working

together.

So, another area I'm very passionate

about is again design optimization of

aerodynamic uh shapes and surfaces. In

this case, you know, some kind of a

vehicle you want to design.

In reality,

we don't just have simulation data of

low and high fidelity. We also have

laboratory tests, wind tunnel tests,

field tests, sometimes you actually have

to build it and fly it.

All of this data contributes um you

know, to a surrogate model, whether this

is an AI model or some kind of

statistical model, you know, there's

lots of options for how to build this

model. But, we want to eventually start

doing our design optimization again on

this less expensive surrogate model.

Uh and that surrogate model can be

informed by hydrogen environments. We

can start including cyber-physical

systems and and actual laboratory data

into these hydrogen environments. So, I

think moving also uh

from control to the broader problem of

design optimization is a really great

opportunity for all of us.

Okay. Um that's the thumbnail sketch.

Really again, this is just a teaser for

a much deeper effort with a lot of

brilliant uh collaborators. And so, I

just want to end here. I want to thank

you for uh paying attention, for getting

involved. If you have flow environments,

physics problems, hard engineering

problems, please work with us to wrap

them into hydrogen. You'll get people

working on your problem uh if you can

get it in this environment. If you're a

machine learning or a controls expert

and you have a new algorithm, you want

to test it on the hardest problems there

are, try your algorithms in our hydrogen

environments. Again, huge shout out to

Christian Lagaman and the whole team,

brilliant researchers from across the

the globe. Truly a pleasure to work with

all of all of you.

Thanks to Boeing and the NSF and thanks

to all of you. Thank you.