HomeVideos

Get started with the Interactions API

Now Playing

Get started with the Interactions API

Transcript

149 segments

0:00

The interactions API is now generally available.

0:05

Now, this is exciting because it's

0:07

a unified interface for all the Gemini models and agents,

0:13

allowing you easy access through a single API.

0:17

And this is really, really powerful

0:19

because you can see here you can create

0:22

an interaction with a model.

0:24

So this will be synchronous.

0:27

You make this call and you'll get back the response.

0:32

Or you can do this where you can have

0:36

an anti-gravity agent perform something really,

0:40

really difficult. For example, in a remote environment,

0:43

multi-step installation, setup all that single API call

0:49

on the interactions API.

0:52

So there's tons of things that you can do now.

0:55

Managed agents is the most recent release

0:58

that's available on the interactions API.

1:01

You have background execution tool improvements.

1:04

You can mix in build tools with your own tools as well.

1:08

You have Deep Research built into this

1:11

as well because you can run it in the background media

1:15

generation.

1:15

You can generate images with Nano banana.

1:18

You can generate music with lyric 3,

1:21

you can generate speech with TTS.

1:25

So tons of good things that are in here,

1:28

and it really makes the development so much easier

1:31

across all the Gemini models.

1:34

And also it is an agent first ecosystem.

1:38

So you can simply grab the Gemini interactions API skill

1:44

and get running with your favorite coding agent.

1:47

So let's do exactly that.

1:49

So here in my anti-gravity Ide, I can simply go ahead

1:54

and I can install the Gemini interactions API skills.

1:59

I will install it here in the project so that you can see it.

2:03

And so now we can also install it globally of course.

2:07

So here we have our reference.

2:08

We have our skill.

2:10

And so our coding agent will know all the good things

2:14

to do when building with the interactions API.

2:17

And what's also really exciting is that it is a stateful API.

2:21

So if you're building something that

2:23

builds upon the previous interaction.

2:26

You can very easily just pass in the previous interaction ID.

2:30

So let's do exactly that.

2:33

Please build a simple application

2:35

that allows me to edit my photos using the Gemini interactions

2:41

API.

2:42

I want to be able to put in a photo of mine,

2:45

as well as a text or a speech prompt,

2:50

and then I want to iteratively get image generations

2:56

and just iterate on them with the previous interactions ID.

3:02

All right, there we go.

3:03

And now, anti-gravity obviously also

3:07

has audio understanding built in, which is really cool.

3:10

And so you can now see that we're looking at our skills

3:14

here.

3:14

Our Gemini interactions API skills and anti-gravity

3:19

is going off and building our application here.

3:23

So yes let's allow this.

3:29

And so first of all we get our integration plan here.

3:34

So we can see yeah we'll need an API key env file.

3:39

We want Gemini 3.1 Flash image.

3:42

Yep That's good 3.5 Flash as well.

3:46

Yeah we'll just use simple NPM.

3:50

Yep That's good.

3:51

Server JS fantastic.

3:54

Yep that looks good.

3:55

So let's proceed with that.

3:59

OK go.

4:00

So we got a server JS file here.

4:03

And we can see we have an API backend here

4:08

that's being built for us.

4:11

And that has our previous interaction ID.

4:16

We'll see here our model that we'll use.

4:20

And yeah.

4:25

And then we can see our interactions create here

4:30

with our payload.

4:32

Fantastic our model.

4:36

Yeah that looks good.

4:41

OK fantastic.

4:43

So let's accept all these changes.

4:47

We will need to set up our env variable here.

4:53

So what we can do is we just can simply go to ai.dev.

4:58

So in ai.dev we can go to our API keys

5:03

and we can grab maybe one of our API keys here.

5:08

Great So I put in the API key.

5:11

Let's do a quick NPM install.

5:17

And then NPM run dev.

5:24

OK cool.

5:25

So here we go.

5:26

Anti-gravity Canvas.

5:29

We can drag or drop our photo in.

5:33

We can select the aspect ratio the resolution and then

5:38

our prompt OK sweet.

5:41

So let's drop my photo in here.

5:43

And then we're just going to say change the color of the cap

5:47

to be green.

5:50

And so now we are going to generate our hour edit.

5:53

So in the first step, we're transcribing the voice note

5:57

and then we are doing our image edit.

6:01

And so if we look into the logs here,

6:04

we can see OK we've got the input.

6:07

So that's our image input.

6:09

And then we have our transcribed text.

6:13

So in the first step, you can see here

6:16

where having an A transcribe API endpoint.

6:22

So that is also using the interactions API.

6:26

Just Gemini 3.5 Flash here to transcribe what we said.

6:31

And then we're passing that into our image editing transaction.

6:36

So Nano banana does currently not support audio as an input.

6:41

So we'll first need to transcribe it.

6:44

And then we can edit our image using the text prompt.

6:50

So you can see here change the color of the cap to be green.

6:53

And so now this is nice.

6:55

It gave us this little slider to see the outcome.

6:59

And so now here the real powerful thing

7:02

of the interactions API.

7:04

Now change the shirt to be pink.

7:10

And so now this time we actually don't need to pass in our image

7:16

anymore.

7:17

But we can simply just pass in the previous interaction ID

7:22

here.

7:23

And we will see, in combination with the prompt we have access

7:28

to the cached tokens.

7:31

And now you can see the cap is still green.

7:33

We only wanted to change the shirt to be pink.

7:38

And this is really cool Yeah Really awesome.

7:43

So you can see here and then obviously you

7:45

could go back to this, actually didn't like this.

7:50

And you can branch from here.

7:52

So you could say.

7:56

Actually change the color of the shirt to be green as well.

8:01

And so now I'd hope because it's the cash tokens,

8:05

it would use the same green as the cap.

8:08

Let's see if that is the case.

8:15

There we go.

8:16

No, it is using a different green.

8:19

OK, maybe that's a bit too much green.

8:21

But yeah, this is really, really cool.

8:25

Here the interactions API.

8:27

Try it out now.

8:28

Really recommend the coding agent skill.

8:32

It's a really, really good way to get started

8:34

with the interactions API.

8:37

Awesome let me know if this was helpful

8:40

and I can't wait to see what you will build with the interactions

8:43

API.

8:44

Cheers

Interactive Summary

The Interactions API is now generally available, offering a unified interface for all Gemini models and agents for easy access. It supports both synchronous calls and complex multi-step tasks through a single API. The API is agent-first, integrates tools, offers Deep Research, and enables media generation like images, music, and speech. A key feature is its stateful nature, allowing developers to build applications that iteratively refine previous interactions by simply passing a prior interaction ID. A demonstration showcased building a photo editing application where edits could be continuously refined without re-uploading the original image, leveraging the API's state management.

Suggested questions

4 ready-made prompts