When Google started Twins three years agothe goal was to create a multimodal large language model — a single neural network that was trained on text, image, audio, and video and could generate content in any of these formats.
Today at the Google I/O developer conference, the company took a concrete step toward that goal with the Gemini Omni, a new family of multimodal models that Google CEO Sundar Pichai says will be able to “create anything from any input.”
Omni will launch with video. Users can now combine images, audio, video, and text, and instead of simply stitching these inputs together, Omni has a word in all of them to produce a consistent output. The result is high quality videos that reflect an understanding of physics, culture, history and science.
Omni also allows users to edit photos with simple text commands instead of complex editing software, similar to Google’s Nano Banana.
Google already has a dedicated video model, Veo, that lets users turn text and images into video, and even direct and customize avatars. But Google DeepMind director of product management Nicole Brichtova says today’s release is more than just a Veo update: “It’s the next step in the evolution of combining the intelligence of Gemini with the rendering capabilities of our media models.”
An example that Koray Kavukcuoglu, DeepMind’s chief technologist, gave to reporters during a media briefing on Monday: When Omni received a simple prompt, such as “an explanation of how protein folding starts,” it quickly rendered a video of a stop-motion explanation with a voice saying, “Proteins start out as chains of amino acids called leaf patterns. beta sheets, forming a perfect three-dimensional shape.”
The long-term vision for Omni is broader, including the model used to do things like create images from audio or audio from video.
“When we first announced Gemini, it was our first AI model that was inherently multimodal,” Pichai said during the briefing. “We knew that training him in a combination of text, code, audio, images and video would give him a deeper understanding of the world. With world models, AI moves from predicting text to simulating reality. Gemini Omni is the next step in that direction.”
As part of the release, users will also be able to create videos of their own digital avatars — something OpenAI made popular in the now-defunct Sora app with Cameos. To avoid deepfakes, users will have to go through a dedicated product onboarding, which involves registering themselves and speaking a series of numbers, according to Brichtova. The avatar is then saved for future use.
Additionally, all videos created with Omni will include Google’s SynthID digital watermark, which allows users to verify whether videos were created through Gemini products.
The first model in the family is the Gemini Omni Flash, which will be released today on the Gemini app, YouTube Shorts and the creative studio AI Flow. Flash will be able to render 10-second videos, which Brichtova says isn’t a model limitation, but rather a decision based both on the desire to get it into more hands and the expectation that most users won’t want to make much longer videos yet. However, longer video durations are in the works for the near future.
Google seems to be pitching Omni Flash as more of a consumer tool. The examples Brichtova and Gabe Barth-Maron, a research engineer at DeepMind, gave on a call with TechCrunch about uses for digital avatars were all personal: making a video of yourself winning an award or going to the moon, or removing a bystander from the background of a video you took on vacation.
Barth-Maron put it more simply: “They are like personalized memes.”
“We definitely focused on making this easy for consumers to use,” Brichtova said. “There aren’t many video models that have breached that gap with consumers, so this is our game to do.”
The ease of use comes with a caveat: Brichtova and Barth-Maron noted that editing prompts should be very specific, otherwise the Omni risks over-editing or inadvertently changing elements the user wanted to keep—a problem Nano Banana users would face.
Despite the short-term consumer focus, the business and creative implications of Omni are obvious, and Google will make Omni available via API in the coming weeks. The avatar creator — a feature available today in Shorts — is something Google is hoping content creators will pick up on. But more broadly, an integrated multimodal workflow could be transformative for advertisers and filmmakers.
Startup Luma AI is building something similar, a tool that can create an entire ad campaign based on a short brief and a product image, backed by its own “unified” model.
“We’re really, really proud of the text rendering capabilities of the model, which is really useful for things like advertising,” Brichtova said. “If you want a product somewhere, or even a slogan, it has to be accurate… We certainly expect filmmakers and other kinds of creators to use this model as well.”
More professional use cases may be better served by the Omni Pro model, which should perform better in all Omni tasks. Google hasn’t yet said when Pro will be released, but Brichtova said it will happen when “we feel like we’re at a point where we have an incremental change over Flash.”
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
