Google Gemini: Everything you need to know about the new AI creation platform

Google is trying to make waves with Gemini, a flagship suite of AI models, applications and services. But while the Gemini looks promising in some aspects, it falls short in others – as our informal review revealed.

So what is Gemini? How can you use it? And how does it measure up to the competition?

To make it easier to keep up with the latest Gemini developments, we’ve put together this handy guide, which we’ll update as new Gemini models and features are released.

What is Gemini?

Gemini is from Google long promised, a family of next-generation GenAI models, developed by Google’s AI research labs DeepMind and Google Research. It comes in three flavors:

• Gemini Ultrathe iconic Gemini model.
• Gemini Proa “lite” Gemini model.
• Twins Nanoa smaller “distilled” model that runs on mobile devices like the Pixel 8 Pro.

All Gemini models were trained to be “innately multimodal” — in other words, able to work and use more than words. They were pre-trained and enhanced on a variety of audio, image and video, a large set of codebases and text in different languages.

This sets Gemini apart from models like Google’s LaMDA, which was trained exclusively on text data. LaMDA can’t understand or create anything other than text (eg essays, email drafts), but that’s not the case with the Gemini models.

What is the difference between Gemini apps and Gemini models?

Google, proving once again that it has no talent for branding, didn’t make it clear from the start that Gemini is separate and distinct from the Gemini web and mobile apps (formerly Bard). Gemini apps are just an interface through which some Gemini models can be accessed — think of it as a client for Google’s GenAI.

Incidentally, Gemini apps and models are also completely independent of Imagen 2, Google’s text-to-image model available in some of the company’s programming tools and environments. Don’t worry – you’re not the only one confused by this.

What can Gemini do?

Because Gemini models are multimodal, they can theoretically perform a range of multimodal tasks, from transcribing speech to captioning images and videos to creating artwork. Few of these features have made it to the product stage yet (more on that later), but Google is promising all of them—and more—at some point in the not-too-distant future.

Of course, it’s a little hard to take the company at its word.

Google seriously underdelivered with the initial release of Bard. And more recently he ruffled feathers with a video purporting to show Gemini’s abilities that turned out to be heavily edited and a little too ambitious.

However, assuming Google is more or less honest with its claims, here’s what the different tiers of Gemini will be able to do once they reach their full potential:

Gemini Ultra

Google says that the Gemini Ultra — thanks to its versatility — can be used to help with things like physics homework, solving step-by-step problems on a worksheet, and pointing out potential mistakes in already filled-in answers.

Gemini Ultra can also be applied to tasks such as locating scientific papers relevant to a particular problem, Google says — extracting information from those papers and “updating” a graph from one by creating the formulas necessary to recreate it chart with more recent data.

Gemini Ultra technically supports image creation, as mentioned earlier. But this feature hasn’t made it to the production version of the model yet—perhaps because the mechanism is more complex than how apps like ChatGPT generate images. Instead of asking for feed to an image generator (like DALL-E 3, in the case of ChatGPT), Gemini outputs images “natively”, without an intermediate step.

Gemini Ultra is available as an API through Vertex AI, Google’s fully managed AI developer platform, and AI Studio, Google’s online tool for app and platform developers. It also powers Gemini apps — but not for free. Access to Gemini Ultra through what Google calls Gemini Advanced requires a subscription to the Google One AI Premium Program, priced at $20 per month.

The AI ​​Premium plan also connects Gemini to your broader Google Workspace account — think emails in Gmail, documents in Docs, presentations in Sheets, and recordings in Google Meet. This is useful, for example, for summarizing emails or taking notes from Gemini during a video call.

Gemini Pro

Google says Gemini Pro is an improvement over LaMDA in reasoning, planning and understanding capabilities.

An independent study Carnegie Mellon and BerriAI researchers found that Gemini Pro is indeed better than OpenAI’s GPT-3.5 at handling larger and more complex reasoning chains. But the study also found that, like all major language models, Gemini Pro particularly struggles with math problems involving lots of digits, and users have found many examples of poor reasoning and mistakes.

However, Google’s promised improvements — and the first one arrived in the form of Gemini 1.5 Pro.

Designed to replace Gemini 1.5 Pro (currently in preview) it has improved in many areas compared to its predecessor, perhaps most significantly in the amount of data it can process. Gemini 1.5 Pro can (in a limited private preview) take ~700,000 words or ~30,000 lines of code — 35 times the amount that Gemini 1.0 Pro can handle. And — the model is multimodal — it’s not limited to text. Gemini 1.5 Pro can analyze up to 11 hours of audio or an hour of video in a variety of different languages, albeit slowly (eg, searching for a scene in an hour-long video takes 30 seconds to a minute to process).

Gemini Pro is also available via API in Vertex AI to accept text as input and generate text as output. An additional endpoint, Gemini Pro Vision, can process text and images — including photos and videos — and text extraction according to OpenAI’s GPT-4 model with Vision.

Using Gemini Pro in Vertex AI. Image Credits: Gemini

In Vertex AI, developers can adapt Gemini Pro to specific environments and use cases using a fine-tuning or “grounding” process. Gemini Pro can also connect to external third-party APIs to perform certain actions.

In AI Studio, there are workflows for creating structured chat messages using Gemini Pro. Developers have access to both Gemini Pro and Gemini Pro Vision endpoints and can adjust model temperature to control the creative range of the production and provide examples to guide tone and style — and also tune the security settings.

Once Gemini Pro is out of preview at Vertex, however, the model will cost $0.0025 per character while the output will cost $0.00005 per character. Vertex customers pay per 1,000 characters (approximately 140 to 250 words) and, in the case of models like the Gemini Pro Vision, per image ($0.0025).

Let’s say a 500-word article contains 2,000 characters. Summarizing this article with Gemini Pro would cost $5. Meanwhile, creating an article of similar length would cost $0.1.

Ultra pricing has yet to be announced.