What Google Gemini AI updates mean for software developers

1 million tokens -- so what?

The major LLM providers have been in an arms race to expand multiple attributes of their models for the last year, and especially over the last few months, said Ian Beaver, chief scientist at Verint Systems, a contact center-as-a-service provider in Melville, N.Y. He cited examples such as Anthropic's Claude 3 Opus launch two months ago, which surpassed OpenAI's GPT-4 in LLM benchmarks from GPT-4; in April, Meta boasted higher benchmark performance for Llama 3 against the preview version of Gemini 1.5 Pro Just yesterday, OpenAI announced GPT-4o and an update to ChatGPT that supported text, audio and image input and included higher benchmarks than both Llama 3 and Gemini 1.5 Pro.

All of these models have made big leaps in input token limits as well, Beaver said: GPT-4 has grown from 16,000 to 128,000 tokens; Claude has grown from 100,000 to 200,000; and Gemini has grown from 32,000 to 1 million.

Larger context windows might help for some applications, such as video prompts and generation. Still, Beaver said he isn't sure how widely useful 1 million tokens will be.

The fact that you can now comfortably send in the entire text of War and Peace may be useful for generating reviews on large novels, but it remains to be seen how effective these models are.

"The fact that you can now comfortably send in the entire text of War and Peace may be useful for generating reviews on large novels, but it remains to be seen how effective these models are at maintaining long-distance dependencies in context data over that large a search space," he said. "In our experience, once you surpass a few hundred thousand tokens, it is generally unhelpful to the quality of the model response to include more, since there is typically some selection pipeline going on before the LLM, such as a database query or a search."

Bigger isn't necessarily better, wrote Torsten Volk, an analyst at Enterprise Management Associates, in a blog post last month.

"While the impressive 1 million token context window of Google's Gemini 1.5 Pro offers a theoretical advantage in handling extensive data, the practical effectiveness of a language model like GPT-4 often surpasses it due to more sophisticated mechanisms … [that] efficiently manage smaller context windows by focusing computational resources on the most relevant information, thus optimizing performance," Volk wrote in the post.

Google AI Studio, Vertex AI updates

Meanwhile, updates to the Google Gemini API and services such as Google AI Studio and Vertex AI added new features this week specifically for developers. One, context caching, might be more effective than large context windows alone, according to Volk. The feature, pitched by Google as a means to make model training and prompting more efficient by not having to resend large data sets repeatedly, can also assist with recurring queries on large document sets.

"Totally coincidentally, OpenAI said that GPT-4o also now has context caching across conversations," Volk said in an online interview this week, referring to OpenAI's news event the day before Google I/O.

Another Google Gemini developer update unveiled this week is parallel function calling, which means the model can invoke more than one function at once.

This will feed into an emerging trend toward deploying AI agents that carry out multi-step workflows; Google's Vertex AI last month added an Agent Builder tool, while Atlassian added support for AI agents, or virtual teammates, with its Atlassian Rovo product.

Gemini 1.5 Flash, Gemma add cost flexibility

A new version of Gemini rolled out this week, Gemini 1.5 Flash, uses a technique called distillation to impart the data analysis capabilities of the larger Pro model to a lighter-weight, lower-cost LLM optimized to give faster answers than the larger version.

With Flash, Google added new pay-as-you-go pricing for AI Studio and Vertex AI. Gemini Flash 1.5 is priced at $0.35 per million tokens for 128,000-token prompts and $0.70 per million tokens for prompts larger than that. By comparison, Gemini 1.5 Pro costs $3.50 per million tokens for up to 128,000 and $7.00 per million tokens for larger prompts. In general, early adopters of hosted LLM services have said controlling cloud costs has been one of their biggest challenges so far.

"We haven't done anything with substantial enough scale on Vertex to have [cost] become a focus, but I will say that a lot of the Vertex products seem to have real utility billing," Strauss said. "I like that because it means we can potentially provision it by default in an isolated way for customers and only pay for actual usage."

In the rapidly growing open source AI world, two fresh permutations of Google's Gemma will significantly increase the size of the open source LLM with the 27 billion-parameter Gemma 2 and add a fine-tuned model for video generation with PaliGemma -- the first vision-language open model from Google.

As with performance benchmarks and token input limits, all the major model providers have launched cheaper and faster versions of their flagship models, according to Verint's Beaver.

"What used to require the largest, most expensive model to perform can now be done by a cost-effective smaller model," he said. "The AI arms race is also rapidly dropping the cost to entry for high-performing LLMs. It is only getting cheaper to deploy applications using Generative AI."

Multi-modal support for a broader array of models will also lower the cost to produce various types of media content, Beaver predicted.

Trust, safety and quality remain top AI concerns

The natively multi-modal Gemini models are capable of processing various forms of data, such as images and videos along with text, and producing multi-format outputs, but don't quite function that way in production-ready practices just yet.

Google is working on a fresh version of Imagen "rebuilt from the ground up," according to a keynote presentation by Douglas Eck, senior research director at Google, following a recent controversy that forced Google to suspend the image generation tool in February. Imagen 3 is now available to trial on Google's ImageFX AI test kitchen and coming soon to Vertex AI. Truly multi-modal functions will be more widely available later this year, I/O keynote speakers said.

Multiple keynote speakers also emphasized work Google is doing on trust and safety for AI to prevent further controversial results, including red-teaming model updates, consulting with a panel of human experts from multiple fields of study, and a watermarking tool called SynthID.

However, slow adoption for generative AI tools such as Ansible Lightspeed so far indicate that enterprises have yet to dive into production use with any gusto. Strauss said early Vertex projects have had mixed results, though he attributes that in part to not having integrated data sets properly.

"We've used Vertex AI in prototypes for recommendations to tag written content and for the Vertex AI search system," he said. "The former is actively in production, and we saw middling results from the latter but need to put more effort into the integration to truly test it."

Beth Pariseau, senior news writer for TechTarget Editorial, is an award-winning veteran of IT journalism covering DevOps. Have a tip? Email her or reach out @PariseauTT.