Unveiling Gemini: Google DeepMind’s Multimodal AI Family Challenges GPT-4 with Superior Capabilities

Google has introduced a new family of large multimodal models called Gemini. These models feature enhanced capabilities and outperform the GPT-4 in many areas. Let’s take a look at their new superpowers.

This article is a synthesis based on the DeepMind Team and Google report “Gemini: A Family of Highly Capable Multimodal Models” and other sources (see references).

On Thursday, December 7, 2023, Google DeepMind announced its Gemini family of artificial intelligence models, which are in close competition with OpenAI’s GPT series.

Gemini can enable new approaches in areas like education, everyday problem solving, multilingual communication, information summarization, extraction, and creativity. Source: DeepMind report.

DeepMind is one of the most popular and recognized artificial intelligence companies, founded in 2010 by Demis Hassabis, Shane Legg, and Mustafa Suleyman and acquired by Alphabet Inc. in 2014. Renowned for its breakthroughs in deep learning and reinforcement learning, DeepMind gained widespread notoriety when its AlphaGo program beat the world Go champion in 2016, highlighting the AI’s ability to master complex tasks, much to the astonishment of the Chinese, who were not expecting this breakthrough so soon. DeepMind’s research revolves around the development of algorithms and models, particularly deep neural networks, with a particular focus on advancing the frontiers of artificial intelligence.

Gemini 1.0 comes in three sizes:

• Gemini Ultra: the largest, most powerful model for highly complex tasks. This is the new model that outperforms OpenAI’s best-performing model, GPT-4, in a number of text-, image-, coding- and reasoning-based tasks. It achieved new best results in 30 out of 32 benchmarks (text and reasoning, image comprehension, video comprehension, speech recognition, and speech translation) in performance tests. Gemini Ultra will be available via a new AI chat feature called Bard Advanced from early 2023: “a new state-of-the-art AI experience that gives you access to our best models and capabilities”.
• Gemini Pro: the best model for scaling across a wide range of tasks. It is already available to the public via Google’s Bard chat interface, but only in English and in 170 countries and territories for the time being. From December 13, developers will have access to this model via the Gemini API in Google AI Studio or Google Cloud Vertex AI.
• Gemini Nano-1 and Nano-2: the most effective model for memory constraint tasks on the device. Both models are designed for summarization, reading comprehension, and text generation tasks, and boast impressive capabilities in reasoning, STEM, coding, multimodal, and multilingual tasks relative to their sizes. They will run on Google’s Pixel 8 Pro smartphone powering new features such as Summarize in the Recorder app and rolling out to Smart Reply in Gboard, starting with WhatsApp.

All these models are called multimodal models because they can process text, images, audio, and video at the same time. They can produce text and images natively. Inputs are ingested without the need for intermediate description in natural language, making understanding more important. DeepMind’s qualitative evaluation highlights the model’s remarkable multimodal reasoning capabilities, enabling it to inherently understand and reason about mixed input sequences composed of audio, images, and text.

Gemini Ultra excels in a number of coding tests, including HumanEval and Natural2Code (private internal held-out dataset), which uses author-generated sources instead of web-based information.

Google’s Gemini family is also a reaction to the criticism levelled at the company following analysts’ negative assessment of Bard’s first presentation in February 2023, and the release of the PaLM 2 large language model in May. Bard and PaLM 2 were deemed inferior to GPT-4, causing Google’s share price to plummet.

Model Architecture

As you would expect, the Gemini models are built on top of the Transformer architecture revealed to the world by Vaswani in 2017. They admit 32,768 context-length tokens and 16kHz audio data. Google has trained two versions of the Nano: a 1.8B and a 3.25B settings to accommodate different phone memory constraints. These are distilled from the larger Gemini models and quantized to 4 bits.

The visual encoding of Gemini models is inspired by their fundamental work on Flamingo, CoCa and PaLI, with the important distinction that the models are multimodal from the start and can produce images using discrete image tokens. Video understanding is performed by encoding the video as a sequence of images in the large contextual window.

The training dataset contains data from web documents, books, and codes, as well as image, audio, and video data without further information. They have carried out security filtering beforehand to remove harmful content.

The Gemini 1.0 models have been trained on Google’s AI-optimized infrastructure using Google’s in-house v4 and v5e tensor processing units (TPUs).

The report does not specify how Gemini is architected internally, as OpenAI did previously in its report on GPT-4. Is there one or more models? How many layers are there? In short, no architectural information is divulged, as is the recipe for Coca-Cola.

Evaluation

Gemini Ultra, the largest model, is the first model to exceed the human expert performance. It reached a score of 90.04% in the MMLU benchmark against 89.8% for humans. This benchmark measures knowledge across a set of fifty-seven subjects. Gemini Ultra shows strong performance on both elementary exams and competition-grade problem sets. In HumanEval, a standard code-completion benchmark, Gemini Ultra correctly implements 74.4% of problems. In Natural2Code, it achieves the highest score of 74.9%.

…Gemini Ultra surpasses human-expert performance on the exam benchmark MMLU, scoring 90.0%, which has been a defacto measure of progress for LLMs ever since it was first released in 2020. Source: DeepMind report.

As you would expect, performance for the various tasks depends on the size of the model. The following figure shows the performance of the Pro model, as a reference, compared with the other models. In summary tasks, the Nano1 model is about 70% less accurate than the Pro model. This small model will be used on low-memory Pixel phones, so expect results to be much less accurate than with the larger models.

When it comes to automatic speech recognition (ASR), the Pro models and even the Gemini Nano-1 outperform OpenAI’s Whisper model. This bodes well for speech recognition and the path towards a complete voice experience between user and assistant. As a demonstration, the report presents a culinary scenario in which a person tries to make an omelette. This novice in the kitchen gets help by adding photos of his preparation as he goes along, and asking questions by voice. We are very close to a complete personal assistant.

In the field of video, these models have acquired a new ability to understand and reason about a temporal sequence of images. As an example, they present a soccer player kicking a ball and the result of the question “How could this person improve their technique?”

As good news, it seems that DeepMind has resolved the “lost in the middle” problem. They have found that the Ultra and Pro models effectively use the full length of the context. This will make it possible to search for information along the entire length of the context without any loss of accuracy.

Mitigations

To be sure the results may not be affected by data contamination (a kind of cheating by learning the results of the competition), they performed an extensive leaked data analysis.

They have done considerable work to mitigate potential downstream damage at the curation and data collection stage. On the other hand, they have developed model impact assessments to identify, evaluate, and document their Gemini models in the areas of factuality, child safety, harmful content, cybersecurity, biohazards, representation, and inclusivity. They developed policies and model classifications to guide model development and evaluation.

They also used benchmarks such as Real Toxicity Prompts, a set of 100,000 prompts with varying degrees of varying degrees of toxicity drawn from the web and developed by experts at the Allen Institute for AI. Testers from all over the world will no doubt be evaluating (hacking) these models to their limits, to show everyone the scabrous discoveries they will have made.

As a “nod” to OpenAI’s criticism of the exploitation of underpaid workers, DeepMind added the following sentence to its report: “ensuring all data enrichment workers are paid at least a local living wage.”

Conclusion

Google DeepMind’s Gemini models mark an important milestone, following in the footsteps of OpenAI, by introducing their first multimodal models to rival OpenAI’s GPT series. With Gemini Ultra boasting unrivalled performance, exceeding the benchmarks set by human experts, and Gemini Pro and Nano meeting a variety of task requirements at smaller sizes, these models represent the future for Google’s language and image processing applications. During evaluation tests, these new models demonstrated text, sound, image and video processing prowess that saturates existing benchmarks.

Google DeepMind’s Gemini family is now a formidable competitor in the evolving landscape of large multi-modal models, pushing boundaries and setting new benchmarks. Google’s constant research, combined with its ethical considerations, makes it once again a leading player in the personal assistant competition. They will integrate Gemini into their products and services as they go along, such as Search, Ads, Chrome, and Duet AI, the code generation application.

In the discussion and conclusion chapter, they point out that there is a continuing need for research and development into “hallucinations”, and that LLMs also have difficulties in tasks requiring high-level reasoning skills such as causal understanding, logical deduction, and counterfactual reasoning. This paves the way for further improvements and future sensational announcements.

References

• Introduction of Gemini models: https://deepmind.google/technologies/gemini/#introduction
• A note from Google and Alphabet CEO Sundar Pichai and by Demis Hassabis, CEO and Co-Founder of Google DeepMind, on behalf of the Gemini team: https://blog.google/technology/ai/google-gemini-ai/
• Link to the paper: https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay on the loop with the latest AI stories. Let’s shape the future of AI together!