First Hands-On With Google’s New Gemini Ultra

How does it compare to ChatGPT and GPT-4?

Yesterday, Google took a massive step forward for the generative AI world by releasing Gemini Ultra, the company’s most powerful chatbot ever.

Gemini is the company’s much-anticipated response to ChatGPT and OpenAI’s GPT-4 model, which was the industry leader to this point.

Based on my early testing, Gemini Ultra matches or exceeds GPT-4’s performance. In particular, there’s one thing that Gemini Ultra does way better than ChatGPT.

Gemini’s release is Google’s biggest AI move in at least a year, and Gemini is now ChatGPT’s only true competitor.

On their official launch page, Google provides three examples of the new things Gemini Ultra can do. I decided to test the model’s performance by trying each of those things.

Here are the results from my first hands-on tests of Gemini Ultra and Google’s new Gemini Advanced platform.

A Personal Tutor

Google writes that Gemini “can be your personal tutor — creating step-by-step instructions, sample quizzes or back-and-forth discussions tailored to your learning style.”

I’m a professional photographer, and I often teach photography courses. I decided to test Gemini’s ability to teach about a specific photographic concept.

A student recently asked me about what “aperture” means in photography. I asked Gemini to tutor me about it.

I was amazed! Gemini used the human eye’s pupil as a metaphor for describing how a camera aperture works. That’s exactly the same metaphor that I use with my students to explain the concept.

Gemini then went on to accurately explain how aperture decisions affect the appearance of photos, and how aperture is used for different photographic styles.

Next, I asked Gemini to create a quiz about aperture. Again, the results were impressive.

If you know about photography, you’ll notice one potential misstep — all the answers to the quiz are A! A real tutor would probably switch them up a bit to keep students on their toes.

Still, in terms of the information that Gemini presents, the results are extremely impressive.

What I like about Gemini Ultra as a tutor is that it doesn’t simply spit out information — it tries to make that information understandable to the student through the use of metaphors, real-world examples, and the like.

In comparison, ChatGPT’s answer is highly technical.

The information is correct, but it’s presented in a way that a beginning photographer would never understand.

Gemini Ultra’s response is impressive both because it provides accurate information, and because it works to make that information understandable and relatable, just as a human tutor would.

Evaluating Code

For its second new skill, Google says that Gemini can act “as a sounding board for ideas and helping you evaluate different coding approaches.” LLMs have always done a good job at working with code, so I was excited to see how Gemini Ultra would perform this extremely helpful task.

To test Gemini, I decided to give it a chunk of Python code, and ask it what the code does.

This specific code is a function that uses an old OCR library to read text from an image file, and then counts the number of words it finds.

Gemini immediately understood what the code was up to:

It then launched into an extremely detailed explanation of each part of the code, breaking it down line by line. Here’s a taste:

That kind of detailed walkthrough is extremely helpful for a coder — especially if they inherited this code as part of a legacy codebase and needed to understand it.

Next, I asked Gemini to re-write the code to make it more efficient and better. The results are solid. It did things like using a built-in list of stopwords instead of the words I had defined, consolidating the regular expressions, and other things a human coder would also do.

Again, Gemini’s ability to write code is impressive. But what’s even more impressive is its ability to explain its actions, showing each step of its process and describing — clearly — what the code does and how it can improve.

Creating Content

Finally, Google says that Gemini Ultra “can help digital creators go from idea to creation by generating fresh content, analyzing recent trends and brainstorming improved ways to grow their audiences.”

To test this, I decided to have Gemini help me write a blog post for my Bichon Frise website.

I asked it for a post about “5 Ways to Celebrate Valentine’s Day With Your Bichon Frise.”

In about 10 seconds, it spat out a 500-word listicle post. Many of the ideas are spot on, and the tone is perfect for a Bichon-lover’s audience.

Next, I asked Gemini for a cute illustration to go with the post:

It delivered!

Finally, I asked for an HTML Meta description, which is an important part of SEO for blog posts. Gemini wrote a very compelling one, and even gave a few options for directions I could take:

The post is live on my blog (I disclose that I use AI).

The post is genuinely full of great ideas! Again, I love that Gemini not only wrote the text, but also gave me options for each piece of content it created.

That allowed me to exercise my own human judgment in making the final choice of what to publish. It felt like Gemini was working with me to write the post and generate the image and description, not writing it for me.

An AI Partner

Overall, that was my biggest impression from my first day of testing Google Gemini Ultra.

Yes, Ultra is an impressive chatbot, powered by an equally impressive LLM. That alone would be enough — just having another option beyond ChatGPT is a huge step forward for the generative AI space, and competition makes everyone better.

But Gemini Ultra goes beyond that. It doesn’t just provide information — it works to make that information understandable and genuinely useful. It explains its choices, involves its human compatriot in its process, and provides multiple options whenever it can.

Whereas ChatGPT feels like a very smart computer, Gemini feels like a true AI partner. It feels like the system is coding with you or writing with you, not just spitting content in your face.

For some industries, that may not matter. But for the vast majority of users, Gemini’s ability to work as a partner is a major step forward — and a big incentive to experiment with and adopt generative AI tech.

I’ve tested thousands of ChatGPT prompts over the last year. As a full-time creator, there are a handful I come back to every day. I compiled them into a free guide, 7 Enormously Useful ChatGPT Prompts For Creators. Grab a copy today!