GROK: Did Elon Just Drop An AI Game-Changer?
The Open-Source Era Of GROK Begins
How a new open-source model is offering unparalleled scale
The large language model (LLM) “Grok” caused quite a buzz during its initial development and testing phase back in 2023.
Elon Musk’s X.ai has now released its code, providing us with possibly the largest open-source model EVER, boasting over 300 billion parameters (for comparison, GPT-3 had 175 billion).
This is so large that it cannot fit anymore on a single GPU — nothing unusual for LLMs, but quite unique for an open-source model. However, there’s no need to worry: we will still be able to utilize Grok without needing to set up a GPU farm in our bedrooms. Here’s how:
What We Know About Grok
It may take another few days before we can actually use Grok. But this is worth the wait considering
- the immense size (314 billion parameters)
- the fact that it is a non-fine-tuned model without any specialization so far (for example, GPT-4 has been fine-tuned to follow instructions and thus work in a dialogue-style chatbox interface)
Grok is particularly intriguing to keep an eye on, given the potential developments that will arise from further fine-tuning and quantization (we’ll clarify what that means in the next section).
So, what do we know about Grok? Here’s what X.ai tells us:
We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI.
This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue.
Besides the size and the fact it’s a “raw base model” without any fine-tuning, there’s another interesting detail: Grok uses the successful Mixture Of Experts (MoE) architecture, the same architecture that powers GPT-4. MoE seems to significantly increase the capabilities of models, especially when they are as large as Grok with its 300+ billion parameters.
You can read more about how MoE supports language models here:
We do know how big Grok is, but unfortunately we don’t know anything about the training data yet. However, if you look at the benchmarks that X.ai published, Grok appears to perform within the capability range of GPT-3.5 or Mistral.
In a few days, I think, we can expect Grok (and its derivates) to show up on the usual LLM leaderboards here or here.
What’s Next?
It usually only takes a few days or weeks for the open source community to publish quantized versions of major open source models.
In what is known as “quantization”, the original model is simplified to reduce its size and shorten its inference time. This allows it to be used on devices with limited computing power or memory.
How does this simplification work?
Remember the parameters, of which Grok has about 300+ billion? Each individual parameter is actually a “computation point”, if you will, a place where the model calculates numerical values that add up to the final prediction at the end: which is the most likely word to follow an input sequence of words.
Quantization reduces the precision of these numerical values (usually by converting floating point numbers to lower precision formats).
Once we have quantized versions of Grok, we’ll see more people experimenting with it. In the meantime, we can expect the first online demos for Grok to show up on your favorite machine learning platform, such as Huggingface or Replicate.
Links
Grok Announcement (2023): https://x.ai/blog/grok
Grok Open Source Announcement (2024): https://x.ai/blog/grok-os
Grok on Huggingface: https://huggingface.co/xai-org/grok-1
Grok on Github: https://github.com/xai-org/grok-1
➡️ Follow me to stay up to date on “AI &Creativity”. If you want to support my work, become a Medium member using my referral link and get full access to all my articles (170+ and growing) and those of thousands of other writers. 🙏
➡️ If you like my content, why not leave a “clap” at the end of this article, so more people can see it?
This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!
