Mixtral MOE 8x7b — A new open-source giant
The french company Mixtral really isn’t staying behind the american ones, her model Mistral 7b is considered one of the best open-source models released surpassing even bigger models and now it launches one of the first successful implementations of the MOE (Mix of Experts) architecture in a open-source model.
NA: To those not familiar with this architecture it is the same rumored used by GPT4 by the leak earlier this year and by the results we see on this model sure look like it’s true, to know more about this you can read my earlier post:
This weekend they released the model to anyone to download on torrent causing a commotion in the community. Today we see various fine-tuned versions of hugginface user’s that can be tried out.
But first let’s take a step back:
What is the MOE Architecture?
The “Mixture of Experts” (MoE) architecture in machine learning is like having a team of specialists where each member is good at solving a specific type of problem. Imagine you have a big, complex problem to solve. Instead of asking one person (or one model) to do everything, you divide the problem into smaller parts and assign each part to a different expert who is really good at that specific thing.
So basically instead of one big model running, we have a set of smaller specialists on a task model and an entering gateway that decides which model to use depending on the prompt.
A few advantages of using this is a faster inference (since the smaller model executes faster than a big one), more efficient and effective problem-solving, can be scaled easily by adding more experts to the model and an efficient dynamic resources allocation.
Some disadvantages can be complexity in the implementation and difficulty in training.
How was Mixtral MOE implemented?
Mistral as appointed in his name has 8 experts of 7b parameters, it’s a decoder only model (as is more common to llms) where the feedforward block (the gate) chooses one or at least two of those each time and then combines their output to generate the final answer.
Mixtral has 46.7B total parameters but only uses 12.9B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12.9B model.
It has a context window of 32K, can understand English, French, Italian, German and Spanish and shows strong performance in code generation.
How it performed?
Here are some comparison benchmarks of the model with other famous ones, the images speak for themselves:
Where can you try it out?
The model is available in two versions Mixtral 8X7B v0.1 and Mixtral 8X7B v0.1 Instruct.
Here we have the original versionw by Mixtral on hugginfaces:
But it is a very big model and to who may have difficulty to have access to some very powerful GPUs here are some other options:
- Quantized:
The Bloke added quantized versions of the model in the formats GPTQ, AWQ and GGUF on his hugginface. The GGUF version that runs on CPU has a smaller RAM requirement of 18.14G, a medium of 28.94G and the bigger with 49.62G.
And this hugginface user has made available a version of the model in a chat tuned version:
Now, if even the quantized version is to big for you there is a few other options.
- Mixtral API
Mixtral has made avaliable versions of the model in their api that you can asked for access.
- Perplexity Labs
Perplexity made available the model on the chatbot Perplexity Labs for testing purposes.
- Vercel
The model is also available for testing on Vercel online SDK.
To who may interest, i am making a Generative AI Basic Course where i teach how to use diverse online tools to make all type of generations: text, image, video and audio. The course is still being developed but it already has the whole curriculum that will be covered displayed and the complete text, image and video generation are complete with audio following son after.
For those who join in this earlier stages i am giving a discount of 35% on the price with the cupom EARLIERADOPTERS. Follow the link to access it:
