Everything About MISTRAL’S Mixtral-8x7B: The Best Open LLM

Every December, machine learning experts gather at the annual NeurIPS conference to discuss the latest and greatest achievements in ML. This influential event makes late November and early December an ideal time for AI startups to launch their products.

ChatGPT revolutionized the AI landscape with its launch on November 30, 2022. Following this, Google introduced its controversial model, Gemini, on December 6th 2023. Just a few days later, Mistral announced their latest model. They shared a torrent magnet link on Twitter, a genius marketing strategy that made the team very popular with the community.

Best Open Source Model So Far?

When it comes to performance, Mixtral seemingly matches or surpasses GPT-3.5 and Llama 2 70B on most benchmarks, while offering 6X faster inference compared to Llama 2.

Its notable features include:

• A context window of 32,000 tokens
• Compatibility with English, French, Italian, German, & Spanish

• Excellent code generation capabilities
• Capability for fine-tuning into an instruction-following model
• Less hallucination and biases

Furthermore, Mixtral has been released under the permissive Apache 2.0 license. Unlike some of OpenAI’s closed-source models, we can see the total parameter count of this model.

With its impressive performance and features, various newsletters, AI influencers, and blog posts have heralded Mixtral as the ‘Best Open-Source Model Ever’ often using hype-inducing titles like ‘Open-source AI on the rise.’

But is it REALLY open source? Let’s take a step back and examine.

In their blog, Mistral’s team defines Mixtral 8x7B as:

“a high-quality sparse mixture of experts model (SMoE) with open weights”.

Interestingly, Mistral doesn’t label their model as ‘open-source,’ but rather as an ‘open-weights’ model. As Andrej Karpathy, an AI expert at OpenAI, points out for a model to be truly open-source, it should also include access to the training code, dataset, and documentation — elements Mistral hasn’t shared.

Furthermore, Mistral’s Terms of Service initially prohibited using Mixtral’s outputs for anything other than fine-tuning their models. This restriction did not sit well with the public, leading to backlash and prompting Arthur Mensch, CEO of Mistral, to remove the problematic clause.

Lastly, to address the elephant in the room: Mistral isn’t a GitHub repository that drew a vast number of programmers tirelessly working to release a smart open-source LLM. Instead, it’s a company that raised over $500 million in 2023 alone, with Bloomberg valuing the company at roughly $2 billion. Their choice not to release the training code, documentation, and dataset — in other words, keeping the model relatively closed-source — is a strategy for Mistral to maintain its competitive edge.

So to sum things up, Mixtral-8x7B is the best open-weights model that outperforms GPT3.5 and Llama 2 70B on most benchmarks.

What is Mixture of Experts (MoE) Anyway?

Let me remind you again of Mistral’s definition of the model:

“a high-quality sparse mixture of experts model (SMoE) with open weights”.

A lot of us (myself included) encountered this term for the first time thanks to the release of Mixtral. To TLDR it for you:

MoEs originated from a 1991 research paper where a system uses multiple neural networks. These networks, or ‘experts’, are selected by a gating network that assigns their roles and weights. Both experts and the gating network are trained together to effectively handle diverse training cases.

For anyone interested going into more depth, I suggest this HuggingFace article.

Sparsity in MoEs means that only parts of the network are active for specific inputs, allowing scaling without increasing total computation.

Pros of sparse MoEs:

1. Efficient Pretraining: MoEs allow for faster and more compute-efficient. This means larger models or datasets can be trained with the same computing budget.
2. Faster Inference: Although MoEs have many parameters, only a subset is active during inference, leading to quicker processing.

Cons of sparse MoEs:

1. Fine-Tuning: sparse MoEs historically struggle with fine-tuning, often leading to overfitting, although recent research gives a lot of hope.
2. Memory Requirements: All parameters of MoEs need to be loaded into memory, requiring substantial VRAM.
3. Load Balancing: Ensuring even distribution of computation across experts is a challenge, often addressed by auxiliary loss functions and expert capacity thresholds.

How To Run it Locally

2 most popular ways are:

VLLM

and LM Studio

Mixtral-8x7B Demos

If you’d like to test out Mixtral but you don’t have an option to run it locally, here’s a list of the best platforms that offers Mixtral inference:

Perplexity labs

Vercel

Replicate

Platform API endpoints

Mistral AI currently provides access to Large Language Models through an pay-as-you-go API. It’s in beta right now and however wants access has to join the waitlist first.

Mistral offers 3 models for endpoints:

1. mistral-tiny — Mistral-7B-v0.2, a better fine-tuning of the initial Mistral-7B released. The most affordable model
2. mistral-small — Mixtral-8X7B-v0.1, excellent choice for code generation and multilanguage output
3. mistral-medium — This model is a mystery and many speculate that its performance might match GPT4
4. mistral-embed — Embeddings models for retrieval and RAG

Mistral AI provides client codes in both Python and Javascript. For the sake of simplicity I’ll share the client code in Python, but you can find javascript code in their docs:

Chat Completion

from mistralai.client import MistralClient
from mistralai.models.chat_completion import ChatMessage

api_key = os.environ["MISTRAL_API_KEY"]
model = "mistral-tiny"

client = MistralClient(api_key=api_key)

messages = [
    ChatMessage(role="user", content="What is the best French cheese?")
]

# No streaming
chat_response = client.chat(
    model=model,
    messages=messages,
)

# With streaming
for chunk in client.chat_stream(model=model, messages=messages):
    print(chunk)

Embeddings

from mistralai.client import MistralClient

api_key = os.environ["MISTRAL_API_KEY"]
client = MistralClient(api_key=api_key)

embeddings_batch_response = client.embeddings(
      model="mistral-embed",
      input=["Embed this sentence.", "As well as this one."],
  )

Conclusion

Mixtral 8x7B stands out as an exceptional open-weights model. The Mistral team appears deeply attuned to their customers’ needs, consistently delivering products that resonate with a wide audience through clever, yet relatively low-key marketing strategies.

After several years of OpenAI’s market dominance, Mistral’s API access is set to offer millions of developers greater choice, enabling them to select the model that best meets their needs. With substantial funding behind them, we can anticipate not only amazing models from Mistral but also a potential shift towards more closed-source LLMs. I’m genuinely excited about Mistral and can’t wait to see what’s next!