Tiny Tweaks, Big Impact: Mastering the parameters of your Mini Language Model

Bake off the hidden potential of small LLMs through careful “ingredients” tuning.

Have you ever tried baking a cake and ended up with something more akin to a brick? We’ve all been there, staring at a disappointing culinary outcome, wondering where we went wrong. Turns out, the answer might be lurking in the measurements! Just like baking, the delicate art of crafting successful text with a small language model (LLM) hinges on fine-tuning its parameters.

But before we dive into the flour and parameters, let’s take a step back.

The secret recipe

Imagine an LLM as a highly skilled chef, equipped with a vast recipe book of words and phrases. To whip up the perfect text dish, however, they need more than just ingredients. They need precise instructions, the right amount of each element, and maybe even a dash of creativity. That’s where parameters come in — they act as the recipe guidelines, telling the LLM how to blend and bake words into something delicious.

Now, why is this especially crucial for small LLMs? Think of them as your petite sous chefs, eager to please but still learning the ropes. With fewer ingredients (parameters) at their disposal, every tweak counts. A sprinkle too much of one element can ruin the whole dish, while a timid pinch might leave it bland and uninspired.

So, how do we master these tiny tweaks and unlock the hidden potential of our mini LLMs? Let’s break it down into three key steps.

Know your ingredients

Understanding what each parameter does is fundamental. Some adjust the model’s temperature, influencing its creativity and risk-taking. Others control the structure and coherence of the generated text.

Imagine you’re a chef writing a recipe using a fancy AI cookbook of words and phrases. You want to choose the best ingredients (words) to create a delicious dish (text), but there are different ways to pick them! Here are three special tools that help you fine-tune your recipe:

Temperature: This is like the oven’s heat setting. A low temperature plays it safe, sticking to common words and grammar, resulting in familiar, predictable text. A high temperature cranks up the creativity, making the AI take bigger risks and potentially surprise you with unexpected or even wacky words.

Top_p: Think of this as picking ingredients from a hat. With high top_p, you grab any word with a decent chance of fitting, leading to more diverse but sometimes nonsensical results. Low top_p focuses on the most likely choices, making the text smooth and consistent, but possibly less exciting.

Repeat_penalty: This discourages using the same word repeatedly. Imagine oversalting your dish! High repeat_penalty puts the brakes on redundancy, ensuring variety and fresh flavors in your text. Low repeat_penalty allows for some repetition, which can be useful for emphasis or rhythm.

By adjusting these tools, you can control the style, creativity, and overall taste of your AI-generated text, just like a chef tailoring a recipe to perfection. So, experiment, have fun, and remember, a pinch of the unexpected can add a delicious dash of magic to your words!

Practical examples?

Here out of the box tips, to give you an idea of how these parameters can be used in different scenarios: taken from OpenAI and Cohere’s blogs.

Experiment with precision

Don’t be afraid to get your hands dirty! Small changes can yield surprising results. Start with subtle tweaks, testing the impact on your text’s quality and accuracy. Remember, sometimes a dash less is more.

So, let’s try making our AI chef bake the perfect text pastry.

Imagine these two dials: Temperature controls the heat of the oven, how adventurous the AI gets with words. Top_p decides how picky the AI is when choosing ingredients, whether it grabs anything decent or only the top choices.

1. Hot oven, picky eater (High temperature and low top_p): The AI is a bit wild with words thanks to the hot temperature, but keeps things mostly sensible by only grabbing likely ingredients. This can be good for a surprise twist without going off the rails.

2. Cool oven, top gourmet (Low temperature and high top_p): The AI plays it safe with familiar words, like a fancy restaurant picking only the finest ingredients. This makes the text smooth and clear, but maybe a bit boring.

3. Hot oven, free-for-all (High temperature and high top_p): The AI throws caution to the wind, grabbing any word that catches its eye! This can lead to wild, creative dishes, but they might be a bit strange or hard to understand.

4. Cool oven, super picky (Low temperature and low top_p): The AI is like a grandma sticking to tried-and-true recipes, choosing only the most common words and never taking risks. This results in safe, predictable text, but it might lack excitement.

By playing with these two dials, you can find the perfect recipe for your text, whether you want a zany surprise, a reliable classic, or something in between. So go forth, experiment, and enjoy the delicious possibilities!

Find your sweet spot — Use a Playground and experiment prompt formats

Every text generation task and desired outcome has its own perfect parameter recipe. Practice makes perfect, so keep experimenting and fine-tuning until you achieve the text that sings to your specific needs.

I tried in the last 2 weeks more than 100 Tiny LLMs. These little guys have a lot of potential, but they are like little children:

1. they need clear instructions
2. they cannot inference with flexibility, so the instructions must like in the primary school textbooks
3. temperature and top_p must be used with care

Furthermore, remember that every model is trained with special tokens: Large Language Models accept instructions only with a special template.

Here 2 examples from this amazing source:

VICUNA STYLE PROMPT TEMPLATE

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. 
USER: hello, who are you? 
ASSISTANT: 


ALPACA STYLE PROMPT TEMPLATE

Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n

### Instruction:\n{instruction}

### Response:"

Have a look at few of my previous articles to see how it is important to feed the model with the correct prompt template structure.

Remember, also, that every instructed base model treats special delimiters like “-” or “>” in a different way: they can help structure the prompt and guide the model’s output.

Delimiters serve as crucial tools in prompt engineering, helping distinguish specific segments of text within a larger prompt. For example, they make it explicit for the language model what text needs to be translated, paraphrased, summarized, and so forth.

Delimiters can take various forms such as triple quotes (“””), triple backticks (“`), triple dashes ( — ), angle brackets (< >), XML tags ( ), or section titles. Their purpose is to clearly delineate a section as separate from the rest.

Here two good examples from https://www.topbots.com/prompt-engineering-chatgpt-llm-applications/

As you can understand, even though small/tiny LLMs have a great potential, most of their performance lies on how good we are in finding the perfect recipe balancing all the ingredients.

So you MUST build your own Playground. I will leave an example in the suggested articles at the end: feel free to take inspiration and do it yourself.

Conclusions

Mastering the art of parameter tuning might seem daunting, but don’t be discouraged! Just like baking, it’s a process of exploration and discovery. With a little patience, practice, and perhaps a dash of curiosity, you’ll be able to confidently guide your mini LLM chefs, transforming them from brick-makers to language maestros, churning out delectable text creations fit for any occasion.

So, go forth and tweak! Remember, in the world of small LLMs, every parameter adjustment holds the potential to unlock a universe of possibilities. Bon appétit!

Hope you enjoyed the article. If this story provided value and you wish to show a little support, you could:

1. Clap a lot of times for this story
2. Highlight the parts more relevant to be remembered (it will be easier for you to find it later, and for me to write better articles)
3. Learn how to start to Build Your Own AI, download This Free eBook
4. Sign up for a Medium membership using my link — ($5/month to read unlimited Medium stories)
5. Follow me on Medium
6. Read my latest articles https://medium.com/@fabio.matricardi

If you want to read more here some ideas:

PlainEnglish.io 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer️
• Learn how you can also write for In Plain English️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter
• Visit our other platforms: Stackademic | CoFeed | Venture