Tiny-Vicuna-1B is the lightweight champion of the Tiny Models

Command and Conquer: the smallest Vicuna flavor is the Tiny Master of Instruction, answers your every call (Flawlessly!)

Forget the hulking behemoths of the AI arena, lumbering through processing power like a tank through mud. The battlefield has shifted, and the reigning champion isn’t Goliath, but David — in this case, the tiny and mighty Tiny-Vicuna-1b.

In a world obsessed with ever-larger language models, the quest for micro warriors who can dance on mobile devices and still pack a linguistic punch has become the new holy grail. And Vicuna-1b, the smallest flavor of Vicuna, stands triumphant — a Tiny Master of Instruction, ready to answer your every command (flawlessly!) without weighing down your devices.

But, will it be good enough? What can we do in real life scenarios with a model so small?

Let’s have a look together!

The promises

The past year saw a general trend towards giant Language Models: it was also leaked, later in 2023 (with no big surprises as I mentioned in several articles…) that GPT-4 is a Model of Experts.

So it seems that if you want to do something with AI you need the computational power of IBM or NASA: how can you ran 8 models of 220Billion parameters each making it a monster 1.76 Trillion parameter effective size?

Against transformers that guzzle terabytes like water, nano Models (like the new born GeminiNano), Tiny Models (like the TinyLlama family) and Microsoft Phi1 and 2 are rewriting the rules.

But all these actors have in common one goal: produce models Small enough to fit the hardware specifications of Mobile devices and Good enough to be useful in the business.

Tiny-Vicuna-1B

Don’t be fooled by the name only. Tiny-Vicuna is indeed a Llama model (but anyway Vicuna has always been an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT).

This small model is part of the TinyLlama project, that aims to pretrain a 1.1B Llama model on 3 trillion tokens with some proper optimization. But since Tiny Vicuna 1B is a TinyLLama 1.1B finetuned with WizardVicuna dataset it has been called Tiny-Vicuna!

This lightweight champion proves that size doesn’t matter when it comes to brains. It sprints through NLP tasks with the grace of a cheetah, leaving bulkier models gasping in its dust. Think flawless summarization, precise question answering.

Running it in quantized version it requires no more than 700 Mb of RAM! Tiny-Vicuna-1b whispers promises of a future where AI isn’t confined to data centers, but lives in your pocket, ready to be your personal language ninja at a moment’s notice.

So, buckle up, fellow AI adventurers, because we’re about to dive into the fascinating world of Tiny-Vicuna-1B-GGUF.

Tiny-Vicuna-1B-GGUF on the test bench

We will use the Quantized GGUF model files for Tiny-Vicuna-1B from Jiayi-Pan. This is mainly because even if it is a barely 1.1 Billion parameters model, running it on CPU in full precision will require almost 10 GB of RAM.

According to the original model card…

This model is a fine-tuned version of TinyLlama on WizardVicuna Dataset. It should be fully compatible with Vicuna-v1.5 series.

As a bonus… this model is under the Apache 2.0 license, and that means that 🥳

The Apache software license gives users permission to reuse code for nearly any purpose, including using the code as part of proprietary software. As with other open source licenses, the Apache license governs how end-users can utilize the software in their own projects. This license is a widely-used open source license, and like other permissive licenses, it continues to grow in popularity because it encourages the use of open source software within proprietary projects.

Required packages

We don’t need to much dependencies. You can find all the code on my Github repo.

Create a virtual environment in a new directory and activate it:

mkdir TinyVicuna
cd TinyVicuna

python3.10 -m venv venv #I am using python 3.10
python -m venv venv  #if you are on Windows

#to activate the Virtual Environment
source venv/bin/activate  #for mac
venv\Scripts\activate     #for windows users

As a minimum, since we have also a Gradio UI we have to install the following pip packages

pip install llama-cpp-python
pip install gradio
pip install psutil
pip install plotly

The last 2 dependencies are used only to give CPU/RAM usage statistics during the inference time. I think it is good to see all of this with your own eyes 😉.

Next is to download in the same directory the GGUF file. You can choose the quantization method, but do not go below q4. In my case I decided to use the q5 version: a little heavier but with minor quality loss. You can download them from here:

The core code for running the model is in 2 different sections of the python file. I will explain them here below. You can directly download the python file from the repo.

from llama_cpp import Llama
modelfile = "./tiny-vicuna-1b.q5_k_m.gguf"
contextlength=2048
stoptoken = '<s>'

################ LOADING THE MODELS  ###############################
# Set gpu_layers to the number of layers to offload to GPU. 
# Set to 0 if no GPU acceleration is available on your system.
####################################################################
llm = Llama(
  model_path=modelfile,  # Download the model file first
  n_ctx=contextlength,  # The max sequence length to use - note that longer sequence lengths require much more resources
  #n_threads=2,            # The number of CPU threads to use, tailor to your system and the resulting performance
)

######### INFERENCE #######################
response = llm(prompt, 
                max_tokens=max_new_tokens, 
                stop=['Q:', stoptoken], 
                temperature = temperature,
                repeat_penalty = repeat_penalty,
                top_p = top_p,
                echo=False)
print(response)

We import the Llama class from llama.cpp and instantiate it into the llm variable. As you can see we need to pass very little parameters here: the model path (that includes the GGUF filename) and the context window.

NOTE 1: if you are running on Windows the modelfile text does not need ./ so it should simply be modelfile = “tiny-vicuna-1b.q5_k_m.gguf”

NOTE 2: Every model is trained with a specific context window. If it is not mentioned in the model card you can always peek at the terminal console while loading the model the first time

You can read more about Model Cards in this article

Then we run the inference calling the llm with additional parameters. The only required one, really, is the prompt: the others are optional.

Note that the prompt must follow the specific Format of the Model. If you are unsure, and nothing is mentioned in the model card (like in this case) you can have a look to this amazing repository.

We search for the Vicuna version 1.5 because the original model states that This model is a fine-tuned version of TinyLlama on WizardVicuna Dataset. It should be fully compatible with Vicuna-v1.5 series.

Selecting the model you can verify what is the expected prompt format.

So our prompt template is as follows (be aware that there are no new lines, I created them for readability):

A chat between a curious user and an artificial intelligence assistant. 
The assistant gives helpful, detailed, and polite answers to the 
user's questions. USER: {prompt} ASSISTANT:

So our prompt can be something like this, in the code:

prompt = f"USER: What is Science? ASSISTANT:"

Real life Use Cases

Ok then, but how good can this little model possibly be?

In a previous article I compared many of the little LLM performances, and the GPT series was not that good

Considering that the base model is still the Tiny-Llama-1.1B we should not expect too much… but here is the catch!

The WizardVicuna dataset realty boosted the capabilities of the bases model. Let’s explore 3 main stream use cases:

1. general questions
2. Extract information from a context (RAG)
3. Format the output following the instructions.

1.General questions

What is science?

Science is the study of knowledge through observation, experimentation, analysis, and interpretation. It involves using logical reasoning, critical thinking, and empirical evidence to understand phenomena in the natural world or human experience.

Write a daily schedule for a busy professional man. Describe a typical daily routine from morning to bed time. Consider he is a man with a wife and kids.

Note: here I had to specify better what i wanted, becuase asking only for daily routine was giving me correct but inconsistent answers (never ending list)

 Morning Routine: 
- Wake up at around 6am, shower and get dressed.
- Have breakfast (breakfast can be anything from eggs, bacon or pancakes) and coffee/tea.
- Check emails and work on important tasks for the day.
- Take a quick walk outside to clear his mind before starting work.
- Start working on his daily routine: 
   - Opening email and answering any urgent messages.
   - Creating to-do lists for the day's activities.
   - Checking social media accounts for updates or breaking news.
   - Reviewing important documents/files.
   - Making phone calls if necessary.
   - Attending meetings or appointments as needed.
   - Taking breaks and stretching to avoid burnout.
- Lunch: 
   - Eating a light lunch, such as sandwiches or salads with hummus or guacamole.
   - Drinking water or tea/coffee.
- Afternoon Routine: 
   - Working on any important tasks from the morning's work.
   - Checking emails and answering urgent messages.
   - Taking breaks to stretch, walk around, or do some light exercise.
   - Spending time with family/friends if possible.
- Evening Routine: 
   - Watching TV or Netflix before going to bed.
   - Reading a book or watching an episode of his favorite show.
   - Taking a warm bath or reading a book while relaxing in bed.
- Bedtime Routine: 
   - Winding down by taking a hot bath, reading a book, and listening to soothing music.
   - Going back to sleep with the lights off.

2.Extract information from a context (RAG)

I took the extract from a youtube video and used as context to reply user questions.

NOTE: this method is applicable also for chatbots that must reply to specific data only and not use external knowledge

Reply to the question only using the provided context.
[context]WHAT IS HIERARCHY 4.0? whether you own build manage maintain or operate an oil plant inevitably issues arise that require immediate action and resolution. With big data flowing in constantly from all sectors making sense of everything while troubleshooting
issues without wasting time can be a huge challenge.  So what's the solution?
introducing hierarchy 4.0 and Innovative software solution for control Safety Systems . Hierarchy 4.0 presents an interactive diagram of the entire plant revealing cause and effect Behavior with readings provided in a hierarchical view allowing for a deep understanding of the system's strategy
All data is collected from multiple sources visualized as a diagram and optimized through a customized dashboard allowing users to run a logic simulation from live data or pick a moment from their history.  Your simulation is based on actual safety Logics not just on a math model
Now every users can prepare an RCA report 90 percent faster in just a few minutes. Hierarchy can be used for any project phase starting from engineering to commissioning and up to operation and maintenance while supporting hazop Hazard analysis by reducing human error and avoiding incorrect documentation. Hierarchy 4.0 supports plant operators in decision making taking into account both the safety and the operability of their assets.  Hierarchy 4.0 Embraces a block log approach: it automatically calculates all Logics affected by an
override and gives a full understanding of constraints.  Not convinced let's look at the data!
During its first project hierarchy 4.0 prevented a revenue loss of 45 million dollars.  Plants that utilize hierarchy 4.0 save up to 95 of their time and deliver a return on investment up to five times in value from day one and experience a Personnel utilization and plant efficiency increase by 20 percent per year.Try our demo and make the move to hierarchy 4.0 today[end of context]

Question: what is Hierarchy 4.0?

The model processed the 500 tokens in the prompt in 40 seconds with this reply 👍🥳:

Hierarchy 4.0 is an innovative software solution for control safety systems that provides a deep understanding of the system’s strategy through interactive diagrams and optimized dashboards. It allows users to run a logic simulation from live data or pick a moment from their history, reducing human error and avoiding incorrect documentation. The RCA report can be prepared in just a few minutes using Hierarchy 4.0’s block log approach. This supports plant operators in decision making taking into account both the safety and operability of their assets.

For punctual/factual RAG usually we use smaller chunks (250/350) after similarity search on a database, so the speed will be good enough.

An more importantly, the information is correct!

⚠️ remember to limit the reply of the chatbot in your system prompt, to make sure that it replies only to the provided knowledge and say “I don’t know” for all other scenarios.

3.Format the output following the instructions.

Here I am going to use 2 main tasks: have a summary from the text and ask the LLM to provide the main topics of a text formatting it into a list.

The prompt is quite simple, but I had to try several version of it before finding the one giving me only the Summary without any introduction words…

[text]... insert here your text...
... the move to hierarchy 4.0 today[end of text]

- SUMMARY:

It did a good job here!

Hierarchy 4.0 is an innovative software solution for control safety systems that provides a deep understanding of the system’s strategy, visualizes data in a hierarchical view, and supports plant operators in decision making taking into account both the safety and operability of their assets. It also saves time and delivers a return on investment by preventing revenue loss due to errors or overlooked hazards.

Also for the second tasks I had to find the good wording and order to get a markdown list as an output. The model followed it correctly

Write the main points of the following text.
[text]WHAT IS HIERARCHY 4.0? whether you ... the move to 
hierarchy 4.0 today[end of text]

format the output as a markdown list.

And as expected the result is good!

- Hierarchy 4.0 is an innovative software solution for control safety systems that provides a deep understanding of the system's strategy through interactive diagrams, optimized with a customized dashboard and supported by RCA reports.
- All data is collected from multiple sources visualized as a hierarchical view allowing users to run a logic simulation from live data or pick a moment from their history.
- Your simulation is based on actual safety logics not just on a math model, making it faster and more accurate for any project phase starting from engineering to commissioning and up to operation and maintenance.
- Hierarchy can be used for any project phase starting from engineering to commissioning and up to operation and maintenance while supporting hazop Hazard analysis by reducing human error and avoiding incorrect documentation.
- Hierarchy 4.0 supports plant operators in decision making taking into account both the safety and the operability of their assets, with a block log approach that calculates all Logics affected by an override automatically.
- Not convinced let's look at the data! During its first project hierarchy 4.0 prevented a revenue loss of 45 million dollars, saving up to 95 of their time and delivering a return on investment up to five times in value from day one and experiencing a Personnel utilization and plant efficiency increase by 20 percent per year.
- Try our demo and make the move to hierarchy 4.0 today!

NOTE: To my surprise, the list generation was accurate and always on the topic.

Test it yourself

To run with the same Gradio User Interface I used you can easily:

• prepare the virtual environment and activate it
• install the dependencies as per the Required Packages section above
• download the q5 GGUF model file from here
• download the python file from here
• fix line 39 as described above

#only "tiny-vicuna-1b.q5_k_m.gguf" if you are on Windows
39|  modelfile = "./tiny-vicuna-1b.q5_k_m.gguf"  

• save the file and run this command on your terminal from the project directory

python 40-vicuna1B_PG_MEM.py

Your default browser will open a new tab with the Gradio Interface

To close the application:

• close the Browser Tab
• type ^C on the Terminal window

Conclusions

Lately it may seem that I am focusing too much on Small Language Models. And it is true.

But if you look around there are really tons of resources about private/paid AI services and really little for common users. And this is really frustrating if we consider how MANY open source models are out there for free…

The best SOTA (State Of The Art) benchmark will always be you! I mean, we know what we want to build, and we need a digital partner good enough to do a good job.

Hope you enjoyed the article. If this story provided value and you wish to show a little support, you could:

1. Clap a lot of times for this story
2. Highlight the parts more relevant to be remembered (it will be easier for you to find it later, and for me to write better articles)
3. Learn how to start to Build Your Own AI, download This Free eBook
4. Sign up for a Medium membership using my link — ($5/month to read unlimited Medium stories)
5. Follow me on Medium
6. Read my latest articles https://medium.com/@fabio.matricardi

All the code and screenshots are in my GitHub Repository

If you want to read more here some ideas using Small Language models and Hugging Face Hub:

Stackademic

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us on Twitter(X), LinkedIn, and YouTube.
• Visit Stackademic.com to find out more about how we are democratizing free programming education around the world.