[RAG Guide] Creating a Q&A Chatbot on Colab: Integrating Embeddings and Mistral-7B
This post is about how to transform a 400-word dummy company profile into an interactive Q&A chatbot. When someone asks question, the chatbot will answer based on the dummy company profile.
The text will be broken down into chunks. Embeddings and a chroma database will then be created to store the chunks. Also a retriever will be created pinpoint the most relevant text segments in the database in response to user queries. At the end, using a question-answering chain, the LLM Mistral-7B and the retriever are ready to answer questions based on the dummy company profile.
Table of Contents
- Use Case
2. Dataflow
3. Source Code
1. Use Case
When someone asks the chatbot, “What are your products?”, it responds with information directly from the dummy company profile.
As demonstrated in the screenshot, the answer given is accurate and matches the details specified in the dummy company profile.
Following are the dummy company profile i used:
Company Profile:
Company Name: Dummy-Gpt2-Datatec-Studio Inc
At Dummy-Gpt2-Datatec-Studio Inc, we are a cutting-edge technology
company committed to transforming visionary ideas into tangible
realities. Our mission is to pioneer innovation that makes a
meaningful impact on the world, from extending human longevity to
addressing critical environmental challenges and revolutionizing the
way we travel.
Portfolio:
Life-Enhancing Pharmaceuticals: Our portfolio includes
groundbreaking pharmaceutical research aimed at developing vaccines
and treatments to prevent diseases, potentially adding an additional
20 years to the human lifespan.
Environmental Sustainability: We are leaders in developing
innovative solutions to combat climate change and ensure a
sustainable future for the planet, encompassing green energy, waste
reduction, and carbon footprint mitigation.
Advanced Transportation: In our transportation division, we engineer
state-of-the-art terrestrial transportation solutions, promising
ultra-fast and efficient modes of travel that will reshape the way
we move.
Products:
Green Energy Devices: Our product line features advanced devices
that empower individuals and businesses to generate clean and
renewable energy, rendering traditional power plants and fossil
fuels obsolete.
Space Exploration Services: Our space exploration services offer
accessible and safe travel options to explore colonies on celestial
bodies such as solar planets and planetary moons, opening new
frontiers for exploration.
Quantum Computing Solutions: We provide cutting-edge quantum
computing solutions, exponentially boosting computational power and
network speeds, thus ushering in a new era of technological
advancement.
Holographic Communication: Our holographic communication technology
replaces traditional televisions and phones with immersive,
interactive experiences. Integrating the senses of touch and smell,
we bring the virtual world closer to reality.
Services:
Genetic Engineering: Our genetic engineering services explore
revolutionary concepts like Wi-Fi-emitting plants, potentially
transforming the way we connect and communicate in a lush and
connected world.
Futuristic Planning Consultation: We specialize in assisting
individuals and organizations in planning for the distant future.
Leveraging transformative technologies, we help shape and anticipate
possible outcomes to achieve long-term goals.
Contact:
For inquiries and collaborations, please reach out to us at:
Email: [email protected]
Phone: +1-123-456-7890
Address: 123 Innovator's Lane, Futuroville, Earth
Community Engagement:
At Dummy-Gpt2-Datatec-Studio Inc, we take our commitment to
community involvement seriously. We actively support STEM education
programs, empowering future generations to be at the forefront of
scientific and technological advancements that will shape our
collective future.2. Dataflow
Since the dataset we need isn’t very large, I create embeddings directly from it instead of using only a portion that might be selected through a query or prompt. If such query is used, it’s important to ensure that this dataset portion includes all relevant information. otherwise, the final result could be incorrect, even when using a large language model (LLM) at the end.
In the method described in this post, I’ve opted to divide the dataset into large chunks to keep the information as intact as possible.
3. Source Code
Following are the related Colab Code. The colab project and the output of each steps can be found in my github repo.
The code should work for both GPU and CPU. I just ran it on GPU.
# AI MVP Project from datatec.studio
!pip install transformers torch accelerate bitsandbytes langchain
!pip install -U sentence-transformers chromadb
# Import Libraries
from transformers import AutoModelForCausalLM, AutoTokenizer
from langchain.llms import HuggingFacePipeline
from langchain.docstore.document import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
import torch
import transformers
# Create model and tokenizer
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", load_in_4bit=True, device_map='auto')
device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
# Create text generation pipeline with Mistral model
text_generation_pipeline = transformers.pipeline(
model=model,
tokenizer=tokenizer,
task="text-generation",
temperature=0.1,
repetition_penalty=1.2,
return_full_text=True,
max_new_tokens=1000
)
mistral_llm = HuggingFacePipeline(pipeline=text_generation_pipeline)
# Create dummy company profile
dummy_company_profile = """
Company Profile:
Company Name: Dummy-Gpt2-Datatec-Studio Inc
At Dummy-Gpt2-Datatec-Studio Inc, we are a cutting-edge technology company committed to transforming visionary ideas into tangible realities. Our mission is to pioneer innovation that makes a meaningful impact on the world, from extending human longevity to addressing critical environmental challenges and revolutionizing the way we travel.
Portfolio:
Life-Enhancing Pharmaceuticals: Our portfolio includes groundbreaking pharmaceutical research aimed at developing vaccines and treatments to prevent diseases, potentially adding an additional 20 years to the human lifespan.
Environmental Sustainability: We are leaders in developing innovative solutions to combat climate change and ensure a sustainable future for the planet, encompassing green energy, waste reduction, and carbon footprint mitigation.
Advanced Transportation: In our transportation division, we engineer state-of-the-art terrestrial transportation solutions, promising ultra-fast and efficient modes of travel that will reshape the way we move.
Products:
Green Energy Devices: Our product line features advanced devices that empower individuals and businesses to generate clean and renewable energy, rendering traditional power plants and fossil fuels obsolete.
Space Exploration Services: Our space exploration services offer accessible and safe travel options to explore colonies on celestial bodies such as solar planets and planetary moons, opening new frontiers for exploration.
Quantum Computing Solutions: We provide cutting-edge quantum computing solutions, exponentially boosting computational power and network speeds, thus ushering in a new era of technological advancement.
Holographic Communication: Our holographic communication technology replaces traditional televisions and phones with immersive, interactive experiences. Integrating the senses of touch and smell, we bring the virtual world closer to reality.
Services:
Genetic Engineering: Our genetic engineering services explore revolutionary concepts like Wi-Fi-emitting plants, potentially transforming the way we connect and communicate in a lush and connected world.
Futuristic Planning Consultation: We specialize in assisting individuals and organizations in planning for the distant future. Leveraging transformative technologies, we help shape and anticipate possible outcomes to achieve long-term goals.
Contact:
For inquiries and collaborations, please reach out to us at:
Email: [email protected]
Phone: +1-123-456-7890
Address: 123 Innovator's Lane, Futuroville, Earth
Community Engagement:
At Dummy-Gpt2-Datatec-Studio Inc, we take our commitment to community involvement seriously. We actively support STEM education programs, empowering future generations to be at the forefront of scientific and technological advancements that will shape our collective future.
"""
# Create Document object from text documents
docs = [Document(page_content=post) for post in [dummy_company_profile]]
# Split documents into chunks, create embedding model and retriever
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1500, chunk_overlap=10, separators=['\n\n', '\n', '.']
)
document_chunks = text_splitter.split_documents(docs)
embedding_model = SentenceTransformerEmbeddings(model_name='BAAI/bge-large-en-v1.5')
chroma_db = Chroma.from_documents(document_chunks, embedding_model)
retriever = chroma_db.as_retriever()
# Create question answer chain
qa_chain = RetrievalQA.from_chain_type(mistral_llm, retriever=retriever)
# Ask questions to chatbot
question = input("Please enter your question: ")
response = qa_chain({"query": question})
print(response['result'])In the end, the AI chatbot answered the question correctly within 14 seconds, as the screenshot at the begin of this page shows.
Although the speed could be improved, the quality of the result really impressed me.
I hope you enjoyed today’s content.
Your claps 👏 and engagement keep me inspired!
