Free AI web copilot to create summaries, insights and extended knowledge, download it at here

1538

Abstract

Data</b>: Load data into a standard format using LangChain Document object.</li></ol><div id="0ca0"><pre><span class="hljs-attr">loader</span> = UnstructuredFileLoader(<span class="hljs-string">"state_of_the_union.txt"</span>) <span class="hljs-attr">raw_documents</span> = loader.load()</pre></div><ol><li><b>Split Text</b>: Break the loaded text into smaller chunks for efficient processing.</li></ol><div id="a4e1"><pre><span class="hljs-attr">text_splitter</span> = RecursiveCharacterTextSplitter() <span class="hljs-attr">documents</span> = text_splitter.split_documents(raw_documents)</pre></div><ol><li><b>Create Embeddings and Store in Vectorstore</b>: Generate embeddings for each text chunk and store them in a vectorstore.</li></ol><div id="83fa"><pre><span class="hljs-attr">embeddings</span> = OpenAIEmbeddings() <span class="hljs-attr">vectorstore</span> = FAISS.from_documents(documents, embeddings)</pre></div><h2 id="7a4b">Querying of Data</h2><p id="d6d6">The querying process involves the following steps:</p><ol><li><b>Condense Question Prompt</b>: Create a standalone question by combining chat history and a new question.</li><li><b>Question-Answering Prompt</b>: Customize the prompt for answering questions based on relevant documents and the standalone question.</li><li><b>Language Model</b>: Choose a language model to power your chatbot, such as the OpenAI LLM.</li></ol><p id="f41c">After making customizations, run the ingestion script using <code>python ingest_data.py</code> to generate a <code>vectorstore

Options

.pkl</code> file.</p><h2 id="13c8">Putting it all together</h2><p id="94f5">The chatbot can be interacted with using a simple interface:</p><div id="57b1"><pre><span class="hljs-keyword">python</span> cli_app.<span class="hljs-keyword">py</span></pre></div><p id="028d">This opens a terminal interface to ask questions and receive answers. Additionally, the app can be deployed via Gradio using <code>python app.py</code> and to Hugging Face spaces.</p><p id="a6b6">This is a high-level overview of setting up ChatGPT over your data. For detailed code and examples, refer to the accompanying GitHub repo mentioned in this post.</p><div id="10be" class="link-block"> <a href="https://readmedium.com/langchain-langchains-support-for-streaming-d250f08100e4"> <div> <div> <h2>LANGCHAIN — LangChains Support for Streaming</h2> <div><h3>Software is a great combination between artistry and engineering. — Bill Gates</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*nu7ZXSdSXeo6aCLEJYoZpg.jpeg)"></div> </div> </div> </a> </div><p id="914f">This tutorial provides a comprehensive overview of setting up ChatGPT over your data. By following the detailed steps for ingestion and querying, you can create a customized chatbot that leverages your specific corpus of data.</p></article></body>

LANGCHAIN — Can GPT-3 be Used for Chat Over Your Data?

Computers are good at following instructions, but not at reading your mind. — Donald Knuth

LANGCHAIN — Async API

The function of good software is to make the complex appear to be simple. — Grady Booch

medium.com

ChatGPT Over Your Data

ChatGPT has gained popularity for general purpose knowledge, but it’s limited to pre-2021 internet data. This tutorial demonstrates how to set up your version of ChatGPT over a specific corpus of data. Let’s dive into the high-level walkthrough and the detailed steps for ingestion of data and querying of data.

High Level Walkthrough

Ingestion of Data

The ingestion process includes several steps to prepare the data for the chatbot:

Load Data: Load data into a standard format using LangChain Document object.

loader = UnstructuredFileLoader("state_of_the_union.txt")
raw_documents = loader.load()

Split Text: Break the loaded text into smaller chunks for efficient processing.

text_splitter = RecursiveCharacterTextSplitter()
documents = text_splitter.split_documents(raw_documents)

Create Embeddings and Store in Vectorstore: Generate embeddings for each text chunk and store them in a vectorstore.

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)

Querying of Data

The querying process involves the following steps:

Condense Question Prompt: Create a standalone question by combining chat history and a new question.
Question-Answering Prompt: Customize the prompt for answering questions based on relevant documents and the standalone question.
Language Model: Choose a language model to power your chatbot, such as the OpenAI LLM.

After making customizations, run the ingestion script using python ingest_data.py to generate a vectorstore.pkl file.

Putting it all together

The chatbot can be interacted with using a simple interface:

python cli_app.py

This opens a terminal interface to ask questions and receive answers. Additionally, the app can be deployed via Gradio using python app.py and to Hugging Face spaces.

This is a high-level overview of setting up ChatGPT over your data. For detailed code and examples, refer to the accompanying GitHub repo mentioned in this post.

LANGCHAIN — LangChains Support for Streaming

Software is a great combination between artistry and engineering. — Bill Gates

medium.com

This tutorial provides a comprehensive overview of setting up ChatGPT over your data. By following the detailed steps for ingestion and querying, you can create a customized chatbot that leverages your specific corpus of data.