LANGCHAIN — Can You Chat with Your Data Using OpenAI, Pinecone, Airbyte, and Langchain?
The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. — Mark Weiser
In this tutorial, we will walk through a real-world use case of leveraging vector databases and Language Model Models (LLMs) to make sense out of unstructured data. By the end of this tutorial, you will learn how to:
- Extract unstructured data from various sources using Airbyte
- Efficiently load data into a vector database and prepare the data for LLM usage
- Integrate a vector database into your LLM to ask questions about proprietary data
Step 1 — Fetch Github issues
To fetch Github issues, create a new source using the Github connector. If you are using Airbyte Cloud, you can easily authenticate using the “Authenticate your GitHub account”. Otherwise, follow the instructions in the documentation on how to set up a personal access token in the Github UI.
Step 2 — Load into vector database
Configure the destination by picking the “Pinecone” connector. After signing up for a free trial account and creating an index using a starter pod, set the dimensions to 1536, as that’s the size of the OpenAI embeddings we will be using.
Step 3 — Create a connection
Set up a connection from the Github source to the vector database destination. Click the stream name to select the individual fields to sync. The sync mode can be used to sync issues incrementally while deduplicating the records in the vector database.
Step 4 — Chat interface
Install relevant pip packages and create a script to implement the basic functionality of the chatbot. Set up OpenAI and Pinecone credentials as environment variables.
Step 5 — Put it on Slack
Set up a Slack “App” and create a new token. Extend the existing chatbot script with a Slack integration using the python slack sdk.
Step 6 — Additional data source: Scrape documentation website
Use the Apify service to scrape the documentation website and turn it into a dataset. Set up a source in Airbyte to connect to the Apify dataset.
Step 7 — Additional data source: Fetch Slack messages
Create a new source using the Slack connector to fetch Slack messages. Add text to the “text fields” of the destination to ensure the relevant data gets embedded properly.
By following these steps, you can set up a pipeline to load unstructured data from multiple sources into a vector database and implement an application that can answer plain text questions about the unstructured data in a general way. Expose this application as a Slack bot to make it available for wider use.
This setup offers flexibility and extensibility, allowing you to easily integrate specialized sources and leverage LLMs in different ways beyond this simple application.
If you are interested in leveraging Airbyte to ship data to your LLM-based applications, take a moment to fill out the survey to prioritize the most important features.
This tutorial provides a comprehensive guide to leveraging OpenAI, Pinecone, Airbyte, and Langchain to create a chat interface for querying unstructured data.
