LANGCHAIN — Constructing Knowledge Graphs from Text Using OpenAI Functions
The great myth of our times is that technology is communication. — Libby Larsen
In this tutorial, we will explore how to construct a knowledge graph from unstructured text using OpenAI functions in combination with LangChain. Knowledge graphs are extremely useful for multi-hop question-answering, real-time analytics, and combining structured and unstructured data in a single database.
To get started, we need to set up a Neo4j environment. You can start a free instance on Neo4j Aura or set up a local instance using the Neo4j Desktop application.
Next, we will instantiate a LangChain wrapper to connect to the Neo4j Database:
from langchain.graphs import Neo4jGraph
url = "neo4j+s://databases.neo4j.io"
username ="neo4j"
password = ""
graph = Neo4jGraph(
url=url,
username=username,
password=password
)After setting up the environment, we can begin the information extraction pipeline using OpenAI functions:
class Node(Serializable):
id: Union[str, int]
type: str = "Node"
properties: dict = Field(default_factory=dict)
class Relationship(Serializable):
source: Node
target: Node
type: str
properties: dict = Field(default_factory=dict)To overcome the limitations of the API, we can use a list of Property classes instead of a dictionary:
class Property(BaseModel):
key: str = Field(..., description="key")
value: str = Field(..., description="value")
class Node(BaseNode):
properties: Optional[List[Property]] = Field(
None, description="List of node properties")
class Relationship(BaseRelationship):
properties: Optional[List[Property]] = Field(
None, description="List of relationship properties")We can then define the KnowledgeGraph class to combine the nodes and relationships:
class KnowledgeGraph(BaseModel):
nodes: List[Node] = Field(
..., description="List of nodes in the knowledge graph")
rels: List[Relationship] = Field(
..., description="List of relationships in the knowledge graph"
)With the prompt engineering in place, we can now define the information extraction pipeline as a single function:
llm = ChatOpenAI(model="gpt-3.5-turbo-16k", temperature=0)
def get_extraction_chain(
allowed_nodes: Optional[List[str]] = None,
allowed_rels: Optional[List[str]] = None
):
prompt = ChatPromptTemplate.from_messages(
[("system", f"""# Knowledge Graph Instructions for GPT-4
## 1. Overview
You are a top-tier algorithm designed for extracting information in structured formats to build a knowledge graph.
- **Nodes** represent entities and concepts. They're akin to Wikipedia nodes.
- The aim is to achieve simplicity and clarity in the knowledge graph, making it accessible for a vast audience.
...
## 5. Strict Compliance
Adhere to the rules strictly. Non-compliance will result in termination."""),
("human", "Use the given format to extract information from the following input: {input}"),
("human", "Tip: Make sure to answer in the correct format"),
])
return create_structured_output_chain(KnowledgeGraph, llm, prompt, verbose=False)After defining the extraction chain, we can extract information from a Wikipedia page and construct a knowledge graph to test the pipeline. We can use the WikipediaLoader and text chunking modules provided by LangChain:
from langchain.document_loaders import WikipediaLoader
from langchain.text_splitter import TokenTextSplitter
raw_documents = WikipediaLoader(query="Walt Disney").load()
text_splitter = TokenTextSplitter(chunk_size=2048, chunk_overlap=24)
documents = text_splitter.split_documents(raw_documents[:3])Finally, we can run the documents through the information extraction pipeline:
for i, d in tqdm(enumerate(documents), total=len(documents)):
extract_and_store_graph(d)The extracted subgraph can be visualized using Neo4j Browser or other tools.
In conclusion, constructing knowledge graphs from text using OpenAI functions in combination with LangChain provides a powerful way to extract structured information. By defining the graph schema and incorporating entity disambiguation, we can ensure the accuracy and effectiveness of our RAG applications.
By leveraging OpenAI functions and LangChain, we can unlock the potential of knowledge graphs to power a wide range of applications. Whether it’s multi-hop question-answering, real-time analytics, or combining structured and unstructured data, knowledge graphs offer a
