LANGCHAIN — What Is the Neo4j X LangChain New Vector Index?

Technological change is not additive; it is ecological. A new technology does not merely add something; it changes everything. — Neil Postman

The Neo4j X LangChain new Vector index implementation is a powerful feature that allows efficient semantic search over unstructured text or embedded data modalities. This tutorial will guide you through the customization options available in the Neo4j Vector Index implementation in LangChain, providing detailed code snippets and examples.

Neo4j Environment Setup

Before diving into the customization options, you need to set up a Neo4j 5.11 or greater instance. You can start a free instance on Neo4j Aura or set up a local instance using the Neo4j Desktop application.

Example Dataset

For this tutorial, we will use the WikipediaLoader to fetch text from the Witcher page and split the document into chunks.

from langchain.document_loaders import WikipediaLoader
from langchain.text_splitter import CharacterTextSplitter

# Read the wikipedia article
raw_documents = WikipediaLoader(query="The Witcher").load()
# Define chunking strategy
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=1000, chunk_overlap=20
)
# Chunk the document
documents = text_splitter.split_documents(raw_documents)
# Remove the summary
for d in documents:
    del d.metadata["summary"]

Neo4j Vector Index Customization

The Neo4j Vector index implementation in LangChain represents text chunks using the Chunk node label by default. You can customize the node label, text, and embedding property names according to your requirements.

neo4j_db = Neo4jVector.from_documents(
    documents,
    OpenAIEmbeddings(),
    url=url,
    username=username,
    password=password,
    database="neo4j",
    index_name="wikipedia",
    node_label="WikipediaArticle",
    text_node_property="info",
    embedding_node_property="vector",
    create_id_index=True,
)

Loading Additional Documents

You can use the add_documents method to load additional documents into an instantiated vector index.

neo4j_db.add_documents(
    [
        Document(
            page_content="LangChain is the coolest library since the Library of Alexandria",
            metadata={"author": "Tomaz", "confidence": 1.0}
        )
    ],
    ids=["langchain"],
)

Loading Existing Index

If you have an existing vector index in Neo4j with populated data, you can use the from_existing_method to connect to it.

existing_index = Neo4jVector.from_existing_index(
    OpenAIEmbeddings(),
    url=url,
    username=username,
    password=password,
    index_name="wikipedia",
    text_node_property="info",
)

Custom Retrieval Queries

The vector index implementation in LangChain allows customization and enrichment of the returned information. You can define custom retrieval queries to collect, transform, or calculate additional graph information.

existing_index.query(
    """MATCH (w:WikipediaArticle {id:'langchain'})
       MERGE (w)<-[:EDITED_BY]-(:Person {name:"Galileo"})
    """
)

retrieval_query = """
OPTIONAL MATCH (node)<-[:EDITED_BY]-(p)
WITH node, score, collect(p) AS editors
RETURN node.info AS text,
       score, 
       node {.*, vector: Null, info: Null, editors: editors} AS metadata
"""
existing_index_return = Neo4jVector.from_existing_index(
    OpenAIEmbeddings(),
    url=url,
    username=username,
    password=password,
    database="neo4j",
    index_name="wikipedia",
    text_node_property="info",
    retrieval_query=retrieval_query,
)

Summary

The new vector index implementation in Neo4j, with LangChain integration, enables support for RAG applications that rely on both structured and unstructured data, making it suitable for highly complex and connected datasets.

By following this tutorial, you can leverage the capabilities of the Neo4j X LangChain new Vector Index implementation to customize and optimize your semantic search and data retrieval operations.