LANGCHAIN — What is Semi-Structured Multi-Modal RAG?
The computer was born to solve problems that did not exist before. — Bill Gates
Semi-Structured Multi-Modal RAG (Retrieval Augmented Generation) is a framework designed to enable question-answering across diverse data types, including images, text, and tables. This framework allows for seamless integration of multi-modal data into the retrieval and generation process. In this tutorial, we will explore the implementation of Semi-Structured Multi-Modal RAG using code snippets and examples.
Multi-Vector Retriever
The Multi-Vector Retriever is a key component of the Semi-Structured Multi-Modal RAG framework. It decouples documents for answer synthesis from a reference used for retrieval. Let’s take a look at how to use the Multi-Vector Retriever to support semi-structured RAG as well as multi-modal RAG.
Document Loading
The first step is to partition a document into its various types using an ELT tool like Unstructured. Unstructured can extract elements (tables, images, text) from various file types. For example, it can partition PDF files by removing embedded image blocks, identifying tables using layout models, and extracting text under different sections of the document.
# Example of using Unstructured to partition a PDF document
from unstructured import UnstructuredPDF
# Load the PDF document
pdf_document = UnstructuredPDF('example.pdf')
# Extract tables
tables = pdf_document.extract_tables()
# Extract images
images = pdf_document.extract_images()
# Extract text under different sections
section_texts = pdf_document.extract_text_by_section()Semi-Structured Data
To support RAG on semi-structured data, we can generate summaries of table elements and use the Multi-Vector Retriever to retrieve these summaries based on semantic similarity to a user question. The raw table can then be passed to the LLM for answer synthesis.
# Example of using the Multi-Vector Retriever to retrieve table summaries
from multi_vector_retriever import MultiVectorRetriever
# Retrieve table summaries based on semantic similarity
retrieved_table_summary = MultiVectorRetriever.retrieve_table_summary(question)
# Pass raw table to LLM for answer synthesis
answer = LLM.generate_answer(retrieved_table_summary)Multi-Modal Data
For multi-modal RAG, we can consider images and use multi-modal LLMs to produce text summaries from images. These text summaries can be embedded and retrieved using a text embedding model. The raw images and text chunks are then passed to the multi-modal LLM for answer synthesis.
# Example of multi-modal RAG with image summaries
from multimodal_llm import MultiModalLLM
# Produce text summaries from images
text_summaries = MultiModalLLM.produce_text_summaries(images)
# Embed and retrieve text summaries
retrieved_text_summary = TextEmbeddingModel.retrieve_text_summary(question, text_summaries)
# Pass raw images and text chunks to multi-modal LLM for answer synthesis
answer = MultiModalLLM.generate_answer(retrieved_text_summary, raw_images, raw_text_chunks)Conclusion
In this tutorial, we have explored the implementation of Semi-Structured Multi-Modal RAG using code snippets and examples. We have demonstrated how the Multi-Vector Retriever can be used to support semi-structured RAG as well as semi-structured RAG with multi-modal data. Additionally, we have shown that this full pipeline can be run locally on consumer laptops using open source components. The Semi-Structured Multi-Modal RAG framework provides a powerful approach to enable question-answering across diverse data types.
For further details and advanced features, please refer to the official documentation and additional cookbooks provided by LangChain.
By following the above code snippets and examples, you can implement Semi-Structured Multi-Modal RAG to enable question-answering across diverse data types using a multi-vector retriever, multi-modal LLMs, and text embedding models. This framework provides a robust solution for handling semi-structured and multi-modal data in question-answering systems.
