Summary

The web content outlines five levels of chunking strategies for enhancing the efficiency of Large Language Model (LLM) applications, as discussed in Greg Kamradt's video, and provides insights into their implementation using Langchain and Llamaindex frameworks.

Abstract

The article discusses the importance of text chunking in LLM applications to improve data retrieval and application efficiency. It details five levels of chunking strategies, starting from the basic Fixed Size Chunking, which segments text by character count, to the more sophisticated Agentic Chunking, which uses LLM to determine text inclusion based on context. Intermediate levels include Recursive Chunking, which iteratively splits text using separators; Document Based Chunking, which respects the document's inherent structure; and Semantic Chunking, which groups text based on semantic similarity using embeddings. The article emphasizes the need for practitioners to choose the right chunking strategy and provides resources and tools within the Langchain and Llamaindex frameworks to implement these strategies. It also encourages following the author on LinkedIn for updates on Generative AI and Machine learning and promotes an AI service, ZAI.chat, as a cost-effective alternative to ChatGPT Plus.

Opinions

The author suggests that the choice of chunking strategy is crucial for the effectiveness of LLM applications.
Greg Kamradt's video is recommended as a valuable resource for understanding different chunking strategies.
The Langchain and Llamaindex frameworks are highlighted as useful tools for implementing chunking strategies.
The author emphasizes the importance of considering the structure and semantic meaning of text when chunking, not just the size.
The use of multi-modal models like GPT-4 vision is recommended for handling documents with images.
The article promotes the concept of Agentic Chunking as an advanced method that leverages LLM to create contextually relevant chunks.
The author advocates for the continuous evolution of Generative AI and Machine learning, indicating a commitment to updating the article with new developments.
A call to action is included for readers to follow the author on LinkedIn and try out the recommended AI service, ZAI.chat, for an affordable alternative to ChatGPT Plus.

Five Levels of Chunking Strategies in RAG| Notes from Greg’s Video

Introduction

Breaking down your large data files into more manageable segments is one of the most crucial steps for enhancing the efficiency of LLM applications. The goal is to provide the LLM with precisely the information needed for the specific task, and nothing more.

“What should be the right chunking strategy in my solution” is one of the initial and fundamental decision a LLM practitioner must make while building advance RAG solution.

In his video, Greg Kamradt provides overview of different chunking strategies. These strategies can be leveraged as starting points to develop RAG based LLM application. They have been classified into five levels based on the complexity and effectiveness.

Your goal is not to chunk for chunking sake, our goal is to get our data in a format where it can be retrieved for value later.

Level 1 : Fixed Size Chunking

This is the most crude and simplest method of segmenting the text. It breaks down the text into chunks of a specified number of characters, regardless of their content or structure.

Langchain and llamaindex framework offer CharacterTextSplitter and SentenceSplitter (default to spliting on sentences) classes for this chunking technique. A few concepts to remember -

How the text is split: by single character
How the chunk size is measured: by number of characters
chunk_size: the number of characters in the chunks
chunk_overlap: the number of characters that are being overlap in sequential chunks. keep duplicate data across chunks
separator: character(s) on which the text would be split on (default “”)

Level 2: Recursive Chunking

While Fixed size chunking is easier to implement, it doesn’t consider the structure of text. Recursive chunking offers an alternative.

In this method, we divide the text into smaller chunk in a hierarchical and iterative manner using a set of separators. If the initial attempt at splitting the text doesn’t produce chunks of the desired size, the method recursively calls itself on the resulting chunks with a different separator until the desired chunk size is achieved.

Langchain framework offers RecursiveCharacterTextSplitter class, which splits text using default separators (“\n\n”, “\n”, “ “,””)

Level 3 : Document Based Chunking

In this chunking method, we split a document based on its inherent structure. This approach considers the flow and structure of content but may not be as effective documents lacking clear structure.

Document with Markdown: Langchain provides MarkdownTextSplitter class to split document that consist markdown as way of separator.
Document with Python/JS: Langchain provides PythonCodeTextSplitter to split the python program based on class, function etc. and We can provide language into from_language method of RecursiveCharacterTextSplitter class.
Document with tables: When dealing with tables, splitting based on levels 1 and 2 might lose the tabular relationship between rows and columns. To preserve this relationship, format the table content in a way that the language model can understand (e.g., using <table> tags in HTML, CSV format separated by ';', etc.). During semantic search, matching on embeddings directly from the table can be challenging. Developers often summarize the table after extraction, generate an embedding of that summary, and use it for matching.
Document with images (Multi- Modal): Embeddings for images and text could be contents different (Though CLIP model support this). The ideal tactic is to use multi-modal model (like GPT-4 vision) to generate summaries of the images and store embeddings of it. Unstructured.io provides partition_pdf method to extract images from pdf document.

Level 4: Semantic Chunking

All above three levels deals with content and structure of documents and necessitate maintaining constant value of chunk size. This chunking method aims to extract semantic meaning from embeddings and then assess the semantic relationship between these chunks. The core idea is to keep together chunks that are semantic similar.

Llamindex has SemanticSplitterNodeParse class that allows to split the document into chunks using contextual relationship between chunks. This adaptively picks the breakpoint in-between sentences using embedding similarity.

few concepts to know

buffer_size: configurable parameter that decides the initial window for chunks
breakpoint_percentile_threshold: another configurable parameter. The threshold value to decide where to split the chunk
embed_mode: the embedding model used.

Level 5: Agentic Chunking

This chunking strategy explore the possibility to use LLM to determine how much and what text should be included in a chunk based on the context.

To generate initial chunks, it uses concept of Propositions based on paper that extracts stand alone statements from a raw piece of text. Langchain provides propositional-retrieval template to implement this.

After generating propositions, these are being feed to LLM-based agent. This agent determine whether a proposition should be included in an existing chunk or if a new chunk should be created.

Propositions are being created or included in existing chunk

Conclusion

In this article, we explored different chunking strategies and methods to implement them in Langchain and Llamaindex framework.

To find the code from Greg: https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb

As Generative AI and Machine learning is evolving faster, so I will keep updating this article. I frequently write about developments in Generative AI and Machine learning, so feel free to follow me on LinkedIn (https://www.linkedin.com/in/anurag-mishra-660961b7/)