LANGCHAIN — LangChains Support for Streaming

Software is a great combination between artistry and engineering. — Bill Gates

LangChain has recently introduced streaming support, a feature that is essential in improving the user experience for LLM applications. With this update, developers can now leverage streaming to reduce perceived latency, making it possible to display progress to the user as the LLM generates tokens. In this article, we’ll explore LangChain’s streaming support and provide code snippets to help you get started.

Streaming Support Introduction

The LangChain team has made significant improvements to the LangChain platform, including the introduction of streaming support. With this update, developers can now implement streaming in their LLM applications, allowing for a more seamless and interactive user experience.

Motivation

Streaming helps reduce perceived latency by returning the output of the LLM token by token, instead of all at once. This means that as a token is generated by the LLM, it can be served immediately to the user, providing a more responsive user interface.

Usage

To start using LangChain’s streaming support, the first step is to implement streaming support for the OpenAI implementation of LLM. Below is an example of how to use the on_llm_new_token callback when the streaming parameter of OpenAI is set to True:

from langchain import OpenAI

# Initialize OpenAI with streaming support
llm = OpenAI(streaming=True)

# Implement the on_llm_new_token callback
def on_llm_new_token(token):
    # Handle the new token
    print(token)

# Set the implemented callback
llm.set_callback("on_llm_new_token", on_llm_new_token)

Web Application Template

LangChain has also provided a sample web application template, chat-langchain, to demonstrate best practices for integrating streaming and other LangChain features into a ready-to-deploy application that can support multiple users. The example below showcases the integration of LangChain streaming and the implementation of callback handlers for sending messages back to the client:

Streaming

# Initialize streaming support for the OpenAI implementation of LLM
llm = OpenAI(streaming=True)

# Implement the StreamingLLMCallbackHandler to send each token back to the client via websocket
class StreamingLLMCallbackHandler:
    def on_new_token(self, token):
        # Send the token back to the client via websocket
        websocket.send(token)

Async Execution

The application also leverages asyncio support for select chains and LLMs to support concurrent execution without having to spawn multiple threads. This is essential when there are multiple client connections to the application:

# Leveraging asyncio support
async def process_request(request):
    # Processing the request asynchronously
    ...

Conclusion

LangChain’s support for streaming provides developers with the tools to create more responsive and interactive LLM applications. By implementing streaming, developers can significantly enhance the user experience and reduce perceived latency in their applications. With continuous updates and improvements, LangChain is committed to providing developers with the necessary features to build best-in-class applications.