LANGCHAIN — Can You Introduce LangServe?

Social media is not about the exploitation of technology but service to community. — Simon Mainwaring

LangServe is a powerful tool that makes it easier to deploy any LangChain chain/agent/runnable. It provides a production-ready API, making it simple to ship your LangChain expression language (LCEL) prototype to your users and gather feedback. LangServe offers a range of features, including support for streaming, async calls, parallel execution, retries and fallbacks, intermediate results access, and input/output schemas. In this tutorial, we will guide you through the process of creating a scalable Python web server using LangServe and deploying it on different platforms.

To start, we need to create a chain. Here’s an example of a conversational retrieval chain:

"""A conversational retrieval chain."""

from langchain.chains import ConversationalRetrievalChain
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS

vectorstore = FAISS.from_texts(
    ["cats like fish", "dogs like sticks"],
    embedding=OpenAIEmbeddings()
)
retriever = vectorstore.as_retriever()

model = ChatOpenAI()

chain = ConversationalRetrievalChain.from_llm(model, retriever)

Next, we need to add routes to our server. Create a file named server.py and include the following code:

#!/usr/bin/env python
"""A server for the chain above."""

from fastapi import FastAPI
from langserve import add_routes

from my_package.chain import chain

app = FastAPI(title="Retrieval App")

add_routes(app, chain)

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="localhost", port=8000)

By running this, you will have a scalable Python web server with various endpoints, including those for input and output schemas, documentation, invocation, batching, streaming, logging, and more.

Once the API is created, the next step is to deploy it on a hosting platform. LangServe offers examples for deployment on Google Cloud Platform (GCP) and Replit. You can deploy to GCP Cloud Run with a single command or use the provided template to deploy on Replit.

To see it in action, consider deploying the example repository provided by LangServe. You can use the API docs to interact with the deployed API and even stream a response using a curl command.

Looking ahead, LangChain is continuously improving and adding new features to LangServe. These include a playground to experiment with different prompts/retrievers for deployed chains and a way to save multiple configurations for the same chain.

In this tutorial, we explored the capabilities of LangServe and how it simplifies the process of deploying LangChain chains/agents/runnables. You can leverage the power of LangServe to take your LLM ideas to scalable LLM applications in production.