LANGCHAIN — What Are Gradio LLM Agents?
I’m not a great programmer; I’m just a good programmer with great habits. — Kent Beck
Large Language Models (LLMs) are impressive, but they can be made even more powerful by equipping them with specialized skills. The gradio_tools library can turn any Gradio application into a tool that an agent can use to complete tasks. For example, an LLM could transcribe a voice recording found online and then summarize it, or apply OCR to a document on Google Drive and answer questions about it using Gradio tools.
In this article, we will focus on how to use gradio_tools to grant your LLM Agent access to cutting-edge Gradio applications. The article will cover how to get started with gradio_tools, create your own tool, and an example tool.
Getting Started with gradio_tools
To get started with gradio_tools, import and initialize your tools and pass them to the LangChain agent. The code snippet below demonstrates the initialization of tools and setting up the LangChain agent using gradio_tools.
import os
if not os.getenv("OPENAI_API_KEY"):
raise ValueError("OPENAI_API_KEY must be set")
from langchain.agents import initialize_agent
from langchain.llms import OpenAI
from gradio_tools import (StableDiffusionTool, ImageCaptioningTool, StableDiffusionPromptGeneratorTool, TextToVideoTool)
from langchain.memory import ConversationBufferMemory
llm = OpenAI(temperature=0)
memory = ConversationBufferMemory(memory_key="chat_history")
tools = [StableDiffusionTool().langchain, ImageCaptioningTool().langchain, StableDiffusionPromptGeneratorTool().langchain, TextToVideoTool().langchain]
agent = initialize_agent(tools, llm, memory=memory, agent="conversational-react-description", verbose=True)
output = agent.run(input=("Please create a photo of a dog riding a skateboard "
"but improve my prompt prior to using an image generator. "
"Please caption the generated image and create a video for it using the improved prompt."))Creating Your Own Tool
To create your own tool using gradio_tools, implement a standard interface as shown in the code snippet below. It includes the GradioTool class, which defines the necessary methods for creating a new tool for your LLM. Also, it provides a sample implementation for a tool called StableDiffusionTool.
class GradioTool(BaseTool):
def __init__(self, name: str, description: str, src: str) -> None:
@abstractmethod
def create_job(self, query: str) -> Job:
pass
@abstractmethod
def postprocess(self, output: Tuple[Any] | Any) -> str:
passclass StableDiffusionTool(GradioTool):
"""Tool for calling stable diffusion from llm"""
def __init__(
self,
name="StableDiffusion",
description=(
"An image generator. Use this to generate images based on "
"text input. Input should be a description of what the image should "
"look like. The output will be a path to an image file."
),
src="gradio-client-demos/stable-diffusion",
hf_token=None,
) -> None:
super().__init__(name, description, src, hf_token)
def create_job(self, query: str) -> Job:
return self.client.submit(query, "", 9, fn_index=1)
def postprocess(self, output: str) -> str:
return [os.path.join(output, i) for i in os.listdir(output) if not i.endswith("json")][0]
def _block_input(self, gr) -> "gr.components.Component":
return gr.Textbox()
def _block_output(self, gr) -> "gr.components.Component":
return gr.Image()Conclusion
By following the steps outlined in this guide, you can extend the abilities of your LLM with the thousands of Gradio spaces running in the wild. Additionally, once you have created your tool, open a pull request to the gradio_tools repository to contribute to the community.
We hope this tutorial has been helpful in understanding how to integrate Gradio tools with LLM agents. We look forward to seeing the tools you all build!
This tutorial provides a comprehensive guide on using gradio_tools to equip LLM agents with specialized skills. It includes examples of initializing tools, creating a custom tool, and demonstrates how to integrate Gradio tools with LangChain agents.