LANGCHAIN — How Mendable Leverages LangSmith to Debug Tools Actions

Programming isn’t about what you know; it’s about what you can figure out. — Chris Pine

Mendable.ai, a platform helping enterprise teams answer technical questions with AI, has been leveraging LangSmith, a debugging tool provided by LangChain, to ensure the precision and efficiency of their AI-driven functionalities, particularly in their Tools & Actions module. In this tutorial, we will explore how Mendable leverages LangSmith to debug Tools & Actions.

LangSmith Integration

Mendable.ai integrates LangSmith into their development process to gain visibility into the behavior of their AI agents. LangSmith allows them to debug, evaluate, and manage LLM apps swiftly and seamlessly. By enabling tracing in LangChain, Mendable.ai captures and visualizes the detailed hierarchy of their AI agent runs, including inputs, outputs, response times, and critical metrics.

// Enable tracing in LangChain
LangChain.enableTracing();

Debugging Process

Once connected to their Tools & Actions module, Mendable.ai quickly spotted issues that were previously invisible. By inspecting the detailed visualization of their AI agent runs, they were able to identify and address problems such as slow response times and prompt optimization.

// Inspect detailed visualization of agent runs
LangSmith.inspectAgentRuns();

Inspecting Tools

Mendable.ai found it crucial to have easy access to traces for inspecting tool inputs, especially for custom tools created by users. They ran experiments and quickly scrolled through LangSmith to ensure the accuracy of tool outputs and schemas, enabling them to improve the AI-generated part of their product.

// Inspect tool inputs and outputs
LangSmith.inspectToolInputsOutputs();

Building Datasets

With the need to save inputs/outputs for further evaluation, Mendable.ai utilized LangSmith to select specific runs and add them to their dataset. This allowed them to centralize their data and evaluate it using LangSmith, streamlining their optimization experiments.

// Add specific runs to the dataset
LangSmith.addToDataset(runs);

Conclusion

LangSmith’s debugging tools have been instrumental for Mendable.ai in gaining insight into how their Tools and Action AI agents think and act, ultimately improving their debugging process. By leveraging LangSmith, Mendable.ai has enhanced the precision and efficiency of their AI-driven functionalities. If you’re interested in accelerating your LLM development process, consider trying out LangSmith, especially if you’re already using LangChain in your pipeline.

In conclusion, with LangSmith’s debugging tools, Mendable.ai has been able to gain a clear understanding of their AI agents’ behavior, addressing issues and optimizing their Tools & Actions module effectively.

Remember, when working with LangSmith, enabling tracing and inspecting agent runs and tool inputs/outputs are crucial steps in gaining visibility and optimizing your AI-driven functionalities.