LANGCHAIN — Public Langsmith Benchmarks

Information technology and business are becoming inextricably interwoven. I don’t think anybody can talk meaningfully about one without the talking about the other. — Bill Gates.

LangSmith has introduced the ability to share evaluation datasets and results, enabling community-driven evaluation and benchmarks. The langchain-benchmarks package has been released to reproduce these results and experiment with LLM architectures. Let's dive into the details and see how to use the LangChain Benchmarks package.

LangChain Docs Q&A Dataset

The first benchmark task is a Q&A dataset over LangChain’s documentation. Various implementations have been evaluated differing across dimensions such as the language model used and the “cognitive architecture” used. To experiment with your own architectures on the Q&A dataset, the new langchain-benchmarks package has been published. This package facilitates experimentation and benchmarking for key functionality when building with LLMs. Let's explore the LangChain Benchmarks package and how to use it.

LangChain Benchmarks

The LangChain Benchmarks package provides functionality to easily test different LLMs, prompts, indexing techniques, and other tooling. It includes benchmarks for extraction, agent tool use, and retrieval-based question answering. Let’s see how to use the LangChain Benchmarks package for experimentation.

import langchain_benchmarks as lb

# Retrieval-based question answering
qa_results = lb.retrieval_based_qa('langchain_docs_qa_dataset.json')
print(qa_results)

The retrieval_based_qa function takes the Q&A dataset as input and returns the results for retrieval-based question answering. Similarly, you can use other functions provided by the package to experiment with different functionalities.

Comparing Simple RAG Approaches

The package also allows comparing different LLM architectures based on performance metrics. The comparison views make it easy to manually review the outputs to get a better sense of how the models behave. Let’s review some results from one of the question-answering tasks to see how it works.

Reviewing the Results

The comparison views also allow manual review of the outputs to get a better sense of how the models behave. The LangSmith’s evaluation and tracing experience helps easily compare approaches in aggregate and on a sample level, and it makes it easy to drill down into each step to identify the root cause for changes in behavior.

By using the LangChain Benchmarks package, you can experiment with different LLM architectures and easily weigh the tradeoffs in different design decisions to pick the best solution for your application.

In conclusion, the LangChain Benchmarks package provides a comprehensive set of tools to experiment with and benchmark LLM architectures. It enables easy comparison of different approaches and empowers developers to make informed decisions when building with LLMs.