LANGCHAIN — Comparing Test Run Results

Before software can be reusable it first has to be usable. — Ralph Johnson

Evaluating the performance of LLM applications can be challenging. It’s difficult to quantitatively measure the impact of changes to prompts, chains, or agents. At LangChain, we recognize the value of manually inspecting data to gain intuition about the problem. To assist in this process, we are excited to introduce Test Run Comparisons.

In the initial release of LangSmith, we introduced support for running tests and scoring them with LLM-assisted feedback. However, we observed that users were hesitant to fully trust the LLM-assisted feedback and often wanted to compare test runs with previous iterations.

Test Run Comparisons is designed to address these insights. It allows users to easily visualize and compare multiple test runs side-by-side. Additionally, it provides the flexibility to use LLM-assisted evals or other evaluation methods to obtain initial scores and then manually explore the data for further insights.

To get started with Test Run Comparisons, follow the steps below.

Setting Up a Dataset and Running Tests

First, set up a dataset and run some tests. If you’ve already done this for an existing project, you’re all set. If not, refer to the documentation for instructions on how to do so.

Comparing Test Runs

Inside a dataset, you can select two or more test runs and click Compare. This action will bring you to the Test Run Comparison view, where you can easily see the inputs, reference output, actual output, evaluation metrics, time, and latency for each test run.

# Selecting test runs and comparing them
selected_test_runs = [test_run1, test_run2]
test_run_comparison_view = compare_test_runs(selected_test_runs)

Exploring Test Run Comparison View

The Test Run Comparison view allows you to quickly compare test runs across the same inputs. You can click on specific datapoints to drill down into the details of each run. Additionally, a sidebar is available to navigate between runs using up and down carets (▲ and ▼).

# Exploring the Test Run Comparison view
explore_test_run_comparison(test_run_comparison_view)

Filtering Data

The Test Run Comparison view includes filters for each column, similar to Excel. These filters enable you to narrow down the data based on specific criteria. For example, you can filter one test run to show correct datapoints and the other to show incorrect datapoints, allowing you to identify significant differences between the two test runs.

# Filtering data in the Test Run Comparison view
filter_test_run_comparison_data(test_run_comparison_view, criteria='correct')
filter_test_run_comparison_data(test_run_comparison_view, criteria='incorrect')

By enabling users to compare test runs and explore the details of each run, Test Run Comparisons in LangSmith aims to facilitate a deeper understanding of LLM applications and improve their development. We are continuously working to enhance LangSmith and will be rolling out more features like this in the future.

To access LangSmith, sign up for the private beta here. We welcome any feedback you have regarding Test Run Comparisons or other features of LangSmith.

Start leveraging Test Run Comparisons to gain valuable insights into your LLM applications and drive improvements based on data-driven comparisons.