LANGCHAIN — Extraction Benchmarking

The function of good software is to make the complex appear to be simple. — Grady Booch

When working with large language models (LLMs), it’s essential to benchmark their performance to ensure their ability to infer correct structured information from different types of data. In this article, we’ll explore the LangChain Extraction Benchmarking project, which provides a new dataset to measure LLMs’ ability to extract and categorize relevant information from chat logs.

Creating the Dataset

The LangChain team settled on a data model to represent the structured output, seeded it with Q&A pairs, generated candidate answers using an LLM, and manually reviewed the results in the annotation queue. They used synthetic dataset generation utilities to bootstrap some initial data. Once the initial dataset was ready, they utilized labeled data as few-shot examples within the seed-generation model to improve the quality of data given to humans for review.

Extraction Schema

The dataset was designed to offer a challenge for many common models today. The schema was nested, and it combined classification, summarization, and structured output generation in a single task, making it challenging for an LLM to address in a single generation.

Evaluation

Custom LangSmith evaluators were used to measure structure verification, classification tasks, and overall difference. Metrics such as json_schema, classification accuracy, and json_edit_distance were used to evaluate the LLMs’ performance.

Experiments

The LangChain team conducted multiple experiments to compare the performance of different LLMs, both closed-source and open-source models, and to test various prompting strategies and structured decoding techniques.

Code Snippets and Examples

To see how the Claude-2 and GPT-4 models compare, you can review the individual predictions side-by-side using the provided link. Below, you can find the code snippet for comparing the two models:

# Compare GPT-4 and Claude
# Using the provided link, review the individual predictions side-by-side
# The summary graph and table below can also be checked for comparison

# GPT-4 performance
gpt_4_metrics = {
    "confidence_level_similarity": 0.94,
    "json_edit_distance": 0.28,
    "json_schema": 1.00,
    "off_topic_similarity": 0.89,
    "programming_language_similarity": 0.59,
    "question_category": 0.56,
    "sentiment_similarity": 1.00,
    "toxicity_similarity": 0.0
}

For benchmarking open-source models, the LangChain team compared the performance of three different LLMs and provided a link to see the outputs in LangSmith. Below, you can find the code snippet for comparing the open-source models:

# Compare Baseline OSS Models Test
# Using the provided link, see the outputs in LangSmith or reference the aggregate metrics below

# Llama-v2-34b-code-instruct performance
llama_metrics = {
    "confidence_level_similarity": 0.93,
    "json_edit_distance": 0.41,
    "json_schema": 0.89,
    "off_topic_similarity": 0.89,
    "programming_language_similarity": 0.44,
    "question_category": 0.07,
    "sentiment_similarity": 0.59,
    "toxicity_similarity": 1.00
}

The LangChain team also tested various prompting strategies and structured decoding techniques. Below, you can find the code snippet for comparing the prompt strategies for OSS models:

# Compare Prompt Strategies for OSS Models Test
# Using the provided link, see the outputs in LangSmith or reference the aggregate metrics below

# Llama-v2-34b-code-instruct-bcce-v1 performance
llama_prompt_metrics = {
    "Prompt": "baseline",
    "confidence_level_similarity": 0.93,
    "json_edit_distance": 0.41,
    "json_schema": 0.89,
    "off_topic_similarity": 0.89,
    "programming_language_similarity": 0.44,
    "question_category": 0.07,
    "sentiment_similarity": 0.59,
    "toxicity_similarity": 1.00
}

For the experiment on structured decoding, the LangChain team compared the baseline with grammar-based decoding. Below, you can find the code snippet for comparing the baseline vs. grammar-based decoding test:

# Compare Baseline vs. Grammar-based Decoding Test
# Using the provided link, see the outputs in LangSmith or reference the aggregate metrics below

# Llama-v2-70b-chat-28a7-v1 performance
llama_baseline_metrics = {
    "Decoding": "baseline",
    "confidence_level_similarity": 0.30,
    "json_edit_distance": 0.</p>