LANGCHAIN — How Correct Are LLM Evaluators?

Artificial intelligence is growing up fast, as are robots whose facial expressions can elicit empathy and make your mirror neurons quiver. — Diane Ackerman.

Evaluating language model applications can be a challenging task. Traditional automated metrics like ROUGE or BLEU may not capture what makes a “good” response. LangChain has developed LLM-assisted evaluators to address this challenge. This tutorial will provide an overview of the LangChain evaluators and share guidelines on their usage based on experimental results.

LangChain’s Evaluators

LangChain provides three evaluators designed to grade the correctness of a predicted output relative to a label. These evaluators are: QAEvalChain, CoT evaluator, and Criteria evaluator. The QAEvalChain prompts a model to grade the prediction similar to a teacher grading a quiz, CoT evaluator instructs step-by-step reasoning, and Criteria evaluator tests whether a prediction meets a custom criterion provided.

Running Experiments

To test the reliability of these evaluators, benchmark datasets were created for three common tasks: Q & A, Translation, and Extraction. These datasets were then used to run experiments to measure the quality of LangChain’s evaluators in determining “correctness” of outputs relative to a label.

Results and Recommendations

The experiments have shown that GPT-4 outperforms other models in tasks requiring structured reasoning. GPT-3.5 and Claude-2 showed reliability in simpler tasks but struggled in tasks requiring additional reasoning. It was also observed that prompt-tuning might improve performance, but GPT-4 remains the most dependable general-purpose model for tasks requiring structured data reasoning.

Additionally, it was found that a single evaluator most consistently produced the expected answers. The default “qa” prompt was particularly effective when compared to other evaluators across different tasks.

Additional Insights

The experiments also revealed some important observations. At the time of testing, Claude-2 was sometimes prone to inconsistencies, and zero-shot language models, like GPT-4 and Claude-2, carry inherent biases, which can lead to unreliable results.

Conclusion

The experiments and analysis provide valuable insights into the effectiveness of LangChain’s LLM-assisted evaluators. By understanding the strengths and limitations of different language models and evaluators, practitioners can make informed decisions regarding their usage for specific tasks.

In conclusion, it is important to spot check evaluation results to ensure they correspond with intuition, especially for tasks involving names or concepts where the model may have a “high confidence” in its trained knowledge.

By considering the experimental results and recommendations, practitioners can make informed decisions when selecting and utilizing LLM evaluators for their specific tasks.