LANGCHAIN — Evaluating Rag Pipelines with Ragas Langsmith
Technology makes it possible for people to gain control over everything, except over technology. — John Tudor.
Ragas and LangSmith offer a robust combination for evaluating LLM-powered QA systems. Ragas provides metrics to evaluate retrieval and generation and calculates a single score, while LangSmith facilitates visualization and continuous evaluations.
To evaluate a QA chain built with Langchain over the NYC Wikipedia page, the following code can be used:
from langchain.document_loaders import WebBaseLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI
# load the Wikipedia page and create index
loader = WebBaseLoader("https://en.wikipedia.org/wiki/New_York_City")
index = VectorstoreIndexCreator().from_loaders([loader])
# create the QA chain
llm = ChatOpenAI()
qa_chain = RetrievalQA.from_chain_type(
llm, retriever=index.vectorstore.as_retriever(), return_source_documents=True
)
# testing it out
question = "How did New York City get its name?"
result = qa_chain({"query": question})
result["result"]Once the QA chain is ready, Ragas metrics can be imported and evaluators can be set up. The following code helps to create evaluator chains and run the evaluations:
from ragas.metrics import faithfulness, answer_relevancy, context_relevancy, context_recall
from ragas.langchain import RagasEvaluatorChain
# make eval chains
eval_chains = {
m.name: RagasEvaluatorChain(metric=m)
for m in [faithfulness, answer_relevancy, context_relevancy, context_recall]
}
# evaluate
for name, eval_chain in eval_chains.items():
score_name = f"{name}_score"
print(f"{score_name}: {eval_chain(result)[score_name]}")To visualize the evaluations with LangSmith, the following code can be used to create a test dataset, run the evaluations, and view the results:
from langsmith import Client
from langchain.smith import RunEvalConfig, run_on_dataset
# test dataset
eval_questions = [
"What is the population of New York City as of 2020?",
"Which borough of New York City has the highest population?",
"What is the economic significance of New York City?",
"How did New York City get its name?",
"What is the significance of the Statue of Liberty in New York City?",
]
eval_answers = [
"8,804,000", # incorrect answer
"Queens", # incorrect answer
"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations... ",
# other answers
]
examples = [{"query": q, "ground_truths": [eval_answers[i]]} for i, q in enumerate(eval_questions)]
# dataset creation
client = Client()
dataset_name = "NYC test"
dataset = client.create_dataset(dataset_name=dataset_name, description="NYC test dataset")
for example in examples:
client.create_example(inputs=example, dataset_id=dataset.id)
# factory function that returns a new qa chain
def create_qa_chain(return_context=True):
qa_chain = RetrievalQA.from_chain_type(
llm,
retriever=index.vectorstore.as_retriever(),
return_source_documents=return_context,
)
return qa_chain
# configure and run the evaluation
evaluation_config = RunEvalConfig(
custom_evaluators=[eval_chains.values()],
prediction_key="result",
)
result = run_on_dataset(
client,
dataset_name,
create_qa_chain,
evaluation=evaluation_config,
input_mapper=lambda x: x,
)These code snippets illustrate how to leverage Ragas and LangSmith for evaluating QA systems, ensuring robustness and readiness for real-world applications.
