LANGCHAIN — Is It Effective to Test Fine-Tuned Open Source Models in Langsmith?

The most dangerous phrase in the language is, ‘We’ve always done it this way.’ — Grace Hopper.

Open source models have become increasingly capable for use in various applications, and the ability to fine-tune these models further expands their utility. LangSmith offers a solution for evaluating and comparing fine-tuned open source models, providing a convenient UI and API for creating evaluation datasets and running tests on multiple models. In this tutorial, we will walk through the process of testing fine-tuned open source models in LangSmith using code snippets and examples.

Uploading Data to LangSmith

LangSmith simplifies the process of uploading data via Python or the user interface. Below is an example of how to upload a dataset in CSV format using the LangSmith Python client:

from langsmith import Client

# Create a dataset
def create_dataset(dataset_name=None):
    client = Client()
    dataset_name = dataset_name
    client.create_dataset(dataset_name=dataset_name)
    return dataset_name

# Add data to the dataset
def add_to_dataset(dataset_name, validation_file_path):
    client = Client()
    dataset = client.read_dataset(dataset_name=dataset_name)
    
    # Open and process the validation file
    with open(validation_file_path, 'r') as f:
        for line in f:
            # Process the data and add to the dataset using client API
            client.create_chat_example(
                messages=[
                    {"type": "system", "data": {"content": "You are a helpful assistant that is knowledgeable about SQL. Only output the SQL."}},
                    {"type": "human", "data": {"content": example}}
                ],
                generations={"type": "ai", "data": {"content": assistant_content}},
                dataset_id=dataset.id
            )

The above code snippet demonstrates the process of creating and adding data to a dataset in LangSmith.

Fine-Tuning and Assessment

Once the data is uploaded, the next step involves fine-tuning and assessing the models. Below is an example of fine-tuning a model and evaluating it using LangSmith:

import replicate

# Fine-tune a model
training = replicate.trainings.create(
  version="meta/llama-2-13b-chat:f4e2de70d66816a838a89eeeb621910adffb0dd0baba3976c96980970978018d",
  input={
    "train_data": "https://storage.googleapis.com/chatopensource-replicate-demo/selected_sql_create_context_v4.jsonl",
    "num_train_epochs": 3
  },
  destination="papermoose/test"
)

# Evaluate the model using LangSmith
async def evaluate_dataset(dataset_name=None, num_repetitions=1, model="gpt-4-0613", project_name=None):
    # Define the chat model to test
    model_to_test = Replicate(
    model=model,
    model_kwargs={"temperature": 0.75, "max_length": 500, "top_p": 1},
    )
    
    # Run evaluation on the dataset using LangSmith
    chain_results = await arun_on_dataset(
        client,
        dataset_name=dataset_name,
        llm_or_chain_factory=model_to_test,
        evaluation=eval_config,
        num_repetitions=num_repetitions,
        project_name=project_name
    )

The above code snippet showcases the process of fine-tuning a model and evaluating it using LangSmith.

Observations and Findings

After evaluating the models, observations and findings can be made based on the results. The data obtained from LangSmith can be visualized and analyzed to draw meaningful conclusions. Here’s an example of visualizing the results using ChatGPT advanced data analysis:

# Visualize the results using ChatGPT advanced data analysis
# Code for graphing out the results

The above code snippet demonstrates how to visualize the results obtained from LangSmith using advanced data analysis.

Conclusion

In this tutorial, we have demonstrated how to test fine-tuned open source models in LangSmith using code snippets and examples. We covered the process of uploading data to LangSmith, fine-tuning and assessing models, and visualizing the results. LangSmith provides a convenient platform for evaluating and comparing fine-tuned open source models and can be a valuable tool for developers and researchers working with language models.

By leveraging LangSmith, developers and researchers can streamline the evaluation and comparison of fine-tuned open source models, enabling them to make informed decisions when choosing the best tool for the job.