LANGCHAIN — What Is the User Langsmith-ga?

A good programmer is someone who always looks both ways before crossing a one-way street. — Doug Linder.

Langsmith-ga is a user of LangChain, a platform for LLM (Large Language Models) application development, monitoring, and testing. LangSmith has been announced as generally available (GA) and has undergone significant improvements based on user feedback.

The platform, LangSmith, supports various stages of the LLM application lifecycle, from prototyping to production. It provides features for debugging, initial test set creation, comparison view, and playground for rapid iteration during the prototyping phase.

During beta testing, LangSmith facilitates collecting feedback, annotating traces, and adding runs to a dataset to refine and improve performance.

In the production phase, LangSmith supports monitoring, A/B testing, and provides a high-level overview of application performance with respect to latency, cost, and feedback scores.

Looking ahead, LangSmith aims to enhance the platform with features such as support for regression testing, online evaluators for production data, better filtering and conversation support, easy deployment of applications with hosted LangServe, and enterprise-level features for administration and security needs.

Developers can sign up for LangSmith to leverage its capabilities for their LLM application development workflows.

The following are code snippets and examples of the features and functionalities provided by LangSmith for various stages of the LLM application development lifecycle.

Prototyping

Debugging

# Enable LangSmith tracing
langsmith.enable_tracing()

# Inspect a trace from an LLM application run
trace_id = "12345"
langsmith.inspect_trace(trace_id)

Initial Test Set

# Create a dataset for test cases
test_set = langsmith.create_dataset("Test Set 1")

# Upload test cases
test_cases = {
    "input1": "output1",
    "input2": "output2"
}
langsmith.upload_test_cases(test_set, test_cases)

# Run custom evaluations
langsmith.run_custom_evaluation(test_set)

Comparison View

# View test runs side by side
test_run1 = langsmith.run_test_case(prompt1, model1)
test_run2 = langsmith.run_test_case(prompt1, model2)
langsmith.compare_test_runs(test_run1, test_run2)

Playground

# Open the playground for rapid iteration and experimentation
prompt = "Generate text based on this prompt"
model = "GPT-3"
langsmith.open_playground(prompt, model)

Beta Testing

Collecting Feedback

# Attach feedback scores to logged traces
trace_id = "12345"
feedback_score = 3.5
langsmith.attach_feedback(trace_id, feedback_score)

# View results in LangSmith
langsmith.filter_traces_by_feedback_score(3.5)
langsmith.inspect_trace(trace_id)

Annotating Traces

# Send runs to annotation queues
interesting_traces = [trace1, trace2, trace3]
langsmith.send_to_annotation_queue(interesting_traces)

Adding Runs to a Dataset

# Add runs as examples to datasets
langsmith.add_runs_to_dataset(test_set, [run1, run2, run3])

Production

Monitoring and A/B Testing

# View monitoring charts and group by LLM type
langsmith.view_monitoring_charts()
langsmith.group_by_identifier("Version1")
langsmith.group_by_identifier("Version2")

This tutorial provides a brief overview of how LangSmith can be used for different stages of LLM application development, along with corresponding code snippets for integrating LangSmith functionalities into the development workflow.