LangSmith

About LangSmith

LangSmith is a comprehensive evaluation and observability platform designed for continuously improving AI agent quality. Built by LangChain, it supports both offline evaluations on datasets for benchmarking and regression testing, as well as online evaluations on production traffic to monitor real world performance.

The platform enables teams to score agent runs using multiple evaluator types: LLM-as-judge, heuristic checks, pairwise comparisons, and human review. LangSmith's annotation queues allow subject-matter experts to review and label runs systematically, providing structured feedback for improving prompts, evaluators, and datasets.

LangSmith includes a Prompt Playground for experimenting with models and prompts, comparing outputs across versions or providers, and using the Prompt Canvas UI to auto improve prompts. The platform is designed to work across frameworks and providers, not just LangChain, and uses asynchronous trace collection to avoid adding latency to applications.

The platform offers flexible deployment options including managed cloud, bring your own cloud (BYOC), and self hosted for enterprises with strict data residency requirements.

Key Features

• Offline Dataset Evals: Run benchmarking and regression tests on curated datasets.
• Online Production Evals: Evaluate real traffic in near real time to monitor deployed agents.
• Multiple Evaluator Types: LLM-as-judge, heuristic, pairwise, and human evaluation methods.
• Annotation Queues: Collect expert feedback via structured review workflows and labeling.
• Prompt Iteration Tools: Playground and Canvas for experimenting with and improving prompts.
• Zero-Latency Tracing: Asynchronous trace collection that doesn't impact application performance.