Editorial illustration for LangSmith adds reusable LLM-as-judge and rule-based code evaluator templates
LangSmith Launches AI Eval Templates for Developer Testing
LangSmith adds reusable LLM-as-judge and rule-based code evaluator templates
Every production AI application is a live experiment. Until now, keeping that experiment under control meant duplicating evaluators across projects, rewriting prompts, and hoping nothing slipped through. LangSmith changes that.
With the new Evaluators tab and a library of prebuilt templates, LLM-as-judge evaluators with tuned prompts and rule-based code evaluators, you can stop reinventing safety checks and start standardizing them. Use the templates as-is for online traffic classification: catching prompt injections, flagging odd user behavior, or routing traces for human review. Tweak them when you need to.
For offline experiments, these templates give you a clear starting point, run evaluations across your datasets, filter down to failures, and diagnose the root cause. And because they’re now reusable, you build an evaluator once, attach it to every production project from a single dashboard, and let your team apply consistent quality metrics without copy-paste. Today’s open release of openevals v0.2.0 adds multimodal support for voice and image outputs, so the same evaluator logic scales to every modality.
The result? Less boilerplate, more insight, and one source of truth for how your agent performs.
Templates include LLM-as-judge evaluators with tuned prompts and rule-based code evaluators. Use them as-is or customize for your agent. They work for both online and offline evaluation.
For online evaluation, templates help you categorize production traffic: detecting prompt injections, flagging unexpected user behavior, or surfacing traces that need human review. You can use your corrections to tune the evaluator prompt so it performs better next time For offline evaluation, templates give you a starting point for running experiments across your datasets. Run the evaluator, check scores, filter down to failures, and understand what went wrong.
These templates are also available in openevals v0.2.0, released today, with new multimodal support for evaluating voice and image outputs. You can use them directly in code or through the LangSmith UI. Reusable evaluators Once you've built evaluators that work well, you need a way to manage them centrally.
A new Evaluators tab surfaces every evaluator in your workspace, regardless of which project it's attached to. You can filter by tracing project and attach an existing evaluator to a new project in seconds. If your team owns evaluation quality across the org (defining safety checks, standardizing quality metrics), you can build evaluators once and apply them everywhere.
No more maintaining separate copies of the same safety evaluator across every tracing project. For individual engineers working in a specific tracing project, the experience stays simple: you can quickly add and configure evaluators scoped to your project from the tracing view. As an example, say you build a prompt injection evaluator from a template.
You tune the prompt, validate it against sample data, and it works well. With reusable evaluators, you attach it to every production tracing project from one place.
The real power here isn’t just the templates themselves, it’s the shift in how you think about evaluation. You move from ad hoc checks scattered across projects to a disciplined, shared practice. One team builds the safety evaluator.
Another tunes it for their edge cases. Everyone benefits. That single prompt injection check, hardened against real traffic, becomes a standard across the org.
No duplication. No drift. Evaluation stops being a bottleneck and starts being infrastructure.
You still keep the flexibility. Engineers in the trenches can grab a template, tweak it, validate it in an afternoon. But when it works, it’s not locked in a single project.
It surfaces in the Evaluators tab, ready to serve every trace, every dataset, every experiment. Online, offline, voice, image, code, the same reusable building blocks. This is where evaluation matures.
Not as an afterthought. As a lever.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Same Week 91% of businesses now use video marketing — AI cut the cost of keeping up by 91% too
Market Trends
From Archives AI made up over a third of new sites by 2025; Pope warning flagged as AI
From Archives Sergey Brin pushes DeepMind to match Claude, unveils agent skills catalog
Common Questions Answered
How do LangSmith's new evaluation templates support both online and offline agent assessment?
LangSmith's templates enable online evaluation by helping developers categorize production traffic, detect prompt injections, and flag unexpected user behaviors. For offline evaluation, the templates provide pre-tuned prompts that can be used to assess response quality, safety, and performance without writing custom evaluation code from scratch.
What types of evaluation modules are included in LangSmith's new toolkit?
LangSmith offers more than thirty templates covering various evaluation scenarios, including safety checks, response quality assessment, trajectory analysis, user behavior monitoring, and multimodal evaluation. The toolkit includes both LLM-as-judge evaluators with tuned prompts and rule-based code evaluators that can be used as-is or customized for specific agent needs.
How can developers customize LangSmith's evaluation templates for their specific use cases?
Developers can use the pre-built templates as a starting point and then customize them to better suit their specific agent requirements. For online evaluations, users can apply corrections to tune the evaluator prompts, improving performance over time and creating more precise assessment mechanisms for their language model agents.
Further Reading
- Reusable Evaluators and Evaluator Templates in LangSmith — LangChain Blog
- How to define an LLM-as-a-judge evaluator — LangChain Documentation
- LLM-as-Judge: How to Calibrate with Human Corrections — LangChain
- Set up LLM-as-a-judge online evaluators — LangChain Documentation
- LLM-As-Judge: 7 Best Practices & Evaluation Templates — Monte Carlo Data
