Evaluation
LangChainRust provides 10 built-in evaluators and an LLM-as-judge for quantifying prompt and model changes.
Evaluators
| Evaluator | Type | Description |
|---|---|---|
ExactMatch | String | Returns 1.0 for exact match, 0.0 otherwise |
StringDistance | String | Levenshtein distance normalized to 0.0-1.0 |
ContainsKeyword | Rule | Checks for keyword presence (any or all) |
RegexMatch | Rule | Matches against a regex pattern |
LengthCheck | Rule | Validates output length within min/max bounds |
Bleu | NLP | BLEU score with character-level Chinese support |
EmbeddingSimilarity | Semantic | Cosine similarity of embeddings mapped to 0.0-1.0 |
LLMAsJudge | LLM | LLM scores predictions on a configurable rubric |
Faithfulness | LLM | Verifies claims against reference context |
PairwiseJudge | LLM | Compares two predictions with position-bias mitigation |
Core Types
#![allow(unused)]
fn main() {
pub struct Score { pub value: f64, pub label: Option<String> } // 0.0-1.0
pub struct Example {
pub input: String,
pub reference: String,
pub contexts: Vec<String>, // retrieved RAG contexts, rank order; [] for non-RAG tasks
}
pub struct Dataset { pub examples: Vec<Example> }
// Report 携带原文 + 汇总(均值/标准差)+ 失败清单,可反序列化(落盘二次分析)
pub struct Report {
pub per_example: Vec<ExampleReport>, // 每条含 input/reference/prediction 原文
pub summary: HashMap<String, ScoreSummary>, // 每个评测器: mean / std / count
pub failures: Vec<FailureRecord>, // 逐条容错记录(为空 = 全部成功)
}
pub struct ExampleReport {
pub index: usize,
pub input: String,
pub reference: String,
pub prediction: String,
pub scores: HashMap<String, Score>,
}
pub struct ScoreSummary { pub mean: f64, pub std: f64, pub count: usize }
}
Batch Evaluation
#![allow(unused)]
fn main() {
use langchainrust::{EvalRunner, ExactMatch, StringDistance, Dataset, Example, Evaluator};
let dataset = Dataset::new(vec![
Example::new("2+2?", "4"),
Example::new("Capital of France?", "Paris"),
]);
let runner = EvalRunner::new(vec![
Box::new(ExactMatch),
Box::new(StringDistance),
]);
let report = runner.run(&dataset, &predictor).await?;
// report.summary: {"exact_match": ScoreSummary { mean: 0.5, .. }, "string_distance": { .. }}
// 单条 predict / 打分失败不中断整批,记入 report.failures(P1-3)
}
Individual Evaluators
#![allow(unused)]
fn main() {
use langchainrust::{ContainsKeyword, RegexMatch, LengthCheck, Bleu, LLMAsJudge, Faithfulness};
// Rule-based
let eval = ContainsKeyword::new(vec!["Rust".into()]).all_required(true);
let score = eval.eval("What language?", "Rust is great", "").await?;
let eval = RegexMatch::new(r"\d+")?;
let score = eval.eval("Calculate", "The answer is 42", "").await?;
let eval = LengthCheck::new().min(10).max(100);
// BLEU
let eval = Bleu::new().with_char_level(true); // for Chinese
// LLM-as-judge
let eval = LLMAsJudge::new(judge_llm)
.with_rubric("Score accuracy on a scale of 0-10")
.with_max_score(10);
// Faithfulness (hallucination detection)
let eval = Faithfulness::new(judge_llm)
.with_llm_split(true) // LLM-based claim splitting
.with_empty_score(0.0);
}
Pairwise Comparison
#![allow(unused)]
fn main() {
use langchainrust::{PairwiseJudge, Verdict};
let judge = PairwiseJudge::new(llm).with_rubric("Which answer is more accurate?");
let verdict = judge.compare("What is X?", "Answer A", "Answer B").await?;
// Verdict::AWins, BWins, or Tie
}
PairwiseJudge 也实现了 PairwiseEvaluator trait(P1-1),可以直接进 EvalRunner
统一报告:成对评测器以 (prediction, reference) 作为 A/B 两个候选(竞技场用法把待比
答案放进 reference 槽),得分 1.0 = prediction 优、0.5 = 平局、0.0 = reference 优:
#![allow(unused)]
fn main() {
let runner = EvalRunner::new(vec![/* 单点评测器 */])
.with_pairwise(vec![Box::new(judge)]);
let report = runner.run(&dataset, &predictor).await?;
}
Trace → Golden → Regression Gate
Offline datasets go stale; the companion lc-testkit crate closes the loop from
production traces to a checked-in golden set to a CI regression gate. It is a
direct/dev dependency (lc-testkit = "0.24") rather than part of the facade.
The mapping from a recorded model exchange to one Example is fixed:
input← the last human message in the recorded request history (agent-loop re-requests still carry the real user turn there);reference← the recorded assistant text;contexts← non-empty tool-result messages in order, so RAGAS-style context precision/recall evaluators can score retrieval too.
Agent loops record several exchanges per user turn, including pure tool-call responses
with empty text — is_scoring_candidate filters those out (GoldenError names the row
that has no human message or empty reference).
// One recording file (from RecordingProvider) -> golden JSONL checked into the repo.
let exchanges = read_exchanges("traces/recorded.jsonl")?;
let scored: Vec<_> = exchanges.iter().filter(|e| is_scoring_candidate(e)).cloned().collect();
let dataset = golden_dataset(&scored)?;
write_golden_jsonl(&dataset, "eval/golden.jsonl")?;
// Fully offline smoke/baseline run: a FIFO ReplayPredictor over the same file
// reproduces every recorded reference exactly — zero network, no API key.
let (dataset, replay) = replay_golden_from_file("traces/recorded.jsonl", ReplayStrategy::Fifo)?;
let report = EvalRunner::new(vec![Box::new(ExactMatch)])
.run(&dataset, &replay)
.await?;
For the real regression run, load the checked-in JSONL (Dataset::from_jsonl) with a
predictor wrapping the new prompt/model, then compare the two reports:
#![allow(unused)]
fn main() {
use langchainrust::{compare_reports, Report};
// baseline = serialized Report from the known-good build (Report round-trips JSON).
let cmp = compare_reports(&baseline, &candidate, 0.02);
assert!(!cmp.is_regressed(), "regressed:\n{}", cmp.to_table());
}
A metric regresses when its mean drops by strictly more than the tolerance —
delta < -tolerance, so a boundary move of exactly -0.02 still passes. ReportComparison
also lists per-evaluator MetricDeltas plus added / dropped evaluator names, so a
candidate that silently stops running an evaluator is visible in the same table.
Verifier / Best-of-N
A judge answers “is this good?”; a verifier answers “how strongly does this satisfy the criteria?” with a score fine-grained enough to compare candidates.
#![allow(unused)]
fn main() {
use langchainrust::evaluation::{best_of_n, LogprobVerifier};
let verifier = LogprobVerifier::new(openai_chat) // requires a logprob-capable provider
.with_max_score(9)
.with_top_logprobs(5);
let candidates = vec![candidate_a, candidate_b, candidate_c];
let result = best_of_n(&verifier, "task description", &candidates).await?;
}
LogprobVerifier expands the first score-bearing token into its top_logprobs
distribution and takes the expectation instead of the argmax. For providers without
logprobs (Anthropic, most local servers), JudgeVerifier parses a single integer and
reports confidence = 0.25 so callers do not over-trust it.
Judge Reliability
Judge output needs position-bias and chance correction before it can gate a release:
#![allow(unused)]
fn main() {
use langchainrust::evaluation::{
cohen_kappa, order_swap_consistent, reliability_report, same_family, Verdict,
};
// A winner must survive both A/B and B/A.
let winner = order_swap_consistent(Verdict::AWins, Verdict::BWins); // AWins
// Chance-corrected agreement.
let kappa = cohen_kappa(&[true, false, true], &[true, false, false]);
// Same-family self-preference risk.
assert!(same_family("gpt-5", "gpt-4o"));
assert!(!same_family("gpt-5", "claude-3"));
let report = reliability_report(&original, &swapped);
}
cohen_kappa returns None for empty, mismatched or degenerate (single-label)
samples. reliability_report aggregates agreement, kappa and ties.