Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Evaluation

LangChainRust provides 10 built-in evaluators and an LLM-as-judge for quantifying prompt and model changes.

Evaluators

EvaluatorTypeDescription
ExactMatchStringReturns 1.0 for exact match, 0.0 otherwise
StringDistanceStringLevenshtein distance normalized to 0.0-1.0
ContainsKeywordRuleChecks for keyword presence (any or all)
RegexMatchRuleMatches against a regex pattern
LengthCheckRuleValidates output length within min/max bounds
BleuNLPBLEU score with character-level Chinese support
EmbeddingSimilaritySemanticCosine similarity of embeddings mapped to 0.0-1.0
LLMAsJudgeLLMLLM scores predictions on a configurable rubric
FaithfulnessLLMVerifies claims against reference context
PairwiseJudgeLLMCompares two predictions with position-bias mitigation

Core Types

#![allow(unused)]
fn main() {
pub struct Score { pub value: f64, pub label: Option<String> }  // 0.0-1.0
pub struct Example {
    pub input: String,
    pub reference: String,
    pub contexts: Vec<String>,  // retrieved RAG contexts, rank order; [] for non-RAG tasks
}
pub struct Dataset { pub examples: Vec<Example> }
// Report 携带原文 + 汇总(均值/标准差)+ 失败清单,可反序列化(落盘二次分析)
pub struct Report {
    pub per_example: Vec<ExampleReport>,          // 每条含 input/reference/prediction 原文
    pub summary: HashMap<String, ScoreSummary>,   // 每个评测器: mean / std / count
    pub failures: Vec<FailureRecord>,             // 逐条容错记录(为空 = 全部成功)
}
pub struct ExampleReport {
    pub index: usize,
    pub input: String,
    pub reference: String,
    pub prediction: String,
    pub scores: HashMap<String, Score>,
}
pub struct ScoreSummary { pub mean: f64, pub std: f64, pub count: usize }
}

Batch Evaluation

#![allow(unused)]
fn main() {
use langchainrust::{EvalRunner, ExactMatch, StringDistance, Dataset, Example, Evaluator};

let dataset = Dataset::new(vec![
    Example::new("2+2?", "4"),
    Example::new("Capital of France?", "Paris"),
]);

let runner = EvalRunner::new(vec![
    Box::new(ExactMatch),
    Box::new(StringDistance),
]);

let report = runner.run(&dataset, &predictor).await?;
// report.summary: {"exact_match": ScoreSummary { mean: 0.5, .. }, "string_distance": { .. }}
// 单条 predict / 打分失败不中断整批,记入 report.failures(P1-3)
}

Individual Evaluators

#![allow(unused)]
fn main() {
use langchainrust::{ContainsKeyword, RegexMatch, LengthCheck, Bleu, LLMAsJudge, Faithfulness};

// Rule-based
let eval = ContainsKeyword::new(vec!["Rust".into()]).all_required(true);
let score = eval.eval("What language?", "Rust is great", "").await?;

let eval = RegexMatch::new(r"\d+")?;
let score = eval.eval("Calculate", "The answer is 42", "").await?;

let eval = LengthCheck::new().min(10).max(100);

// BLEU
let eval = Bleu::new().with_char_level(true); // for Chinese

// LLM-as-judge
let eval = LLMAsJudge::new(judge_llm)
    .with_rubric("Score accuracy on a scale of 0-10")
    .with_max_score(10);

// Faithfulness (hallucination detection)
let eval = Faithfulness::new(judge_llm)
    .with_llm_split(true)  // LLM-based claim splitting
    .with_empty_score(0.0);
}

Pairwise Comparison

#![allow(unused)]
fn main() {
use langchainrust::{PairwiseJudge, Verdict};

let judge = PairwiseJudge::new(llm).with_rubric("Which answer is more accurate?");
let verdict = judge.compare("What is X?", "Answer A", "Answer B").await?;
// Verdict::AWins, BWins, or Tie
}

PairwiseJudge 也实现了 PairwiseEvaluator trait(P1-1),可以直接进 EvalRunner 统一报告:成对评测器以 (prediction, reference) 作为 A/B 两个候选(竞技场用法把待比 答案放进 reference 槽),得分 1.0 = prediction 优、0.5 = 平局、0.0 = reference 优:

#![allow(unused)]
fn main() {
let runner = EvalRunner::new(vec![/* 单点评测器 */])
    .with_pairwise(vec![Box::new(judge)]);
let report = runner.run(&dataset, &predictor).await?;
}

Trace → Golden → Regression Gate

Offline datasets go stale; the companion lc-testkit crate closes the loop from production traces to a checked-in golden set to a CI regression gate. It is a direct/dev dependency (lc-testkit = "0.24") rather than part of the facade.

The mapping from a recorded model exchange to one Example is fixed:

  • input ← the last human message in the recorded request history (agent-loop re-requests still carry the real user turn there);
  • reference ← the recorded assistant text;
  • contexts ← non-empty tool-result messages in order, so RAGAS-style context precision/recall evaluators can score retrieval too.

Agent loops record several exchanges per user turn, including pure tool-call responses with empty text — is_scoring_candidate filters those out (GoldenError names the row that has no human message or empty reference).

// One recording file (from RecordingProvider) -> golden JSONL checked into the repo.
let exchanges = read_exchanges("traces/recorded.jsonl")?;
let scored: Vec<_> = exchanges.iter().filter(|e| is_scoring_candidate(e)).cloned().collect();
let dataset = golden_dataset(&scored)?;
write_golden_jsonl(&dataset, "eval/golden.jsonl")?;

// Fully offline smoke/baseline run: a FIFO ReplayPredictor over the same file
// reproduces every recorded reference exactly — zero network, no API key.
let (dataset, replay) = replay_golden_from_file("traces/recorded.jsonl", ReplayStrategy::Fifo)?;
let report = EvalRunner::new(vec![Box::new(ExactMatch)])
    .run(&dataset, &replay)
    .await?;

For the real regression run, load the checked-in JSONL (Dataset::from_jsonl) with a predictor wrapping the new prompt/model, then compare the two reports:

#![allow(unused)]
fn main() {
use langchainrust::{compare_reports, Report};

// baseline = serialized Report from the known-good build (Report round-trips JSON).
let cmp = compare_reports(&baseline, &candidate, 0.02);
assert!(!cmp.is_regressed(), "regressed:\n{}", cmp.to_table());
}

A metric regresses when its mean drops by strictly more than the tolerance — delta < -tolerance, so a boundary move of exactly -0.02 still passes. ReportComparison also lists per-evaluator MetricDeltas plus added / dropped evaluator names, so a candidate that silently stops running an evaluator is visible in the same table.

Verifier / Best-of-N

A judge answers “is this good?”; a verifier answers “how strongly does this satisfy the criteria?” with a score fine-grained enough to compare candidates.

#![allow(unused)]
fn main() {
use langchainrust::evaluation::{best_of_n, LogprobVerifier};

let verifier = LogprobVerifier::new(openai_chat)  // requires a logprob-capable provider
    .with_max_score(9)
    .with_top_logprobs(5);

let candidates = vec![candidate_a, candidate_b, candidate_c];
let result = best_of_n(&verifier, "task description", &candidates).await?;
}

LogprobVerifier expands the first score-bearing token into its top_logprobs distribution and takes the expectation instead of the argmax. For providers without logprobs (Anthropic, most local servers), JudgeVerifier parses a single integer and reports confidence = 0.25 so callers do not over-trust it.

Judge Reliability

Judge output needs position-bias and chance correction before it can gate a release:

#![allow(unused)]
fn main() {
use langchainrust::evaluation::{
    cohen_kappa, order_swap_consistent, reliability_report, same_family, Verdict,
};

// A winner must survive both A/B and B/A.
let winner = order_swap_consistent(Verdict::AWins, Verdict::BWins); // AWins

// Chance-corrected agreement.
let kappa = cohen_kappa(&[true, false, true], &[true, false, false]);

// Same-family self-preference risk.
assert!(same_family("gpt-5", "gpt-4o"));
assert!(!same_family("gpt-5", "claude-3"));

let report = reliability_report(&original, &swapped);
}

cohen_kappa returns None for empty, mismatched or degenerate (single-label) samples. reliability_report aggregates agreement, kappa and ties.