VoltAgent buildScorer 自定义评分器实战:四步流水线、参数化与加权融合全解析

• 人工智能
• AI Agent
• Agent 框架
• 后端
• 多智能体
• RAG
• 工具调用
• Agent 记忆

在 VoltAgent 的评测体系中,自定义评分器(custom scorer)是连接"评估标准"与"Agent 实际输出"的核心机制。本文基于仓库文档 building-custom-scorers 展开,系统讲解buildScorer提供的 Prepare → Analyze → Score → Reason 四步流水线、三类评分器形态(启发式、LLM 判别、混合)、在离线实验与 Agent 实时评估中的接入方式,以及参数化、weightedBlend加权融合等高级模式;并结合 评分器构建器源码 与 底层管道实现,还原每一步的上下文结构、错误处理与采样跳过逻辑,帮助你从"会抄示例"进阶到"能读懂执行细节"。

一、何时需要自定义评分器

VoltAgent 已提供一批预置评分器(详见 prebuilt-scorers),但以下场景必须自己实现:

• 内置评分器与你的评估标准不匹配;
• 需要领域特定的评估逻辑(如客服语气合规、代码答案可编译性);
• 想把多种评估方法组合成一个复合指标;
• 需要自定义阈值或打分刻度(例如 0–1 之外的 0–10 分制再归一化)。

判断依据很简单:评分器的职责是"给定 payload,输出 0~1 的分数及解释",只要这个映射无法用现成评分器表达,就适合走buildScorer流水线。

二、四步评分器流水线:概念与源码级执行顺序

buildScorer返回一个链式构建器(Builder),依次注册四个可选步骤,score为必填。文档给出的流水线如下:

Input Payload ↓ ┌─────────────┐ │ Prepare │ → Transform & validate input └─────────────┘ ↓ ┌─────────────┐ │ Analyze │ → Extract features & insights └─────────────┘ ↓ ┌─────────────┐ │ Score │ → Calculate numeric score (0-1) └─────────────┘ ↓ ┌─────────────┐ │ Reason │ → Generate explanation └─────────────┘ ↓ Final Result

从源码结构看,这条流水线由两层实现协作完成:

1. 构建层(builder.ts):ScorerBuilderImpl以私有状态保存已注册的prepare/analyze/score/reason步骤。build()时若缺少score步骤会直接抛出Scorer '...' is missing a required 'score' step.错误,并且每次重新注册步骤都会使缓存的定义失效,保证构建结果与最终注册集合一致;
2. 执行层(create-scorer.ts 中的createScorer):构建器把用户步骤适配为底层的preprocess → analyze → generateScore → generateReason顺序调用。每步输出会被写入results记录,score结果经过normalizeGenerateScore归一化(只接受有限数字,否则置为null),reason若返回字符串对象则同时合并其metadata;整条管道用try/catch包裹,任一步抛错时返回{ status: "error", score, metadata, error }而不是让整个评估崩溃,错误对象上挂的metadata也会被合并进结果。

这套设计解释了文档中"各步骤可访问payload/params/results"的说法:构建器传给每一步的上下文是 builder.ts 中定义的BuilderPrepareContext等类型,统一携带:

每个步骤都支持同步或异步实现(返回unknown | Promise<unknown>),这为 LLM 类异步评估留出了空间。

2.1 Step 1:Prepare(可选)

在评分前转换或校验输入 payload,典型工作是清洗文本、解析类型、设置默认值:

.prepare(({ payload }) => { // Clean and validate inputs const text = String(payload.output || "").trim(); const minWords = Number(payload.minWords || 5); return { text, minWords }; })

该步返回值会被存为results.prepare,供后续步骤引用。

2.2 Step 2:Analyze(可选)

对已准备的数据做特征提取或更重的分析(包括调用外部 LLM):

.analyze(({ prepared }) => { // Extract features from prepared data const wordCount = prepared.text.split(/\s+/).length; const hasMinWords = wordCount >= prepared.minWords; return { wordCount, hasMinWords }; })

注意各步骤函数接收的是解构上下文;若想在analyze中引用 prepare 的输出,标准写法是从results.prepare读取(见下文完整示例)。

2.3 Step 3:Score(必填)

基于前面结果计算 0.0~1.0 的分数,并可返回metadata携带诊断信息:

.score(({ payload, prepared, analysis }) => { // Calculate score (0.0 to 1.0) const score = analysis.hasMinWords ? 1.0 : 0.0; return { score, metadata: { wordCount: analysis.wordCount } }; })

从 create-scorer.ts 的GenerateScoreResult类型看,score步既可以返回纯数字,也可以返回{ score, metadata? }对象——对象形式是附带元数据的推荐写法,metadata会合并进最终结果。

2.4 Step 4:Reason(可选)

为分数生成人类可读的解释,输出字符串即可:

.reason(({ payload, score, metadata }) => { // Provide explanation const passed = score >= 0.5; return passed ? `Output meets minimum word requirement (${metadata.wordCount} words)` : `Output too short (${metadata.wordCount} words, need ${payload.minWords})`; })

reason步骤的上下文额外带有score字段(见 builder.ts 的BuilderReasonContext),因此解释文案可以直接依据分数分支。

三、完整示例:情感倾向评分器

下面构建一个评估"回复是否维持了目标情感倾向"的评分器,完整覆盖四步:

import { buildScorer } from "@voltagent/core"; const sentimentScorer = buildScorer({ id: "sentiment-analyzer", label: "Sentiment Analyzer", description: "Evaluates response sentiment and positivity", }) .prepare(({ payload }) => { // Step 1: Clean and prepare the text const text = String(payload.output || "") .toLowerCase() .trim(); const targetSentiment = String(payload.targetSentiment || "positive"); return { text, targetSentiment }; }) .analyze(({ results }) => { // Step 2: Analyze sentiment indicators const prepared = results.prepare as { text: string; targetSentiment: string }; const positiveWords = ["great", "excellent", "happy", "wonderful", "fantastic"]; const negativeWords = ["bad", "terrible", "awful", "horrible", "poor"]; const positiveCount = positiveWords.filter((word) => prepared.text.includes(word)).length; const negativeCount = negativeWords.filter((word) => prepared.text.includes(word)).length; const sentiment = positiveCount > negativeCount ? "positive" : negativeCount > positiveCount ? "negative" : "neutral"; return { sentiment, positiveCount, negativeCount, matchesTarget: sentiment === prepared.targetSentiment, }; }) .score(({ results }) => { // Step 3: Calculate score based on sentiment match const analysis = results.analyze as { sentiment: string; positiveCount: number; negativeCount: number; matchesTarget: boolean; }; const score = analysis.matchesTarget ? 1.0 : 0.0; return { score, metadata: { detectedSentiment: analysis.sentiment, positiveWords: analysis.positiveCount, negativeWords: analysis.negativeCount, }, }; }) .reason(({ score, results }) => { // Step 4: Explain the scoring decision const prepared = results.prepare as { text: string; targetSentiment: string }; const metadata = results.raw as any; if (score === 1.0) { return ( `Sentiment matches target (${prepared.targetSentiment}). ` + `Found ${metadata.positiveWords} positive and ${metadata.negativeWords} negative indicators.` ); } return ( `Sentiment mismatch. Expected ${prepared.targetSentiment} but detected ${metadata.detectedSentiment}. ` + `Found ${metadata.positiveWords} positive and ${metadata.negativeWords} negative indicators.` ); }) .build();

构建选项除必填的id外,还支持label、description、metadata、sampling与params(见 builder.ts 的BuildScorerCustomOptions),其中sampling用于控制该评分器的采样执行策略。

运行结果示例

输入 1:正面回复

await sentimentScorer.run({ payload: { output: "This is a fantastic solution! Great work on the implementation.", targetSentiment: "positive" }, params: {} }); // Result: { score: 1.0, metadata: { detectedSentiment: "positive", positiveWords: 2, negativeWords: 0 }, reason: "Sentiment matches target (positive). Found 2 positive and 0 negative indicators." }

输入 2:情感不匹配

await sentimentScorer.run({ payload: { output: "This approach seems problematic and could cause terrible issues.", targetSentiment: "positive" }, params: {} }); // Result: { score: 0.0, metadata: { detectedSentiment: "negative", positiveWords: 0, negativeWords: 1 }, reason: "Sentiment mismatch. Expected positive but detected negative. Found 0 positive and 1 negative indicators." }

从 builder.ts 的BuildScorerRunResult类型看,run()实际返回的结构比上面示例更丰富:除score、reason、metadata外,还有id、status("success" | "error" | "skipped"三态)、durationMs(本次执行耗时)、sampling(采样元数据)与steps(各步骤输出快照)。steps快照让你可以在调试时直接查看 prepare/analyze 的中间产物,这正是文档中results.raw用于调试的底层来源。

四、三类评分器形态

4.1 启发式评分器(Heuristic)

纯规则、零外部依赖,执行成本最低,适合长度、格式、关键词这类确定性检查:

const lengthScorer = buildScorer({ id: "length-check", label: "Length Validator", }) .score(({ payload }) => { const length = String(payload.output || "").length; const maxLength = Number(payload.maxLength || 100); return { score: length <= maxLength ? 1.0 : 0.0, metadata: { length, maxLength }, }; }) .build();

注意这里只注册了必填的score步——四步中任意一步都可省略,prepare/analyze/reason均为可选。

4.2 LLM 判别评分器(LLM-Based)

把重量级的语言模型调用放进analyze步(该步原生支持 async),用结构化输出约束打分尺度:

import { Agent } from "@voltagent/core"; import { openai } from "@ai-sdk/openai"; import { z } from "zod"; const QUALITY_SCHEMA = z.object({ score: z.number().min(0).max(10), reason: z.string(), }); const qualityScorer = buildScorer({ id: "quality-check", label: "Response Quality", }) .analyze(async ({ payload }) => { const agent = new Agent({ name: "quality-evaluator", model: openai("gpt-4o-mini"), instructions: "You evaluate response quality on a scale of 0-10", }); const prompt = `Rate the quality of this response: ${payload.output}`; const result = await agent.generateObject(prompt, QUALITY_SCHEMA); return result.object; }) .score(({ results }) => { const analysis = results.analyze as z.infer<typeof QUALITY_SCHEMA>; return { score: analysis.score / 10, metadata: { rating: analysis.score, reason: analysis.reason }, }; }) .build();

这个示例展示了流水线的典型分工:analyze承担 I/O 密集的重活(调用模型),score只做轻量归一化(0–10 分制换算成 0–1)。这与后文性能优化建议"keepscorelightweight"完全一致。

4.3 混合评分器(Hybrid)

同一评分器内组合多种判据,用加权求和得到综合分:

const hybridScorer = buildScorer({ id: "hybrid-validator", label: "Comprehensive Validator", }) .analyze(({ payload }) => { // Heuristic checks const hasProperLength = String(payload.output || "").length >= 50; const hasNoErrors = !String(payload.output || "").includes("error"); // Could add LLM analysis here return { hasProperLength, hasNoErrors }; }) .score(({ results }) => { // Combine multiple criteria const analysis = results.analyze as { hasProperLength: boolean; hasNoErrors: boolean }; const lengthScore = analysis.hasProperLength ? 0.5 : 0; const errorScore = analysis.hasNoErrors ? 0.5 : 0; return { score: lengthScore + errorScore, metadata: analysis, }; }) .build();

与 4.2 的 LLM 版相比,混合评分器把"规则判据 + 可选模型判据"放在同一条流水线内,便于用单一score字段对外呈现。

五、把评分器接进评估系统

5.1 离线评估(Offline Evaluations)

通过 @voltagent/evals 的 createExperiment 将自定义评分器挂载到实验上,可传评分器实例本身,也可传"实例 + 默认参数 + 阈值"的配置对象:

import { createExperiment } from "@voltagent/evals"; export default createExperiment({ dataset: { name: "customer-support" }, experiment: { name: "sentiment-test" }, runner: async ({ item }) => ({ output: await generateResponse(item.input), }), scorers: [ sentimentScorer, { scorer: lengthScorer, params: { maxLength: 200 }, threshold: 1.0, }, ], });

threshold用于在实验汇总中计算通过率(pass rate)。更多实验机制见 offline-evaluations 文档。

5.2 Agent 实时评估(Live Evaluations)

把评分器注册进 Agent 的eval配置后,框架会在请求完成后调度评分器执行(实现见 agent/eval.ts,配置类型见 agent/types.ts 中的scorers: Record<string, AgentEvalScorerConfig>):

import { Agent } from "@voltagent/core"; const agent = new Agent({ name: "support-agent", model: openai("gpt-4o-mini"), eval: { scorers: { sentiment: { scorer: sentimentScorer, params: { targetSentiment: "positive" }, }, }, sampling: { rate: 0.1 }, // Sample 10% of requests }, });

sampling: { rate: 0.1 }表示只对 10% 的请求触发评分,控制在线成本;实时评估的整体机制见 live-evaluations。

六、最佳实践

6.1 类型安全

为 payload 定义接口,并通过buildScorer<Payload>泛型注入,让 TypeScript 在payload.targetSentiment这类字段访问上给出完整提示:

interface SentimentPayload { output: string; targetSentiment: "positive" | "negative" | "neutral"; } const typedScorer = buildScorer<SentimentPayload>({ id: "typed-sentiment", label: "Typed Sentiment", }) .score(({ payload }) => { // TypeScript knows payload structure const isPositive = payload.targetSentiment === "positive"; return { score: isPositive ? 1.0 : 0.0 }; }) .build();

buildScorer的两个泛型参数分别是Payload与Params,默认均为Record<string, unknown>(见 builder.ts),同时约束两者可以进一步收紧params的类型。

6.2 错误处理

让评分器对意外输入保持鲁棒。有两点源码依据值得注意:其一,底层管道对步骤抛错有兜底(返回status: "error"),但显式处理能产出更有信息量的分数;其二,构建器在run()外层还叠加了采样判定,若被采样跳过则直接返回status: "skipped"、score: null(见 builder.ts)。

.prepare(({ payload }) => { try { const text = String(payload.output || ""); if (!text) throw new Error("Empty output"); return { text }; } catch (error) { return { text: "", error: error.message }; } })

6.3 性能优化

• 用prepare一次性完成校验与清洗,避免后续步骤重复处理;
• 在analyze中缓存昂贵计算(如模型调用结果),供score/reason复用;
• 保持score轻量,只做数值运算;
• reason只在确实需要解释时注册——它虽然开销小,但每多一步就多一次上下文快照。

6.4 用测试验证评分器

评分器本身是一个纯函数式的可测单元,用 vitest 直接断言分数与元数据即可:

import { describe, it, expect } from "vitest"; describe("sentimentScorer", () => { it("detects positive sentiment", async () => { const result = await sentimentScorer.run({ payload: { output: "This is excellent!", targetSentiment: "positive", }, params: {}, }); expect(result.score).toBe(1.0); expect(result.metadata.detectedSentiment).toBe("positive"); }); it("handles empty input", async () => { const result = await sentimentScorer.run({ payload: { output: "", targetSentiment: "positive", }, params: {}, }); expect(result.score).toBeDefined(); expect(result.reason).toContain("neutral"); }); });

仓库中同样风格的构建器行为测试可参考 builder.spec.ts,底层管道测试见 create-scorer.spec.ts。

七、高级模式

7.1 静态与动态参数

params既可以是构建时的静态默认值,也可以是一个由payload动态派生的函数。从 builder.ts 的#resolveParams实现看,参数解析遵循"函数式/对象式基础参数 → 展开 → 用run()传入的params覆盖"的合并顺序:

interface KeywordParams { keyword: string; caseSensitive?: boolean; } const keywordScorer = buildScorer<Record<string, unknown>, KeywordParams>({ id: "keyword-match", params: { caseSensitive: false }, // default }) .score(({ payload, params }) => { const output = String(payload.output); const keyword = params.keyword; const caseSensitive = params.caseSensitive ?? false; const match = caseSensitive ? output.includes(keyword) : output.toLowerCase().includes(keyword.toLowerCase()); return match ? 1 : 0; }) .build();

动态参数:参数也可以完全由 payload 推导,让同一个评分器在不同数据行上使用不同判据:

const dynamicScorer = buildScorer({ id: "dynamic-params", params: (payload) => ({ expectedCategory: payload.category, threshold: payload.confidence ?? 0.8, }), }) .score(({ payload, params }) => { const match = payload.output === params.expectedCategory; return match ? 1 : 0; }) .build();

7.2 加权复合评分器(weightedBlend)

weightedBlend把多个打分函数组合成一个score步骤,是官方提供的组合工具。结合 create-scorer.ts 的实现,有几个值得注意的语义:

• 每个组件含id、weight与可选的step;若未提供step,则从context.results[id]读取既有结果,便于复用analyze阶段产出的分量;
• 只有分数为有限数字的组件参与加权,最终分数按有效权重之和归一化计算,即某个组件缺失或失败不会让总分失真;
• 全部缺失或总权重为 0 时返回score: 0;
• 每个组件的得分都会写入context.results[component.id],并在metadata.blend下记录components(含normalizedWeight)与totalWeight,天然可审计。

import { weightedBlend } from "@voltagent/core"; const compositeScorer = buildScorer({ id: "composite", }) .score( weightedBlend([ { id: "length", weight: 0.3, step: ({ payload }) => { const length = String(payload.output).length; return Math.min(length / 500, 1); }, }, { id: "quality", weight: 0.7, step: async ({ payload }) => { // Call LLM judge const result = await evaluateQuality(payload.output); return result.score; }, }, ]) ) .build();

八、小结与延伸

自定义评分器的核心心智模型是"四步各负其责,结果层层可见":prepare管输入卫生,analyze管特征与重计算,score管数值判定,reason管解释;底层由 createScorer 管道 保证顺序执行、错误隔离与元数据合并,构建器再提供类型上下文、参数解析与采样跳过。掌握这套机制后,你可以按以下路径继续深入仓库:

• 常用评估需求先看 预置评分器;
• 批量离线回归参见 离线评估;
• 线上实时监控参见 Agent 实时评估;
• 评分器聚合与汇总逻辑可进一步阅读 实验聚合器 与 实验评分器解析。

• 人工智能
• AI Agent
• Agent 框架
• 后端
• 多智能体
• RAG
• 工具调用
• Agent 记忆