使用 UpTrain 评估 LlamaIndex RAG 管线：Callback Handler 与 EvalLlamaIndex 双路径实战指南

使用 UpTrain 评估 LlamaIndex RAG 管线：Callback Handler 与 EvalLlamaIndex 双路径实战指南

【免费下载链接】llama_indexLlamaIndex is the leading document agent and OCR platform项目地址: https://gitcode.com/GitHub_Trending/ll/llama_index

导读：本文基于 LlamaIndex 仓库中的 UpTrain 集成文档，系统讲解如何将开源评估平台 UpTrain 接入 LlamaIndex RAG 管线。你将掌握两种集成方式——零侵入的UpTrainCallbackHandler（自动评估查询引擎、子问题分解与重排序全过程）与更灵活的EvalLlamaIndex（手动控制评估流程），并理解回调处理器在源码层面的事件捕获与评分触发机制，从而为自己的 RAG 系统建立可量化的质量度量体系。

背景：RAG 管线越复杂，越需要系统化评估

随着越来越多公司将 LLM 原型推向生产环境，其 RAG 管线也日趋复杂。开发者普遍引入 QueryRewrite、Context ReRank 等模块来提升系统准确率，而复杂度上升的同时也意味着更多潜在失效点：

1. 需要更高级的评估手段：判断这些新增模块是否真正提升了系统准确率，需要能覆盖"中间步骤"的评估检查（而不只是对最终回答打分）；
2. 需要稳健的实验框架：系统化地对比不同模块组合，用数据而非直觉做决策。

UpTrain 正是针对这两个诉求设计的开源平台：它提供20+ 个预配置评估检查（覆盖语言、代码、嵌入等用例），支持对失败案例做根因分析并给出改进建议。对应到 RAG 场景，其核心检查包括 ContextRelevance（上下文相关性）、SubQueryCompleteness（子查询完整性）、ContextReranking（上下文重排质量）、ContextConciseness（上下文精简度）、FactualAccuracy（事实准确性）、ContextUtilization（上下文利用）、ResponseCompleteness（回答完整性）、ResponseConciseness（回答简洁性）等。

UpTrain 与 LlamaIndex 集成的两种方式

下文将分别给出完整实战流程。

方式一：使用 UpTrainCallbackHandler

该方式对应仓库中的集成包 llama-index-callbacks-uptrain，核心实现位于 base.py，完整的可运行 Notebook 见 docs/examples/observability/UpTrainCallback.ipynb。

安装依赖与导入库

%pip install llama-index-readers-web %pip install llama-index-callbacks-uptrain %pip install -q html2text llama-index pandas tqdm uptrain torch sentence-transformers

from getpass import getpass from llama_index.core import Settings, VectorStoreIndex from llama_index.core.node_parser import SentenceSplitter from llama_index.readers.web import SimpleWebPageReader from llama_index.core.callbacks import CallbackManager from llama_index.callbacks.uptrain.base import UpTrainCallbackHandler from llama_index.core.query_engine import SubQuestionQueryEngine from llama_index.core.tools import QueryEngineTool, ToolMetadata from llama_index.core.postprocessor import SentenceTransformerRerank from llama_index.llms.openai import OpenAI import os

需要说明的是，llama-index-callbacks-uptrain包的依赖约束可在其 pyproject.toml 中确认：要求uptrain>=0.7.1、llama-index-core>=0.13.0,<0.15，且requires-python = ">=3.10,<4.0"。

两种评估后端配置（Setup）

UpTrain 提供两类后端供选择：

1. UpTrain 开源软件（OSS）：使用开源评估服务，需要提供 OpenAI API Key。若要在本地仪表盘查看评估结果，需先启动 UpTrain：

git clone https://github.com/uptrain-ai/uptrain cd uptrain bash run_uptrain.sh

启动后通过http://localhost:3000/dashboard访问本地仪表盘。

对应参数：

• key_type="openai"
• api_key="OPENAI_API_KEY"
• project_name="PROJECT_NAME"

2. UpTrain 托管服务与仪表盘：使用云端托管服务，无需在本地搭建仪表盘，且可直接使用多种 LLM 而无需各自申请 API Key。评估结果在https://dashboard.uptrain.ai/dashboard查看。

对应参数：

• key_type="uptrain"
• api_key="UPTRAIN_API_KEY"
• project_name="PROJECT_NAME"

注意：project_name是评估结果在 UpTrain 仪表盘中归属的项目名称。

创建 UpTrainCallbackHandler 并挂载

os.environ["OPENAI_API_KEY"] = getpass() callback_handler = UpTrainCallbackHandler( key_type="openai", api_key=os.environ["OPENAI_API_KEY"], project_name="uptrain_llamaindex", ) Settings.callback_manager = CallbackManager([callback_handler])

将 handler 挂到全局Settings.callback_manager后，LlamaIndex 运行时产生的事件便会自动流向该回调处理器。

加载并解析文档

以 Paul Graham 的《What I Worked On》一文为例：

documents = SimpleWebPageReader().load_data( [ "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt" ] )

将文档解析为节点：

parser = SentenceSplitter() nodes = parser.get_nodes_from_documents(documents)

场景 1：RAG Query Engine 评估

挂载回调后，处理器会自动捕获 query、context 与 response，并对每条回答运行三项评估（分数范围 0～1）：

• Context Relevance：判定检索到的上下文是否包含足够信息来回答用户问题；
• Factual Accuracy：判定 LLM 的回答能否被检索上下文所验证；
• Response Completeness：判定回答是否完整覆盖了用户问题所需的全部信息。

index = VectorStoreIndex.from_documents( documents, ) query_engine = index.as_query_engine() max_characters_per_line = 80 queries = [ "What did Paul Graham do growing up?", "When and how did Paul Graham's mother die?", "What, in Paul Graham's opinion, is the most distinctive thing about YC?", "When and how did Paul Graham meet Jessica Livingston?", "What is Bel, and when and where was it written?", ] for query in queries: response = query_engine.query(query)

运行后控制台会打印每条查询对应的评分，例如：

Question: What did Paul Graham do growing up? Response: Paul Graham wrote short stories and started programming on the IBM 1401 in 9th grade using an early version of Fortran. Later, he convinced his father to buy a TRS-80, where he wrote simple games, a program to predict rocket heights, and a word processor. Context Relevance Score: 0.0 Factual Accuracy Score: 1.0 Response Completeness Score: 1.0

Question: When and how did Paul Graham meet Jessica Livingston? Response: Paul Graham met Jessica Livingston at a big party at his house in October 2003. Context Relevance Score: 1.0 Factual Accuracy Score: 0.5 Response Completeness Score: 1.0

从上面两条结果可以看到评估的区分度：当检索上下文与问题不匹配时 Context Relevance 会给出低分；当回答部分无法被上下文佐证时 Factual Accuracy 会降低。随后即可在 UpTrain 仪表盘中按分数筛选失败案例、下钻定位问题，并获取失败案例的共性洞察。

说明：文档示例使用的是基础 RAG query engine，同样的评估也适用于高级 RAG query engine。

场景 2：Sub-Question Query Engine 评估

子问题查询引擎用于处理跨多个数据源的复杂问题：先将复杂问题拆解为针对各数据源的子问题，汇总中间回答后再综合生成最终答案。除上述三项评估外，回调处理器还会对子问题分解本身追加：

• Sub Query Completeness：确保子问题准确、完整地覆盖了原始查询。

# build index and query engine vector_query_engine = VectorStoreIndex.from_documents( documents=documents, use_async=True, ).as_query_engine() query_engine_tools = [ QueryEngineTool( query_engine=vector_query_engine, metadata=ToolMetadata( name="documents", description="Paul Graham essay on What I Worked On", ), ), ] query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=query_engine_tools, use_async=True, ) response = query_engine.query( "How was Paul Grahams life different before, during, and after YC?" )

运行输出示例：

Generated 3 sub questions. [documents] Q: What did Paul Graham work on before Y Combinator? [documents] Q: What did Paul Graham work on during Y Combinator? [documents] Q: What did Paul Graham work on after Y Combinator? [documents] A: Paul Graham worked on a project with Robert and Trevor after Y Combinator. [documents] A: Paul Graham worked on projects with his colleagues Robert and Trevor before Y Combinator. [documents] A: Paul Graham worked on writing essays and working on Y Combinator during his time at Y Combinator. Question: What did Paul Graham work on after Y Combinator? Response: Paul Graham worked on a project with Robert and Trevor after Y Combinator. Context Relevance Score: 0.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.5

对每个子问题都会打印独立的 Context Relevance / Factual Accuracy / Response Completeness 评分，同时对父问题输出 Sub Query Completeness 评分：

Question: How was Paul Grahams life different before, during, and after YC? Sub Query Completeness Score: 1.0

仪表盘会以柱状图等方式可视化各子问题的得分分布。

场景 3：Re-ranking 评估

重排序根据节点与查询的相关性对节点重新排序并选取 top-n。这里以SentenceTransformerRerank为例（同样的评估也适用于 LlamaIndex 提供的其他 re-ranker）。依据重排后返回节点数的变化，会触发不同的评估：

3a. 重排后节点数不变 → Context Reranking

当top_n与原始节点数相同时，重排器仅改变顺序而不增减节点，此时评估Context Reranking——检查重排后的节点顺序是否比原顺序与查询更相关。

callback_handler = UpTrainCallbackHandler( key_type="openai", api_key=os.environ["OPENAI_API_KEY"], project_name_prefix="llama", ) Settings.callback_manager = CallbackManager([callback_handler]) rerank_postprocessor = SentenceTransformerRerank( top_n=3, # number of nodes after reranking keep_retrieval_score=True, ) index = VectorStoreIndex.from_documents( documents=documents, ) query_engine = index.as_query_engine( similarity_top_k=3, # number of nodes before reranking node_postprocessors=[rerank_postprocessor], ) response = query_engine.query( "What did Sam Altman do in this essay?", )

输出示例：

Question: What did Sam Altman do in this essay? Context Reranking Score: 0.0 Question: What did Sam Altman do in this essay? Response: Sam Altman was asked to become the president of Y Combinator after the original founders decided to step back and reorganize the company for long-term sustainability. Context Relevance Score: 1.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.5

3b. 重排后节点数减少 → Context Conciseness

当top_n小于原始节点数时，重排器会裁剪节点，此时评估Context Conciseness——检查精简后的节点集是否仍包含回答问题所需的全部信息。

callback_handler = UpTrainCallbackHandler( key_type="openai", api_key=os.environ["OPENAI_API_KEY"], project_name_prefix="llama", ) Settings.callback_manager = CallbackManager([callback_handler]) rerank_postprocessor = SentenceTransformerRerank( top_n=2, # Number of nodes after re-ranking keep_retrieval_score=True, ) index = VectorStoreIndex.from_documents( documents=documents, ) query_engine = index.as_query_engine( similarity_top_k=5, # Number of nodes before re-ranking node_postprocessors=[rerank_postprocessor], ) # Use your advanced RAG response = query_engine.query( "What did Sam Altman do in this essay?", )

输出示例：

Question: What did Sam Altman do in this essay? Context Conciseness Score: 0.0 Question: What did Sam Altman do in this essay? Response: Sam Altman offered unsolicited advice to the author during a visit to California for interviews. Context Relevance Score: 1.0 Factual Accuracy Score: 1.0 Response Completeness Score: 0.5

参数提示：similarity_top_k决定重排前的节点数量，top_n决定重排后保留的节点数量；两者相等走 Context Reranking 评估，前者大于后者则走 Context Conciseness 评估。

切换到 UpTrain 托管服务

通过回调处理器使用托管服务时，只需更换key_type与api_key两个参数，其余代码完全不变：

callback_handler = UpTrainCallbackHandler( key_type="uptrain", api_key="up-******************************", project_name_prefix="llama", )

源码视角：Callback 如何捕获事件并触发评估

理解事件机制有助于你在真实项目中排查问题。base.py 中UpTrainCallbackHandler继承自BaseCallbackHandler（该继承关系亦被 tests/test_uptrain_callback.py 中的test_handler_callable用例验证），其工作流可以概括为"捕获事件 → 组装数据 → 批量送评"：

• 数据暂存（UpTrainDataSchema）：维护project_name、question、context、response，以及重排场景的old_context/new_context/reranking_type和子问题场景的sub_question_map/sub_question_parent_id等状态（base.py#L13-L41）；
• 事件起点捕获（on_event_start）：QUERY事件记录问题文本；TEMPLATING事件从template_vars["context_str"]捕获检索上下文；RERANKING事件记录重排前的旧节点；SUB_QUESTION事件记录父问题与父事件 ID（base.py#L133-L161）；
• 事件终点触发评估（on_event_end）：
  • 父事件结束时，对全部子问题一次性执行sub_question_answering（Context Relevance / Factual Accuracy / Response Completeness），并对父问题执行sub_query_completeness（Sub Query Completeness）（base.py#L181-L213）；
  • SYNTHESIZE事件结束且当前不在子问题上下文时，执行question_answering三项评估；若该查询经历重排，则根据节点数变化命名为question_answering_rerank或question_answering_resize（base.py#L215-L243）；
  • RERANKING事件结束时，对比新旧节点数量：相等则送context_reranking（Context Reranking），不等则送context_conciseness（Context Conciseness）（base.py#L245-L293）。

评分结果通过uptrain_evaluate打印到控制台，其中的分数列名映射（score_context_relevance→Context Relevance Score等六项）见 base.py#L109-L116。此外，key_type="uptrain"时内部使用APIClient.log_and_evaluate，key_type="openai"时使用EvalLLM.evaluate，两种后端共用同一套评估数据组装逻辑（base.py#L86-L107）。

版本注意：文档示例中出现的project_name_prefix参数，在当前仓库源码的构造函数签名中已不存在——UpTrainCallbackHandler.__init__仅接受api_key、key_type与project_name三个参数（base.py#L52-L57），并以project_name="uptrain_llamaindex"为默认值。实际使用时请以仓库源码为准，统一使用project_name命名评估项目。

方式二：使用 UpTrain 的 EvalLlamaIndex

该方式通过EvalLlamaIndex对象先为查询生成响应，再对响应执行评估，控制粒度更细。完整 Notebook 见 docs/examples/evaluation/UpTrain.ipynb。

安装与导入

pip install uptrain llama_index

import httpx import os import openai import pandas as pd from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings from uptrain import Evals, EvalLlamaIndex, Settings as UpTrainSettings

准备数据集与查询列表

以维基百科中关于纽约市（NYC）的数据为例。可替换为任意自有文档：

url = "https://uptrain-assets.s3.ap-south-1.amazonaws.com/data/nyc_text.txt" if not os.path.exists("nyc_wikipedia"): os.makedirs("nyc_wikipedia") dataset_path = os.path.join("./nyc_wikipedia", "nyc_text.txt") if not os.path.exists(dataset_path): r = httpx.get(url) with open(dataset_path, "wb") as f: f.write(r.content)

构建查询列表（与 NYC 主题相关）：

data = [ {"question": "What is the population of New York City?"}, {"question": "What is the area of New York City?"}, {"question": "What is the largest borough in New York City?"}, {"question": "What is the average temperature in New York City?"}, {"question": "What is the main airport in New York City?"}, {"question": "What is the famous landmark in New York City?"}, {"question": "What is the official language of New York City?"}, {"question": "What is the currency used in New York City?"}, {"question": "What is the time zone of New York City?"}, {"question": "What is the famous sports team in New York City?"}, ]

本 Notebook 同时使用 OpenAI API 生成提示词文本与构建向量索引，因此需要设置openai.api_key：

openai.api_key = "sk-************************" # your OpenAI API key

用 LlamaIndex 构建查询引擎

Settings.chunk_size = 512 documents = SimpleDirectoryReader("./nyc_wikipedia/").load_data() vector_index = VectorStoreIndex.from_documents( documents, ) query_engine = vector_index.as_query_engine()

评估后端配置

与回调方式一致，EvalLlamaIndex也支持两种后端：

方式 A：使用 OSS 开源评估服务（需 OpenAI API Key；本地仪表盘搭建方式同前，通过git clone+bash run_uptrain.sh启动）：

settings = UpTrainSettings( openai_api_key=openai.api_key, )

方式 B：使用 UpTrain 托管服务（使用uptrain_access_token参数替代openai_api_key）：

UPTRAIN_API_KEY = "up-**********************" # your UpTrain API key settings = UpTrainSettings( uptrain_access_token=UPTRAIN_API_KEY, )

创建 EvalLlamaIndex 对象并运行评估

llamaindex_object = EvalLlamaIndex( settings=settings, query_engine=query_engine )

运行评估，选择与教程最相关的两项检查：

• Context Relevance：检查检索到的上下文是否与查询相关——检索上下文是生成回答的依据，若上下文不相关，回答必然不相关；
• Response Conciseness：检查回答是否简洁、不包含无关冗余信息。

results = llamaindex_object.evaluate( project_name="uptrain-llama-index", evaluation_name="nyc_wikipedia", # adding project and evaluation names allow you to track the results in the UpTrain dashboard data=data, checks=[Evals.CONTEXT_RELEVANCE, Evals.RESPONSE_CONCISENESS], )

结果可直接转为 DataFrame 查看：

pd.DataFrame(results)

project_name与evaluation_name用于在 UpTrain 仪表盘中追踪、归集本次评估结果。仪表盘会展示"分数-案例数"直方图，并支持过滤失败案例、聚合生成共性主题，从而定位核心问题并加以修复。

参数与关键点速查

实践要点归纳：

1. 回调方式是首选：只需在现有管线中挂载一个 handler，即可自动覆盖 RAG 查询引擎、子问题分解、重排序等环节的评估，并免费获得仪表盘、失败洞察、生产数据可观测性与 CI/CD 回归测试能力；
2. EvalLlamaIndex 适合定制化场景：当你需要精确控制"评估哪些查询、跑哪些检查、如何处理结果"时，用它替代回调方式；
3. 两种方式共享同一套 UpTrain 检查体系（Context Relevance、Factual Accuracy、Response Completeness、Sub Query Completeness、Context Reranking、Context Conciseness、Response Conciseness 等），选型差异仅在接入形态，不影响评估语义。

进一步阅读

• 集成包说明与能力概览：llama-index-callbacks-uptrain README
• 回调处理器源码（事件捕获与评分触发）：base.py
• 回调方式完整 Notebook：docs/examples/observability/UpTrainCallback.ipynb
• EvalLlamaIndex 完整 Notebook：docs/examples/evaluation/UpTrain.ipynb
• 集成包继承关系测试：tests/test_uptrain_callback.py

【免费下载链接】llama_indexLlamaIndex is the leading document agent and OCR platform项目地址: https://gitcode.com/GitHub_Trending/ll/llama_index