guidellm LLM大模型性能评测工具

Ref

https://github.com/vllm-project/guidellm

这是一个比较专业的LLM性能评测工具，里面工程实现也比较优雅，技术深度非常深。

其他评测工具参考：

sglang/VLLM性能评测: bench_serving工具

安装

pip install guidellm --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple

git clone源码安装

pip uninstall -y guidellm git clone https://github.com/vllm-project/guidellm.git cd guidellm/ pip install -e ./

docker images

https://github.com/vllm-project/guidellm/pkgs/container/guidellm

使用样例

guidellm run \ --config chat \ --backend kind=openai_http,target=http://localhost:30000,model=Qwen3-30B-A3B \ --profile kind=concurrent,streams=20 \ --constraint kind=max_requests,count=320 \ --tokenizer kind=hf_auto,model=/data0/models/Qwen3-30B-A3B-Thinking-2507-FP8 \ --data '{"kind":"synthetic_text","prefix_buckets":[{"bucket_weight":100,"prefix_count":1,"prefix_tokens":11428}],"prompt_tokens":860,"prompt_tokens_stdev":500,"output_tokens":512,"output_tokens_stdev":100}' \ --seed kind=static,value=1

Key parameters:

• --profile kind=<type>: Defines the traffic pattern —synchronous,concurrent,throughput,constant,poisson, orsweep
• --profile kind=constant,rate=10: Forconstant/poisson, set requests per second in the profile config; forconcurrent, usestreams=; forthroughput, usemax_concurrency=
• --constraint kind=max_duration,seconds=<seconds>or--constraint kind=max_requests,count=<count>: Limit each strategy by time or request count

• prefix_count=1：只生成一个共享前缀，所有请求重复使用。
• prefix_tokens=8192：共享前缀长度约 8192 tokens。
• prompt_tokens=8192：不包含前缀的独立 prompt 长度。

多轮对话评测

https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.md

Key parameters:

- turns=3 : Number of turns per conversation
- prompt_tokens=200 : Input tokens per turn
- output_tokens=100 : Output tokens per turn
- prefix_tokens=100 : (Optional) Add a system prompt
With System Prompts (Prefixes)

guidellm benchmark run \ --target "http://localhost:8000" \ --model "meta-llama/Llama-3.1-8B-Instruct" \ --request-format /v1/chat/completions \ --profile constant \ --rate 2.0 \ --max-requests 100 \ --data "prompt_tokens=150,output_tokens=75,turns=4,prefix_tokens=100"

设置共享前缀：

guidellm benchmark run \ --target "http://localhost:30000" \ --model "models/DeepSeek-V3.2" \ --request-format /v1/chat/completions \ --data "prompt_tokens=1024,output_tokens=1,turns=1,prefix_count=1,prefix_tokens=1024,prompt_tokens_stdev=1000,output_tokens_stdev=1" \ --rate-type concurrent \ --random-seed 1 \ --rate 30 \ --max-requests 30

guidellm benchmark \ --target "http://x.x.x.x" \ --model deepseek-v3.1 \ --processor /path/DeepSeek-V3.1-Terminus \ --request-type "chat_completions" \ --backend-args '{"validate_backend": false}' \ --data "prompt_tokens=2048,output_tokens=512,prompt_tokens_stdev=200,output_tokens_stdev=100" \ --rate-type concurrent \ --rate 100 \ --max-requests 128 # --data-sampler "random" \ # can reference OpenAIHTTPBackend args --backend-kwargs '{"api_key": "sk-..."}' \ --backend-kwargs '{"http2": false, "timeout": 120}' \

guidellm benchmark \ --target "http://localhost:30000" \ --model DeepSeek-V3.1-Terminus \ --processor model/DeepSeek-V3.1-Terminus \ --processor-args '{"trust_remote_code": true}' \ --data "prompt_tokens=2048,output_tokens=512,prompt_tokens_stdev=200,output_tokens_stdev=100" \ --rate-type poisson \ --rate 1 \ --max-requests 1536

guidellm benchmark \ --target "http://localhost:30000" \ --model DeepSeek-V3.1-Terminus \ --processor model/DeepSeek-V3.1-Terminus \ --processor-args '{"trust_remote_code": true}' \ --data "prompt_tokens=2048,output_tokens=512,prompt_tokens_stdev=200,output_tokens_stdev=50" \ --rate-type concurrent \ --rate 2560 \ --max-requests 3200 # --random-seed 1

设置前缀长度prefix_tokens_max

结果样例

ℹ Request Latency Statistics (Completed Requests) |===========|======|======|======|========|========|========|======|======|======|======|======|======| | Benchmark | Request Latency ||| TTFT ||| ITL ||| TPOT ||| | Strategy | Sec ||| ms ||| ms ||| ms ||| | | Mean | Mdn | p99 | Mean | Mdn | p99 | Mean | Mdn | p99 | Mean | Mdn | p99 | |-----------|------|------|------|--------|--------|--------|------|------|------|------|------|------| | poisson | 11.2 | 11.4 | 12.9 | 2136.1 | 2256.4 | 2326.5 | xxx | xxx | xxx | xxx | xxx |xxx | |===========|======|======|======|========|========|========|======|======|======|======|======|======| ℹ Server Throughput Statistics |===========|=====|======|=======|======|========|=========|========|=======|=======|========| | Benchmark | Requests |||| Input Tokens || Output Tokens || Total Tokens || | Strategy | Per Sec || Concurrency || Per Sec || Per Sec || Per Sec || | | Mdn | Mean | Mdn | Mean | Mdn | Mean | Mdn | Mean | Mdn | Mean | |-----------|-----|------|-------|------|--------|---------|--------|-------|-------|--------| | poisson | 0.2 | 0.8 | 12.0 | 9.0 | 5435.5 | 25898.4 | 418.1 | 546.6 | 421.6 | 2687.4 | |===========|=====|======|=======|======|========|=========|========|=======|=======|========|

注意与国内的evalscope对比性能时，guidellm的ITL = evalscope的TPOT。但是guidellm的TPOT != evalscope的ITL。注意最新版这块计算交换了。

当前0.4.0默认ITL/TTFT输出median和P95，没有找到配置方法。

要改成mean, mdn, p99，可以自行修改代码

vim /usr/local/lib/python3.12/dist-packages/guidellm/benchmark/outputs/console.py +100

_get_stat_type_name_val和add_stats默认参数：

def add_stats( self, xxx types: Sequence[StatTypesAlias] = ("mean", "median", "p99"), ): xxx @classmethod def _get_stat_type_name_val( cls, stat_type: StatTypesAlias, stats: DistributionSummary | None ) -> tuple[str, float | None]: if stat_type == "mean": return "Mean", stats.mean if stats else None elif stat_type == "median": return "Mdn", stats.median if stats else None elif stat_type == "p95": return "p95", stats.percentiles.p95 if stats else None elif stat_type == "p99": return "p99", stats.percentiles.p99 if stats else None else: raise ValueError(f"Unsupported stat type: {stat_type}")

以及print_server_throughput_table等调用add_stats所设置的参数。

自定义数据集

创建jsonl数据集文件，例如：

{"prompt": "Hello, how are you?", "output_tokens_count": 5, "additional_column": "foo"} {"prompt": "What is your name?", "output_tokens_count": 3, "additional_column": "baz"}

Key Fields

• prompt(required): The main text/input. GuideLLM also recognizes common aliases:instruction,input,inputs,question,context,text,content, orbody
• output_tokens_count(optional): Expected output token count. Aliases:output_tokens,completion_tokens
• prompt_tokens_count(optional): Input token count. Aliases:prompt_tokens,input_tokens
• Additional custom columns can be included and referenced as needed

Using Your Custom JSONL Dataset

guidellm benchmark \ --target "http://localhost:30000" \ --data custom_dataset.jsonl \ --backend-args '{"validate_backend": false}' \ --request-type "chat_completions" \ --rate-type concurrent \ --rate 10 \ --max-requests 10 # --processor model_path \ # --processor-args '{"trust_remote_code": true}' \ # --data-sampler shuffle \

参数设置

参数设置方法

命令行参数设置以及环境变量

例如

参考docs\guides\configuration.md

export GUIDELLM__OPENAI__API_KEY="your-api-key"

GUIDELLM__REQUEST_TIMEOUT

等等。

重要参数

https://github.com/vllm-project/guidellm/blob/main/README.md

评测方法rate-type通常采用并发模式或者poisson模式。

然后针对性设置--rate参数

--rate:

"Benchmark rate(s) to test. Meaning depends on profile: "

"sweep=number of benchmarks, concurrent=concurrent requests, "

"async/constant/poisson=requests per second."

poisson模式除了rate，看上去还可以通过环境变量设置max_concurrency。

--request-type可以选择评测类型和端口，例如/v1/completions 还是 /v1/chat/completions
GenerativeRequestType = Literal[
"text_completions",
"chat_completions",
"audio_transcriptions",
"audio_translations",
]

To benchmark the text completions endpoint ( /v1/completions ) instead of the default chat completions endpoint ( /v1/chat/completions ), you need to use the --request-type text_completions CLI option.

代码逻辑

参数定义和运行入口

src\guidellm\__main__.py

调用src\guidellm\benchmark\entrypoints.py定义的

benchmark_generative_text()

async def benchmark_generative_text( args: BenchmarkGenerativeTextArgs, progress: GenerativeConsoleBenchmarkerProgress | None = None, console: Console | None = None, **constraints: dict[str, ConstraintInitializer | Any], ) -> tuple[GenerativeBenchmarksReport, dict[str, Any]]:

backend, model = await resolve_backend()

创建评测backend，当前默认为注册名为"openai_http"的OpenAIHTTPBackend

model为评测的模型id，例如DeepSeek

processor = await resolve_processor(processor=args.processor, model=model, console=console)

args.processor: "Tokenizer path"

request_loader = await resolve_request_loader( data=args.data, model=model, data_args=args.data_args, data_samples=args.data_samples, processor=processor, processor_args=args.processor_args,

profile = await resolve_profile( profile=args.profile, rate=args.rate, random_seed=args.random_seed, constraints=constraints, max_seconds=args.max_seconds, max_requests=args.max_requests,

benchmarker = Benchmarker()

核心评测调用

async for benchmark in benchmarker.run( benchmark_class=args.benchmark_cls, requests=request_loader, backend=backend, profile=profile, environment=NonDistributedEnvironment(), data=args.data, progress=progress, sample_requests=args.sample_requests, warmup=args.warmup, cooldown=args.cooldown, prefer_response_metrics=args.prefer_response_metrics, ): if benchmark: report.benchmarks.append(benchmark)

数据收集

GenerativeBenchmark.compile()

GenerativeMetrics.compile()

time_per_output_token_ms=StatusDistributionSummary.from_values( value_types=request_types, values=[req.time_per_output_token_ms or 0.0 for req in requests], ), inter_token_latency_ms=StatusDistributionSummary.from_values( value_types=request_types, values=[req.inter_token_latency_ms or 0.0 for req in requests], ),

TTFT/ITL/TOPT等计算逻辑

class GenerativeRequestStats(StandardBaseDict): def request_latency(self) -> float | None: """ End-to-end request processing latency in seconds. :return: Duration from request start to completion, or None if unavailable. """ return self.info.timings.request_end - self.info.timings.request_start def time_to_first_token_ms(self) -> float | None: """ Time to first token generation in milliseconds. :return: Latency from request start to first token, or None if unavailable. """ return 1000 * ( self.info.timings.first_iteration - self.info.timings.request_start ) def time_per_output_token_ms(self) -> float | None: """ Average time per output token in milliseconds. Includes time for first token and all subsequent tokens. :return: Average milliseconds per output token, or None if unavailable. """ return ( 1000 * (self.info.timings.last_iteration - self.info.timings.request_start) / self.output_metrics.total_tokens ) def inter_token_latency_ms(self) -> float | None: """ Average inter-token latency in milliseconds. Measures time between token generations, excluding first token. :return: Average milliseconds between tokens, or None if unavailable. """ return ( 1000 * (self.info.timings.last_iteration - self.info.timings.first_iteration) / (self.output_metrics.total_tokens - 1) ) @computed_field # type: ignore[misc] @property def tokens_per_second(self) -> float | None: """ Overall token throughput including prompt and output tokens. :return: Total tokens per second, or None if unavailable. """ if not (latency := self.request_latency) or self.total_tokens is None: return None return self.total_tokens / latency @computed_field # type: ignore[misc] @property def output_tokens_per_second(self) -> float | None: """ Output token generation throughput. :return: Output tokens per second, or None if unavailable. """ return self.output_tokens / latency @computed_field # type: ignore[misc] @property def output_tokens_per_iteration(self) -> float | None: """ Average output tokens generated per iteration. :return: Output tokens per iteration, or None if unavailable. """ return self.output_tokens / self.info.timings.iterations

结果生成和打印

output_format_results = {} for key, output in output_formats.items(): output_result = await output.finalize(report) output_format_results[key] = output_result # print to console

@GenerativeBenchmarkerOutput.register("console") class GenerativeBenchmarkerConsole(GenerativeBenchmarkerOutput): async def finalize(self, report: GenerativeBenchmarksReport) -> str: """ Print the complete benchmark report to the console. :param report: The completed benchmark report. :return: """ self._print_benchmarks_metadata(report.benchmarks) self._print_benchmarks_info(report.benchmarks) self._print_benchmarks_stats(report.benchmarks)

Benchmarker.run()

strategies_generator = profile.strategies_generator() strategy, constraints = next(strategies_generator) scheduler: Scheduler[RequestT, ResponseT] = Scheduler() while strategy is not None: async for ( response,request,request_info,scheduler_state, ) in scheduler.run( requests=requests, backend=backend, strategy=strategy, startup_duration=warmup if warmup and warmup >= 1 else 0.0, env=environment, **constraints or {}, ): try: benchmark_class.update_estimate( args, estimated_state, response, request, request_info, scheduler_state, )

strategies_generator = profile.strategies_generator()

strategy, constraints = next(strategies_generator)

创建通过while创建多个并发的benchmark

Scheduler

不同评测方案，possion, concurrency等设置

Profile

SynchronousProfile: "synchronous"

ConcurrentProfile:"concurrent"

ThroughputProfile: "throughput"

AsyncProfile: ["async", "constant", "poisson"]

Backend - OpenAIHTTPBackend

process_startup
validate
process_shutdown

available_models

resolve

response_handler = self._resolve_response_handler(

request_type=request.request_type

)

src\guidellm\backends\response_handlers.py