270 lines
14 KiB
Markdown
270 lines
14 KiB
Markdown
# ❓ FAQ
|
|
|
|
This section compiles common issues and solutions encountered when using EvalScope.
|
|
|
|
```{important}
|
|
We recommend updating to the latest `main` branch code when encountering issues. Many problems may have already been fixed in the latest version.
|
|
```
|
|
|
|
## Quick Navigation
|
|
|
|
- [❓ FAQ](#-faq)
|
|
- [Quick Navigation](#quick-navigation)
|
|
- [Installation \& Environment](#installation--environment)
|
|
- [Model Evaluation](#model-evaluation)
|
|
- [Evaluation Configuration \& Parameters](#evaluation-configuration--parameters)
|
|
- [Result Anomalies \& Troubleshooting](#result-anomalies--troubleshooting)
|
|
- [Model \& Dataset Support](#model--dataset-support)
|
|
- [Framework Usage \& Extension](#framework-usage--extension)
|
|
- [Performance Testing (perf)](#performance-testing-perf)
|
|
- [Basic Usage \& Configuration](#basic-usage--configuration)
|
|
- [Performance Metrics \& Troubleshooting](#performance-metrics--troubleshooting)
|
|
- [Citing Us](#citing-us)
|
|
|
|
## Installation & Environment
|
|
|
|
**Q: How to use EvalScope through Docker?**
|
|
|
|
**A:** You can use the official ModelScope image which includes EvalScope. For details, please refer to the [Environment Setup Documentation](https://modelscope.cn/docs/intro/environment-setup#%E6%9C%80%E6%96%B0%E9%95%9C%E5%83%8F).
|
|
|
|
**Q: What to do when compilation fails during `pip install evalscope[all]`?**
|
|
|
|
**A:** Try installing `pip install python-dotenv` separately first, then execute `pip install evalscope[all]`.
|
|
|
|
**Q: Environment conflicts between `evalscope[service]` and other libraries (like `bfcl-eval`)?**
|
|
|
|
**A:** Please try installing these libraries separately rather than in a single command. For `bfcl-eval`, try version `2025.6.16`.
|
|
|
|
**Q: How to handle the `trust_remote_code=True` warning during evaluation?**
|
|
|
|
**A:** This is an informational warning that doesn't affect the evaluation process. EvalScope framework has `trust_remote_code=True` set by default, so you can use it safely.
|
|
|
|
**Q: Error in Notebook environment: `RuntimeError: Cannot run the event loop while another loop is running`?**
|
|
|
|
**A:** Please write the evaluation code in a Python script file (`.py`) and execute it in the terminal, avoiding running it in Notebook.
|
|
|
|
## Model Evaluation
|
|
|
|
### Evaluation Configuration & Parameters
|
|
|
|
**Q: How to conduct pass@k evaluation or generate multiple answers for a single sample?**
|
|
|
|
**A:** The `pass@k` metric supports all datasets with `mean` as the aggregation method. Specific setup instructions:
|
|
1. Set `repeats=k` in `TaskConfig`
|
|
2. Set in `dataset_args`:
|
|
`'<dataset_name>': {'aggregation': 'mean_and_pass_at_k'}`
|
|
1. When running the evaluation, each sample will be repeated `k` times, and all `pass@n, 1<=n<=k` metrics will be calculated.
|
|
|
|
For details, please [refer to](https://github.com/modelscope/evalscope/pull/964) the examples provided. Additionally, `mean_and_vote_at_k` and `mean_and_pass_hat_k` aggregation methods are also supported.
|
|
|
|
**Q: How to remove the "thinking process" (such as `<think>...</think>`) from model outputs?**
|
|
|
|
**A:** For `evalscope >= v1.0` versions, the `<think>...</think>` is automatically handled by default. If you need custom handling, use the `--dataset-args` parameter to add `filters` for specific datasets. For example, to remove content before `</think>` when evaluating ifeval dataset:
|
|
```shell
|
|
--dataset-args '{"ifeval": {"filters": {"remove_until": "</think>"}}}'
|
|
```
|
|
If your model uses different thinking tags like `<|end_of_thinking|>`, simply replace it accordingly.
|
|
|
|
**Q: How to use a local model as a Judge Model?**
|
|
|
|
**A:** You can deploy the local model as an API service using frameworks like vLLM, then specify its service address in `--judge-model-args`.
|
|
|
|
**Q: How to set timeout for judge models?**
|
|
|
|
**A:** Set the `timeout` parameter in the `generation_config` of `--judge-model-args`.
|
|
|
|
**Q: How to add custom request headers when evaluating API services?**
|
|
|
|
**A:** Set `extra_headers` in `generation_config`.
|
|
```python
|
|
# Example
|
|
task_config = TaskConfig(
|
|
# ...
|
|
generation_config={'extra_headers': {'Authorization': 'Bearer YOUR_TOKEN'}}
|
|
)
|
|
```
|
|
|
|
**Q: How to set up multi-GPU evaluation?**
|
|
|
|
**A:** EvalScope currently doesn't support Data Parallel. However, you can achieve model parallelism through:
|
|
1. **Using inference services**: Start a multi-GPU inference service with frameworks like vLLM (e.g., set `--tensor-parallel-size`), then evaluate through API.
|
|
2. **Local loading**: Specify `device_map=auto` in `--model-args` to automatically distribute model weights across multiple devices.
|
|
|
|
**Q: Error with `stream` parameter: `unrecognized arguments: --stream True`?**
|
|
|
|
**A:** `--stream` is a switch parameter, use it directly without appending `True`. Correct usage: `--stream`.
|
|
|
|
### Result Anomalies & Troubleshooting
|
|
|
|
**Q: How to troubleshoot obviously abnormal evaluation results (like extremely low accuracy)?**
|
|
|
|
**A:** Please follow these steps:
|
|
1. **Check model interface**: Confirm that the model service or local model can generate responses normally.
|
|
2. **Review prediction files**: Check JSONL files in the `outputs/<timestamp>/predictions/` directory to confirm model outputs meet expectations.
|
|
3. **Visualization analysis**: Use `evalscope service` to start the visualization interface for intuitive result viewing and analysis.
|
|
|
|
**Q: Unstable evaluation results - inconsistent across runs?**
|
|
|
|
**A:** Inconsistent results are usually caused by sampling randomness. Try these methods to stabilize results:
|
|
1. Set `temperature=0` in `generation_config`.
|
|
2. Set a fixed `seed` parameter.
|
|
|
|
**Q: When evaluating `humaneval` and other code generation tasks, scores are much lower than expected?**
|
|
|
|
**A:**
|
|
1. **Check model type**: Use Instruct or Chat models. Base models may have issues like repetition due to not following instructions.
|
|
2. **Remove thinking process**: Some models generate thinking process before code. Please refer to [this method](#evaluation-configuration--parameters) for filtering.
|
|
3. **Adjust generation length**: Default maximum generation length may be insufficient. Please appropriately increase `max_tokens` in `generation_config`.
|
|
|
|
**Q: Inaccurate answer extraction or misjudgments when evaluating `MATH-500` dataset?**
|
|
|
|
**A:** Math problem answer formats are complex, and rule-based parsing can't cover all cases. We recommend using LLM as an auxiliary judge to improve accuracy:
|
|
```python
|
|
# Set in TaskConfig or DataAdapter
|
|
judge_strategy=JudgeStrategy.LLM_RECALL,
|
|
judge_model_args={
|
|
'model_id': 'qwen2.5-72b-instruct',
|
|
'api_url': '...',
|
|
'api_key': '...'
|
|
}
|
|
```
|
|
Reference documentation: [Judge Model Parameters](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html#judge).
|
|
|
|
**Q: `Connection error` when evaluating `alpaca_eval`?**
|
|
|
|
**A:** `alpaca_eval` requires specifying a Judge Model for scoring. It uses OpenAI API by default, and connection will fail if related keys aren't configured. Please specify an available judge model through `--judge-model-args`.
|
|
|
|
**Q: How to continue from checkpoint after evaluation interruption?**
|
|
|
|
**A:** Checkpoint resumption is supported. Use the `--use-cache` parameter and specify the output directory path from the previous evaluation to reuse completed model predictions and evaluation results.
|
|
|
|
**Q: `evalscope service` visualization interface is inaccessible or charts display abnormally?**
|
|
|
|
**A:**
|
|
- **Inaccessible**: Make sure the service is running and open the Dashboard URL printed on startup in your browser. If the port is in use, specify another port with `--port`.
|
|
- **Chart anomalies**: Try upgrading `plotly` to the latest version.
|
|
|
|
**Q: Error when loading models locally: `Expected all tensors to be on the same device`?**
|
|
|
|
**A:** This is usually caused by insufficient GPU memory. `device_map='auto'` may allocate some weights to CPU. Please ensure sufficient GPU memory or try running on smaller models.
|
|
|
|
### Model & Dataset Support
|
|
|
|
**Q: How to evaluate multimodal models (like Qwen-VL, Gemma3)?**
|
|
|
|
**A:** The recommended approach is the Native backend with API-based evaluation: deploy the multimodal model as an OpenAI-compatible service using vLLM, ms-swift, LMDeploy, etc., then run `evalscope eval` against multimodal benchmarks (e.g. `ocr_bench`, `mmmu`, `math_vista` — see the full list in [VLM Benchmarks](../get_started/supported_dataset/vlm.md)). The Native backend does not support loading multimodal model checkpoints locally for evaluation. If you need the VLMEvalKit backend, see [VLMEvalKit Backend](../user_guides/backend/vlmevalkit_backend.md).
|
|
|
|
**Q: Error when evaluating `embeddings` models via API service: `dimensions is currently not supported`?**
|
|
|
|
**A:** Set `'dimensions': None` in `generation_config` or don't pass this parameter.
|
|
|
|
**Q: Error when loading datasets locally (like missing `dtype`)?**
|
|
|
|
**A:** This is a known issue. Temporary solution: manually delete the `dataset_infos.json` file in the dataset cache directory, then retry.
|
|
|
|
**Q: How to use custom datasets for evaluation?**
|
|
|
|
**A:** EvalScope supports custom datasets. Please refer to the following documentation:
|
|
- **LLM Custom Datasets**: [Link](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/llm.html)
|
|
- **Multimodal Custom Datasets**: [Link](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/custom.html#id3)
|
|
|
|
**Q: What do `-f`, `-p`, `-r` represent in `rouge` metrics?**
|
|
|
|
**A:** They are three variants of the ROUGE metric:
|
|
- **-r (Recall)**: Measures how much of the reference text content is covered by generated text.
|
|
- **-p (Precision)**: Measures how much content in generated text is accurate and relevant.
|
|
- **-f (F-measure)**: F1-score, harmonic mean of precision and recall, serving as a comprehensive metric.
|
|
|
|
### Framework Usage & Extension
|
|
|
|
**Q: How to add a custom Benchmark?**
|
|
|
|
**A:** You can inherit from `DataAdapter` and implement its methods, then register through the `@register_benchmark` decorator. For detailed steps, please refer to [Adding Benchmark Documentation](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html).
|
|
|
|
**Q: How to adapt custom model APIs that are not OpenAI-style?**
|
|
|
|
**A:** You can implement your own `Model` class to interface with specific API formats. Please refer to [Custom Model Tutorial](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_model.html).
|
|
|
|
**Q: How to debug or modify EvalScope source code?**
|
|
|
|
**A:** Please use source code installation. After cloning the project locally, execute `pip install -e .` in the project root directory. This way, any modifications to the code will take effect immediately.
|
|
|
|
**Q: Why can't evaluation metrics align between EvalScope and OpenCompass?**
|
|
|
|
**A:** Different evaluation frameworks have differences in implementation details (like prompt templates, post-processing logic, metric calculation methods), making it difficult to achieve complete alignment. We recommend conducting horizontal comparisons between models within the same framework.
|
|
|
|
## Performance Testing (perf)
|
|
|
|
### Basic Usage & Configuration
|
|
|
|
**Q: What should I fill in for the `--url` parameter in `evalscope perf`?**
|
|
|
|
**A:**
|
|
- **General scenarios**: For most OpenAI API-compatible services, use the `/v1/chat/completions` endpoint.
|
|
- **`speed_benchmark` dataset**: This specific dataset is used for testing completion performance and should be used with the `/v1/completions` endpoint to avoid overhead from Chat Template processing.
|
|
|
|
**Q: How to use local files as stress test datasets?**
|
|
|
|
**A:** Specify `--dataset-path` and set `--dataset line_by_line`. The program will read file content line by line as prompts.
|
|
|
|
**Q: How to use pre-downloaded datasets in offline environments?**
|
|
|
|
**A:** Use `--dataset-path` to point to the local dataset path. Two approaches are supported:
|
|
|
|
- **Point to a directory** (for Arrow/Parquet format datasets like `flickr8k`, `kontext_bench`, `longalpaca`):
|
|
```bash
|
|
evalscope perf --dataset kontext_bench --dataset-path /path/to/local/kontext-bench ...
|
|
```
|
|
- **Point to a file** (for JSONL format datasets like `openqa`, `share_gpt_zh`):
|
|
```bash
|
|
evalscope perf --dataset openqa --dataset-path /path/to/open_qa.jsonl ...
|
|
```
|
|
|
|
You can also use `--data-source` to specify the data source (defaults to `modelscope`, can be switched to `huggingface`).
|
|
|
|
**Q: How to stress test multimodal models?**
|
|
|
|
**A:** Currently supports the `flickr8k` dataset for multimodal stress testing. Set `--dataset flickr8k`.
|
|
|
|
**Q: How to set System Prompt during stress testing?**
|
|
|
|
**A:** Pass it as a JSON string in the `model` parameter, for example:
|
|
```shell
|
|
--model '{"model": "my-model", "system_prompt": "You are a helpful assistant."}'
|
|
```
|
|
|
|
### Performance Metrics & Troubleshooting
|
|
|
|
**Q: When stress testing Ollama, concurrency won't increase - what to do?**
|
|
|
|
**A:** Try setting the environment variable `export OLLAMA_NUM_PARALLEL=10` (or other appropriate value) before executing the stress test command to increase Ollama's parallel processing capability.
|
|
|
|
**Q: Why is TTFT (Time To First Token) the same as Latency (Total Latency)?**
|
|
|
|
**A:** To accurately measure TTFT, you must add the `--stream` parameter in the stress test command to enable streaming output. Otherwise, TTFT will equal the total latency when receiving complete responses.
|
|
|
|
**Q: TTFT metric is too high during stress testing, far exceeding single request response time?**
|
|
|
|
**A:** When concurrency exceeds service processing capacity, requests enter a queue to wait. TTFT includes queuing time, so increased TTFT under high concurrency is normal and reflects the service's real performance under high load.
|
|
|
|
**Q: Stress test results show `nan`?**
|
|
|
|
**A:** Please check if input data format is correct. For example, the `openqa` dataset uses the `question` field in JSONL files as prompts by default. If fields don't match or file format is incorrect, it may cause inability to process requests properly.
|
|
|
|
**Q: How to visualize stress test results?**
|
|
|
|
**A:** Results from the `perf` subcommand are not suitable for `evalscope service`. However, visualization through `wandb` or `swanlab` is supported. Please refer to [Stress Test Result Visualization Guide](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/quick_start.html#id6).
|
|
|
|
## Citing Us
|
|
|
|
**Q: I used EvalScope in my work - how should I cite it?**
|
|
|
|
**A:** Thank you very much! You can use the following BibTeX format to cite our work:
|
|
```bibtex
|
|
@misc{evalscope_2024,
|
|
title={{EvalScope}: Evaluation Framework for Large Models},
|
|
author={ModelScope Team},
|
|
year={2024},
|
|
url={https://github.com/modelscope/evalscope}
|
|
}
|
|
``` |