183 lines
5.3 KiB
Markdown
183 lines
5.3 KiB
Markdown
# BFCL-v4
|
|
|
|
|
|
## Overview
|
|
|
|
BFCL-v4 (Berkeley Function-Calling Leaderboard V4) is a comprehensive benchmark for evaluating agentic function-calling capabilities of LLMs. It tests web search, memory operations, and format sensitivity as building blocks for agentic applications.
|
|
|
|
## Task Description
|
|
|
|
- **Task Type**: Agentic Function Calling Evaluation
|
|
- **Input**: Scenarios requiring function calls for web search, memory, etc.
|
|
- **Output**: Correct function calls with proper arguments
|
|
- **Capabilities**: Web search, memory read/write, format handling
|
|
|
|
## Key Features
|
|
|
|
- Holistic agentic evaluation framework
|
|
- Tests building blocks for LLM agents (search, memory, formatting)
|
|
- Multiple scoring categories for detailed analysis
|
|
- Supports both FC models and non-FC models
|
|
- Optional web search with SerpAPI integration
|
|
|
|
## Evaluation Notes
|
|
|
|
- **Installation Required**: `pip install bfcl-eval==2025.10.27.1`
|
|
- Primary metric: **Accuracy** per category
|
|
- Configure `is_fc_model` based on model capabilities
|
|
- Optional: Set `SERPAPI_API_KEY` for web search tasks
|
|
- [Usage Example](https://evalscope.readthedocs.io/en/latest/third_party/bfcl_v4.html)
|
|
|
|
|
|
## Properties
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Benchmark Name** | `bfcl_v4` |
|
|
| **Dataset ID** | [berkeley-function-call-leaderboard](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) |
|
|
| **Paper** | N/A |
|
|
| **Tags** | `Agent`, `FunctionCalling` |
|
|
| **Metrics** | `acc` |
|
|
| **Default Shots** | 0-shot |
|
|
| **Evaluation Split** | `train` |
|
|
|
|
|
|
## Data Statistics
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Samples | 5,106 |
|
|
| Prompt Length (Mean) | 350.4 chars |
|
|
| Prompt Length (Min/Max) | 36 / 10805 chars |
|
|
|
|
**Per-Subset Statistics:**
|
|
|
|
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|
|
|--------|---------|-------------|------------|------------|
|
|
| `irrelevance` | 240 | 85.98 | 55 | 203 |
|
|
| `live_irrelevance` | 884 | 198.72 | 36 | 10805 |
|
|
| `live_multiple` | 1,053 | 149.16 | 42 | 4828 |
|
|
| `live_parallel` | 16 | 163.88 | 88 | 365 |
|
|
| `live_parallel_multiple` | 24 | 217.67 | 72 | 650 |
|
|
| `live_relevance` | 16 | 385.94 | 71 | 1830 |
|
|
| `live_simple` | 258 | 171.91 | 47 | 1247 |
|
|
| `memory_kv` | 155 | 646.46 | 585 | 826 |
|
|
| `memory_rec_sum` | 155 | 646.46 | 585 | 826 |
|
|
| `memory_vector` | 155 | 646.46 | 585 | 826 |
|
|
| `multi_turn_base` | 200 | 808.43 | 106 | 1756 |
|
|
| `multi_turn_long_context` | 200 | 808.41 | 106 | 1756 |
|
|
| `multi_turn_miss_func` | 200 | 811.88 | 110 | 1760 |
|
|
| `multi_turn_miss_param` | 200 | 863.88 | 167 | 1791 |
|
|
| `multiple` | 200 | 120.75 | 65 | 229 |
|
|
| `parallel` | 200 | 297.93 | 69 | 781 |
|
|
| `parallel_multiple` | 200 | 432.17 | 88 | 1249 |
|
|
| `simple_java` | 100 | 228.93 | 138 | 531 |
|
|
| `simple_javascript` | 50 | 222.34 | 118 | 637 |
|
|
| `simple_python` | 400 | 119.66 | 57 | 303 |
|
|
| `web_search_base` | 100 | 831.01 | 748 | 963 |
|
|
| `web_search_no_snippet` | 100 | 831.01 | 748 | 963 |
|
|
|
|
## Sample Example
|
|
|
|
**Subset**: `irrelevance`
|
|
|
|
```json
|
|
{
|
|
"input": [
|
|
{
|
|
"id": "4e05e910",
|
|
"content": "[[{\"role\": \"user\", \"content\": \"Calculate the area of a triangle given the base is 10 meters and height is 5 meters.\"}]]"
|
|
}
|
|
],
|
|
"target": "{}",
|
|
"id": 0,
|
|
"group_id": 0,
|
|
"metadata": {
|
|
"id": "irrelevance_0",
|
|
"question": [
|
|
[
|
|
{
|
|
"role": "user",
|
|
"content": "Calculate the area of a triangle given the base is 10 meters and height is 5 meters."
|
|
}
|
|
]
|
|
],
|
|
"function": [
|
|
{
|
|
"name": "determine_body_mass_index",
|
|
"description": "Calculate body mass index given weight and height.",
|
|
"parameters": {
|
|
"type": "dict",
|
|
"properties": {
|
|
"weight": {
|
|
"type": "float",
|
|
"description": "Weight of the individual in kilograms."
|
|
},
|
|
"height": {
|
|
"type": "float",
|
|
"description": "Height of the individual in meters."
|
|
}
|
|
},
|
|
"required": [
|
|
"weight",
|
|
"height"
|
|
]
|
|
}
|
|
}
|
|
],
|
|
"category": "irrelevance",
|
|
"ground_truth": {}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Prompt Template
|
|
|
|
*No prompt template defined.*
|
|
|
|
## Extra Parameters
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `underscore_to_dot` | `bool` | `True` | Convert underscores to dots in function names for evaluation. |
|
|
| `is_fc_model` | `bool` | `True` | Indicates the evaluated model natively supports function calling. |
|
|
| `SERPAPI_API_KEY` | `str | null` | `None` | SerpAPI key enabling web-search capability in BFCL V4. Null disables web search. |
|
|
|
|
## Usage
|
|
|
|
### Using CLI
|
|
|
|
```bash
|
|
evalscope eval \
|
|
--model YOUR_MODEL \
|
|
--api-url OPENAI_API_COMPAT_URL \
|
|
--api-key EMPTY_TOKEN \
|
|
--datasets bfcl_v4 \
|
|
--limit 10 # Remove this line for formal evaluation
|
|
```
|
|
|
|
### Using Python
|
|
|
|
```python
|
|
from evalscope import run_task
|
|
from evalscope.config import TaskConfig
|
|
|
|
task_cfg = TaskConfig(
|
|
model='YOUR_MODEL',
|
|
api_url='OPENAI_API_COMPAT_URL',
|
|
api_key='EMPTY_TOKEN',
|
|
datasets=['bfcl_v4'],
|
|
dataset_args={
|
|
'bfcl_v4': {
|
|
# subset_list: ['irrelevance', 'live_irrelevance', 'live_multiple'] # optional, evaluate specific subsets
|
|
# extra_params: {} # uses default extra parameters
|
|
}
|
|
},
|
|
limit=10, # Remove this line for formal evaluation
|
|
)
|
|
|
|
run_task(task_cfg=task_cfg)
|
|
```
|
|
|
|
|