BFCL-v3

Overview

BFCL (Berkeley Function Calling Leaderboard) v3 is the first comprehensive and executable function call evaluation benchmark for assessing LLMs' ability to invoke functions. It evaluates various forms of function calls, diverse scenarios, and executability.

Task Description

Task Type: Function Calling / Tool Use Evaluation
Input: User query with available function definitions
Output: Correct function call with parameters
Categories: AST_NON_LIVE, AST_LIVE, RELEVANCE, MULTI_TURN

Key Features

Comprehensive function call evaluation
Tests simple, multiple, and parallel function calls
Multi-language support (Python, Java, JavaScript)
Relevance detection (irrelevance handling)
Multi-turn conversation function calling
Live API endpoints for execution testing

Evaluation Notes

Requires pip install bfcl-eval==2025.10.27.1 before evaluation
Set is_fc_model=True for models with native function calling support
underscore_to_dot parameter controls function name formatting
See the usage documentation for detailed setup
Results broken down by category (AST, Relevance, Multi-turn)

Properties

Property	Value
Benchmark Name	`bfcl_v3`
Dataset ID	AI-ModelScope/bfcl_v3
Paper	N/A
Tags	`Agent`, `FunctionCalling`
Metrics	`acc`
Default Shots	0-shot
Evaluation Split	`train`

Data Statistics

Metric	Value
Total Samples	4,441
Prompt Length (Mean)	291.25 chars
Prompt Length (Min/Max)	36 / 10805 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
`simple`	400	119.66	57	303
`multiple`	200	120.75	65	229
`parallel`	200	298.92	69	781
`parallel_multiple`	200	432.17	88	1249
`java`	100	228.93	138	531
`javascript`	50	222.34	118	637
`live_simple`	258	161.74	47	1247
`live_multiple`	1,053	138.76	42	4828
`live_parallel`	16	150.44	88	365
`live_parallel_multiple`	24	217.67	72	650
`irrelevance`	240	85.96	55	203
`live_relevance`	18	263.83	71	1214
`live_irrelevance`	882	188.26	36	10805
`multi_turn_base`	200	803.33	106	1756
`multi_turn_miss_func`	200	806.75	110	1760
`multi_turn_miss_param`	200	858.2	167	1791
`multi_turn_long_context`	200	803.29	106	1756

Sample Example

Subset: simple

{
  "input": [
    {
      "id": "15e84989",
      "content": "[[{\"role\": \"user\", \"content\": \"Find the area of a triangle with a base of 10 units and height of 5 units.\"}]]"
    }
  ],
  "target": "[{\"calculate_triangle_area\": {\"base\": [10], \"height\": [5], \"unit\": [\"units\", \"\"]}}]",
  "id": 0,
  "group_id": 0,
  "subset_key": "simple",
  "metadata": {
    "id": "simple_0",
    "multi_turn": false,
    "functions": [
      {
        "name": "calculate_triangle_area",
        "description": "Calculate the area of a triangle given its base and height.",
        "parameters": {
          "type": "dict",
          "properties": {
            "base": {
              "type": "integer",
              "description": "The base of the triangle."
            },
            "height": {
              "type": "integer",
              "description": "The height of the triangle."
            },
            "unit": {
              "type": "string",
              "description": "The unit of measure (defaults to 'units' if not specified)"
            }
          },
          "required": [
            "base",
            "height"
          ]
        }
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "calculate_triangle_area",
          "description": "Calculate the area of a triangle given its base and height. Note that the provided function is in Python 3 syntax.",
          "parameters": {
            "type": "object",
            "properties": {
              "base": {
                "type": "integer",
                "description": "The base of the triangle."
              },
              "height": {
                "type": "integer",
                "description": "The height of the triangle."
              },
              "unit": {
                "type": "string",
                "description": "The unit of measure (defaults to 'units' if not specified)"
              }
            },
            "required": [
              "base",
              "height"
            ]
          }
        }
      }
    ],
    "missed_functions": "{}",
    "initial_config": {},
    "involved_classes": [],
    "turns": [
      [
        {
          "role": "user",
          "content": "Find the area of a triangle with a base of 10 units and height of 5 units."
        }
      ]
    ],
    "language": "Python",
    "test_category": "simple",
    "subset": "simple",
    "ground_truth": [
      {
        "calculate_triangle_area": {
          "base": [
            10
          ],
          "height": [
            5
          ],
          "unit": [
            "units",
            ""
          ]
        }
      }
    ],
    "should_execute_tool_calls": false,
    "missing_functions": {},
    "is_fc_model": true
  }
}

Prompt Template

No prompt template defined.

Extra Parameters

Parameter	Type	Default	Description
`underscore_to_dot`	`bool`	`True`	Convert underscores to dots in function names for evaluation.
`is_fc_model`	`bool`	`True`	Indicates the evaluated model natively supports function calling.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets bfcl_v3 \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['bfcl_v3'],
    dataset_args={
        'bfcl_v3': {
            # subset_list: ['simple', 'multiple', 'parallel']  # optional, evaluate specific subsets
            # extra_params: {}  # uses default extra parameters
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

6.6 KiB Raw Permalink Blame History