{
  "meta": {
    "pretty_name": "ACEBench",
    "dataset_id": "evalscope/acebench",
    "paper_url": null,
    "tags": [
      "FunctionCalling",
      "Agent",
      "MultiTurn"
    ],
    "metrics": [
      "acc",
      "process_acc",
      "end_state_acc"
    ],
    "few_shot_num": 0,
    "eval_split": "normal",
    "train_split": "",
    "subset_list": [
      "normal",
      "special",
      "agent"
    ],
    "description": "\n## Overview\n\nACEBench is a tool-use benchmark for evaluating whether large language models can select APIs, fill\narguments, handle abnormal requests, and complete realistic agent tasks.\n\n## Task Description\n\n- **Task Type**: Function calling and agentic tool use\n- **Input**: Conversation history, API specifications, optional time/profile context, and agent task context\n- **Output**: Function calls or diagnostic text for special cases\n- **Subsets**: normal, special, and agent\n\n## Evaluation Notes\n\n- The adapter passes ACEBench API specifications as EvalScope tools and also includes concise text\n  instructions for text-only models.\n- Normal samples are scored by matching function names and arguments.\n- Special samples are scored against ACEBench's diagnostic text contract.\n- Agent samples report `process_acc` against ACEBench milestones. If a model returns a final-state JSON object,\n  `end_state_acc` is also reported and used as `acc`; otherwise `acc` follows `process_acc`.\n",
    "prompt_template": "",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 1023,
    "subset_stats": [
      {
        "name": "normal",
        "sample_count": 823,
        "prompt_length_mean": 4859.68,
        "prompt_length_min": 1070,
        "prompt_length_max": 10555,
        "prompt_length_std": 2230.89,
        "target_length_mean": 260.37
      },
      {
        "name": "special",
        "sample_count": 150,
        "prompt_length_mean": 3449.69,
        "prompt_length_min": 1007,
        "prompt_length_max": 8692,
        "prompt_length_std": 1734.54,
        "target_length_mean": 92.68
      },
      {
        "name": "agent",
        "sample_count": 50,
        "prompt_length_mean": 5346.72,
        "prompt_length_min": 4454,
        "prompt_length_max": 5656,
        "prompt_length_std": 332.98,
        "target_length_mean": 2020.6
      }
    ],
    "prompt_length": {
      "mean": 4676.74,
      "min": 1007,
      "max": 10555,
      "std": 2171.82
    },
    "target_length_mean": 321.82,
    "computed_at": "2026-05-31T08:27:27.526066"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "134dd42a",
          "content": "You are evaluating ACEBench tool-use tasks.\n\nUse the available tool schemas when native function calling is supported.\n\nFor text-only output, return API calls as [ApiName(key1='value1', key2=2)]. Return only the call list and no extra explana ... [TRUNCATED 2553 chars] ... \"}, \"effects\": {\"description\": \"List of audio effects to apply.\", \"type\": \"array\", \"items\": {\"type\": \"string\", \"enum\": [\"reverb\", \"echo\", \"distortion\"]}}}, \"required\": [\"frequency\", \"gain\"]}}}, \"required\": [\"microphone\", \"performanceTime\"]}}]"
        },
        {
          "id": "da9681d9",
          "content": "I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.",
          "role": "user"
        }
      ],
      "target": "{\"ground_truth\": {\"NightSkyAnalysis_performEclipseAnalysis\": {\"dateRange\": {\"startDate\": \"2023-01-01\", \"endDate\": \"2028-01-01\"}, \"location\": {\"latitude\": 37.9838, \"longitude\": 23.7275}, \"eclipseType\": \"total\"}}, \"mile_stone\": []}",
      "id": 0,
      "group_id": 0,
      "tools": [
        {
          "name": "NightSkyAnalysis_performEclipseAnalysis",
          "description": "Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.",
          "parameters": {
            "type": "object",
            "properties": {
              "dateRange": {
                "type": "object",
                "description": "The range of dates for which to analyze solar eclipses.",
                "properties": {
                  "startDate": {
                    "type": "string",
                    "description": "The starting date for the analysis in YYYY-MM-DD format."
                  },
                  "endDate": {
                    "type": "string",
                    "description": "The ending date for the analysis in YYYY-MM-DD format."
                  }
                },
                "required": [
                  "startDate",
                  "endDate"
                ]
              },
              "location": {
                "type": "object",
                "description": "Geographical coordinates to focus the eclipse analysis.",
                "properties": {
                  "latitude": {
                    "type": "number",
                    "description": "Latitude of the location."
                  },
                  "longitude": {
                    "type": "number",
                    "description": "Longitude of the location."
                  }
                },
                "required": [
                  "latitude",
                  "longitude"
                ]
              },
              "eclipseType": {
                "type": "string",
                "description": "The type of solar eclipse to specifically analyze.",
                "enum": [
                  "total",
                  "annular",
                  "partial"
                ]
              }
            },
            "required": [
              "dateRange",
              "location"
            ]
          }
        },
        {
          "name": "AudioPerformanceOptimizer_optimizeMicrophoneSettings",
          "description": "Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.",
          "parameters": {
            "type": "object",
            "properties": {
              "microphone": {
                "type": "object",
                "description": "Details of the microphone used.",
                "properties": {
                  "type": {
                    "type": "string",
                    "description": "Type of the microphone.",
                    "enum": [
                      "dynamic",
                      "condenser",
                      "ribbon"
                    ]
                  },
                  "model": {
                    "type": "string",
                    "description": "Model of the microphone."
                  }
                },
                "required": [
                  "type",
                  "model"
                ]
              },
              "performanceTime": {
                "type": "string",
                "description": "Scheduled time for the performance.",
                "enum": [
                  "morning",
                  "afternoon",
                  "evening",
                  "night"
                ]
              },
              "environment": {
                "type": "object",
                "description": "Environmental conditions of the performance area.",
                "properties": {
                  "humidity": {
                    "type": "integer",
                    "description": "Humidity level as a percentage."
                  },
                  "temperature": {
                    "type": "integer",
                    "description": "Temperature in Celsius."
                  }
                }
              },
              "soundSettings": {
                "type": "array",
                "description": "Specific sound settings to apply.",
                "items": {
                  "type": "object",
                  "properties": {
                    "frequency": {
                      "type": "integer",
                      "description": "Frequency adjustments in Hz."
                    },
                    "gain": {
                      "type": "integer",
                      "description": "Gain adjustments in dB."
                    },
                    "effects": {
                      "type": "array",
                      "description": "List of audio effects to apply.",
                      "items": {
                        "type": "string",
                        "enum": [
                          "reverb",
                          "echo",
                          "distortion"
                        ]
                      }
                    }
                  },
                  "required": [
                    "frequency",
                    "gain"
                  ]
                }
              }
            },
            "required": [
              "microphone",
              "performanceTime"
            ]
          }
        }
      ],
      "metadata": {
        "id": "normal_single_turn_single_function_0",
        "sub_category": "data_normal_single_turn_single_function",
        "question": "user: I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.\n",
        "time": "",
        "profile": "",
        "functions": [
          {
            "name": "NightSkyAnalysis_performEclipseAnalysis",
            "description": "Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.",
            "parameters": {
              "type": "object",
              "properties": {
                "dateRange": {
                  "description": "The range of dates for which to analyze solar eclipses.",
                  "type": "object",
                  "properties": {
                    "startDate": {
                      "description": "The starting date for the analysis in YYYY-MM-DD format.",
                      "type": "string"
                    },
                    "endDate": {
                      "description": "The ending date for the analysis in YYYY-MM-DD format.",
                      "type": "string"
                    }
                  },
                  "required": [
                    "startDate",
                    "endDate"
                  ]
                },
                "location": {
                  "description": "Geographical coordinates to focus the eclipse analysis.",
                  "type": "object",
                  "properties": {
                    "latitude": {
                      "description": "Latitude of the location.",
                      "type": "number",
                      "minimum": -90,
                      "maximum": 90
                    },
                    "longitude": {
                      "description": "Longitude of the location.",
                      "type": "number",
                      "minimum": -180,
                      "maximum": 180
                    }
                  },
                  "required": [
                    "latitude",
                    "longitude"
                  ]
                },
                "eclipseType": {
                  "description": "The type of solar eclipse to specifically analyze.",
                  "type": "string",
                  "enum": [
                    "total",
                    "annular",
                    "partial"
                  ]
                }
              },
              "required": [
                "dateRange",
                "location"
              ]
            }
          },
          {
            "name": "AudioPerformanceOptimizer_optimizeMicrophoneSettings",
            "description": "Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.",
            "parameters": {
              "type": "object",
              "properties": {
                "microphone": {
                  "description": "Details of the microphone used.",
                  "type": "object",
                  "properties": {
                    "type": {
                      "description": "Type of the microphone.",
                      "type": "string",
                      "enum": [
                        "dynamic",
                        "condenser",
                        "ribbon"
                      ]
                    },
                    "model": {
                      "description": "Model of the microphone.",
                      "type": "string"
                    }
                  },
                  "required": [
                    "type",
                    "model"
                  ]
                },
                "performanceTime": {
                  "description": "Scheduled time for the performance.",
                  "type": "string",
                  "enum": [
                    "morning",
                    "afternoon",
                    "evening",
                    "night"
                  ]
                },
                "environment": {
                  "description": "Environmental conditions of the performance area.",
                  "type": "object",
                  "properties": {
                    "humidity": {
                      "description": "Humidity level as a percentage.",
                      "type": "integer",
                      "minimum": 0,
                      "maximum": 100
                    },
                    "temperature": {
                      "description": "Temperature in Celsius.",
                      "type": "integer"
                    }
                  }
                },
                "soundSettings": {
                  "description": "Specific sound settings to apply.",
                  "type": "array",
                  "items": {
                    "type": "object",
                    "properties": {
                      "frequency": {
                        "description": "Frequency adjustments in Hz.",
                        "type": "integer"
                      },
                      "gain": {
                        "description": "Gain adjustments in dB.",
                        "type": "integer"
                      },
                      "effects": {
                        "description": "List of audio effects to apply.",
                        "type": "array",
                        "items": {
                          "type": "string",
                          "enum": [
                            "reverb",
                            "echo",
                            "distortion"
                          ]
                        }
                      }
                    },
                    "required": [
                      "frequency",
                      "gain"
                    ]
                  }
                }
              },
              "required": [
                "microphone",
                "performanceTime"
              ]
            }
          }
        ],
        "ground_truth": {
          "NightSkyAnalysis_performEclipseAnalysis": {
            "dateRange": {
              "startDate": "2023-01-01",
              "endDate": "2028-01-01"
            },
            "location": {
              "latitude": 37.9838,
              "longitude": 23.7275
            },
            "eclipseType": "total"
          }
        },
        "mile_stone": [],
        "initial_config": {},
        "involved_classes": []
      }
    },
    "subset": "normal",
    "truncated": false
  },
  "readme": {
    "en": "# ACEBench\n\n\n## Overview\n\nACEBench is a tool-use benchmark for evaluating whether large language models can select APIs, fill\narguments, handle abnormal requests, and complete realistic agent tasks.\n\n## Task Description\n\n- **Task Type**: Function calling and agentic tool use\n- **Input**: Conversation history, API specifications, optional time/profile context, and agent task context\n- **Output**: Function calls or diagnostic text for special cases\n- **Subsets**: normal, special, and agent\n\n## Evaluation Notes\n\n- The adapter passes ACEBench API specifications as EvalScope tools and also includes concise text\n  instructions for text-only models.\n- Normal samples are scored by matching function names and arguments.\n- Special samples are scored against ACEBench's diagnostic text contract.\n- Agent samples report `process_acc` against ACEBench milestones. If a model returns a final-state JSON object,\n  `end_state_acc` is also reported and used as `acc`; otherwise `acc` follows `process_acc`.\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `acebench` |\n| **Dataset ID** | [evalscope/acebench](https://modelscope.cn/datasets/evalscope/acebench/summary) |\n| **Paper** | N/A |\n| **Tags** | `Agent`, `FunctionCalling`, `MultiTurn` |\n| **Metrics** | `acc`, `process_acc`, `end_state_acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `normal` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 1,023 |\n| Prompt Length (Mean) | 4676.74 chars |\n| Prompt Length (Min/Max) | 1007 / 10555 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `normal` | 823 | 4859.68 | 1070 | 10555 |\n| `special` | 150 | 3449.69 | 1007 | 8692 |\n| `agent` | 50 | 5346.72 | 4454 | 5656 |\n\n## Sample Example\n\n**Subset**: `normal`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"134dd42a\",\n      \"content\": \"You are evaluating ACEBench tool-use tasks.\\n\\nUse the available tool schemas when native function calling is supported.\\n\\nFor text-only output, return API calls as [ApiName(key1='value1', key2=2)]. Return only the call list and no extra explana ... [TRUNCATED 2553 chars] ... \\\"}, \\\"effects\\\": {\\\"description\\\": \\\"List of audio effects to apply.\\\", \\\"type\\\": \\\"array\\\", \\\"items\\\": {\\\"type\\\": \\\"string\\\", \\\"enum\\\": [\\\"reverb\\\", \\\"echo\\\", \\\"distortion\\\"]}}}, \\\"required\\\": [\\\"frequency\\\", \\\"gain\\\"]}}}, \\\"required\\\": [\\\"microphone\\\", \\\"performanceTime\\\"]}}]\"\n    },\n    {\n      \"id\": \"da9681d9\",\n      \"content\": \"I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.\",\n      \"role\": \"user\"\n    }\n  ],\n  \"target\": \"{\\\"ground_truth\\\": {\\\"NightSkyAnalysis_performEclipseAnalysis\\\": {\\\"dateRange\\\": {\\\"startDate\\\": \\\"2023-01-01\\\", \\\"endDate\\\": \\\"2028-01-01\\\"}, \\\"location\\\": {\\\"latitude\\\": 37.9838, \\\"longitude\\\": 23.7275}, \\\"eclipseType\\\": \\\"total\\\"}}, \\\"mile_stone\\\": []}\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"tools\": [\n    {\n      \"name\": \"NightSkyAnalysis_performEclipseAnalysis\",\n      \"description\": \"Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"dateRange\": {\n            \"type\": \"object\",\n            \"description\": \"The range of dates for which to analyze solar eclipses.\",\n            \"properties\": {\n              \"startDate\": {\n                \"type\": \"string\",\n                \"description\": \"The starting date for the analysis in YYYY-MM-DD format.\"\n              },\n              \"endDate\": {\n                \"type\": \"string\",\n                \"description\": \"The ending date for the analysis in YYYY-MM-DD format.\"\n              }\n            },\n            \"required\": [\n              \"startDate\",\n              \"endDate\"\n            ]\n          },\n          \"location\": {\n            \"type\": \"object\",\n            \"description\": \"Geographical coordinates to focus the eclipse analysis.\",\n            \"properties\": {\n              \"latitude\": {\n                \"type\": \"number\",\n                \"description\": \"Latitude of the location.\"\n              },\n              \"longitude\": {\n                \"type\": \"number\",\n                \"description\": \"Longitude of the location.\"\n              }\n            },\n            \"required\": [\n              \"latitude\",\n              \"longitude\"\n            ]\n          },\n          \"eclipseType\": {\n            \"type\": \"string\",\n            \"description\": \"The type of solar eclipse to specifically analyze.\",\n            \"enum\": [\n              \"total\",\n              \"annular\",\n              \"partial\"\n            ]\n          }\n        },\n        \"required\": [\n          \"dateRange\",\n          \"location\"\n        ]\n      }\n    },\n    {\n      \"name\": \"AudioPerformanceOptimizer_optimizeMicrophoneSettings\",\n      \"description\": \"Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"microphone\": {\n            \"type\": \"object\",\n            \"description\": \"Details of the microphone used.\",\n            \"properties\": {\n              \"type\": {\n                \"type\": \"string\",\n                \"description\": \"Type of the microphone.\",\n                \"enum\": [\n                  \"dynamic\",\n                  \"condenser\",\n                  \"ribbon\"\n                ]\n              },\n              \"model\": {\n                \"type\": \"string\",\n                \"description\": \"Model of the microphone.\"\n              }\n            },\n            \"required\": [\n              \"type\",\n              \"model\"\n            ]\n          },\n          \"performanceTime\": {\n            \"type\": \"string\",\n            \"description\": \"Scheduled time for the performance.\",\n            \"enum\": [\n              \"morning\",\n              \"afternoon\",\n              \"evening\",\n              \"night\"\n            ]\n          },\n          \"environment\": {\n            \"type\": \"object\",\n            \"description\": \"Environmental conditions of the performance area.\",\n            \"properties\": {\n              \"humidity\": {\n                \"type\": \"integer\",\n                \"description\": \"Humidity level as a percentage.\"\n              },\n              \"temperature\": {\n                \"type\": \"integer\",\n                \"description\": \"Temperature in Celsius.\"\n              }\n            }\n          },\n          \"soundSettings\": {\n            \"type\": \"array\",\n            \"description\": \"Specific sound settings to apply.\",\n            \"items\": {\n              \"type\": \"object\",\n              \"properties\": {\n                \"frequency\": {\n                  \"type\": \"integer\",\n                  \"description\": \"Frequency adjustments in Hz.\"\n                },\n                \"gain\": {\n                  \"type\": \"integer\",\n                  \"description\": \"Gain adjustments in dB.\"\n                },\n                \"effects\": {\n                  \"type\": \"array\",\n                  \"description\": \"List of audio effects to apply.\",\n                  \"items\": {\n                    \"type\": \"string\",\n                    \"enum\": [\n                      \"reverb\",\n                      \"echo\",\n                      \"distortion\"\n                    ]\n                  }\n                }\n              },\n              \"required\": [\n                \"frequency\",\n                \"gain\"\n              ]\n            }\n          }\n        },\n        \"required\": [\n          \"microphone\",\n          \"performanceTime\"\n        ]\n      }\n    }\n  ],\n  \"metadata\": {\n    \"id\": \"normal_single_turn_single_function_0\",\n    \"sub_category\": \"data_normal_single_turn_single_function\",\n    \"question\": \"user: I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.\\n\",\n    \"time\": \"\",\n    \"profile\": \"\",\n    \"functions\": [\n      {\n        \"name\": \"NightSkyAnalysis_performEclipseAnalysis\",\n        \"description\": \"Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"dateRange\": {\n              \"description\": \"The range of dates for which to analyze solar eclipses.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"startDate\": {\n                  \"description\": \"The starting date for the analysis in YYYY-MM-DD format.\",\n                  \"type\": \"string\"\n                },\n                \"endDate\": {\n                  \"description\": \"The ending date for the analysis in YYYY-MM-DD format.\",\n                  \"type\": \"string\"\n                }\n              },\n              \"required\": [\n                \"startDate\",\n                \"endDate\"\n              ]\n            },\n            \"location\": {\n              \"description\": \"Geographical coordinates to focus the eclipse analysis.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"latitude\": {\n                  \"description\": \"Latitude of the location.\",\n                  \"type\": \"number\",\n                  \"minimum\": -90,\n                  \"maximum\": 90\n                },\n                \"longitude\": {\n                  \"description\": \"Longitude of the location.\",\n                  \"type\": \"number\",\n                  \"minimum\": -180,\n                  \"maximum\": 180\n                }\n              },\n              \"required\": [\n                \"latitude\",\n                \"longitude\"\n              ]\n            },\n            \"eclipseType\": {\n              \"description\": \"The type of solar eclipse to specifically analyze.\",\n              \"type\": \"string\",\n              \"enum\": [\n                \"total\",\n                \"annular\",\n                \"partial\"\n              ]\n            }\n          },\n          \"required\": [\n            \"dateRange\",\n            \"location\"\n          ]\n        }\n      },\n      {\n        \"name\": \"AudioPerformanceOptimizer_optimizeMicrophoneSettings\",\n        \"description\": \"Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"microphone\": {\n              \"description\": \"Details of the microphone used.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"type\": {\n                  \"description\": \"Type of the microphone.\",\n                  \"type\": \"string\",\n                  \"enum\": [\n                    \"dynamic\",\n                    \"condenser\",\n                    \"ribbon\"\n                  ]\n                },\n                \"model\": {\n                  \"description\": \"Model of the microphone.\",\n                  \"type\": \"string\"\n                }\n              },\n              \"required\": [\n                \"type\",\n                \"model\"\n              ]\n            },\n            \"performanceTime\": {\n              \"description\": \"Scheduled time for the performance.\",\n              \"type\": \"string\",\n              \"enum\": [\n                \"morning\",\n                \"afternoon\",\n                \"evening\",\n                \"night\"\n              ]\n            },\n            \"environment\": {\n              \"description\": \"Environmental conditions of the performance area.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"humidity\": {\n                  \"description\": \"Humidity level as a percentage.\",\n                  \"type\": \"integer\",\n                  \"minimum\": 0,\n                  \"maximum\": 100\n                },\n                \"temperature\": {\n                  \"description\": \"Temperature in Celsius.\",\n                  \"type\": \"integer\"\n                }\n              }\n            },\n            \"soundSettings\": {\n              \"description\": \"Specific sound settings to apply.\",\n              \"type\": \"array\",\n              \"items\": {\n                \"type\": \"object\",\n                \"properties\": {\n                  \"frequency\": {\n                    \"description\": \"Frequency adjustments in Hz.\",\n                    \"type\": \"integer\"\n                  },\n                  \"gain\": {\n                    \"description\": \"Gain adjustments in dB.\",\n                    \"type\": \"integer\"\n                  },\n                  \"effects\": {\n                    \"description\": \"List of audio effects to apply.\",\n                    \"type\": \"array\",\n                    \"items\": {\n                      \"type\": \"string\",\n                      \"enum\": [\n                        \"reverb\",\n                        \"echo\",\n                        \"distortion\"\n                      ]\n                    }\n                  }\n                },\n                \"required\": [\n                  \"frequency\",\n                  \"gain\"\n                ]\n              }\n            }\n          },\n          \"required\": [\n            \"microphone\",\n            \"performanceTime\"\n          ]\n        }\n      }\n    ],\n    \"ground_truth\": {\n      \"NightSkyAnalysis_performEclipseAnalysis\": {\n        \"dateRange\": {\n          \"startDate\": \"2023-01-01\",\n          \"endDate\": \"2028-01-01\"\n        },\n        \"location\": {\n          \"latitude\": 37.9838,\n          \"longitude\": 23.7275\n        },\n        \"eclipseType\": \"total\"\n      }\n    },\n    \"mile_stone\": [],\n    \"initial_config\": {},\n    \"involved_classes\": []\n  }\n}\n```\n\n## Prompt Template\n\n*No prompt template defined.*\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets acebench \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['acebench'],\n    dataset_args={\n        'acebench': {\n            # subset_list: ['normal', 'special', 'agent']  # optional, evaluate specific subsets\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# ACEBench\n\n\n## 概述\n\nACEBench 是一个工具使用基准测试，用于评估大语言模型是否能够选择 API、填充参数、处理异常请求，并完成真实的智能体任务。\n\n## 任务描述\n\n- **任务类型**：函数调用与智能体工具使用\n- **输入**：对话历史、API 规范、可选的时间/用户资料上下文，以及智能体任务上下文\n- **输出**：函数调用或特殊情况下的诊断文本\n- **子集**：normal（常规）、special（特殊）、agent（智能体）\n\n## 评估说明\n\n- 适配器将 ACEBench 的 API 规范作为 EvalScope 工具传入，同时也为纯文本模型提供简洁的文本指令。\n- 常规样本通过匹配函数名称和参数进行评分。\n- 特殊样本根据 ACEBench 的诊断文本规范进行评分。\n- 智能体样本根据 ACEBench 的里程碑报告 `process_acc`。如果模型返回最终状态的 JSON 对象，则还会报告 `end_state_acc` 并将其作为 `acc`；否则 `acc` 与 `process_acc` 一致。\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `acebench` |\n| **数据集ID** | [evalscope/acebench](https://modelscope.cn/datasets/evalscope/acebench/summary) |\n| **论文** | N/A |\n| **标签** | `Agent`, `FunctionCalling`, `MultiTurn` |\n| **指标** | `acc`, `process_acc`, `end_state_acc` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `normal` |\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 1,023 |\n| 提示词长度（平均） | 4676.74 字符 |\n| 提示词长度（最小/最大） | 1007 / 10555 字符 |\n\n**各子集统计数据：**\n\n| 子集 | 样本数 | 提示词平均长度 | 提示词最小长度 | 提示词最大长度 |\n|--------|---------|-------------|------------|------------|\n| `normal` | 823 | 4859.68 | 1070 | 10555 |\n| `special` | 150 | 3449.69 | 1007 | 8692 |\n| `agent` | 50 | 5346.72 | 4454 | 5656 |\n\n## 样例示例\n\n**子集**: `normal`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"134dd42a\",\n      \"content\": \"You are evaluating ACEBench tool-use tasks.\\n\\nUse the available tool schemas when native function calling is supported.\\n\\nFor text-only output, return API calls as [ApiName(key1='value1', key2=2)]. Return only the call list and no extra explana ... [TRUNCATED 2553 chars] ... \\\"}, \\\"effects\\\": {\\\"description\\\": \\\"List of audio effects to apply.\\\", \\\"type\\\": \\\"array\\\", \\\"items\\\": {\\\"type\\\": \\\"string\\\", \\\"enum\\\": [\\\"reverb\\\", \\\"echo\\\", \\\"distortion\\\"]}}}, \\\"required\\\": [\\\"frequency\\\", \\\"gain\\\"]}}}, \\\"required\\\": [\\\"microphone\\\", \\\"performanceTime\\\"]}}]\"\n    },\n    {\n      \"id\": \"da9681d9\",\n      \"content\": \"I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.\",\n      \"role\": \"user\"\n    }\n  ],\n  \"target\": \"{\\\"ground_truth\\\": {\\\"NightSkyAnalysis_performEclipseAnalysis\\\": {\\\"dateRange\\\": {\\\"startDate\\\": \\\"2023-01-01\\\", \\\"endDate\\\": \\\"2028-01-01\\\"}, \\\"location\\\": {\\\"latitude\\\": 37.9838, \\\"longitude\\\": 23.7275}, \\\"eclipseType\\\": \\\"total\\\"}}, \\\"mile_stone\\\": []}\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"tools\": [\n    {\n      \"name\": \"NightSkyAnalysis_performEclipseAnalysis\",\n      \"description\": \"Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"dateRange\": {\n            \"type\": \"object\",\n            \"description\": \"The range of dates for which to analyze solar eclipses.\",\n            \"properties\": {\n              \"startDate\": {\n                \"type\": \"string\",\n                \"description\": \"The starting date for the analysis in YYYY-MM-DD format.\"\n              },\n              \"endDate\": {\n                \"type\": \"string\",\n                \"description\": \"The ending date for the analysis in YYYY-MM-DD format.\"\n              }\n            },\n            \"required\": [\n              \"startDate\",\n              \"endDate\"\n            ]\n          },\n          \"location\": {\n            \"type\": \"object\",\n            \"description\": \"Geographical coordinates to focus the eclipse analysis.\",\n            \"properties\": {\n              \"latitude\": {\n                \"type\": \"number\",\n                \"description\": \"Latitude of the location.\"\n              },\n              \"longitude\": {\n                \"type\": \"number\",\n                \"description\": \"Longitude of the location.\"\n              }\n            },\n            \"required\": [\n              \"latitude\",\n              \"longitude\"\n            ]\n          },\n          \"eclipseType\": {\n            \"type\": \"string\",\n            \"description\": \"The type of solar eclipse to specifically analyze.\",\n            \"enum\": [\n              \"total\",\n              \"annular\",\n              \"partial\"\n            ]\n          }\n        },\n        \"required\": [\n          \"dateRange\",\n          \"location\"\n        ]\n      }\n    },\n    {\n      \"name\": \"AudioPerformanceOptimizer_optimizeMicrophoneSettings\",\n      \"description\": \"Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.\",\n      \"parameters\": {\n        \"type\": \"object\",\n        \"properties\": {\n          \"microphone\": {\n            \"type\": \"object\",\n            \"description\": \"Details of the microphone used.\",\n            \"properties\": {\n              \"type\": {\n                \"type\": \"string\",\n                \"description\": \"Type of the microphone.\",\n                \"enum\": [\n                  \"dynamic\",\n                  \"condenser\",\n                  \"ribbon\"\n                ]\n              },\n              \"model\": {\n                \"type\": \"string\",\n                \"description\": \"Model of the microphone.\"\n              }\n            },\n            \"required\": [\n              \"type\",\n              \"model\"\n            ]\n          },\n          \"performanceTime\": {\n            \"type\": \"string\",\n            \"description\": \"Scheduled time for the performance.\",\n            \"enum\": [\n              \"morning\",\n              \"afternoon\",\n              \"evening\",\n              \"night\"\n            ]\n          },\n          \"environment\": {\n            \"type\": \"object\",\n            \"description\": \"Environmental conditions of the performance area.\",\n            \"properties\": {\n              \"humidity\": {\n                \"type\": \"integer\",\n                \"description\": \"Humidity level as a percentage.\"\n              },\n              \"temperature\": {\n                \"type\": \"integer\",\n                \"description\": \"Temperature in Celsius.\"\n              }\n            }\n          },\n          \"soundSettings\": {\n            \"type\": \"array\",\n            \"description\": \"Specific sound settings to apply.\",\n            \"items\": {\n              \"type\": \"object\",\n              \"properties\": {\n                \"frequency\": {\n                  \"type\": \"integer\",\n                  \"description\": \"Frequency adjustments in Hz.\"\n                },\n                \"gain\": {\n                  \"type\": \"integer\",\n                  \"description\": \"Gain adjustments in dB.\"\n                },\n                \"effects\": {\n                  \"type\": \"array\",\n                  \"description\": \"List of audio effects to apply.\",\n                  \"items\": {\n                    \"type\": \"string\",\n                    \"enum\": [\n                      \"reverb\",\n                      \"echo\",\n                      \"distortion\"\n                    ]\n                  }\n                }\n              },\n              \"required\": [\n                \"frequency\",\n                \"gain\"\n              ]\n            }\n          }\n        },\n        \"required\": [\n          \"microphone\",\n          \"performanceTime\"\n        ]\n      }\n    }\n  ],\n  \"metadata\": {\n    \"id\": \"normal_single_turn_single_function_0\",\n    \"sub_category\": \"data_normal_single_turn_single_function\",\n    \"question\": \"user: I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.\\n\",\n    \"time\": \"\",\n    \"profile\": \"\",\n    \"functions\": [\n      {\n        \"name\": \"NightSkyAnalysis_performEclipseAnalysis\",\n        \"description\": \"Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"dateRange\": {\n              \"description\": \"The range of dates for which to analyze solar eclipses.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"startDate\": {\n                  \"description\": \"The starting date for the analysis in YYYY-MM-DD format.\",\n                  \"type\": \"string\"\n                },\n                \"endDate\": {\n                  \"description\": \"The ending date for the analysis in YYYY-MM-DD format.\",\n                  \"type\": \"string\"\n                }\n              },\n              \"required\": [\n                \"startDate\",\n                \"endDate\"\n              ]\n            },\n            \"location\": {\n              \"description\": \"Geographical coordinates to focus the eclipse analysis.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"latitude\": {\n                  \"description\": \"Latitude of the location.\",\n                  \"type\": \"number\",\n                  \"minimum\": -90,\n                  \"maximum\": 90\n                },\n                \"longitude\": {\n                  \"description\": \"Longitude of the location.\",\n                  \"type\": \"number\",\n                  \"minimum\": -180,\n                  \"maximum\": 180\n                }\n              },\n              \"required\": [\n                \"latitude\",\n                \"longitude\"\n              ]\n            },\n            \"eclipseType\": {\n              \"description\": \"The type of solar eclipse to specifically analyze.\",\n              \"type\": \"string\",\n              \"enum\": [\n                \"total\",\n                \"annular\",\n                \"partial\"\n              ]\n            }\n          },\n          \"required\": [\n            \"dateRange\",\n            \"location\"\n          ]\n        }\n      },\n      {\n        \"name\": \"AudioPerformanceOptimizer_optimizeMicrophoneSettings\",\n        \"description\": \"Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"microphone\": {\n              \"description\": \"Details of the microphone used.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"type\": {\n                  \"description\": \"Type of the microphone.\",\n                  \"type\": \"string\",\n                  \"enum\": [\n                    \"dynamic\",\n                    \"condenser\",\n                    \"ribbon\"\n                  ]\n                },\n                \"model\": {\n                  \"description\": \"Model of the microphone.\",\n                  \"type\": \"string\"\n                }\n              },\n              \"required\": [\n                \"type\",\n                \"model\"\n              ]\n            },\n            \"performanceTime\": {\n              \"description\": \"Scheduled time for the performance.\",\n              \"type\": \"string\",\n              \"enum\": [\n                \"morning\",\n                \"afternoon\",\n                \"evening\",\n                \"night\"\n              ]\n            },\n            \"environment\": {\n              \"description\": \"Environmental conditions of the performance area.\",\n              \"type\": \"object\",\n              \"properties\": {\n                \"humidity\": {\n                  \"description\": \"Humidity level as a percentage.\",\n                  \"type\": \"integer\",\n                  \"minimum\": 0,\n                  \"maximum\": 100\n                },\n                \"temperature\": {\n                  \"description\": \"Temperature in Celsius.\",\n                  \"type\": \"integer\"\n                }\n              }\n            },\n            \"soundSettings\": {\n              \"description\": \"Specific sound settings to apply.\",\n              \"type\": \"array\",\n              \"items\": {\n                \"type\": \"object\",\n                \"properties\": {\n                  \"frequency\": {\n                    \"description\": \"Frequency adjustments in Hz.\",\n                    \"type\": \"integer\"\n                  },\n                  \"gain\": {\n                    \"description\": \"Gain adjustments in dB.\",\n                    \"type\": \"integer\"\n                  },\n                  \"effects\": {\n                    \"description\": \"List of audio effects to apply.\",\n                    \"type\": \"array\",\n                    \"items\": {\n                      \"type\": \"string\",\n                      \"enum\": [\n                        \"reverb\",\n                        \"echo\",\n                        \"distortion\"\n                      ]\n                    }\n                  }\n                },\n                \"required\": [\n                  \"frequency\",\n                  \"gain\"\n                ]\n              }\n            }\n          },\n          \"required\": [\n            \"microphone\",\n            \"performanceTime\"\n          ]\n        }\n      }\n    ],\n    \"ground_truth\": {\n      \"NightSkyAnalysis_performEclipseAnalysis\": {\n        \"dateRange\": {\n          \"startDate\": \"2023-01-01\",\n          \"endDate\": \"2028-01-01\"\n        },\n        \"location\": {\n          \"latitude\": 37.9838,\n          \"longitude\": 23.7275\n        },\n        \"eclipseType\": \"total\"\n      }\n    },\n    \"mile_stone\": [],\n    \"initial_config\": {},\n    \"involved_classes\": []\n  }\n}\n```\n\n## 提示模板\n\n*未定义提示模板。*\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets acebench \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['acebench'],\n    dataset_args={\n        'acebench': {\n            # subset_list: ['normal', 'special', 'agent']  # 可选，用于评估特定子集\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "c78d1165208e91b46c98720809902907",
    "needs_translation": false
  },
  "updated_at": "2026-06-01T15:05:41.862420",
  "translation_updated_at": "2026-06-01T15:06:01"
}