36 lines
2.1 KiB
JSON
36 lines
2.1 KiB
JSON
{
|
|
"name": "qwen-plus@gsm8k",
|
|
"dataset_name": "gsm8k",
|
|
"dataset_pretty_name": "GSM8K",
|
|
"dataset_description": "\n## Overview\n\nGSM8K (Grade School Math 8K) is a high-quality dataset of 8.5K linguistically diverse grade school math word problems created by human problem writers. The dataset is specifically designed to evaluate and improve the multi-step mathematical reasoning capabilities of language models.\n\n## Task Description\n\n- **Task Type**: Mathematical Word Problem Solving\n- **Input**: Natural language math word problem\n- **Output**: Numerical answer derived through step-by-step reasoning\n- **Difficulty**: Grade school level (2-8 reasoning steps required)\n\n## Key Features\n\n- Problems require basic arithmetic operations (addition, subtraction, multiplication, division)\n- Solutions involve 2 to 8 sequential reasoning steps\n- High linguistic diversity in problem formulations\n- Human-written problems ensuring natural language quality\n- Clear numerical answers for objective evaluation\n\n## Evaluation Notes\n\n- Default configuration uses **4-shot** examples with Chain-of-Thought (CoT) prompting\n- Answers should be formatted within `\\boxed{}` for proper extraction\n- The metric extracts numerical values for accuracy comparison\n- Supports both zero-shot and few-shot evaluation modes\n",
|
|
"model_name": "qwen-plus",
|
|
"score": 1.0,
|
|
"metrics": [
|
|
{
|
|
"name": "mean_acc",
|
|
"num": 10,
|
|
"score": 1.0,
|
|
"macro_score": 1.0,
|
|
"categories": [
|
|
{
|
|
"name": [
|
|
"default"
|
|
],
|
|
"num": 10,
|
|
"score": 1.0,
|
|
"macro_score": 1.0,
|
|
"subsets": [
|
|
{
|
|
"name": "main",
|
|
"score": 1.0,
|
|
"num": 10
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"analysis": "N/A",
|
|
"perf_metrics": null,
|
|
"num": 10
|
|
} |