{
  "meta": {
    "pretty_name": "AIR-Bench-Foundation",
    "dataset_id": "evalscope/AIR-Bench",
    "paper_url": "https://aclanthology.org/2024.acl-long.109/",
    "tags": [
      "Audio",
      "MCQ",
      "Knowledge"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "Speaker_Age_Prediction_common_voice_13.0_en",
      "Speaker_Emotion_Recontion_iemocap",
      "Speaker_Emotion_Recontion_meld",
      "Speaker_Gender_Recognition_common_voice_13_en",
      "Speaker_Gender_Recognition_meld",
      "Speaker_Intent_Classification_slurp",
      "Speaker_Number_Verification_voxceleb1",
      "Speech_Entity_Reconition_slurp",
      "Speech_Grounding_librispeech",
      "Spoken_Language_Identification_covost2",
      "Synthesized_Voice_Detection_fake_or_real",
      "Acoustic_Scene_Classification_CochlScene",
      "Acoustic_Scene_Classification_TUT2017",
      "Audio_Grounding_AudioGrounding",
      "Sound_AQA_avqa",
      "Sound_AQA_clothoaqa",
      "vocal_sound_classification_VocalSound",
      "Music_AQA_music_avqa",
      "Music_Genre_Recognition_MTJ-Jamendo",
      "Music_Genre_Recognition_fma",
      "Music_Instruments_Classfication_MTJ-Jamendo",
      "Music_Instruments_Classfication_nsynth",
      "Music_Midi_Pitch_Analysis_nsynth",
      "Music_Midi_Velocity_Analysis_nsynth",
      "Music_Mood_Recognition_MTJ-Jamendo"
    ],
    "description": "\n## Overview\n\nAIR-Bench Foundation is the discriminative half of [AIR-Bench](https://arxiv.org/abs/2402.07729) (Audio InstRuction Benchmark, ACL 2024 main conference) — the first instruction-following benchmark for large audio-language models (LALMs), covering **human speech, natural sounds and music**. The Foundation track contains roughly 25k single-choice questions spanning 19 logical tasks across three audio categories.\n\n## Task Description\n\n- **Task Type**: Single-choice question answering grounded on an audio clip.\n- **Input**: One audio clip + a question with up to four candidate answers (A/B/C/D).\n- **Output**: A single letter chosen from the provided options.\n\n## Categories (19 tasks / 25 source-dataset subsets)\n\n- **Speech** (11 dirs / 9 tasks): speech grounding, language ID, gender, emotion (IEMOCAP+MELD), age, speech entity recognition, intent classification, speaker counting, synthesized-voice detection.\n- **Sound** (6 dirs / 4 tasks): audio grounding, vocal sound classification, acoustic scene classification (CochlScene+TUT2017), sound QA (avqa+clothoaqa).\n- **Music** (8 dirs / 6 tasks): instruments, genre, MIDI pitch, MIDI velocity, music QA, music emotion.\n\n## Prompt Template (matches official `Inference_Foundation.py`)\n\n```text\nChoose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\n{question}\nA. {choice_a}\nB. {choice_b}\nC. {choice_c}\nD. {choice_d}\n```\n\n## Dataset Access\n\n- The dataset is hosted on ModelScope: [`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset). It uses an *audiofolder + JSON metadata* layout. evalscope downloads it lazily via `modelscope.dataset_snapshot_download` on first run; the full release is ~49 GB, so it is recommended to limit which subsets are pulled via `extra_params`.\n- If the dataset is already on disk, pass `dataset_args={'air_bench_foundation': {'local_path': '/path/to/AIR-Bench-Dataset'}}`; the local root should contain `Foundation/`.\n- Some Foundation samples are FLAC. For OpenAI-compatible audio input evalscope converts them to cached WAV files, which requires either `soundfile` (`pip install \"evalscope[air_bench]\"`) or a working `ffmpeg` binary.\n\n## Evaluation Notes\n\n- Metric: **accuracy** (per source-dataset subset, plus per-category aggregation).\n- Default prompt follows the official `Inference_Foundation.py` formatting so existing AIR-Bench leaderboard numbers can be reproduced.\n- Set `extra_params={'subsets': [...]}` to limit to a subset of the 25 source directories — useful for partial downloads.\n",
    "prompt_template": "Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {
      "subsets": {
        "type": "list",
        "description": "Optional list of Foundation source-dataset directories to evaluate. Defaults to all 25 directories. Useful when only a subset has been downloaded locally.",
        "value": null
      }
    },
    "sandbox_config": {},
    "category": "vlm"
  },
  "statistics": {
    "total_samples": 21426,
    "subset_stats": [
      {
        "name": "Speaker_Age_Prediction_common_voice_13.0_en",
        "sample_count": 1000,
        "prompt_length_mean": 291.98,
        "prompt_length_min": 278,
        "prompt_length_max": 305,
        "prompt_length_std": 6.24,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      },
      {
        "name": "Speaker_Emotion_Recontion_iemocap",
        "sample_count": 1000,
        "prompt_length_mean": 227.13,
        "prompt_length_min": 218,
        "prompt_length_max": 238,
        "prompt_length_std": 4.91,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Speaker_Emotion_Recontion_meld",
        "sample_count": 1000,
        "prompt_length_mean": 233.21,
        "prompt_length_min": 218,
        "prompt_length_max": 250,
        "prompt_length_std": 5.68,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Speaker_Gender_Recognition_common_voice_13_en",
        "sample_count": 780,
        "prompt_length_mean": 226.19,
        "prompt_length_min": 213,
        "prompt_length_max": 241,
        "prompt_length_std": 6.49,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 780,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      },
      {
        "name": "Speaker_Gender_Recognition_meld",
        "sample_count": 1000,
        "prompt_length_mean": 226.42,
        "prompt_length_min": 213,
        "prompt_length_max": 241,
        "prompt_length_std": 6.54,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Speaker_Intent_Classification_slurp",
        "sample_count": 662,
        "prompt_length_mean": 268.52,
        "prompt_length_min": 232,
        "prompt_length_max": 295,
        "prompt_length_std": 9.81,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 662,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Speaker_Number_Verification_voxceleb1",
        "sample_count": 314,
        "prompt_length_mean": 208.24,
        "prompt_length_min": 194,
        "prompt_length_max": 221,
        "prompt_length_std": 6.32,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 314,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Speech_Entity_Reconition_slurp",
        "sample_count": 1000,
        "prompt_length_mean": 253.14,
        "prompt_length_min": 226,
        "prompt_length_max": 316,
        "prompt_length_std": 11.9,
        "target_length_mean": 1.0,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Speech_Grounding_librispeech",
        "sample_count": 981,
        "prompt_length_mean": 253.92,
        "prompt_length_min": 230,
        "prompt_length_max": 282,
        "prompt_length_std": 8.63,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 981,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Spoken_Language_Identification_covost2",
        "sample_count": 495,
        "prompt_length_mean": 207.12,
        "prompt_length_min": 191,
        "prompt_length_max": 232,
        "prompt_length_std": 9.01,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 495,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      },
      {
        "name": "Synthesized_Voice_Detection_fake_or_real",
        "sample_count": 1000,
        "prompt_length_mean": 224.79,
        "prompt_length_min": 203,
        "prompt_length_max": 242,
        "prompt_length_std": 10.34,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Acoustic_Scene_Classification_CochlScene",
        "sample_count": 1000,
        "prompt_length_mean": 240.97,
        "prompt_length_min": 213,
        "prompt_length_max": 278,
        "prompt_length_std": 12.26,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Acoustic_Scene_Classification_TUT2017",
        "sample_count": 1000,
        "prompt_length_mean": 241.72,
        "prompt_length_min": 207,
        "prompt_length_max": 284,
        "prompt_length_std": 12.76,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Audio_Grounding_AudioGrounding",
        "sample_count": 896,
        "prompt_length_mean": 273.74,
        "prompt_length_min": 249,
        "prompt_length_max": 321,
        "prompt_length_std": 10.91,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 896,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Sound_AQA_avqa",
        "sample_count": 1000,
        "prompt_length_mean": 227.21,
        "prompt_length_min": 193,
        "prompt_length_max": 298,
        "prompt_length_std": 21.15,
        "target_length_mean": 1.0,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Sound_AQA_clothoaqa",
        "sample_count": 1000,
        "prompt_length_mean": 199.89,
        "prompt_length_min": 179,
        "prompt_length_max": 262,
        "prompt_length_std": 12.13,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "vocal_sound_classification_VocalSound",
        "sample_count": 985,
        "prompt_length_mean": 232.62,
        "prompt_length_min": 210,
        "prompt_length_max": 253,
        "prompt_length_std": 7.23,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 985,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Music_AQA_music_avqa",
        "sample_count": 814,
        "prompt_length_mean": 208.7,
        "prompt_length_min": 185,
        "prompt_length_max": 238,
        "prompt_length_std": 13.58,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 814,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Music_Genre_Recognition_MTJ-Jamendo",
        "sample_count": 1000,
        "prompt_length_mean": 223.84,
        "prompt_length_min": 200,
        "prompt_length_max": 248,
        "prompt_length_std": 8.44,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      },
      {
        "name": "Music_Genre_Recognition_fma",
        "sample_count": 1000,
        "prompt_length_mean": 224.59,
        "prompt_length_min": 201,
        "prompt_length_max": 250,
        "prompt_length_std": 8.66,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      },
      {
        "name": "Music_Instruments_Classfication_MTJ-Jamendo",
        "sample_count": 1000,
        "prompt_length_mean": 236.52,
        "prompt_length_min": 218,
        "prompt_length_max": 262,
        "prompt_length_std": 7.27,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 1000,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      },
      {
        "name": "Music_Instruments_Classfication_nsynth",
        "sample_count": 996,
        "prompt_length_mean": 228.12,
        "prompt_length_min": 216,
        "prompt_length_max": 247,
        "prompt_length_std": 4.5,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 996,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Music_Midi_Pitch_Analysis_nsynth",
        "sample_count": 493,
        "prompt_length_mean": 253.88,
        "prompt_length_min": 243,
        "prompt_length_max": 264,
        "prompt_length_std": 4.54,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 493,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Music_Midi_Velocity_Analysis_nsynth",
        "sample_count": 484,
        "prompt_length_mean": 270.6,
        "prompt_length_min": 259,
        "prompt_length_max": 279,
        "prompt_length_std": 4.28,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 484,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "wav"
            ]
          }
        }
      },
      {
        "name": "Music_Mood_Recognition_MTJ-Jamendo",
        "sample_count": 526,
        "prompt_length_mean": 229.67,
        "prompt_length_min": 210,
        "prompt_length_max": 248,
        "prompt_length_std": 7.06,
        "target_length_mean": 1,
        "multimodal": {
          "has_images": false,
          "has_audio": true,
          "has_video": false,
          "audio": {
            "count_total": 526,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "sample_rates": [],
            "formats": [
              "mp3"
            ]
          }
        }
      }
    ],
    "prompt_length": {
      "mean": 236.68,
      "min": 179,
      "max": 321,
      "std": 23.65
    },
    "target_length_mean": 1.0,
    "computed_at": "2026-05-14T13:14:00.943639",
    "multimodal": {
      "has_images": false,
      "has_audio": true,
      "has_video": false,
      "audio": {
        "count_total": 21426,
        "count_per_sample": {
          "min": 1,
          "max": 1,
          "mean": 1
        },
        "duration": null,
        "sample_rates": [],
        "formats": [
          "mp3",
          "wav"
        ]
      }
    }
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "544443c0",
          "content": [
            {
              "audio": "[BASE64_AUDIO: mp3, ~25.9KB]",
              "format": "mp3"
            },
            {
              "text": "Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\nWhich age range do you believe best matches the speaker's voice?\nA. teens to twenties\nB. thirties to fourties\nC. fifties to sixties\nD. seventies to eighties"
            }
          ]
        }
      ],
      "target": "B",
      "id": 0,
      "group_id": 0,
      "subset_key": "Speaker_Age_Prediction_common_voice_13.0_en",
      "metadata": {
        "uniq_id": 5973,
        "task_name": "Speaker_Age_Prediction",
        "dataset_name": "common_voice_13.0_en",
        "category": "speech",
        "answer_gt_text": "thirties to fourties",
        "choices": {
          "A": "teens to twenties",
          "B": "thirties to fourties",
          "C": "fifties to sixties",
          "D": "seventies to eighties"
        }
      }
    },
    "subset": "Speaker_Age_Prediction_common_voice_13.0_en",
    "truncated": false
  },
  "readme": {
    "en": "# AIR-Bench-Foundation\n\n\n## Overview\n\nAIR-Bench Foundation is the discriminative half of [AIR-Bench](https://arxiv.org/abs/2402.07729) (Audio InstRuction Benchmark, ACL 2024 main conference) — the first instruction-following benchmark for large audio-language models (LALMs), covering **human speech, natural sounds and music**. The Foundation track contains roughly 25k single-choice questions spanning 19 logical tasks across three audio categories.\n\n## Task Description\n\n- **Task Type**: Single-choice question answering grounded on an audio clip.\n- **Input**: One audio clip + a question with up to four candidate answers (A/B/C/D).\n- **Output**: A single letter chosen from the provided options.\n\n## Categories (19 tasks / 25 source-dataset subsets)\n\n- **Speech** (11 dirs / 9 tasks): speech grounding, language ID, gender, emotion (IEMOCAP+MELD), age, speech entity recognition, intent classification, speaker counting, synthesized-voice detection.\n- **Sound** (6 dirs / 4 tasks): audio grounding, vocal sound classification, acoustic scene classification (CochlScene+TUT2017), sound QA (avqa+clothoaqa).\n- **Music** (8 dirs / 6 tasks): instruments, genre, MIDI pitch, MIDI velocity, music QA, music emotion.\n\n## Prompt Template (matches official `Inference_Foundation.py`)\n\n```text\nChoose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\n{question}\nA. {choice_a}\nB. {choice_b}\nC. {choice_c}\nD. {choice_d}\n```\n\n## Dataset Access\n\n- The dataset is hosted on ModelScope: [`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset). It uses an *audiofolder + JSON metadata* layout. evalscope downloads it lazily via `modelscope.dataset_snapshot_download` on first run; the full release is ~49 GB, so it is recommended to limit which subsets are pulled via `extra_params`.\n- If the dataset is already on disk, pass `dataset_args={'air_bench_foundation': {'local_path': '/path/to/AIR-Bench-Dataset'}}`; the local root should contain `Foundation/`.\n- Some Foundation samples are FLAC. For OpenAI-compatible audio input evalscope converts them to cached WAV files, which requires either `soundfile` (`pip install \"evalscope[air_bench]\"`) or a working `ffmpeg` binary.\n\n## Evaluation Notes\n\n- Metric: **accuracy** (per source-dataset subset, plus per-category aggregation).\n- Default prompt follows the official `Inference_Foundation.py` formatting so existing AIR-Bench leaderboard numbers can be reproduced.\n- Set `extra_params={'subsets': [...]}` to limit to a subset of the 25 source directories — useful for partial downloads.\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `air_bench_foundation` |\n| **Dataset ID** | [evalscope/AIR-Bench](https://modelscope.cn/datasets/evalscope/AIR-Bench/summary) |\n| **Paper** | [Paper](https://aclanthology.org/2024.acl-long.109/) |\n| **Tags** | `Audio`, `Knowledge`, `MCQ` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 21,426 |\n| Prompt Length (Mean) | 236.68 chars |\n| Prompt Length (Min/Max) | 179 / 321 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `Speaker_Age_Prediction_common_voice_13.0_en` | 1,000 | 291.98 | 278 | 305 |\n| `Speaker_Emotion_Recontion_iemocap` | 1,000 | 227.13 | 218 | 238 |\n| `Speaker_Emotion_Recontion_meld` | 1,000 | 233.21 | 218 | 250 |\n| `Speaker_Gender_Recognition_common_voice_13_en` | 780 | 226.19 | 213 | 241 |\n| `Speaker_Gender_Recognition_meld` | 1,000 | 226.42 | 213 | 241 |\n| `Speaker_Intent_Classification_slurp` | 662 | 268.52 | 232 | 295 |\n| `Speaker_Number_Verification_voxceleb1` | 314 | 208.24 | 194 | 221 |\n| `Speech_Entity_Reconition_slurp` | 1,000 | 253.14 | 226 | 316 |\n| `Speech_Grounding_librispeech` | 981 | 253.92 | 230 | 282 |\n| `Spoken_Language_Identification_covost2` | 495 | 207.12 | 191 | 232 |\n| `Synthesized_Voice_Detection_fake_or_real` | 1,000 | 224.79 | 203 | 242 |\n| `Acoustic_Scene_Classification_CochlScene` | 1,000 | 240.97 | 213 | 278 |\n| `Acoustic_Scene_Classification_TUT2017` | 1,000 | 241.72 | 207 | 284 |\n| `Audio_Grounding_AudioGrounding` | 896 | 273.74 | 249 | 321 |\n| `Sound_AQA_avqa` | 1,000 | 227.21 | 193 | 298 |\n| `Sound_AQA_clothoaqa` | 1,000 | 199.89 | 179 | 262 |\n| `vocal_sound_classification_VocalSound` | 985 | 232.62 | 210 | 253 |\n| `Music_AQA_music_avqa` | 814 | 208.7 | 185 | 238 |\n| `Music_Genre_Recognition_MTJ-Jamendo` | 1,000 | 223.84 | 200 | 248 |\n| `Music_Genre_Recognition_fma` | 1,000 | 224.59 | 201 | 250 |\n| `Music_Instruments_Classfication_MTJ-Jamendo` | 1,000 | 236.52 | 218 | 262 |\n| `Music_Instruments_Classfication_nsynth` | 996 | 228.12 | 216 | 247 |\n| `Music_Midi_Pitch_Analysis_nsynth` | 493 | 253.88 | 243 | 264 |\n| `Music_Midi_Velocity_Analysis_nsynth` | 484 | 270.6 | 259 | 279 |\n| `Music_Mood_Recognition_MTJ-Jamendo` | 526 | 229.67 | 210 | 248 |\n\n**Audio Statistics:**\n\n| Metric | Value |\n|--------|-------|\n| Total Audio Files | 21,426 |\n| Audio per Sample | min: 1, max: 1, mean: 1 |\n| Formats | mp3, wav |\n\n\n## Sample Example\n\n**Subset**: `Speaker_Age_Prediction_common_voice_13.0_en`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"544443c0\",\n      \"content\": [\n        {\n          \"audio\": \"[BASE64_AUDIO: mp3, ~25.9KB]\",\n          \"format\": \"mp3\"\n        },\n        {\n          \"text\": \"Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\\nWhich age range do you believe best matches the speaker's voice?\\nA. teens to twenties\\nB. thirties to fourties\\nC. fifties to sixties\\nD. seventies to eighties\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"B\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"subset_key\": \"Speaker_Age_Prediction_common_voice_13.0_en\",\n  \"metadata\": {\n    \"uniq_id\": 5973,\n    \"task_name\": \"Speaker_Age_Prediction\",\n    \"dataset_name\": \"common_voice_13.0_en\",\n    \"category\": \"speech\",\n    \"answer_gt_text\": \"thirties to fourties\",\n    \"choices\": {\n      \"A\": \"teens to twenties\",\n      \"B\": \"thirties to fourties\",\n      \"C\": \"fifties to sixties\",\n      \"D\": \"seventies to eighties\"\n    }\n  }\n}\n```\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nChoose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\n```\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `subsets` | `list` | `None` | Optional list of Foundation source-dataset directories to evaluate. Defaults to all 25 directories. Useful when only a subset has been downloaded locally. |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets air_bench_foundation \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['air_bench_foundation'],\n    dataset_args={\n        'air_bench_foundation': {\n            # subset_list: ['Speaker_Age_Prediction_common_voice_13.0_en', 'Speaker_Emotion_Recontion_iemocap', 'Speaker_Emotion_Recontion_meld']  # optional, evaluate specific subsets\n            # extra_params: {}  # uses default extra parameters\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# AIR-Bench-Foundation\n\n\n## 概述\n\nAIR-Bench Foundation 是 [AIR-Bench](https://arxiv.org/abs/2402.07729)（Audio InstRuction Benchmark，ACL 2024 主会）的判别任务部分——这是首个面向大型音频-语言模型（LALMs）的指令遵循基准测试，涵盖**人类语音、自然声音和音乐**。Foundation 轨道包含约 2.5 万道单选题，横跨三大音频类别中的 19 项逻辑任务。\n\n## 任务描述\n\n- **任务类型**：基于音频片段的单选问答。\n- **输入**：一段音频 + 一道包含最多四个候选答案（A/B/C/D）的问题。\n- **输出**：从给定选项中选择一个字母。\n\n## 类别（19 项任务 / 25 个源数据集子集）\n\n- **语音**（11 个子集 / 9 项任务）：语音定位、语言识别、性别识别、情感识别（IEMOCAP+MELD）、年龄预测、语音实体识别、意图分类、说话人数量统计、合成语音检测。\n- **声音**（6 个子集 / 4 项任务）：音频定位、人声分类、声学场景分类（CochlScene+TUT2017）、声音问答（avqa+clothoaqa）。\n- **音乐**（8 个子集 / 6 项任务）：乐器识别、流派识别、MIDI 音高分析、MIDI 力度分析、音乐问答、音乐情感识别。\n\n## 提示模板（与官方 `Inference_Foundation.py` 一致）\n\n```text\nChoose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\n{question}\nA. {choice_a}\nB. {choice_b}\nC. {choice_c}\nD. {choice_d}\n```\n\n## 数据集获取\n\n- 该数据集托管于 ModelScope：[`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset)。采用 *audiofolder + JSON metadata* 的布局。evalscope 在首次运行时通过 `modelscope.dataset_snapshot_download` 按需下载；完整数据集约 49 GB，建议通过 `extra_params` 参数限制下载的子集。\n- 如果数据集已存在于本地磁盘，请传入 `dataset_args={'air_bench_foundation': {'local_path': '/path/to/AIR-Bench-Dataset'}}`；本地根目录应包含 `Foundation/` 文件夹。\n- 部分 Foundation 样本为 FLAC 格式。为兼容 OpenAI 风格的音频输入，evalscope 会将其转换为缓存的 WAV 文件，这需要安装 `soundfile`（`pip install \"evalscope[air_bench]\"`）或系统中存在可用的 `ffmpeg` 二进制文件。\n\n## 评估说明\n\n- **指标**：**准确率**（按源数据集子集计算，并按类别聚合）。\n- 默认提示模板遵循官方 `Inference_Foundation.py` 的格式，因此可复现现有的 AIR-Bench 排行榜结果。\n- 设置 `extra_params={'subsets': [...]}` 可限制评估范围至 25 个源目录中的部分子集——适用于仅下载了部分数据的情况。\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `air_bench_foundation` |\n| **数据集 ID** | [evalscope/AIR-Bench](https://modelscope.cn/datasets/evalscope/AIR-Bench/summary) |\n| **论文** | [Paper](https://aclanthology.org/2024.acl-long.109/) |\n| **标签** | `Audio`, `Knowledge`, `MCQ` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `test` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 21,426 |\n| 提示词长度（平均） | 236.68 字符 |\n| 提示词长度（最小/最大） | 179 / 321 字符 |\n\n**各子集统计信息：**\n\n| 子集 | 样本数 | 提示平均长度 | 提示最小长度 | 提示最大长度 |\n|--------|---------|-------------|------------|------------|\n| `Speaker_Age_Prediction_common_voice_13.0_en` | 1,000 | 291.98 | 278 | 305 |\n| `Speaker_Emotion_Recontion_iemocap` | 1,000 | 227.13 | 218 | 238 |\n| `Speaker_Emotion_Recontion_meld` | 1,000 | 233.21 | 218 | 250 |\n| `Speaker_Gender_Recognition_common_voice_13_en` | 780 | 226.19 | 213 | 241 |\n| `Speaker_Gender_Recognition_meld` | 1,000 | 226.42 | 213 | 241 |\n| `Speaker_Intent_Classification_slurp` | 662 | 268.52 | 232 | 295 |\n| `Speaker_Number_Verification_voxceleb1` | 314 | 208.24 | 194 | 221 |\n| `Speech_Entity_Reconition_slurp` | 1,000 | 253.14 | 226 | 316 |\n| `Speech_Grounding_librispeech` | 981 | 253.92 | 230 | 282 |\n| `Spoken_Language_Identification_covost2` | 495 | 207.12 | 191 | 232 |\n| `Synthesized_Voice_Detection_fake_or_real` | 1,000 | 224.79 | 203 | 242 |\n| `Acoustic_Scene_Classification_CochlScene` | 1,000 | 240.97 | 213 | 278 |\n| `Acoustic_Scene_Classification_TUT2017` | 1,000 | 241.72 | 207 | 284 |\n| `Audio_Grounding_AudioGrounding` | 896 | 273.74 | 249 | 321 |\n| `Sound_AQA_avqa` | 1,000 | 227.21 | 193 | 298 |\n| `Sound_AQA_clothoaqa` | 1,000 | 199.89 | 179 | 262 |\n| `vocal_sound_classification_VocalSound` | 985 | 232.62 | 210 | 253 |\n| `Music_AQA_music_avqa` | 814 | 208.7 | 185 | 238 |\n| `Music_Genre_Recognition_MTJ-Jamendo` | 1,000 | 223.84 | 200 | 248 |\n| `Music_Genre_Recognition_fma` | 1,000 | 224.59 | 201 | 250 |\n| `Music_Instruments_Classfication_MTJ-Jamendo` | 1,000 | 236.52 | 218 | 262 |\n| `Music_Instruments_Classfication_nsynth` | 996 | 228.12 | 216 | 247 |\n| `Music_Midi_Pitch_Analysis_nsynth` | 493 | 253.88 | 243 | 264 |\n| `Music_Midi_Velocity_Analysis_nsynth` | 484 | 270.6 | 259 | 279 |\n| `Music_Mood_Recognition_MTJ-Jamendo` | 526 | 229.67 | 210 | 248 |\n\n**音频统计信息：**\n\n| 指标 | 值 |\n|--------|-------|\n| 音频文件总数 | 21,426 |\n| 每样本音频数量 | 最小: 1, 最大: 1, 平均: 1 |\n| 格式 | mp3, wav |\n\n\n## 样例示例\n\n**子集**: `Speaker_Age_Prediction_common_voice_13.0_en`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"544443c0\",\n      \"content\": [\n        {\n          \"audio\": \"[BASE64_AUDIO: mp3, ~25.9KB]\",\n          \"format\": \"mp3\"\n        },\n        {\n          \"text\": \"Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\\nWhich age range do you believe best matches the speaker's voice?\\nA. teens to twenties\\nB. thirties to fourties\\nC. fifties to sixties\\nD. seventies to eighties\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"B\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"subset_key\": \"Speaker_Age_Prediction_common_voice_13.0_en\",\n  \"metadata\": {\n    \"uniq_id\": 5973,\n    \"task_name\": \"Speaker_Age_Prediction\",\n    \"dataset_name\": \"common_voice_13.0_en\",\n    \"category\": \"speech\",\n    \"answer_gt_text\": \"thirties to fourties\",\n    \"choices\": {\n      \"A\": \"teens to twenties\",\n      \"B\": \"thirties to fourties\",\n      \"C\": \"fifties to sixties\",\n      \"D\": \"seventies to eighties\"\n    }\n  }\n}\n```\n\n## 提示模板\n\n**提示模板：**\n```text\nChoose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\n```\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `subsets` | `list` | `None` | 可选的 Foundation 源数据集目录列表，用于指定评估范围。默认为全部 25 个目录。当本地仅下载了部分子集时非常有用。 |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets air_bench_foundation \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['air_bench_foundation'],\n    dataset_args={\n        'air_bench_foundation': {\n            # subset_list: ['Speaker_Age_Prediction_common_voice_13.0_en', 'Speaker_Emotion_Recontion_iemocap', 'Speaker_Emotion_Recontion_meld']  # 可选，评估特定子集\n            # extra_params: {}  # 使用默认额外参数\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "65ab941d81be7057eb6d4052b715a11c",
    "needs_translation": false
  },
  "updated_at": "2026-05-18T10:23:16.650501",
  "translation_updated_at": "2026-05-18T10:23:22"
}