# VLM Benchmarks

Below is the list of supported VLM benchmarks. Click on a benchmark name for details.

| Benchmark Name | Pretty Name | Task Categories |
|------------|----------|----------|
| `a_okvqa` | [A-OKVQA](../../benchmarks/a_okvqa.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `ai2d` | [AI2D](../../benchmarks/ai2d.md) | `Knowledge`, `MultiModal`, `QA` |
| `air_bench_chat` | [AIR-Bench-Chat](../../benchmarks/air_bench_chat.md) | `Audio`, `InstructionFollowing`, `QA` |
| `air_bench_foundation` | [AIR-Bench-Foundation](../../benchmarks/air_bench_foundation.md) | `Audio`, `Knowledge`, `MCQ` |
| `baby_vision` | [BabyVision](../../benchmarks/baby_vision.md) | `MultiModal`, `QA`, `Reasoning` |
| `blink` | [BLINK](../../benchmarks/blink.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `cc_bench` | [CCBench](../../benchmarks/cc_bench.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `chartqa` | [ChartQA](../../benchmarks/chartqa.md) | `Knowledge`, `MultiModal`, `QA` |
| `charxiv` | [CharXiv](../../benchmarks/charxiv.md) | `MultiModal`, `QA`, `Reasoning` |
| `cmmmu` | [CMMMU](../../benchmarks/cmmmu.md) | `Chinese`, `Knowledge`, `MultiModal`, `QA` |
| `cmmu` | [CMMU](../../benchmarks/cmmu.md) | `Knowledge`, `MCQ`, `MultiModal`, `QA` |
| `common_voice_15` | [CommonVoice15](../../benchmarks/common_voice_15.md) | `Audio`, `MultiLingual`, `SpeechRecognition` |
| `docvqa` | [DocVQA](../../benchmarks/docvqa.md) | `Knowledge`, `MultiModal`, `QA` |
| `erqa` | [ERQA](../../benchmarks/erqa.md) | `MCQ`, `MultiModal`, `Reasoning` |
| `fleurs` | [FLEURS](../../benchmarks/fleurs.md) | `Audio`, `MultiLingual`, `SpeechRecognition` |
| `general_vmcq` | [General-VMCQ](../../benchmarks/general_vmcq.md) | `Custom`, `MCQ`, `MultiModal` |
| `general_vqa` | [General-VQA](../../benchmarks/general_vqa.md) | `Custom`, `MultiModal`, `QA` |
| `gsm8k_v` | [GSM8K-V](../../benchmarks/gsm8k_v.md) | `Math`, `MultiModal`, `Reasoning` |
| `hallusion_bench` | [HallusionBench](../../benchmarks/hallusion_bench.md) | `Hallucination`, `MultiModal`, `Yes/No` |
| `infovqa` | [InfoVQA](../../benchmarks/infovqa.md) | `Knowledge`, `MultiModal`, `QA` |
| `librispeech` | [LibriSpeech](../../benchmarks/librispeech.md) | `Audio`, `SpeechRecognition` |
| `maritime_ocr_bench` | [Maritime-OCR-Bench](../../benchmarks/maritime_ocr_bench.md) | `MultiModal`, `QA` |
| `math_verse` | [MathVerse](../../benchmarks/math_verse.md) | `MCQ`, `Math`, `MultiModal`, `Reasoning` |
| `math_vision` | [MathVision](../../benchmarks/math_vision.md) | `MCQ`, `Math`, `MultiModal`, `Reasoning` |
| `math_vista` | [MathVista](../../benchmarks/math_vista.md) | `MCQ`, `Math`, `MultiModal`, `Reasoning` |
| `mia_bench` | [MIA-Bench](../../benchmarks/mia_bench.md) | `InstructionFollowing`, `MultiModal`, `QA` |
| `micro_vqa` | [MicroVQA](../../benchmarks/micro_vqa.md) | `Knowledge`, `MCQ`, `Medical`, `MultiModal` |
| `mm_bench` | [MMBench](../../benchmarks/mm_bench.md) | `Knowledge`, `MultiModal`, `QA` |
| `mm_star` | [MMStar](../../benchmarks/mm_star.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `mmau` | [MMAU](../../benchmarks/mmau.md) | `Audio`, `MCQ` |
| `mmmu` | [MMMU](../../benchmarks/mmmu.md) | `Knowledge`, `MultiModal`, `QA` |
| `mmmu_pro` | [MMMU-PRO](../../benchmarks/mmmu_pro.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `msr_vtt` | [MSR-VTT](../../benchmarks/msr_vtt.md) | `ImageCaptioning`, `MultiModal` |
| `msvd` | [MSVD](../../benchmarks/msvd.md) | `ImageCaptioning`, `MultiModal` |
| `mvbench` | [MVBench](../../benchmarks/mvbench.md) | `MCQ`, `MultiModal` |
| `ocr_bench` | [OCRBench](../../benchmarks/ocr_bench.md) | `Knowledge`, `MultiModal`, `QA` |
| `ocr_bench_v2` | [OCRBench-v2](../../benchmarks/ocr_bench_v2.md) | `Knowledge`, `MultiModal`, `QA` |
| `olympiad_bench` | [OlympiadBench](../../benchmarks/olympiad_bench.md) | `Math`, `Reasoning` |
| `omni_bench` | [OmniBench](../../benchmarks/omni_bench.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `omni_doc_bench` | [OmniDocBench](../../benchmarks/omni_doc_bench.md) | `Knowledge`, `MultiModal`, `QA` |
| `pope` | [POPE](../../benchmarks/pope.md) | `Hallucination`, `MultiModal`, `Yes/No` |
| `real_world_qa` | [RealWorldQA](../../benchmarks/real_world_qa.md) | `Knowledge`, `MultiModal`, `QA` |
| `science_qa` | [ScienceQA](../../benchmarks/science_qa.md) | `Knowledge`, `MCQ`, `MultiModal` |
| `seed_bench_2_plus` | [SEED-Bench-2-Plus](../../benchmarks/seed_bench_2_plus.md) | `Knowledge`, `MCQ`, `MultiModal`, `Reasoning` |
| `simple_vqa` | [SimpleVQA](../../benchmarks/simple_vqa.md) | `MultiModal`, `QA`, `Reasoning` |
| `tir_bench` | [TIR-Bench](../../benchmarks/tir_bench.md) | `MultiModal`, `QA`, `Reasoning` |
| `torgo` | [TORGO](../../benchmarks/torgo.md) | `Audio`, `SpeechRecognition` |
| `videomme_v2` | [Video-MME-v2](../../benchmarks/videomme_v2.md) | `MCQ`, `MultiModal` |
| `visulogic` | [VisuLogic](../../benchmarks/visulogic.md) | `MCQ`, `Math`, `MultiModal`, `Reasoning` |
| `vqav2` | [VQAv2](../../benchmarks/vqav2.md) | `MultiModal`, `QA` |
| `vstar_bench` | [V*Bench](../../benchmarks/vstar_bench.md) | `Grounding`, `MCQ`, `MultiModal` |
| `wenet_speech` | [WenetSpeech](../../benchmarks/wenet_speech.md) | `Audio`, `SpeechRecognition` |
| `world_vqa` | [WorldVQA](../../benchmarks/world_vqa.md) | `Knowledge`, `MultiModal`, `QA` |
| `zerobench` | [ZeroBench](../../benchmarks/zerobench.md) | `Knowledge`, `MultiModal`, `QA` |

:::{toctree}
:hidden:
:maxdepth: 1

../../benchmarks/a_okvqa.md
../../benchmarks/ai2d.md
../../benchmarks/air_bench_chat.md
../../benchmarks/air_bench_foundation.md
../../benchmarks/baby_vision.md
../../benchmarks/blink.md
../../benchmarks/cc_bench.md
../../benchmarks/chartqa.md
../../benchmarks/charxiv.md
../../benchmarks/cmmmu.md
../../benchmarks/cmmu.md
../../benchmarks/common_voice_15.md
../../benchmarks/docvqa.md
../../benchmarks/erqa.md
../../benchmarks/fleurs.md
../../benchmarks/general_vmcq.md
../../benchmarks/general_vqa.md
../../benchmarks/gsm8k_v.md
../../benchmarks/hallusion_bench.md
../../benchmarks/infovqa.md
../../benchmarks/librispeech.md
../../benchmarks/maritime_ocr_bench.md
../../benchmarks/math_verse.md
../../benchmarks/math_vision.md
../../benchmarks/math_vista.md
../../benchmarks/mia_bench.md
../../benchmarks/micro_vqa.md
../../benchmarks/mm_bench.md
../../benchmarks/mm_star.md
../../benchmarks/mmau.md
../../benchmarks/mmmu.md
../../benchmarks/mmmu_pro.md
../../benchmarks/msr_vtt.md
../../benchmarks/msvd.md
../../benchmarks/mvbench.md
../../benchmarks/ocr_bench.md
../../benchmarks/ocr_bench_v2.md
../../benchmarks/olympiad_bench.md
../../benchmarks/omni_bench.md
../../benchmarks/omni_doc_bench.md
../../benchmarks/pope.md
../../benchmarks/real_world_qa.md
../../benchmarks/science_qa.md
../../benchmarks/seed_bench_2_plus.md
../../benchmarks/simple_vqa.md
../../benchmarks/tir_bench.md
../../benchmarks/torgo.md
../../benchmarks/videomme_v2.md
../../benchmarks/visulogic.md
../../benchmarks/vqav2.md
../../benchmarks/vstar_bench.md
../../benchmarks/wenet_speech.md
../../benchmarks/world_vqa.md
../../benchmarks/zerobench.md
:::