VLM评测集#

以下是支持的VLM评测集列表，点击数据集标准名称可跳转详细信息。

数据集名称

标准名称

任务类别

ai2d

AI2D

Knowledge, MultiModal, QA

cc_bench

CCBench

Knowledge, MCQ, MultiModal

math_vista

MathVista

MCQ, Math, MultiModal, Reasoning

mm_bench

MMBench

Knowledge, MultiModal, QA

mm_star

MMStar

Knowledge, MCQ, MultiModal

mmmu

MMMU

Knowledge, MultiModal, QA

mmmu_pro

MMMU-PRO

Knowledge, MCQ, MultiModal

olympiad_bench

OlympiadBench

Math, Reasoning

omni_bench

OmniBench

Knowledge, MCQ, MultiModal

real_world_qa

RealWorldQA

Knowledge, MultiModal, QA

数据集详情#

AI2D#

返回目录

  • 数据集名称: ai2d

  • 数据集ID: lmms-lab/ai2d

  • 数据集描述:

    A Diagram Is Worth A Dozen Images

  • 任务类别: Knowledge, MultiModal, QA

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: default

  • 提示模板:

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

{question}

{choices}

CCBench#

返回目录

  • 数据集名称: cc_bench

  • 数据集ID: lmms-lab/MMBench

  • 数据集描述:

    CCBench is an extension of MMBench with newly design questions about Chinese traditional culture, including Calligraphy Painting, Cultural Relic, Food & Clothes, Historical Figures, Scenery & Building, Sketch Reasoning and Traditional Show.

  • 任务类别: Knowledge, MCQ, MultiModal

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: cc

  • 提示模板:

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

{question}

{choices}

MathVista#

返回目录

  • 数据集名称: math_vista

  • 数据集ID: evalscope/MathVista

  • 数据集描述:

    MathVista is a consolidated Mathematical reasoning benchmark within Visual contexts. It consists of three newly created datasets, IQTest, FunctionQA, and PaperQA, which address the missing visual domains and are tailored to evaluate logical reasoning on puzzle test figures, algebraic reasoning over functional plots, and scientific reasoning with academic paper figures, respectively. It also incorporates 9 MathQA datasets and 19 VQA datasets from the literature, which significantly enrich the diversity and complexity of visual perception and mathematical reasoning challenges within our benchmark. In total, MathVista includes 6,141 examples collected from 31 different datasets.

  • 任务类别: MCQ, Math, MultiModal, Reasoning

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: default

  • 提示模板:

Solve the following problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{question}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \boxed command.

MMBench#

返回目录

  • 数据集名称: mm_bench

  • 数据集ID: lmms-lab/MMBench

  • 数据集描述:

    MMBench is a comprehensive evaluation pipeline comprised of meticulously curated multimodal dataset and a novel circulareval strategy using ChatGPT. It is comprised of 20 ability dimensions defined by MMBench. It also contains chinese version with translated question.

  • 任务类别: Knowledge, MultiModal, QA

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: cn, en

  • 提示模板:

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

{question}

{choices}

MMStar#

返回目录

  • 数据集名称: mm_star

  • 数据集ID: evalscope/MMStar

  • 数据集描述:

    MMStar: an elite vision-indispensible multi-modal benchmark, aiming to ensure each curated sample exhibits visual dependency, minimal data leakage, and requires advanced multi-modal capabilities.

  • 任务类别: Knowledge, MCQ, MultiModal

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: coarse perception, fine-grained perception, instance reasoning, logical reasoning, math, science & technology

  • 提示模板:

Answer the following multiple choice question.
The last line of your response should be of the following format:
'ANSWER: $LETTER' (without quotes)
where LETTER is one of A,B,C,D. Think step by step before answering.

{question}

MMMU#

返回目录

  • 数据集名称: mmmu

  • 数据集ID: AI-ModelScope/MMMU

  • 数据集描述:

    MMMU (A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI) benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures.

  • 任务类别: Knowledge, MultiModal, QA

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: Accounting, Agriculture, Architecture_and_Engineering, Art_Theory, Art, Basic_Medical_Science, Biology, Chemistry, Clinical_Medicine, Computer_Science, Design, Diagnostics_and_Laboratory_Medicine, Economics, Electronics, Energy_and_Power, Finance, Geography, History, Literature, Manage, Marketing, Materials, Math, Mechanical_Engineering, Music, Pharmacy, Physics, Psychology, Public_Health, Sociology

  • 提示模板:

Solve the following problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{question}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \boxed command.

MMMU-PRO#

返回目录

  • 数据集名称: mmmu_pro

  • 数据集ID: AI-ModelScope/MMMU_Pro

  • 数据集描述:

    MMMU-Pro is an enhanced multimodal benchmark designed to rigorously assess the true understanding capabilities of advanced AI models across multiple modalities. It builds upon the original MMMU benchmark by introducing several key improvements that make it more challenging and realistic, ensuring that models are evaluated on their genuine ability to integrate and comprehend both visual and textual information.

  • 任务类别: Knowledge, MCQ, MultiModal

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: Accounting, Agriculture, Architecture_and_Engineering, Art_Theory, Art, Basic_Medical_Science, Biology, Chemistry, Clinical_Medicine, Computer_Science, Design, Diagnostics_and_Laboratory_Medicine, Economics, Electronics, Energy_and_Power, Finance, Geography, History, Literature, Manage, Marketing, Materials, Math, Mechanical_Engineering, Music, Pharmacy, Physics, Psychology, Public_Health, Sociology

  • 额外参数:

{
    "dataset_format": "# choose from ['standard (4 options)', 'standard (10 options)', 'vision'], default 'standard (4 options)'"
}

  • 提示模板:

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

{question}

{choices}

OlympiadBench#

返回目录

  • 数据集名称: olympiad_bench

  • 数据集ID: AI-ModelScope/OlympiadBench

  • 数据集描述:

    OlympiadBench is an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. In the subsets: OE stands for Open-Ended, TP stands for Theorem Proving, MM stands for Multimodal, TO stands for Text-Only, CEE stands for Chinese Entrance Exam, COMP stands for Comprehensive. Note: The TP subsets can't be evaluated with auto-judge for now.

  • 任务类别: Math, Reasoning

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: OE_MM_maths_en_COMP, OE_MM_maths_zh_CEE, OE_MM_maths_zh_COMP, OE_MM_physics_en_COMP, OE_MM_physics_zh_CEE, OE_TO_maths_en_COMP, OE_TO_maths_zh_CEE, OE_TO_maths_zh_COMP, OE_TO_physics_en_COMP, OE_TO_physics_zh_CEE, TP_MM_maths_en_COMP, TP_MM_maths_zh_CEE, TP_MM_maths_zh_COMP, TP_MM_physics_en_COMP, TP_TO_maths_en_COMP, TP_TO_maths_zh_CEE, TP_TO_maths_zh_COMP, TP_TO_physics_en_COMP

  • 提示模板:

{question}
Please reason step by step, and put your final answer within \boxed{{}}.

OmniBench#

返回目录

  • 数据集名称: omni_bench

  • 数据集ID: m-a-p/OmniBench

  • 数据集描述:

    OmniBench, a pioneering universal multimodal benchmark designed to rigorously evaluate MLLMs' capability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.

  • 任务类别: Knowledge, MCQ, MultiModal

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: default

  • 额外参数:

{
    "use_image": true,
    "use_audio": true
}

  • 提示模板:

Answer the following multiple choice question based on the image and audio content. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

{question}

{choices}

RealWorldQA#

返回目录

  • 数据集名称: real_world_qa

  • 数据集ID: lmms-lab/RealWorldQA

  • 数据集描述:

    RealWorldQA is a benchmark designed to evaluate the real-world spatial understanding capabilities of multimodal AI models, contributed by XAI. It assesses how well these models comprehend physical environments. The benchmark consists of 700+ images, each accompanied by a question and a verifiable answer. These images are drawn from real-world scenarios, including those captured from vehicles. The goal is to advance AI models' understanding of our physical world.

  • 任务类别: Knowledge, MultiModal, QA

  • 评估指标: acc

  • 需要LLM Judge: 否

  • 默认提示方式: 0-shot

  • 数据集子集: default

  • 提示模板:

Read the picture and solve the following problem step by step.The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{question}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \boxed command.