MedXpertQA#

概述#

MedXpertQA 是一个专家级医学多项选择基准测试，旨在评估高级医学知识与推理能力。该基准包含独立的纯文本（Text-only）和多模态（Multimodal, MM）两个赛道，题目源自具有挑战性的医学考试试题，并经由持证医师审核。

任务描述#

  • 任务类型：单答案医学多项选择题

  • 输入：一道临床或生物医学问题及其选项，可选附带最多六张图像

  • 输出：一个答案字母（Text 赛道为 A-J，MM 赛道为 A-E）

  • 领域：涵盖17个医学专科和11个人体系统

主要特点#

  • 测试集包含4,450道题目：其中2,450道为Text题目（含十个选项），2,000道为MM题目（含五个选项）

  • MM赛道包含放射影像、病理切片、光学图像、照片、示意图、图表、表格、文档及生命体征图像

  • 所有题目均标注了医学任务类型、人体系统和问题类型；其中3,307道测试题侧重推理能力，1,143道侧重理解能力

  • 题目经过难度筛选、选项增强、数据泄露缓解以及多轮专家评审

评估说明#

  • 主要指标：准确率（Accuracy），通过预测答案字母与标准答案的精确匹配计算

  • 默认提示词采用 EvalScope 的零样本思维链（zero-shot chain-of-thought）模板，保留官方指定的逐步推理指令及严格的答案字母评分格式

  • 应将 max_tokens 设置得足够高，以确保模型能完整输出所需的最终行 ANSWER: [LETTER]；否则，若推理过程被截断，解析器可能回退到提取最后一个有效的大写字母作为答案

  • 结果分别报告 Text 和 MM 子集的表现，并通过样本加权聚合计算整体得分

  • MM图像存储在 images.zip（约517 MB）中，直接从压缩包读取，无需额外解压副本

  • 公开数据集共包含4,460条记录（含10个开发样例）；本集成仅评估其中4,450道预留的测试题

  • 论文 | GitHub

属性#

属性

值

基准测试名称

medxpertqa

数据集ID

evalscope/MedXpertQA

论文

Paper

标签

MCQ, Medical, MultiModal, Reasoning

指标

accuracy

默认示例数

0-shot

评估划分

test

数据统计#

指标

值

总样本数

4,450

提示词长度（平均）

1135.22 字符

提示词长度（最小/最大）

346 / 4771 字符

各子集统计：

子集

样本数

提示词平均长度

提示词最小长度

提示词最大长度

Text

2,450

1337.92

435

4771

MM

2,000

886.91

346

2335

图像统计：

指标

值

图像总数

2,852

每样本图像数

最小: 1, 最大: 6, 平均: 1.43

分辨率范围

323x34 - 4248x2144

格式

jpeg, png

样例示例#

子集: Text

{
  "input": [
    {
      "id": "3f1d2f2a",
      "content": "You are a helpful medical assistant."
    },
    {
      "id": "1a9f9143",
      "content": [
        {
          "text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D,E,F,G,H,I,J. Think step by step before answering.\n\nWhich pat ... [TRUNCATED 885 chars] ... ere posterior wear undergoing shoulder arthroplasty\nI) 58-year-old male with glenoid retroversion of 12-degrees undergoing shoulder arthroplasty\nJ) 55-year-old male with glenoid retroversion of 8-degrees undergoing total shoulder arthroplasty"
        }
      ]
    }
  ],
  "choices": [
    "70-year-old male with glenoid retroversion of 18-degrees undergoing shoulder arthroplasty",
    "70-year-old female with humeral anteversion of 13-degrees undergoing shoulder arthroplasty",
    "63-year-old female with glenoid retroversion of 22-degrees and mild posterior wear undergoing shoulder arthroplasty",
    "65-year-old female with glenoid retroversion of 25-degrees undergoing shoulder arthroplasty",
    "65-year-old female with a glenoid retroversion of 13-degrees undergoing shoulder arthroplasty",
    "68-year-old female with glenoid retroversion of 20-degrees undergoing reverse shoulder arthroplasty",
    "72-year-old male with glenoid retroversion of 15-degrees undergoing shoulder arthroplasty",
    "65-year-old female with glenoid retroversion of 30-degrees and severe posterior wear undergoing shoulder arthroplasty",
    "58-year-old male with glenoid retroversion of 12-degrees undergoing shoulder arthroplasty",
    "55-year-old male with glenoid retroversion of 8-degrees undergoing total shoulder arthroplasty"
  ],
  "target": "E",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "id": "Text-0",
    "medical_task": "Basic Science",
    "body_system": "Skeletal",
    "question_type": "Reasoning",
    "images": []
  }
}

注：部分内容为显示目的已截断。

提示模板#

系统提示：

You are a helpful medical assistant.

提示模板：

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.

{question}

{choices}

使用方法#

使用 CLI#

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets medxpertqa \
    --limit 10  # 正式评估时请删除此行

使用 Python#

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['medxpertqa'],
    dataset_args={
        'medxpertqa': {
            # subset_list: ['Text', 'MM']  # 可选，用于评估特定子集
        }
    },
    limit=10,  # 正式评估时请删除此行
)

run_task(task_cfg=task_cfg)