跳转到主要内容
我们整理并预处理了当前 RAG 研究中最常用的公开评测数据集和语料库,并已在 ModelScopeHuggingface 上同步发布。 用户可直接下载使用,无需额外清洗或转换,即可与 UltraRAG 的评测管线无缝对接。

Benchmark

下表汇总了当前已支持的任务类型及对应数据集的统计信息:
任务类型数据集名称原始数据数量评测采样数量
QANQ3,6101,000
QATriviaQA11,3131,000
QAPopQA14,2671,000
QAAmbigQA2,0021,000
QAMarcoQA55,6361,000
QAWebQuestions2,0321,000
VQAMP-DocVQA591591
VQAChartQA6363
VQAInfoVQA718718
VQAPlotQA863863
Multi-hop QAHotpotQA7,4051,000
Multi-hop QA2WikiMultiHopQA12,5761,000
Multi-hop QAMusique2,4171,000
Multi-hop QABamboogle125125
Multi-hop QAStrategyQA2,2901,000
Multi-hop VQASlideVQA556556
Multiple-choiceARC3,5481,000
Multiple-choiceMMLU14,0421,000
Multiple-choice VQAArXivQA816816
Long-form QAASQA948948
Fact-verificationFEVER13,3321,000
DialogueWoW3,0541,000
Slot-fillingT-REx5,0001,000
数据格式规范 为了确保与 UltraRAG 各模块完全兼容,建议用户将测试数据统一存储为 .jsonl 文件,并遵循以下格式规范。 非选择题数据格式:
/images/json.svg
{
  "id": 0, 
  "question": "where does the karate kid 2010 take place", 
  "golden_answers": ["China", "Beijing", "Beijing, China"], 
  "meta_data": {} 
}
选择题数据格式:
/images/json.svg
{
  "id": 0, 
  "question": "Mast Co. converted from the FIFO method for inventory valuation to the LIFO method for financial statement and tax purposes. During a period of inflation would Mast's ending inventory and income tax payable using LIFO be higher or lower than FIFO? Ending inventory Income tax payable", 
  "golden_answers": ["A"], 
  "choices": ["Lower Lower", "Higher Higher", "Lower Higher", "Higher Lower"], 
  "meta_data": {"subject": "professional_accounting"}
}

Corpus

UltraRAG 提供了多源、高质量的标准化语料库,涵盖文本与图像两种模态,便于构建多场景 RAG 系统。 以下为当前已收录语料的统计信息:
语料库名称文档数量
Wiki-201821,015,324
Wiki-202430,463,973
MP-DocVQA741
ChartQA500
InfoVQA459
PlotQA9,593
SlideVQA1,284
ArXivQA8,066
数据格式规范 文本语料格式:
/images/json.svg
{
  "id": "15106858", 
  "contents": "Arrowhead Stadium 1970s practice would eventually spread to the other NFL stadiums as the 1970s progressed, finally becoming mandatory league-wide in the 1978 season (after being used in Super Bowl XII), and become almost near-universal at the lower levels of football. On January 20, 1974, Arrowhead Stadium hosted the Pro Bowl. Due to an ice storm and brutally cold temperatures the week leading up to the game, the game's participants worked out at the facilities of the San Diego Chargers. On game day, the temperature soared to 41 F, melting most of the ice and snow that accumulated during the week. The AFC defeated the NFC, 15–13."
}
图像语料格式:
/images/json.svg
{
  "id": 0, 
  "image_id": "37313.jpeg", 
  "image_path": "image/37313.jpg"
}
I