> ## Documentation Index > Fetch the complete documentation index at: https://ultrarag.openbmb.cn/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation Data We have organized and preprocessed the most commonly used public evaluation datasets and corpora in current RAG research, and have released them synchronously on [ModelScope](https://modelscope.cn/datasets/UltraRAG/UltraRAG_Benchmark) and [Huggingface](https://huggingface.co/datasets/UltraRAG/UltraRAG_Benchmark). Users can download and use them directly without additional cleaning or conversion, seamlessly interfacing with UltraRAG's evaluation pipeline. ## Benchmark The following table summarizes the currently supported task types and statistical information of corresponding datasets: | Task Type | Dataset Name | Raw Data Count | Eval Sample Count | | :------------------ | :------------------------------------------------------------------------------------------------------------------------------------------ | :------------- | :---------------- | | QA | [NQ](https://huggingface.co/datasets/google-research-datasets/nq_open) | 3,610 | 1,000 | | QA | [TriviaQA](https://nlp.cs.washington.edu/triviaqa/) | 11,313 | 1,000 | | QA | [PopQA](https://huggingface.co/datasets/akariasai/PopQA) | 14,267 | 1,000 | | QA | [AmbigQA](https://huggingface.co/datasets/sewon/ambig_qa) | 2,002 | 1,000 | | QA | [MarcoQA](https://huggingface.co/datasets/microsoft/ms_marco/viewer/v2.1/validation) | 55,636 | 1,000 | | QA | [WebQuestions](https://huggingface.co/datasets/stanfordnlp/web_questions) | 2,032 | 1,000 | | VQA | [MP-DocVQA](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-MP-DocVQA) | 591 | 591 | | VQA | [ChartQA](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-ChartQA) | 63 | 63 | | VQA | [InfoVQA](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-InfoVQA) | 718 | 718 | | VQA | [PlotQA](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-PlotQA) | 863 | 863 | | Multi-hop QA | [HotpotQA](https://huggingface.co/datasets/hotpotqa/hotpot_qa) | 7,405 | 1,000 | | Multi-hop QA | [2WikiMultiHopQA](https://www.dropbox.com/scl/fi/heid2pkiswhfaqr5g0piw/data.zip?e=2\&file_subpath=%2Fdata\&rlkey=ira57daau8lxfj022xvk1irju) | 12,576 | 1,000 | | Multi-hop QA | [Musique](https://drive.google.com/file/d/1tGdADlNjWFaHLeZZGShh2IRcpO6Lv24h/view) | 2,417 | 1,000 | | Multi-hop QA | [Bamboogle](https://huggingface.co/datasets/chiayewken/bamboogle) | 125 | 125 | | Multi-hop QA | [StrategyQA](https://huggingface.co/datasets/tasksource/strategy-qa) | 2,290 | 1,000 | | Multi-hop VQA | [SlideVQA](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-SlideVQA) | 556 | 556 | | Multiple-choice | [ARC](https://huggingface.co/datasets/allenai/ai2_arc) | 3,548 | 1,000 | | Multiple-choice | [MMLU](https://huggingface.co/datasets/cais/mmlu) | 14,042 | 1,000 | | Multiple-choice VQA | [ArXivQA](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-ArxivQA) | 816 | 816 | | Long-form QA | [ASQA](https://huggingface.co/datasets/din0s/asqa) | 948 | 948 | | Fact-verification | [FEVER](https://fever.ai/dataset/fever.html) | 13,332 | 1,000 | | Dialogue | [WoW](https://huggingface.co/datasets/facebook/kilt_tasks) | 3,054 | 1,000 | | Slot-filling | [T-REx](https://huggingface.co/datasets/facebook/kilt_tasks) | 5,000 | 1,000 | **Data Format Specification** To ensure full compatibility with UltraRAG modules, it is recommended that users store test data uniformly as .jsonl files and follow the format specifications below. Non-multiple-choice data format: ```json icon="https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/json.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=81a8c440100333f3454ca984a5b0fe5a" theme={null} { "id": 0, "question": "where does the karate kid 2010 take place", "golden_answers": ["China", "Beijing", "Beijing, China"], "meta_data": {} } ``` Multiple-choice data format: ```json icon="https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/json.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=81a8c440100333f3454ca984a5b0fe5a" theme={null} { "id": 0, "question": "Mast Co. converted from the FIFO method for inventory valuation to the LIFO method for financial statement and tax purposes. During a period of inflation would Mast's ending inventory and income tax payable using LIFO be higher or lower than FIFO? Ending inventory Income tax payable", "golden_answers": ["A"], "choices": ["Lower Lower", "Higher Higher", "Lower Higher", "Higher Lower"], "meta_data": {"subject": "professional_accounting"} } ``` ## Corpus UltraRAG provides multi-source, high-quality standardized corpora, covering both text and image modalities, facilitating the construction of multi-scenario RAG systems. The following is statistical information on currently collected corpora: | Corpus Name | Document Count | | :---------- | :------------- | | Wiki-2018 | 21,015,324 | | Wiki-2024 | 30,463,973 | | MP-DocVQA | 741 | | ChartQA | 500 | | InfoVQA | 459 | | PlotQA | 9,593 | | SlideVQA | 1,284 | | ArXivQA | 8,066 | **Data Format Specification** Text corpus format: ```json icon="https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/json.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=81a8c440100333f3454ca984a5b0fe5a" theme={null} { "id": "15106858", "contents": "Arrowhead Stadium 1970s practice would eventually spread to the other NFL stadiums as the 1970s progressed, finally becoming mandatory league-wide in the 1978 season (after being used in Super Bowl XII), and become almost near-universal at the lower levels of football. On January 20, 1974, Arrowhead Stadium hosted the Pro Bowl. Due to an ice storm and brutally cold temperatures the week leading up to the game, the game's participants worked out at the facilities of the San Diego Chargers. On game day, the temperature soared to 41 F, melting most of the ice and snow that accumulated during the week. The AFC defeated the NFC, 15–13." } ``` Image corpus format: ```json icon="https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/json.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=81a8c440100333f3454ca984a5b0fe5a" theme={null} { "id": 0, "image_id": "37313.jpeg", "image_path": "image/37313.jpg" } ```