> ## Documentation Index
> Fetch the complete documentation index at: https://ultrarag.openbmb.cn/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark

## `get_data`

**Signature**

```python theme={null}
@app.tool(output="benchmark->q_ls,gt_ls")
def get_data(benchmark: Dict[str, Any]) -> Dict[str, List[Any]]
```

**Function**

* Multi-format Loading: Supports loading evaluation datasets in `.jsonl`, `.json`, or `.parquet` formats from local storage.
* Dynamic Field Mapping: Uses `key_map` to map different column names in raw data (such as `question`, `answer`) to standardized output keys (usually `q_ls` and `gt_ls`).
* Data Preprocessing: Built-in support for random shuffling (`shuffle`) and quantity truncation (`limit`).
* Used in Demos to receive user input, treating it as a piece of data (`q_ls`).

**Output Format (JSON)**

```json theme={null}
{
  "q_ls": ["Question 1", "Question 2"],
  "gt_ls": [["Answer A1", "Answer A2"], ["Answer B"]]
}
```

***

## Configuration

```yaml servers/benchmark/parameter.yaml icon="https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/yaml.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=69b41e79144bc908039c2ee3abbb1c3b" theme={null}
benchmark:
  name: nq
  path: data/sample_nq_10.jsonl
  key_map:
    q_ls: question
    gt_ls: golden_answers
  shuffle: false
  seed: 42
  limit: -1
```

Parameter Description:

| Parameter | Type    | Description                                                                                                     |                                                                                    |
| --------- | ------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `name`    | str     | Evaluation set name, used only for logging and identification (Example: `nq`)                                   |                                                                                    |
| `path`    | str     | Data file path, supports `.jsonl`, `.json`, `.parquet`                                                          |                                                                                    |
| `key_map` | dict    | Field mapping table, mapping raw fields to tool output keys                                                     |                                                                                    |
|           | `q_ls`  | str                                                                                                             | Raw field name mapped to Question List (e.g., question column in file)             |
|           | `gt_ls` | str                                                                                                             | Raw field name mapped to Golden Answer List (e.g., golden\_answers column in file) |
| `shuffle` | bool    | Whether to shuffle sample order (default `false`)                                                               |                                                                                    |
| `seed`    | int     | Random seed (effective when `shuffle=true`)                                                                     |                                                                                    |
| `limit`   | int     | Upper limit of sampled data items. Default is -1 (load all), positive integer means truncation of first N items |                                                                                    |