Benchmark

`get_data`

Signature

@app.tool(output="benchmark->q_ls,gt_ls")
def get_data(benchmark: Dict[str, Any]) -> Dict[str, List[Any]]

Function

Multi-format Loading: Supports loading evaluation datasets in .jsonl, .json, or .parquet formats from local storage.
Dynamic Field Mapping: Uses key_map to map different column names in raw data (such as question, answer) to standardized output keys (usually q_ls and gt_ls).
Data Preprocessing: Built-in support for random shuffling (shuffle) and quantity truncation (limit).
Used in Demos to receive user input, treating it as a piece of data (q_ls).

Output Format (JSON)

{
  "q_ls": ["Question 1", "Question 2"],
  "gt_ls": [["Answer A1", "Answer A2"], ["Answer B"]]
}

servers/benchmark/parameter.yaml

benchmark:
  name: nq
  path: data/sample_nq_10.jsonl
  key_map:
    q_ls: question
    gt_ls: golden_answers
  shuffle: false
  seed: 42
  limit: -1

Parameter Description:

Parameter	Type	Description
`name`	str	Evaluation set name, used only for logging and identification (Example: `nq`)
`path`	str	Data file path, supports `.jsonl`, `.json`, `.parquet`
`key_map`	dict	Field mapping table, mapping raw fields to tool output keys
	`q_ls`	str	Raw field name mapped to Question List (e.g., question column in file)
	`gt_ls`	str	Raw field name mapped to Golden Answer List (e.g., golden_answers column in file)
`shuffle`	bool	Whether to shuffle sample order (default `false`)
`seed`	int	Random seed (effective when `shuffle=true`)
`limit`	int	Upper limit of sampled data items. Default is -1 (load all), positive integer means truncation of first N items