Skip to main content

​
Function

The Benchmark Server is used to load evaluation datasets, commonly used in the data configuration phase of benchmark testing, Q&A tasks, or generation tasks.
We strongly recommend preprocessing data into .jsonl format.
Example data:
data/sample_nq_10.jsonl

​
Usage Examples

​
Basic Usage

examples/load_data.yaml
Run the following command to compile the Pipeline:
Modify corresponding fields according to the actual situation:
examples/parameters/load_data_parameter.yaml
Run the following command to execute the Pipeline:
After completion, the system will automatically load and output data samples, providing input support for subsequent retrieval and generation tasks.

​
Add Dataset Loading Fields

In some cases, we may not only need to load query and ground_truth fields, but also wish to use other information in the dataset, such as retrieved passage. In this case, you can modify the code of the Benchmark Server to add fields that need to be returned.
You can extend other fields (such as cot, retrieved_passages, etc.) in the same way, just add the corresponding key names synchronously in the decorator output and key_map.
If you have generated results (such as the pred field), you can use it together with Evaluation Server to achieve rapid evaluation.
The following example demonstrates how to add the id_ls field in the get_data function:
servers/prompt/src/benchmark.py
Then, run the following command to recompile the Pipeline:
In the generated parameter file, add the field id_ls and specify its corresponding key name in the original data:
examples/parameters/load_data_parameter.yaml
After completing the modification, rerun the Pipeline to load data samples containing id.