Evaluation

​

evaluate

@app.tool(output="pred_ls,gt_ls,metrics,save_path->eval_res")
def evaluate(
    pred_ls: List[str],
    gt_ls: List[List[str]],
    metrics: List[str] | None,
    save_path: str,
) -> Dict[str, Any]

• Executes automatic metric evaluation for QA/Generation tasks.
• Supported Metrics: acc, em, coverem, stringem, f1, rouge-1, rouge-2, rouge-l.
• Results are automatically saved as .json file and printed as Markdown table.

​

evaluate_trec

@app.tool(output="run_path,qrels_path,ir_metrics,ks,save_path->eval_res")
def evaluate_trec(
    run_path: str,
    qrels_path: str,
    metrics: List[str] | None,
    ks: List[int] | None,
    save_path: str,
)

• Performs IR retrieval metric evaluation based on pytrec_eval.
• Reads standard TREC format:
  • qrels: <qid> <iter> <docid> <rel>
  • run: <qid> Q0 <docid> <rank> <score> <tag>
• Supported Metrics: mrr, map, recall@k, precision@k, ndcg@k.
• Automatically aggregates statistics and outputs as table.

​

evaluate_trec_pvalue

@app.tool(
    output="run_new_path,run_old_path,qrels_path,ir_metrics,ks,n_resamples,save_path->eval_res"
)
def evaluate_trec_pvalue(
    run_new_path: str,
    run_old_path: str,
    qrels_path: str,
    metrics: List[str] | None,
    ks: List[int] | None,
    n_resamples: int | None,
    save_path: str,
)

• Compares significance of two TREC result files using Two-sided Permutation Test to calculate p-value.
• Default resampling count n_resamples=10000.
• Outputs mean, difference, p-value, and significance flag.

​

Configuration

servers/evaluation/parameter.yaml

save_path: output/evaluate_results.json

# QA task
metrics: [ 'acc', 'f1', 'em', 'coverem', 'stringem', 'rouge-1', 'rouge-2', 'rouge-l' ]

# Retrieval task
qrels_path: data/qrels.txt
run_path: data/run_a.txt
ks: [ 1, 5, 10, 20, 50, 100 ]
ir_metrics: [ "mrr", "map", "recall", "ndcg", "precision" ]

# significant
run_new_path: data/run_a.txt
run_old_path: data/run_b.txt
n_resamples: 10000