Skip to main content

​
Overview

The Custom Server is used to store user-defined utility functions that do not belong to standard modules such as Retriever, Generation, or Evaluation.
It provides developers with a flexible extension space for implementing logic components that work in coordination with core RAG modules, such as:
  • Data cleaning and preprocessing
  • Keyword extraction or feature construction
  • Task-specific logic (e.g., answer extraction, formatting, filtering, etc.)
The Custom Server is your personal toolbox — any functionality not belonging to the core Servers can be defined and reused here.

​
Implementation Example

The following example demonstrates how to define and register a custom Tool, using a common function output_extract_from_boxed as an example.
servers/custom/src/custom.py
@app.tool(output="ans_ls->pred_ls")
def output_extract_from_boxed(ans_ls: List[str]) -> Dict[str, List[str]]:
    def extract(ans: str) -> str:
        start = ans.rfind(r"\boxed{")
        if start == -1:
            content = ans.strip()
        else:
            i = start + len(r"\boxed{")
            brace_level = 1
            end = i
            while end < len(ans) and brace_level > 0:
                if ans[end] == "{":
                    brace_level += 1
                elif ans[end] == "}":
                    brace_level -= 1
                end += 1
            content = ans[i : end - 1].strip()
            content = re.sub(r"^\$+|\$+$", "", content).strip()
            content = re.sub(r"^\\\(|\\\)$", "", content).strip()
            if content.startswith(r"\text{") and content.endswith("}"):
                content = content[len(r"\text{") : -1].strip()
            content = content.strip("()").strip()

        content = content.replace("\\", " ")
        content = content.replace("  ", " ")
        return content

    return {"pred_ls": [extract(ans) for ans in ans_ls]}
This Tool extracts the final answer text in the \boxed{...} format from model output strings.
The extracted results are mapped to the variable pred_ls, which can then be used by downstream evaluation or post-processing modules.

​
Usage Example

Once you have defined your custom Tool, simply register the custom module in the Pipeline and call the corresponding Tool:
https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/yaml.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=69b41e79144bc908039c2ee3abbb1c3bexamples/rag_full.yaml
# MCP Server
servers:
  benchmark: servers/benchmark
  retriever: servers/retriever
  prompt: servers/prompt
  generation: servers/generation
  evaluation: servers/evaluation
  custom: servers/custom

# MCP Client Pipeline
pipeline:
- benchmark.get_data
- retriever.retriever_init
- retriever.retriever_embed
- retriever.retriever_index
- retriever.retriever_search
- generation.generation_init
- prompt.qa_rag_boxed
- generation.generate
- custom.output_extract_from_boxed
- evaluation.evaluate
In this example, custom.output_extract_from_boxed is used to extract standardized answers from model outputs,
which are then passed to evaluation.evaluate for further evaluation.