Corpus - UltraRAG

`build_text_corpus`

Signature

@app.tool(output="parse_file_path,text_corpus_save_path->None")
async def build_text_corpus(parse_file_path: str, text_corpus_save_path: str) -> None

Function

Supports .txt / .md;
Supports .docx (reads paragraphs and tables);
Supports .pdf / .xps / .oxps / .epub / .mobi / .fb2 (pure text extraction via pymupdf).
Recursively processes in directory mode.

Output Format (JSONL)

{"id": "<stem>", "title": "<stem>", "contents": "<full text>"}

`build_image_corpus`

Signature

@app.tool(output="parse_file_path,image_corpus_save_path->None")
async def build_image_corpus(parse_file_path: str, image_corpus_save_path: str) -> None

Function

Only supports PDF: Renders each page as JPG (RGB) at 144DPI and validates file validity.
Recursively processes in directory mode.

Output Index (JSONL)

{"id": 0, "image_id": "paper/page_0.jpg", "image_path": "image/paper/page_0.jpg"}

`mineru_parse`

Signature

@app.tool(output="parse_file_path,mineru_dir,mineru_extra_params->None")
async def mineru_parse(
    parse_file_path: str, 
    mineru_dir: str, 
    mineru_extra_params: Optional[Dict[str, Any]] = None
) -> None

Function

Calls CLI mineru to structurally parse PDF/directory and outputs to mineru_dir.

`build_mineru_corpus`

Signature

@app.tool(output="mineru_dir,parse_file_path,text_corpus_save_path,image_corpus_save_path->None")
async def build_mineru_corpus(
    mineru_dir: str, 
    parse_file_path: str, 
    text_corpus_save_path: str, 
    image_corpus_save_path: str
) -> None

Function

Aggregates MinerU parsing artifacts into Text Corpus JSONL and Image Index JSONL.

Output Format (JSONL)

Text:

{"id": "<stem>", "title": "<stem>", "contents": "<markdown full text>"}

Image:

{"id": 0, "image_id": "paper/page_0.jpg", "image_path": "images/paper/page_0.jpg"}

`chunk_documents`

Signature

@app.tool(output="raw_chunk_path,chunk_backend_configs,chunk_backend,tokenizer_or_token_counter,chunk_size,chunk_path,use_title->None")
async def chunk_documents(
    raw_chunk_path: str,
    chunk_backend_configs: Dict[str, Any],
    chunk_backend: str = "token",
    tokenizer_or_token_counter: str = "character",
    chunk_size: int = 256,
    chunk_path: Optional[str] = None,
    use_title: bool = True,
) -> None

Function

Chunks input text corpus (JSONL, containing id/title/contents) into paragraphs using selected backend:
Chunk Backend: Supports token / sentence / recursive.
Tokenizer: tokenizer_or_token_counter can be word, character, or tiktoken encoding name (e.g., gpt2).
Chunk Size: Controls block size via chunk_size (overlap defaults to size/4).
Optionally appends document title to the beginning of each block (use_title).

Output Format (JSONL)

{"id": 0, "doc_id": "paper", "title": "paper", "contents": "Chunked text"}

Configuration

servers/corpus/parameter.yaml

# servers/corpus/parameter.yaml
parse_file_path: data/UltraRAG.pdf
text_corpus_save_path: corpora/text.jsonl
image_corpus_save_path: corpora/image.jsonl

# mineru
mineru_dir: corpora/
mineru_extra_params:
  source: modelscope

# chunking parameters
raw_chunk_path: corpora/text.jsonl
chunk_path: corpora/chunks.jsonl
use_title: false
chunk_backend: sentence # choices=["token", "sentence", "recursive"]
tokenizer_or_token_counter: character
chunk_size: 512
chunk_backend_configs:
  token:
    chunk_overlap: 50
  sentence:
    chunk_overlap: 50
    min_sentences_per_chunk: 1
    delim: "['.', '!', '?', '；', '。', '！', '？', '\\n']"
  recursive:
    min_characters_per_chunk: 12

Parameter Description:

Parameter	Type	Description
`parse_file_path`	str	Input file or directory path
`text_corpus_save_path`	str	Text corpus output path (JSONL)
`image_corpus_save_path`	str	Image corpus index output path (JSONL)
`mineru_dir`	str	MinerU output root directory
`mineru_extra_params`	dict	MinerU extra parameters, such as `source`, `layout`, etc.
`raw_chunk_path`	str	Chunking input file path (JSONL format)
`chunk_path`	str	Chunking output path
`use_title`	bool	Whether to append document title to the beginning of each chunk
`chunk_backend`	str	Select chunking method: `token`, `sentence`, `recursive`
`tokenizer_or_token_counter`	str	Tokenizer or counting method. Options: `word`, `character` or `tiktoken` model name (e.g., `gpt2`)
`chunk_size`	int	Target size for each chunk
`chunk_backend_configs`	dict	Configuration items for each chunking method (see below)

chunk_backend_configs Detailed Parameters:

Backend Type	Parameter	Description
token	`chunk_overlap`	Overlapping tokens between chunks
sentence	`chunk_overlap`	Overlapping count between chunks
	`min_sentences_per_chunk`	Minimum number of sentences per chunk
	`delim`	Sentence delimiter list (Python list in string format)
recursive	`min_characters_per_chunk`	Minimum character unit for recursive splitting

​build_text_corpus

​build_image_corpus

​mineru_parse

​build_mineru_corpus

​chunk_documents

​Configuration

`build_text_corpus`

`build_image_corpus`

`mineru_parse`

`build_mineru_corpus`

`chunk_documents`

Configuration