> ## Documentation Index
> Fetch the complete documentation index at: https://ultrarag.openbmb.cn/llms.txt
> Use this file to discover all available pages before exploring further.

# Corpus

## `build_text_corpus`

**Signature**

```python theme={null}
@app.tool(output="parse_file_path,text_corpus_save_path->None")
async def build_text_corpus(parse_file_path: str, text_corpus_save_path: str) -> None
```

**Function**

* Supports .txt / .md;
* Supports .docx (reads paragraphs and tables);
* Supports .pdf / .xps / .oxps / .epub / .mobi / .fb2 (pure text extraction via pymupdf).
* Recursively processes in directory mode.

**Output Format (JSONL)**

```json theme={null}
{"id": "<stem>", "title": "<stem>", "contents": "<full text>"}
```

***

## `build_image_corpus`

**Signature**

```python theme={null}
@app.tool(output="parse_file_path,image_corpus_save_path->None")
async def build_image_corpus(parse_file_path: str, image_corpus_save_path: str) -> None
```

**Function**

* **Only supports PDF**: Renders each page as JPG (RGB) at 144DPI and validates file validity.
* Recursively processes in directory mode.

**Output Index (JSONL)**

```json theme={null}
{"id": 0, "image_id": "paper/page_0.jpg", "image_path": "image/paper/page_0.jpg"}
```

***

## `mineru_parse`

**Signature**

```python theme={null}
@app.tool(output="parse_file_path,mineru_dir,mineru_extra_params->None")
async def mineru_parse(
    parse_file_path: str, 
    mineru_dir: str, 
    mineru_extra_params: Optional[Dict[str, Any]] = None
) -> None
```

**Function**

* Calls CLI `mineru` to structurally parse PDF/directory and outputs to `mineru_dir`.

***

## `build_mineru_corpus`

**Signature**

```python theme={null}
@app.tool(output="mineru_dir,parse_file_path,text_corpus_save_path,image_corpus_save_path->None")
async def build_mineru_corpus(
    mineru_dir: str, 
    parse_file_path: str, 
    text_corpus_save_path: str, 
    image_corpus_save_path: str
) -> None
```

**Function**

* Aggregates MinerU parsing artifacts into **Text Corpus JSONL** and **Image Index JSONL**.

**Output Format (JSONL)**

* Text:

```json theme={null}
{"id": "<stem>", "title": "<stem>", "contents": "<markdown full text>"}
```

* Image:

```json theme={null}
{"id": 0, "image_id": "paper/page_0.jpg", "image_path": "images/paper/page_0.jpg"}
```

***

## `chunk_documents`

**Signature**

```python theme={null}
@app.tool(output="raw_chunk_path,chunk_backend_configs,chunk_backend,tokenizer_or_token_counter,chunk_size,chunk_path,use_title->None")
async def chunk_documents(
    raw_chunk_path: str,
    chunk_backend_configs: Dict[str, Any],
    chunk_backend: str = "token",
    tokenizer_or_token_counter: str = "character",
    chunk_size: int = 256,
    chunk_path: Optional[str] = None,
    use_title: bool = True,
) -> None
```

**Function**

* Chunks input text corpus (JSONL, containing `id/title/contents`) into paragraphs using selected backend:
* Chunk Backend: Supports `token` / `sentence` / `recursive`.
* Tokenizer: `tokenizer_or_token_counter` can be `word`, `character`, or `tiktoken` encoding name (e.g., `gpt2`).
* Chunk Size: Controls block size via `chunk_size` (overlap defaults to size/4).
* Optionally appends document title to the beginning of each block (`use_title`).

**Output Format (JSONL)**

```json theme={null}
{"id": 0, "doc_id": "paper", "title": "paper", "contents": "Chunked text"}
```

***

## Configuration

```yaml servers/corpus/parameter.yaml icon="https://mintcdn.com/ultrarag/T7GffHzZitf6TThi/images/yaml.svg?fit=max&auto=format&n=T7GffHzZitf6TThi&q=85&s=69b41e79144bc908039c2ee3abbb1c3b" theme={null}
# servers/corpus/parameter.yaml
parse_file_path: data/UltraRAG.pdf
text_corpus_save_path: corpora/text.jsonl
image_corpus_save_path: corpora/image.jsonl

# mineru
mineru_dir: corpora/
mineru_extra_params:
  source: modelscope

# chunking parameters
raw_chunk_path: corpora/text.jsonl
chunk_path: corpora/chunks.jsonl
use_title: false
chunk_backend: sentence # choices=["token", "sentence", "recursive"]
tokenizer_or_token_counter: character
chunk_size: 512
chunk_backend_configs:
  token:
    chunk_overlap: 50
  sentence:
    chunk_overlap: 50
    min_sentences_per_chunk: 1
    delim: "['.', '!', '?', '；', '。', '！', '？', '\\n']"
  recursive:
    min_characters_per_chunk: 12
```

Parameter Description:

| Parameter                    | Type | Description                                                                                        |
| ---------------------------- | ---- | -------------------------------------------------------------------------------------------------- |
| `parse_file_path`            | str  | Input file or directory path                                                                       |
| `text_corpus_save_path`      | str  | Text corpus output path (JSONL)                                                                    |
| `image_corpus_save_path`     | str  | Image corpus index output path (JSONL)                                                             |
| `mineru_dir`                 | str  | MinerU output root directory                                                                       |
| `mineru_extra_params`        | dict | MinerU extra parameters, such as `source`, `layout`, etc.                                          |
| `raw_chunk_path`             | str  | Chunking input file path (JSONL format)                                                            |
| `chunk_path`                 | str  | Chunking output path                                                                               |
| `use_title`                  | bool | Whether to append document title to the beginning of each chunk                                    |
| `chunk_backend`              | str  | Select chunking method: `token`, `sentence`, `recursive`                                           |
| `tokenizer_or_token_counter` | str  | Tokenizer or counting method. Options: `word`, `character` or `tiktoken` model name (e.g., `gpt2`) |
| `chunk_size`                 | int  | Target size for each chunk                                                                         |
| `chunk_backend_configs`      | dict | Configuration items for each chunking method (see below)                                           |

`chunk_backend_configs` Detailed Parameters:

| Backend Type  | Parameter                  | Description                                            |
| ------------- | -------------------------- | ------------------------------------------------------ |
| **token**     | `chunk_overlap`            | Overlapping tokens between chunks                      |
| **sentence**  | `chunk_overlap`            | Overlapping count between chunks                       |
|               | `min_sentences_per_chunk`  | Minimum number of sentences per chunk                  |
|               | `delim`                    | Sentence delimiter list (Python list in string format) |
| **recursive** | `min_characters_per_chunk` | Minimum character unit for recursive splitting         |