Generation

​

generation_init

def generation_init(
    backend_configs: Dict[str, Any],
    sampling_params: Dict[str, Any],
    extra_params: Optional[Dict[str, Any]] = None,
    backend: str = "vllm",
) -> None

• Initializes inference backend and sampling parameters.
• Supports vllm, openai, hf backends.
• extra_params can be used to pass chat_template_kwargs or other backend-specific parameters.

​

generate

async def generate(
    prompt_ls: List[Union[str, Dict[str, Any]]],
    system_prompt: str = "",
) -> Dict[str, List[str]]

• Plain text conversation generation.
• Automatically handles Prompt in list, supports string or OpenAI format dictionary.

{"ans_ls": ["answer for prompt_0", "answer for prompt_1", "..."]}

​

multimodal_generate

async def multimodal_generate(
    multimodal_path: List[List[str]],
    prompt_ls: List[Union[str, Dict[str, Any]]],
    system_prompt: str = "",
    image_tag: Optional[str] = None,
) -> Dict[str, List[str]]

• Text-image multimodal conversation generation.
• multimodal_path: List of image paths corresponding to each Prompt (supports local path or URL).
• image_tag: If specified (e.g., <img>), inserts image at that tag’s position in Prompt; otherwise defaults to appending to end of Prompt.

{"ans_ls": ["answer with images for prompt_0", "..."]}

​

multiturn_generate

async def multiturn_generate(
    messages: List[Dict[str, str]],
    system_prompt: str = "",
) -> Dict[str, List[str]]

• Multi-turn conversation generation.
• Supports only single-call generation, does not handle batch Prompts.

{"ans_ls": ["assistant response"]}

​

vllm_shutdown

def vllm_shutdown() -> None

• Explicitly shuts down vLLM engine and releases VRAM resources.
• Valid only when using vllm backend.

​

Configuration

servers/generation/parameter.yaml

# servers/generation/parameter.yaml
backend: vllm # options: vllm, openai
backend_configs:
  vllm:
    model_name_or_path: openbmb/MiniCPM4-8B
    gpu_ids: "2,3"
    gpu_memory_utilization: 0.9
    dtype: auto
    trust_remote_code: true
  openai:
    model_name: MiniCPM4-8B
    base_url: http://localhost:8000/v1
    api_key: "abc"
    concurrency: 8
    retries: 3
    base_delay: 1.0
  hf:
    model_name_or_path: openbmb/MiniCPM4-8B
    gpu_ids: '2,3'
    trust_remote_code: true
    batch_size: 8
sampling_params:
  temperature: 0.7
  top_p: 0.8
  max_tokens: 2048
extra_params:
  chat_template_kwargs:
    enable_thinking: false
system_prompt: ""
image_tag: null