> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Builder

> Convert and optimize PyTorch models to ONNX format for ONNX Runtime GenAI

The ONNX Runtime GenAI Model Builder allows you to quickly create optimized and quantized ONNX models that run with ONNX Runtime GenAI.

## Supported Model Architectures

The model builder currently supports the following architectures:

* AMD OLMo
* ChatGLM
* DeepSeek
* ERNIE 4.5
* Gemma
* gpt-oss
* Granite
* InternLM2
* Llama
* Mistral
* Nemotron
* Phi
* Qwen
* SmolLM3

## Installation

The model builder is included in the ONNX Runtime GenAI Python package:

```bash theme={null}
pip install onnxruntime-genai
```

## Basic Usage

View all available options:

<CodeGroup>
  ```bash From Wheel theme={null}
  python -m onnxruntime_genai.models.builder --help
  ```

  ```bash From Source theme={null}
  python builder.py --help
  ```
</CodeGroup>

## Converting Models

### PyTorch Model from Hugging Face

Convert a model directly from Hugging Face:

<CodeGroup>
  ```bash From Wheel theme={null}
  python -m onnxruntime_genai.models.builder \
    -m model_name \
    -o path_to_output_folder \
    -p precision \
    -e execution_provider \
    -c cache_dir_to_save_hf_files
  ```

  ```bash From Source theme={null}
  python builder.py \
    -m model_name \
    -o path_to_output_folder \
    -p precision \
    -e execution_provider \
    -c cache_dir_to_save_hf_files
  ```
</CodeGroup>

**Parameters:**

* `-m`: Model name from Hugging Face
* `-o`: Output directory for the ONNX model
* `-p`: Precision (fp16, fp32, int4, etc.)
* `-e`: Execution provider (cpu, cuda, dml, etc.)
* `-c`: Cache directory for Hugging Face files

### PyTorch Model from Disk

Convert a locally downloaded PyTorch model:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -m model_name \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  -c cache_dir_where_hf_files_are_saved
```

### Customized or Finetuned Model

Convert your custom or finetuned PyTorch model:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  -c cache_dir_to_store_temp_files
```

### GGUF Model

Convert a GGUF model to ONNX format:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -m model_name \
  -i path_to_gguf_file \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  -c cache_dir_for_hf_files
```

## Quantization Options

### INT4 Quantization

Convert a pre-quantized INT4 model (AutoGPTQ or AutoAWQ):

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p int4 \
  -e execution_provider \
  -c cache_dir_to_store_temp_files
```

### Shared Embeddings

Enable weight sharing between embedding layer and language modeling head to reduce model size:

<CodeGroup>
  ```bash INT4 with K-Quant theme={null}
  python -m onnxruntime_genai.models.builder \
    -m model_name \
    -o path_to_output_folder \
    -p int4 \
    -e cuda \
    --extra_options shared_embeddings=true int4_algo_config=k_quant
  ```

  ```bash INT4 + INT8 Embeddings theme={null}
  python -m onnxruntime_genai.models.builder \
    -m model_name \
    -o path_to_output_folder \
    -p int4 \
    -e cuda \
    --extra_options shared_embeddings=true int4_algo_config=k_quant_last
  ```

  ```bash FP16 Embeddings theme={null}
  python -m onnxruntime_genai.models.builder \
    -m model_name \
    -o path_to_output_folder \
    -p fp16 \
    -e cuda \
    --extra_options shared_embeddings=true
  ```
</CodeGroup>

<Note>
  Shared embeddings are automatically enabled if `tie_word_embeddings=true` in the model's config.json. Cannot be used with `exclude_embeds=true` or `exclude_lm_head=true`.
</Note>

### QDQ Pattern Quantization

Use the QDQ (Quantize-Dequantize) pattern for 4-bit quantization:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p int4 \
  -e execution_provider \
  --extra_options use_qdq=true
```

## Advanced Options

### Config Only Mode

Generate only the configuration files for an existing ONNX model:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -m model_name \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  -c cache_dir_for_hf_files \
  --extra_options config_only=true
```

After running this, modify the `genai_config.json` file in the output folder as needed.

### Exclude Components

Exclude specific model components:

<CodeGroup>
  ```bash Exclude Embeddings theme={null}
  python -m onnxruntime_genai.models.builder \
    -i path_to_local_folder_on_disk \
    -o path_to_output_folder \
    -p precision \
    -e execution_provider \
    --extra_options exclude_embeds=true
  ```

  ```bash Exclude LM Head theme={null}
  python -m onnxruntime_genai.models.builder \
    -i path_to_local_folder_on_disk \
    -o path_to_output_folder \
    -p precision \
    -e execution_provider \
    --extra_options exclude_lm_head=true
  ```

  ```bash Prune LM Head theme={null}
  python -m onnxruntime_genai.models.builder \
    -i path_to_local_folder_on_disk \
    -o path_to_output_folder \
    -p precision \
    -e execution_provider \
    --extra_options prune_lm_head=true
  ```
</CodeGroup>

### Include Hidden States

Include last hidden states as model output:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  --extra_options include_hidden_states=true
```

<Info>
  The last hidden states are also known as embeddings.
</Info>

### CUDA Graph Support

Enable CUDA graph optimization:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p precision \
  -e cuda \
  --extra_options enable_cuda_graph=true
```

### Disable QKV Fusion

Keep Q/K/V projections separate instead of fusing them:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  --extra_options disable_qkv_fusion=true
```

## LoRA Adapter Support

Convert models with LoRA adapters using PEFT:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -i path_to_local_folder_on_disk \
  -o path_to_output_folder \
  -p fp16 \
  -e execution_provider \
  -c cache_dir_to_store_temp_files \
  --extra_options adapter_path=path_to_adapter_files
```

* Base weights should be in `path_to_local_folder_on_disk`
* Adapter weights should be in `path_to_adapter_files`

See the [Multi-LoRA guide](/guides/multi-lora) for runtime usage.

## Testing Models

Create a model with reduced layers for testing:

### Option 1: Direct Builder Command

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -m model_name \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  --extra_options num_hidden_layers=4
```

### Option 2: Edit config.json

<Steps>
  <Step title="Locate the Model Files">
    Navigate to where the PyTorch model is saved on disk.
  </Step>

  <Step title="Edit config.json">
    Modify `num_hidden_layers` in `config.json` to your desired value (e.g., 4 layers).
  </Step>

  <Step title="Run the Builder">
    ```bash theme={null}
    python -m onnxruntime_genai.models.builder \
      -m model_name \
      -o path_to_output_folder \
      -p precision \
      -e execution_provider \
      -c cache_dir_where_hf_files_are_saved
    ```
  </Step>
</Steps>

## Hugging Face Configuration

### Custom Authentication

Disable or use a different Hugging Face token:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -m model_name \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  --extra_options hf_token=false
```

### Remote Code Trust

Disable trusting remote code from Hugging Face:

```bash theme={null}
python -m onnxruntime_genai.models.builder \
  -m model_name \
  -o path_to_output_folder \
  -p precision \
  -e execution_provider \
  --extra_options hf_remote=false
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Download Models" icon="download" href="/guides/download-models">
    Learn about other ways to obtain models
  </Card>

  <Card title="Runtime Options" icon="sliders" href="/guides/runtime-options">
    Configure your model at runtime
  </Card>

  <Card title="Multi-LoRA" icon="layer-group" href="/guides/multi-lora">
    Use multiple LoRA adapters dynamically
  </Card>

  <Card title="Quickstart" icon="rocket" href="/quickstart">
    Run your first inference
  </Card>
</CardGroup>