> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Phi Vision Models

> Use Microsoft's Phi vision models for multi-modal understanding with ONNX Runtime GenAI

Microsoft's Phi vision models are compact yet powerful multi-modal models that combine visual understanding with language capabilities. ONNX Runtime GenAI supports Phi-3 Vision, Phi-3.5 Vision, and Phi-4 Multi-Modal models.

## Supported Models

<CardGroup cols={3}>
  <Card title="Phi-3 Vision" icon="eye">
    128k context length vision model for image understanding
  </Card>

  <Card title="Phi-3.5 Vision" icon="sparkles">
    Enhanced vision capabilities with improved accuracy
  </Card>

  <Card title="Phi-4 Multi-Modal" icon="wand-magic-sparkles">
    Latest model supporting both vision and audio inputs
  </Card>
</CardGroup>

## Model Architecture

Phi vision models are multi-modal models consisting of several internal components:

* **Vision Encoder**: Processes images and extracts visual features
* **Image Embedding**: Converts visual features into embeddings compatible with the language model
* **Language Model**: Core transformer model for text generation
* **Fusion Layers**: Combine visual and text embeddings

For ONNX Runtime GenAI, each internal component is exported as a separate ONNX model for optimal performance.

## Building Phi Vision Models

<Tabs>
  <Tab title="Phi-3 Vision">
    ### Phi-3 Vision (128k Context)

    <Steps>
      <Step title="Download PyTorch Model">
        ```bash theme={null}
        # Create workspace
        mkdir -p phi3-vision-128k-instruct/pytorch
        cd phi3-vision-128k-instruct/pytorch

        # Download from Hugging Face
        huggingface-cli download microsoft/Phi-3-vision-128k-instruct --local-dir .
        ```
      </Step>

      <Step title="Download Modified Files">
        ```bash theme={null}
        cd ..
        huggingface-cli download microsoft/Phi-3-vision-128k-instruct-onnx \
          --include onnx/* --local-dir .
        ```
      </Step>

      <Step title="Replace Modeling Files">
        ```bash theme={null}
        # Replace config (flash_attention_2 -> eager)
        rm pytorch/config.json
        mv onnx/config.json pytorch/

        # Replace modified modeling file
        rm pytorch/modeling_phi3_v.py
        mv onnx/modeling_phi3_v.py pytorch/

        # Add ONNX export helper
        mv onnx/image_embedding_phi3_v_for_onnx.py pytorch/

        # Move builder script
        mv onnx/builder.py .
        rm -rf onnx/
        ```
      </Step>

      <Step title="Build ONNX Models">
        <CodeGroup>
          ```bash CPU theme={null}
          python3 builder.py \
            --input ./pytorch \
            --output ./cpu \
            --precision fp32 \
            --execution_provider cpu
          ```

          ```bash CUDA theme={null}
          python3 builder.py \
            --input ./pytorch \
            --output ./cuda \
            --precision fp16 \
            --execution_provider cuda
          ```

          ```bash DirectML theme={null}
          python3 builder.py \
            --input ./pytorch \
            --output ./dml \
            --precision fp16 \
            --execution_provider dml
          ```
        </CodeGroup>
      </Step>

      <Step title="Add Configuration Files">
        Download the required JSON configuration files:

        * [`genai_config.json`](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx/blob/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/genai_config.json)
        * [`processor_config.json`](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx/blob/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/processor_config.json)
      </Step>
    </Steps>
  </Tab>

  <Tab title="Phi-3.5 Vision">
    ### Phi-3.5 Vision

    <Steps>
      <Step title="Download PyTorch Model">
        ```bash theme={null}
        mkdir -p phi3.5-vision-instruct/pytorch
        cd phi3.5-vision-instruct/pytorch
        huggingface-cli download microsoft/Phi-3.5-vision-instruct --local-dir .
        ```
      </Step>

      <Step title="Download Modified Files">
        ```bash theme={null}
        cd ..
        huggingface-cli download microsoft/Phi-3.5-vision-instruct-onnx \
          --include onnx/* --local-dir .
        ```
      </Step>

      <Step title="Replace Modeling Files">
        ```bash theme={null}
        rm pytorch/config.json
        mv onnx/config.json pytorch/

        rm pytorch/modeling_phi3_v.py
        mv onnx/modeling_phi3_v.py pytorch/

        mv onnx/builder.py .
        rm -rf onnx/
        ```
      </Step>

      <Step title="Build ONNX Models">
        ```bash theme={null}
        # INT4 quantized models with FP16 inputs/outputs for CUDA
        python3 builder.py \
          --input ./pytorch \
          --output ./cuda \
          --precision fp16 \
          --execution_provider cuda
        ```
      </Step>

      <Step title="Add Configuration Files">
        Download configuration files from the [Phi-3.5-vision-instruct-onnx repository](https://huggingface.co/microsoft/Phi-3.5-vision-instruct-onnx/tree/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4).
      </Step>
    </Steps>
  </Tab>

  <Tab title="Phi-4 Multi-Modal">
    ### Phi-4 Multi-Modal (Vision + Audio)

    <Note>
      Phi-4 requires the latest nightly versions of ONNX Runtime, PyTorch, and related libraries.
    </Note>

    <Steps>
      <Step title="Install Prerequisites">
        ```bash theme={null}
        # Install ONNX Runtime GenAI
        pip install onnxruntime-genai-cuda

        # Uninstall stable ONNX Runtime
        pip uninstall -y onnxruntime-gpu

        # Install nightly ONNX Runtime
        pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ \
          --pre onnxruntime-gpu

        # Install nightly PyTorch
        pip install torch --index-url https://download.pytorch.org/whl/nightly/cu124
        pip install torchvision --index-url https://download.pytorch.org/whl/nightly/cu124
        pip install torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124
        ```
      </Step>

      <Step title="Download and Prepare Model">
        ```bash theme={null}
        mkdir -p phi4-multi-modal/pytorch
        cd phi4-multi-modal/pytorch
        huggingface-cli download microsoft/Phi-4-multimodal-instruct --local-dir .

        cd ..
        huggingface-cli download microsoft/Phi-4-multimodal-instruct-onnx \
          --include onnx/* --local-dir .
        ```
      </Step>

      <Step title="Replace Modeling Files">
        ```bash theme={null}
        # Replace config and modeling files
        rm pytorch/config.json
        mv onnx/config.json pytorch/

        rm pytorch/modeling_phi4mm.py
        mv onnx/modeling_phi4mm.py pytorch/

        rm pytorch/speech_conformer_encoder.py
        mv onnx/speech_conformer_encoder.py pytorch/

        rm pytorch/vision_siglip_navit.py
        mv onnx/vision_siglip_navit.py pytorch/

        rm pytorch/processing_phi4mm.py
        mv onnx/processing_phi4mm.py pytorch/

        mv onnx/builder.py .
        rm -rf onnx/
        ```
      </Step>

      <Step title="Build ONNX Components">
        ```bash theme={null}
        # Build INT4 components for CUDA
        python3 builder.py \
          --input ./pytorch \
          --output ./cuda \
          --precision fp16 \
          --execution_provider cuda
        ```
      </Step>

      <Step title="Add Configuration Files">
        Phi-4 requires three configuration files:

        * `genai_config.json`
        * `speech_processor.json` (for audio)
        * `vision_processor.json` (for vision)

        Download from the [Phi-4-multimodal-instruct-onnx repository](https://huggingface.co/microsoft/Phi-4-multimodal-instruct-onnx/tree/main/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4).
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Using Phi Vision Models

### Basic Image Understanding

```python theme={null}
import onnxruntime_genai as og

# Load model
config = og.Config("./phi3-vision-128k-instruct/cuda")
model = og.Model(config)
processor = model.create_multimodal_processor()
tokenizer = og.Tokenizer(model)

# Load image
images = og.Images.open("image.jpg")

# Create prompt
prompt = "<|user|>\n<|image_1|>\nWhat is shown in this image?<|end|>\n<|assistant|>\n"

# Process inputs
inputs = processor(prompt, images=images)

# Generate response
params = og.GeneratorParams(model)
params.set_search_options(max_length=2048)

generator = og.Generator(model, params)
generator.set_inputs(inputs)

print("Response: ", end="", flush=True)
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end="", flush=True)
print()
```

### Multi-Image Processing

```python theme={null}
import onnxruntime_genai as og

# Load multiple images
images = og.Images.open("image1.jpg", "image2.jpg", "image3.jpg")

# Reference images in prompt
prompt = """
<|user|>
<|image_1|>
<|image_2|>
<|image_3|>
Compare these three images and describe their similarities and differences.
<|end|>
<|assistant|>
"""

# Process and generate
inputs = processor(prompt, images=images)
generator = og.Generator(model, params)
generator.set_inputs(inputs)

while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end="", flush=True)
```

### Chat Template Integration

```python theme={null}
import json
import onnxruntime_genai as og

# Create messages with image
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

# Apply chat template
if hasattr(tokenizer, 'apply_chat_template'):
    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
else:
    # Manual template
    prompt = f"<|user|>\n<|image_1|>\n{messages[0]['content'][1]['text']}<|end|>\n<|assistant|>\n"

images = og.Images.open("image.jpg")
inputs = processor(prompt, images=images)
```

## Image Input Handling

### Supported Image Formats

Phi vision models support common image formats:

* JPEG/JPG
* PNG
* BMP
* TIFF

### Image Preprocessing

The processor automatically handles:

1. **Resizing**: Images are resized to the model's expected dimensions
2. **Normalization**: Pixel values are normalized
3. **Patch Extraction**: Images are divided into patches
4. **Embedding**: Visual patches are converted to embeddings

### Image Resolution

```python theme={null}
# Phi-3 Vision supports high-resolution images
# The model automatically handles various resolutions
images = og.Images.open("high_res_image.jpg")  # Automatically preprocessed
```

## Advanced Usage

### Batch Processing

```python theme={null}
import onnxruntime_genai as og

# Process multiple image-text pairs in batch
image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
prompts = [
    "<|user|>\n<|image_1|>\nDescribe this<|end|>\n<|assistant|>\n",
    "<|user|>\n<|image_1|>\nWhat do you see?<|end|>\n<|assistant|>\n",
    "<|user|>\n<|image_1|>\nAnalyze this image<|end|>\n<|assistant|>\n"
]

for img_path, prompt in zip(image_paths, prompts):
    images = og.Images.open(img_path)
    inputs = processor(prompt, images=images)
    
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)
    
    print(f"Processing {img_path}:")
    while not generator.is_done():
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        print(tokenizer.decode(new_token), end="", flush=True)
    print("\n")
```

### Custom Generation Parameters

```python theme={null}
params = og.GeneratorParams(model)
params.set_search_options(
    max_length=4096,           # Maximum output length
    do_sample=True,            # Enable sampling
    top_p=0.9,                 # Nucleus sampling
    top_k=50,                  # Top-k sampling
    temperature=0.7,           # Sampling temperature
    repetition_penalty=1.1     # Penalize repetition
)
```

## Performance Optimization

<AccordionGroup>
  <Accordion title="Precision Selection">
    Choose the right precision for your hardware:

    * **FP32**: Best accuracy, slower, works on all devices
    * **FP16**: Good balance, requires GPU with FP16 support
    * **INT4**: Fastest, smallest memory footprint, slight accuracy loss

    ```bash theme={null}
    # Build with INT4 quantization
    python3 builder.py --input ./pytorch --output ./cuda \
      --precision fp16 --execution_provider cuda
    ```
  </Accordion>

  <Accordion title="Execution Provider Selection">
    <Tabs>
      <Tab title="CUDA (NVIDIA)">
        ```python theme={null}
        config = og.Config("./model/cuda")
        config.clear_providers()
        config.append_provider("cuda")
        model = og.Model(config)
        ```
      </Tab>

      <Tab title="DirectML (AMD/Intel)">
        ```python theme={null}
        config = og.Config("./model/dml")
        config.clear_providers()
        config.append_provider("dml")
        model = og.Model(config)
        ```
      </Tab>

      <Tab title="CPU">
        ```python theme={null}
        config = og.Config("./model/cpu")
        # CPU provider is default
        model = og.Model(config)
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Memory Management">
    For large images or long sequences:

    ```python theme={null}
    # Monitor token count
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)

    input_tokens = generator.token_count()
    print(f"Input tokens (including image): {input_tokens}")

    # Process in chunks if needed
    max_new_tokens = 1024
    generated = 0

    while not generator.is_done() and generated < max_new_tokens:
        generator.generate_next_token()
        generated += 1
    ```
  </Accordion>
</AccordionGroup>

## Fine-Tuning Support

You can use your own fine-tuned Phi vision models:

<Steps>
  <Step title="Fine-tune with PyTorch">
    Fine-tune the model using your preferred training framework.
  </Step>

  <Step title="Replace Weights">
    ```bash theme={null}
    # After downloading the base model files
    # Replace the *.safetensors files with your fine-tuned weights
    cp /path/to/finetuned/*.safetensors ./phi3-vision-128k-instruct/pytorch/
    ```
  </Step>

  <Step title="Build ONNX Models">
    ```bash theme={null}
    python3 builder.py --input ./pytorch --output ./cuda \
      --precision fp16 --execution_provider cuda
    ```
  </Step>

  <Step title="Update Configurations">
    Modify `genai_config.json` and `processor_config.json` if your fine-tuning changed model architecture or tokenizer.
  </Step>
</Steps>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Image Not Loading">
    ```python theme={null}
    import os

    # Verify image path exists
    image_path = "image.jpg"
    if not os.path.exists(image_path):
        raise FileNotFoundError(f"Image not found: {image_path}")

    # Load with error handling
    try:
        images = og.Images.open(image_path)
    except Exception as e:
        print(f"Error loading image: {e}")
    ```
  </Accordion>

  <Accordion title="Out of Memory">
    If you encounter OOM errors:

    1. Reduce image resolution before processing
    2. Use INT4 quantization instead of FP16
    3. Reduce `max_length` parameter
    4. Process images one at a time instead of batching

    ```python theme={null}
    # Reduce max output length
    params.set_search_options(max_length=1024)  # Instead of 4096
    ```
  </Accordion>

  <Accordion title="Flash Attention Errors">
    If you see flash attention errors:

    ```bash theme={null}
    # Verify config.json has eager attention
    cat pytorch/config.json | grep _attn_implementation
    # Should show: "_attn_implementation": "eager"
    ```
  </Accordion>
</AccordionGroup>

## Example Application

Here's a complete example script for document analysis:

```python theme={null}
import onnxruntime_genai as og
import argparse

def analyze_document(image_path, question):
    # Load model
    config = og.Config("./phi3-vision-128k-instruct/cuda")
    model = og.Model(config)
    processor = model.create_multimodal_processor()
    tokenizer = og.Tokenizer(model)
    
    # Load document image
    images = og.Images.open(image_path)
    
    # Create prompt
    prompt = f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n"
    
    # Process
    inputs = processor(prompt, images=images)
    
    # Generate
    params = og.GeneratorParams(model)
    params.set_search_options(
        max_length=2048,
        do_sample=True,
        top_p=0.9,
        temperature=0.7
    )
    
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)
    
    response = ""
    while not generator.is_done():
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        token_text = tokenizer.decode(new_token)
        response += token_text
        print(token_text, end="", flush=True)
    print()
    
    return response

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image", required=True, help="Path to document image")
    parser.add_argument("--question", required=True, help="Question about the document")
    args = parser.parse_args()
    
    analyze_document(args.image, args.question)
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Qwen Vision" icon="arrow-right" href="/multimodal/qwen-vision">
    Explore Qwen's advanced vision models
  </Card>

  <Card title="Gemma Vision" icon="arrow-right" href="/multimodal/gemma-vision">
    Learn about Google's Gemma vision models
  </Card>

  <Card title="Whisper Audio" icon="arrow-right" href="/multimodal/whisper">
    Add audio processing capabilities
  </Card>

  <Card title="Model Quantization" icon="arrow-right" href="/guides/quantization">
    Optimize models with quantization
  </Card>
</CardGroup>