Phi Vision Models - ONNX Runtime GenAI

Microsoft’s Phi vision models are compact yet powerful multi-modal models that combine visual understanding with language capabilities. ONNX Runtime GenAI supports Phi-3 Vision, Phi-3.5 Vision, and Phi-4 Multi-Modal models.

Supported Models

Phi-3 Vision

128k context length vision model for image understanding

Phi-3.5 Vision

Enhanced vision capabilities with improved accuracy

Phi-4 Multi-Modal

Latest model supporting both vision and audio inputs

Model Architecture

Phi vision models are multi-modal models consisting of several internal components:

Vision Encoder: Processes images and extracts visual features
Image Embedding: Converts visual features into embeddings compatible with the language model
Language Model: Core transformer model for text generation
Fusion Layers: Combine visual and text embeddings

For ONNX Runtime GenAI, each internal component is exported as a separate ONNX model for optimal performance.

Building Phi Vision Models

Phi-3 Vision
Phi-3.5 Vision
Phi-4 Multi-Modal

Phi-3 Vision (128k Context)

Download PyTorch Model

# Create workspace
mkdir -p phi3-vision-128k-instruct/pytorch
cd phi3-vision-128k-instruct/pytorch

# Download from Hugging Face
huggingface-cli download microsoft/Phi-3-vision-128k-instruct --local-dir .

Download Modified Files

cd ..
huggingface-cli download microsoft/Phi-3-vision-128k-instruct-onnx \
  --include onnx/* --local-dir .

Replace Modeling Files

# Replace config (flash_attention_2 -> eager)
rm pytorch/config.json
mv onnx/config.json pytorch/

# Replace modified modeling file
rm pytorch/modeling_phi3_v.py
mv onnx/modeling_phi3_v.py pytorch/

# Add ONNX export helper
mv onnx/image_embedding_phi3_v_for_onnx.py pytorch/

# Move builder script
mv onnx/builder.py .
rm -rf onnx/

Build ONNX Models

python3 builder.py \
  --input ./pytorch \
  --output ./cpu \
  --precision fp32 \
  --execution_provider cpu

Add Configuration Files

Download the required JSON configuration files:

Phi-3.5 Vision

Download PyTorch Model

mkdir -p phi3.5-vision-instruct/pytorch
cd phi3.5-vision-instruct/pytorch
huggingface-cli download microsoft/Phi-3.5-vision-instruct --local-dir .

Download Modified Files

cd ..
huggingface-cli download microsoft/Phi-3.5-vision-instruct-onnx \
  --include onnx/* --local-dir .

Replace Modeling Files

rm pytorch/config.json
mv onnx/config.json pytorch/

rm pytorch/modeling_phi3_v.py
mv onnx/modeling_phi3_v.py pytorch/

mv onnx/builder.py .
rm -rf onnx/

Build ONNX Models

# INT4 quantized models with FP16 inputs/outputs for CUDA
python3 builder.py \
  --input ./pytorch \
  --output ./cuda \
  --precision fp16 \
  --execution_provider cuda

Add Configuration Files

Download configuration files from the Phi-3.5-vision-instruct-onnx repository.

Phi-4 requires the latest nightly versions of ONNX Runtime, PyTorch, and related libraries.

Install Prerequisites

# Install ONNX Runtime GenAI
pip install onnxruntime-genai-cuda

# Uninstall stable ONNX Runtime
pip uninstall -y onnxruntime-gpu

# Install nightly ONNX Runtime
pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ \
  --pre onnxruntime-gpu

# Install nightly PyTorch
pip install torch --index-url https://download.pytorch.org/whl/nightly/cu124
pip install torchvision --index-url https://download.pytorch.org/whl/nightly/cu124
pip install torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124

Download and Prepare Model

mkdir -p phi4-multi-modal/pytorch
cd phi4-multi-modal/pytorch
huggingface-cli download microsoft/Phi-4-multimodal-instruct --local-dir .

cd ..
huggingface-cli download microsoft/Phi-4-multimodal-instruct-onnx \
  --include onnx/* --local-dir .

Replace Modeling Files

# Replace config and modeling files
rm pytorch/config.json
mv onnx/config.json pytorch/

rm pytorch/modeling_phi4mm.py
mv onnx/modeling_phi4mm.py pytorch/

rm pytorch/speech_conformer_encoder.py
mv onnx/speech_conformer_encoder.py pytorch/

rm pytorch/vision_siglip_navit.py
mv onnx/vision_siglip_navit.py pytorch/

rm pytorch/processing_phi4mm.py
mv onnx/processing_phi4mm.py pytorch/

mv onnx/builder.py .
rm -rf onnx/

Build ONNX Components

# Build INT4 components for CUDA
python3 builder.py \
  --input ./pytorch \
  --output ./cuda \
  --precision fp16 \
  --execution_provider cuda

Add Configuration Files

Phi-4 requires three configuration files:

genai_config.json
speech_processor.json (for audio)
vision_processor.json (for vision)

Download from the Phi-4-multimodal-instruct-onnx repository.

Using Phi Vision Models

Basic Image Understanding

import onnxruntime_genai as og

# Load model
config = og.Config("./phi3-vision-128k-instruct/cuda")
model = og.Model(config)
processor = model.create_multimodal_processor()
tokenizer = og.Tokenizer(model)

# Load image
images = og.Images.open("image.jpg")

# Create prompt
prompt = "<|user|>\n<|image_1|>\nWhat is shown in this image?<|end|>\n<|assistant|>\n"

# Process inputs
inputs = processor(prompt, images=images)

# Generate response
params = og.GeneratorParams(model)
params.set_search_options(max_length=2048)

generator = og.Generator(model, params)
generator.set_inputs(inputs)

print("Response: ", end="", flush=True)
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end="", flush=True)
print()

Multi-Image Processing

import onnxruntime_genai as og

# Load multiple images
images = og.Images.open("image1.jpg", "image2.jpg", "image3.jpg")

# Reference images in prompt
prompt = """
<|user|>
<|image_1|>
<|image_2|>
<|image_3|>
Compare these three images and describe their similarities and differences.
<|end|>
<|assistant|>
"""

# Process and generate
inputs = processor(prompt, images=images)
generator = og.Generator(model, params)
generator.set_inputs(inputs)

while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end="", flush=True)

Chat Template Integration

import json
import onnxruntime_genai as og

# Create messages with image
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

# Apply chat template
if hasattr(tokenizer, 'apply_chat_template'):
    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
else:
    # Manual template
    prompt = f"<|user|>\n<|image_1|>\n{messages[0]['content'][1]['text']}<|end|>\n<|assistant|>\n"

images = og.Images.open("image.jpg")
inputs = processor(prompt, images=images)

Image Input Handling

Supported Image Formats

Phi vision models support common image formats:

JPEG/JPG
PNG
BMP
TIFF

Image Preprocessing

The processor automatically handles:

Resizing: Images are resized to the model’s expected dimensions
Normalization: Pixel values are normalized
Patch Extraction: Images are divided into patches
Embedding: Visual patches are converted to embeddings

Image Resolution

# Phi-3 Vision supports high-resolution images
# The model automatically handles various resolutions
images = og.Images.open("high_res_image.jpg")  # Automatically preprocessed

Advanced Usage

Batch Processing

import onnxruntime_genai as og

# Process multiple image-text pairs in batch
image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
prompts = [
    "<|user|>\n<|image_1|>\nDescribe this<|end|>\n<|assistant|>\n",
    "<|user|>\n<|image_1|>\nWhat do you see?<|end|>\n<|assistant|>\n",
    "<|user|>\n<|image_1|>\nAnalyze this image<|end|>\n<|assistant|>\n"
]

for img_path, prompt in zip(image_paths, prompts):
    images = og.Images.open(img_path)
    inputs = processor(prompt, images=images)
    
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)
    
    print(f"Processing {img_path}:")
    while not generator.is_done():
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        print(tokenizer.decode(new_token), end="", flush=True)
    print("\n")

Custom Generation Parameters

params = og.GeneratorParams(model)
params.set_search_options(
    max_length=4096,           # Maximum output length
    do_sample=True,            # Enable sampling
    top_p=0.9,                 # Nucleus sampling
    top_k=50,                  # Top-k sampling
    temperature=0.7,           # Sampling temperature
    repetition_penalty=1.1     # Penalize repetition
)

Performance Optimization

Precision Selection

Choose the right precision for your hardware:

FP32: Best accuracy, slower, works on all devices
FP16: Good balance, requires GPU with FP16 support
INT4: Fastest, smallest memory footprint, slight accuracy loss

# Build with INT4 quantization
python3 builder.py --input ./pytorch --output ./cuda \
  --precision fp16 --execution_provider cuda

Execution Provider Selection

CUDA (NVIDIA)
DirectML (AMD/Intel)
CPU

config = og.Config("./model/cuda")
config.clear_providers()
config.append_provider("cuda")
model = og.Model(config)

config = og.Config("./model/dml")
config.clear_providers()
config.append_provider("dml")
model = og.Model(config)

config = og.Config("./model/cpu")
# CPU provider is default
model = og.Model(config)

Memory Management

For large images or long sequences:

# Monitor token count
generator = og.Generator(model, params)
generator.set_inputs(inputs)

input_tokens = generator.token_count()
print(f"Input tokens (including image): {input_tokens}")

# Process in chunks if needed
max_new_tokens = 1024
generated = 0

while not generator.is_done() and generated < max_new_tokens:
    generator.generate_next_token()
    generated += 1

Fine-Tuning Support

You can use your own fine-tuned Phi vision models:

Fine-tune with PyTorch

Fine-tune the model using your preferred training framework.

Replace Weights

# After downloading the base model files
# Replace the *.safetensors files with your fine-tuned weights
cp /path/to/finetuned/*.safetensors ./phi3-vision-128k-instruct/pytorch/

Build ONNX Models

python3 builder.py --input ./pytorch --output ./cuda \
  --precision fp16 --execution_provider cuda

Update Configurations

Modify genai_config.json and processor_config.json if your fine-tuning changed model architecture or tokenizer.

Troubleshooting

Image Not Loading

import os

# Verify image path exists
image_path = "image.jpg"
if not os.path.exists(image_path):
    raise FileNotFoundError(f"Image not found: {image_path}")

# Load with error handling
try:
    images = og.Images.open(image_path)
except Exception as e:
    print(f"Error loading image: {e}")

Out of Memory

If you encounter OOM errors:

Reduce image resolution before processing
Use INT4 quantization instead of FP16
Reduce max_length parameter
Process images one at a time instead of batching

# Reduce max output length
params.set_search_options(max_length=1024)  # Instead of 4096

Flash Attention Errors

If you see flash attention errors:

# Verify config.json has eager attention
cat pytorch/config.json | grep _attn_implementation
# Should show: "_attn_implementation": "eager"

Example Application

Here’s a complete example script for document analysis:

import onnxruntime_genai as og
import argparse

def analyze_document(image_path, question):
    # Load model
    config = og.Config("./phi3-vision-128k-instruct/cuda")
    model = og.Model(config)
    processor = model.create_multimodal_processor()
    tokenizer = og.Tokenizer(model)
    
    # Load document image
    images = og.Images.open(image_path)
    
    # Create prompt
    prompt = f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n"
    
    # Process
    inputs = processor(prompt, images=images)
    
    # Generate
    params = og.GeneratorParams(model)
    params.set_search_options(
        max_length=2048,
        do_sample=True,
        top_p=0.9,
        temperature=0.7
    )
    
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)
    
    response = ""
    while not generator.is_done():
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        token_text = tokenizer.decode(new_token)
        response += token_text
        print(token_text, end="", flush=True)
    print()
    
    return response

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image", required=True, help="Path to document image")
    parser.add_argument("--question", required=True, help="Question about the document")
    args = parser.parse_args()
    
    analyze_document(args.image, args.question)

Next Steps

Qwen Vision

Explore Qwen’s advanced vision models

Gemma Vision

Learn about Google’s Gemma vision models

Whisper Audio

Add audio processing capabilities

Model Quantization

Optimize models with quantization

​Supported Models

Phi-3 Vision

Phi-3.5 Vision

Phi-4 Multi-Modal

​Model Architecture

​Building Phi Vision Models

​Phi-3 Vision (128k Context)

​Phi-3.5 Vision

​Phi-4 Multi-Modal (Vision + Audio)

​Using Phi Vision Models

​Basic Image Understanding

​Multi-Image Processing

​Chat Template Integration

​Image Input Handling

​Supported Image Formats

​Image Preprocessing

​Image Resolution

​Advanced Usage

​Batch Processing

​Custom Generation Parameters

​Performance Optimization

​Fine-Tuning Support

​Troubleshooting

​Example Application

​Next Steps

Qwen Vision

Gemma Vision

Whisper Audio

Model Quantization

Supported Models

Model Architecture

Building Phi Vision Models

Phi-3 Vision (128k Context)

Phi-3.5 Vision

Phi-4 Multi-Modal (Vision + Audio)

Using Phi Vision Models

Basic Image Understanding

Multi-Image Processing

Chat Template Integration

Image Input Handling

Supported Image Formats

Image Preprocessing

Image Resolution

Advanced Usage

Batch Processing

Custom Generation Parameters

Performance Optimization

Fine-Tuning Support

Troubleshooting

Example Application

Next Steps