> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen Vision Models

> Use Qwen's advanced vision-language models with ONNX Runtime GenAI

Qwen2.5-VL is an advanced vision-language model that supports multi-image understanding, dynamic resolution, and sophisticated spatial reasoning through Multi-Resolution Rotary Position Embedding (MRoPE).

## Overview

Qwen2.5-VL models are state-of-the-art vision-language models from Alibaba Cloud that excel at:

* **Multi-image understanding**: Process and reason across multiple images
* **Dynamic resolution**: Handle images of varying sizes and aspect ratios
* **3D positional encoding**: MRoPE for better spatial understanding
* **Long context**: Support for extended context lengths

<Note>
  Qwen2.5-VL uses a unique 3D position encoding scheme with temporal, height, and width dimensions for superior spatial reasoning.
</Note>

## Architecture Details

### Multi-Resolution Rotary Position Embedding (MRoPE)

Qwen2.5-VL uses MRoPE to encode positional information in three dimensions:

```python theme={null}
# Position IDs shape: [3, batch_size, sequence_length]
# Dimension 0: Temporal dimension
# Dimension 1: Height dimension  
# Dimension 2: Width dimension
```

This 3D encoding allows the model to:

* Better understand spatial relationships in images
* Handle dynamic image resolutions
* Process multi-image inputs with proper position awareness

### Model Components

The Qwen2.5-VL architecture consists of:

1. **Vision Encoder**: Processes images into visual features
   * Patch embedding for image tokenization
   * Vision attention layers
   * Patch merger for feature aggregation

2. **Language Model**: Core text generation model
   * Modified attention with MRoPE
   * Grouped Query Attention (GQA)
   * RMS Layer Normalization (always computed in FP32)

3. **Vision Pipeline**: Multi-stage processing
   ```
   Image → Patch Embed → Vision Attention → Patch Merger → Embeddings
   ```

## Building Qwen2.5-VL Models

<Note>
  Qwen2.5-VL requires specific versions of dependencies. Follow the installation steps carefully.
</Note>

### Prerequisites

<Steps>
  <Step title="Install ONNX Runtime">
    ```bash theme={null}
    # Install nightly ONNX Runtime GenAI for CUDA
    pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ \
      --pre onnxruntime-genai-cuda

    # Uninstall stable ONNX Runtime
    pip uninstall -y onnxruntime-gpu

    # Install nightly ONNX Runtime
    pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ \
      --pre onnxruntime-gpu
    ```
  </Step>

  <Step title="Install PyTorch">
    ```bash theme={null}
    # Ensure PyTorch >= 2.7.0
    pip install torch==2.7.0
    ```
  </Step>

  <Step title="Install Additional Dependencies">
    ```bash theme={null}
    pip install transformers
    pip install pillow
    pip install numpy==1.26.4  # Must be < 2.0.0
    pip install --pre onnxscript
    ```
  </Step>
</Steps>

### Model Export

<Steps>
  <Step title="Download Base Model">
    ```bash theme={null}
    mkdir -p qwen2.5-vl-7b-instruct
    cd qwen2.5-vl-7b-instruct

    # Download from Hugging Face
    huggingface-cli download Qwen/Qwen2.5-VL-7B-Instruct \
      --local-dir ./pytorch
    ```
  </Step>

  <Step title="Export to ONNX">
    ```python theme={null}
    # Use the builder from ONNX Runtime GenAI source
    from onnxruntime_genai.models.builder import build_model

    build_model(
        model_name="Qwen/Qwen2.5-VL-7B-Instruct",
        input_path="./pytorch",
        output_path="./onnx-cuda",
        precision="fp16",
        execution_provider="cuda",
        cache_dir="./cache"
    )
    ```
  </Step>
</Steps>

### Precision Options

<Tabs>
  <Tab title="FP16">
    ```bash theme={null}
    # FP16 for CUDA (recommended for most GPUs)
    python3 -m onnxruntime_genai.models.builder \
      --model_name Qwen/Qwen2.5-VL-7B-Instruct \
      --output ./onnx-fp16 \
      --precision fp16 \
      --execution_provider cuda
    ```
  </Tab>

  <Tab title="BF16">
    ```bash theme={null}
    # BF16 for newer GPUs (A100, H100)
    python3 -m onnxruntime_genai.models.builder \
      --model_name Qwen/Qwen2.5-VL-7B-Instruct \
      --output ./onnx-bf16 \
      --precision bf16 \
      --execution_provider cuda
    ```
  </Tab>

  <Tab title="FP32">
    ```bash theme={null}
    # FP32 for CPU
    python3 -m onnxruntime_genai.models.builder \
      --model_name Qwen/Qwen2.5-VL-7B-Instruct \
      --output ./onnx-fp32 \
      --precision fp32 \
      --execution_provider cpu
    ```
  </Tab>
</Tabs>

## Using Qwen2.5-VL

### Basic Usage

```python theme={null}
import onnxruntime_genai as og

# Load model
config = og.Config("./qwen2.5-vl-7b-instruct/onnx-cuda")
model = og.Model(config)
processor = model.create_multimodal_processor()
tokenizer = og.Tokenizer(model)

# Load image
images = og.Images.open("image.jpg")

# Create prompt
prompt = "Describe this image in detail."

# Process inputs
inputs = processor(prompt, images=images)

# Generate response
params = og.GeneratorParams(model)
params.set_search_options(max_length=2048)

generator = og.Generator(model, params)
generator.set_inputs(inputs)

print("Response: ", end="", flush=True)
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end="", flush=True)
print()
```

### Multi-Image Processing

Qwen2.5-VL excels at reasoning across multiple images:

```python theme={null}
import onnxruntime_genai as og

# Load multiple images
images = og.Images.open(
    "product_front.jpg",
    "product_back.jpg",
    "product_side.jpg"
)

# Ask comparative question
prompt = """
Analyze these three product images:
1. What are the key features visible from different angles?
2. Are there any defects or quality issues?
3. How would you rate the product packaging?
"""

inputs = processor(prompt, images=images)

# Generate detailed analysis
params = og.GeneratorParams(model)
params.set_search_options(
    max_length=4096,
    temperature=0.7,
    top_p=0.9
)

generator = og.Generator(model, params)
generator.set_inputs(inputs)

while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end="", flush=True)
```

### Chat Conversation with Images

```python theme={null}
import json
import onnxruntime_genai as og

# Initialize model
config = og.Config("./qwen2.5-vl-7b-instruct/onnx-cuda")
model = og.Model(config)
processor = model.create_multimodal_processor()
tokenizer = og.Tokenizer(model)

# Chat history
messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant that can see images."
    },
    {
        "role": "user",
        "content": "What objects can you identify in this image?"
    }
]

# Load image
images = og.Images.open("scene.jpg")

# Convert messages to prompt
messages_json = json.dumps(messages)
prompt = messages_json  # Processor handles chat template

# Process and generate
inputs = processor(prompt, images=images)

params = og.GeneratorParams(model)
params.set_search_options(max_length=2048)

generator = og.Generator(model, params)
generator.set_inputs(inputs)

response = ""
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    token_text = tokenizer.decode(new_token)
    response += token_text
    print(token_text, end="", flush=True)

# Add assistant response to history
messages.append({"role": "assistant", "content": response})
```

## Image Preprocessing

### Automatic Resolution Handling

Qwen2.5-VL automatically handles various image resolutions:

```python theme={null}
# High resolution image
high_res = og.Images.open("high_resolution_4k.jpg")

# Low resolution image  
low_res = og.Images.open("thumbnail_128x128.jpg")

# Both are automatically preprocessed to optimal resolution
inputs_high = processor("Analyze this high-res image", images=high_res)
inputs_low = processor("Analyze this thumbnail", images=low_res)
```

### Grid Dimensions

The model uses grid-based image processing with temporal, height, and width dimensions:

```python theme={null}
# Grid dimensions are automatically calculated based on image size
# For a 1024x768 image with patch size 14:
# - Temporal: 1 (single image)
# - Height: 1024 / 14 ≈ 73 patches
# - Width: 768 / 14 ≈ 55 patches
```

## Advanced Features

### Custom Position IDs

For advanced use cases, you can work with 3D position IDs:

```python theme={null}
import torch

# Example: Create custom 3D position IDs
# Shape: [3, batch_size, sequence_length]
batch_size = 1
sequence_length = 100

# Temporal, Height, Width dimensions
position_ids = torch.zeros((3, batch_size, sequence_length), dtype=torch.int64)

# For text tokens, all dimensions use same sequential IDs
for i in range(sequence_length):
    position_ids[0, 0, i] = i  # Temporal
    position_ids[1, 0, i] = i  # Height
    position_ids[2, 0, i] = i  # Width

# For image patches, dimensions vary based on spatial location
# (automatically handled by the processor)
```

### Vision Pipeline Components

Access individual vision pipeline components:

```python theme={null}
# The vision pipeline consists of:
# 1. Patch Embedding: Image -> Patches
# 2. Vision Attention: Process patches with attention
# 3. Patch Merger: Merge patches to reduce sequence length

# These are automatically orchestrated by the processor
images = og.Images.open("image.jpg")
inputs = processor(prompt, images=images)

# The processor handles:
# - Patch extraction from images
# - Window-based attention reordering
# - Spatial merge operations
# - Final embedding generation
```

## Performance Optimization

<AccordionGroup>
  <Accordion title="Execution Provider Selection">
    Choose the right execution provider for your hardware:

    ```python theme={null}
    config = og.Config("./qwen2.5-vl-7b-instruct/onnx-cuda")

    # For NVIDIA GPUs
    config.clear_providers()
    config.append_provider("cuda")

    # For DirectML (AMD/Intel)
    # config.append_provider("dml")

    model = og.Model(config)
    ```
  </Accordion>

  <Accordion title="Precision Trade-offs">
    Different precisions offer different trade-offs:

    | Precision | Speed | Memory | Accuracy  | Hardware    |
    | --------- | ----- | ------ | --------- | ----------- |
    | FP32      | Slow  | High   | Best      | All         |
    | FP16      | Fast  | Medium | Good      | Modern GPUs |
    | BF16      | Fast  | Medium | Very Good | A100, H100  |

    <Note>
      LayerNorm and RoPE are always computed in FP32 internally for numerical stability, regardless of model precision.
    </Note>
  </Accordion>

  <Accordion title="Batch Processing">
    Process multiple images efficiently:

    ```python theme={null}
    # Single batch with multiple images
    images = og.Images.open("img1.jpg", "img2.jpg", "img3.jpg")
    prompt = "Analyze all these images together."
    inputs = processor(prompt, images=images)

    # Batch size is automatically determined from inputs
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)
    ```
  </Accordion>

  <Accordion title="Memory Management">
    For large images or long contexts:

    ```python theme={null}
    # Monitor token count
    generator.set_inputs(inputs)
    total_tokens = generator.token_count()
    print(f"Total input tokens: {total_tokens}")

    # Adjust max_length based on available memory
    available_memory_gb = 16  # Your GPU memory
    if total_tokens > 2048:
        max_new_tokens = 1024  # Reduce for large inputs
    else:
        max_new_tokens = 2048

    params.set_search_options(max_length=total_tokens + max_new_tokens)
    ```
  </Accordion>
</AccordionGroup>

## Implementation Details

### RoPE Computation

Qwen2.5-VL uses a custom MRoPE implementation:

```python theme={null}
# Pseudo-code for MRoPE logic
# 1. Calculate dynamic RoPE caches from 3D position_ids
cos_cache, sin_cache = make_dynamic_rope_caches(position_ids)
# Shape: [3, batch_size, sequence_length, head_dim]

# 2. Flatten and reorder based on MRoPE sections
flat_cos, flat_sin = make_mrope_flattened_caches(cos_cache, sin_cache)
# Shape: [batch_size * sequence_length, head_dim / 2]

# 3. Apply rotation to Q and K
q_rotated = apply_mrope_rotation(q, flat_cos, flat_sin)
k_rotated = apply_mrope_rotation(k, flat_cos, flat_sin)

# 4. Grouped Query Attention
output = grouped_query_attention(q_rotated, k_rotated, v)
```

### Layer Normalization

```python theme={null}
# Qwen2.5-VL uses RMSNorm with forced FP32 computation
# Regardless of model precision (FP16/BF16), normalization uses FP32

# This is configured automatically in the builder:
layernorm_attrs = {
    "cast": {
        "use_fp32": True,      # Compute in FP32
        "root_input": True,    # Cast input to FP32
        "skip_input": True,    # Cast skip connection to FP32
        "output_0": True,      # Cast output back to model dtype
        "output_3": True       # Cast residual output back
    }
}
```

## Troubleshooting

<AccordionGroup>
  <Accordion title="NumPy Version Error">
    If you see NumPy 2.0 compatibility errors:

    ```bash theme={null}
    # Uninstall numpy 2.0+
    pip uninstall -y numpy

    # Install compatible version
    pip install numpy==1.26.4
    ```
  </Accordion>

  <Accordion title="PyTorch Version Error">
    Qwen2.5-VL requires PyTorch >= 2.7.0:

    ```bash theme={null}
    # Check version
    python -c "import torch; print(torch.__version__)"

    # Upgrade if needed
    pip install --upgrade torch==2.7.0
    ```
  </Accordion>

  <Accordion title="Position IDs Shape Error">
    If you encounter position\_ids shape errors:

    ```python theme={null}
    # Verify position_ids shape is [3, batch_size, sequence_length]
    # This is handled automatically by the processor

    # If manually creating inputs, ensure correct shape:
    position_ids = torch.zeros((3, 1, seq_len), dtype=torch.int64)
    ```
  </Accordion>

  <Accordion title="OOM with High Resolution Images">
    For very high resolution images:

    ```python theme={null}
    # The model automatically handles resolution
    # But you can pre-resize large images if needed:

    from PIL import Image

    img = Image.open("very_large_image.jpg")
    max_size = 2048
    if max(img.size) > max_size:
        img.thumbnail((max_size, max_size), Image.LANCZOS)
        img.save("resized.jpg")

    images = og.Images.open("resized.jpg")
    ```
  </Accordion>
</AccordionGroup>

## Example: Document Understanding

```python theme={null}
import onnxruntime_genai as og
import argparse

def analyze_document(image_path, task="summarize"):
    """Analyze a document image with Qwen2.5-VL."""
    
    # Load model
    config = og.Config("./qwen2.5-vl-7b-instruct/onnx-cuda")
    model = og.Model(config)
    processor = model.create_multimodal_processor()
    tokenizer = og.Tokenizer(model)
    
    # Load document
    images = og.Images.open(image_path)
    
    # Create task-specific prompt
    prompts = {
        "summarize": "Summarize the key points from this document.",
        "extract": "Extract all important information including names, dates, and numbers.",
        "translate": "Translate the text in this document to English.",
        "ocr": "Extract all text from this document."
    }
    
    prompt = prompts.get(task, prompts["summarize"])
    
    # Process
    inputs = processor(prompt, images=images)
    
    # Generate with appropriate parameters
    params = og.GeneratorParams(model)
    params.set_search_options(
        max_length=4096,
        temperature=0.3,  # Lower temperature for factual tasks
        top_p=0.8,
        repetition_penalty=1.1
    )
    
    generator = og.Generator(model, params)
    generator.set_inputs(inputs)
    
    print(f"\nTask: {task}")
    print(f"Document: {image_path}")
    print("\nResult:")
    print("-" * 50)
    
    while not generator.is_done():
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        print(tokenizer.decode(new_token), end="", flush=True)
    
    print("\n" + "-" * 50)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--image", required=True, help="Document image path")
    parser.add_argument("--task", default="summarize",
                       choices=["summarize", "extract", "translate", "ocr"],
                       help="Analysis task")
    args = parser.parse_args()
    
    analyze_document(args.image, args.task)
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Phi Vision Models" icon="arrow-right" href="/multimodal/phi-vision">
    Explore Microsoft's Phi vision models
  </Card>

  <Card title="Gemma Vision Models" icon="arrow-right" href="/multimodal/gemma-vision">
    Learn about Google's Gemma vision models
  </Card>

  <Card title="Model Optimization" icon="arrow-right" href="/guides/optimization">
    Optimize inference performance
  </Card>

  <Card title="Custom Models" icon="arrow-right" href="/guides/custom-models">
    Build custom vision models
  </Card>
</CardGroup>