> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Modal Overview

> Learn about multi-modal capabilities in ONNX Runtime GenAI including vision and audio model support

ONNX Runtime GenAI provides comprehensive support for multi-modal models that can process multiple types of inputs such as images, audio, and text. This enables powerful AI applications that can understand and generate responses based on visual and auditory information alongside text.

## Multi-Modal Capabilities

ONNX Runtime GenAI supports two primary categories of multi-modal models:

<CardGroup cols={2}>
  <Card title="Vision Models" icon="eye">
    Process images alongside text for visual understanding and reasoning tasks
  </Card>

  <Card title="Audio Models" icon="microphone">
    Process audio inputs for speech recognition and transcription
  </Card>
</CardGroup>

## Supported Model Types

### Vision Models

ONNX Runtime GenAI supports several state-of-the-art vision-language models:

<AccordionGroup>
  <Accordion title="Phi Vision Models" icon="microsoft">
    * **Phi-3 Vision**: 128k context length vision model
    * **Phi-3.5 Vision**: Enhanced vision capabilities
    * **Phi-4 Multi-Modal**: Latest model with both vision and audio support

    [Learn more about Phi Vision models](/multimodal/phi-vision)
  </Accordion>

  <Accordion title="Qwen Vision Models" icon="image">
    * **Qwen2.5-VL**: Advanced vision-language model with 3D positional encoding
    * Supports dynamic image resolution and multi-image inputs
    * MRoPE (Multi-Resolution Rotary Position Embedding) for better spatial understanding

    [Learn more about Qwen Vision models](/multimodal/qwen-vision)
  </Accordion>

  <Accordion title="Gemma Vision Models" icon="google">
    * **Gemma-3 Vision**: Google's multi-modal model family
    * Available in multiple sizes: 4B, 12B, 27B parameters
    * Supports BF16 and FP16 precision

    [Learn more about Gemma Vision models](/multimodal/gemma-vision)
  </Accordion>
</AccordionGroup>

### Audio Models

<Card title="Whisper" icon="waveform">
  OpenAI's Whisper models for speech recognition and transcription with support for multiple languages and beam search decoding.

  [Learn more about Whisper](/multimodal/whisper)
</Card>

## Multi-Modal Processing Architecture

Multi-modal models in ONNX Runtime GenAI consist of multiple components that work together:

<Steps>
  <Step title="Input Processing">
    Images are preprocessed through vision encoders, and audio through speech encoders to create embeddings.
  </Step>

  <Step title="Embedding Generation">
    Visual or audio data is converted into embeddings that can be combined with text embeddings.
  </Step>

  <Step title="Fusion">
    Multi-modal embeddings are fused with text embeddings using specialized fusion layers.
  </Step>

  <Step title="Language Model Processing">
    The combined embeddings are processed by the language model to generate responses.
  </Step>
</Steps>

## Working with Multi-Modal Models

### Using the Multi-Modal Processor

All multi-modal models use a unified processor interface:

```python theme={null}
import onnxruntime_genai as og

# Load model
model = og.Model("path/to/model")
processor = model.create_multimodal_processor()
tokenizer = og.Tokenizer(model)
```

### Processing Different Input Types

<Tabs>
  <Tab title="Vision">
    ```python theme={null}
    # Load images
    images = og.Images.open("image1.jpg", "image2.png")

    # Process with prompt
    prompt = "What do you see in these images?"
    inputs = processor(prompt, images=images)
    ```
  </Tab>

  <Tab title="Audio">
    ```python theme={null}
    # Load audio files
    audios = og.Audios.open("audio1.wav", "audio2.mp3")

    # Process with prompt
    prompt = "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>"
    inputs = processor(prompt, audios=audios)
    ```
  </Tab>

  <Tab title="Mixed (Phi-4)">
    ```python theme={null}
    # Phi-4 supports both vision and audio
    images = og.Images.open("scene.jpg")
    audios = og.Audios.open("question.wav")

    # Process multi-modal input
    inputs = processor(prompt, images=images, audios=audios)
    ```
  </Tab>
</Tabs>

## Input Preprocessing

### Vision Input Preprocessing

Vision models require images to be preprocessed according to model-specific requirements:

* **Image Resizing**: Images are resized to match the model's expected input dimensions
* **Normalization**: Pixel values are normalized (typically to \[0, 1] or \[-1, 1])
* **Patch Embedding**: Images are divided into patches and embedded
* **Position Encoding**: Spatial position information is encoded

### Audio Input Preprocessing

Audio models process raw audio waveforms:

* **Sampling Rate**: Audio is resampled to the model's expected rate (e.g., 16kHz for Whisper)
* **Feature Extraction**: Spectrograms or mel-frequency features are computed
* **Windowing**: Audio is divided into overlapping windows for processing

## Generation with Multi-Modal Inputs

```python theme={null}
import onnxruntime_genai as og

# Setup model
config = og.Config("path/to/model")
model = og.Model(config)
processor = model.create_multimodal_processor()
tokenizer = og.Tokenizer(model)

# Load inputs
images = og.Images.open("image.jpg")
prompt = "Describe this image in detail."

# Process inputs
inputs = processor(prompt, images=images)

# Create generator
params = og.GeneratorParams(model)
params.set_search_options(max_length=2048, do_sample=True, top_p=0.9, temperature=0.7)

generator = og.Generator(model, params)
generator.set_inputs(inputs)

# Generate response
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(tokenizer.decode(new_token), end='', flush=True)
```

## Execution Providers

Multi-modal models support multiple execution providers for optimal performance:

<CodeGroup>
  ```bash CPU theme={null}
  python model-mm.py -m ./model/cpu -e cpu
  ```

  ```bash CUDA theme={null}
  python model-mm.py -m ./model/cuda -e cuda
  ```

  ```bash DirectML theme={null}
  python model-mm.py -m ./model/dml -e dml
  ```
</CodeGroup>

<Note>
  For best performance with vision models, use CUDA or DirectML execution providers. CPU inference is supported but may be slower for large vision models.
</Note>

## Configuration Files

Multi-modal models require specific configuration files:

### Required Files

* **`genai_config.json`**: Core model configuration including model paths, precision, and execution settings
* **`processor_config.json`**: Vision processor configuration (for vision models)
* **`speech_processor.json`**: Audio processor configuration (for audio-enabled models like Phi-4)
* **`vision_processor.json`**: Vision processor configuration (for Phi-4)

### Example Configuration Structure

```json theme={null}
{
  "model": {
    "decoder": {
      "filename": "model.onnx",
      "session_options": {
        "log_severity_level": 2,
        "provider_options": []
      }
    },
    "vision": {
      "inputs": {
        "pixel_values": "pixel_values"
      },
      "filename": "vision_model.onnx"
    }
  }
}
```

## Best Practices

<CardGroup cols={2}>
  <Card title="Batch Processing" icon="layer-group">
    Process multiple images or audio files in batches for better throughput
  </Card>

  <Card title="Precision Selection" icon="gauge">
    Use FP16 or BF16 for faster inference on compatible hardware
  </Card>

  <Card title="Memory Management" icon="memory">
    Monitor memory usage with large images or long audio files
  </Card>

  <Card title="Input Validation" icon="check">
    Validate image formats and audio sampling rates before processing
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Phi Vision Models" icon="arrow-right" href="/multimodal/phi-vision">
    Learn how to use Microsoft's Phi vision models
  </Card>

  <Card title="Qwen Vision Models" icon="arrow-right" href="/multimodal/qwen-vision">
    Explore Qwen's advanced vision capabilities
  </Card>

  <Card title="Gemma Vision Models" icon="arrow-right" href="/multimodal/gemma-vision">
    Work with Google's Gemma vision models
  </Card>

  <Card title="Whisper Audio" icon="arrow-right" href="/multimodal/whisper">
    Process audio with Whisper models
  </Card>
</CardGroup>