> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenVINO Execution Provider

> Intel hardware optimization for CPU, GPU, and NPU acceleration

The OpenVINO execution provider optimizes inference on Intel hardware including CPUs, integrated GPUs (iGPUs), and Neural Processing Units (NPUs).

## Requirements

### Hardware

* **CPU**: Intel Core, Xeon, or Atom processors
* **GPU**: Intel Iris Xe, Arc, or Data Center GPU Flex/Max
* **NPU**: Intel Core Ultra processors (Meteor Lake and newer)

### Software

* OpenVINO Runtime 2024.0 or later
* Intel Graphics Driver (for GPU acceleration)
* Operating Systems:
  * Windows 10/11
  * Linux (Ubuntu 20.04+, RHEL 8+)
  * macOS 10.15+

<Note>
  OpenVINO provides excellent CPU performance and is the recommended provider for Intel hardware.
</Note>

## Installation

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    # Install OpenVINO
    pip install openvino

    # Install ONNX Runtime GenAI
    pip install onnxruntime-genai --pre
    ```
  </Tab>

  <Tab title="Build from Source">
    ```bash theme={null}
    git clone https://github.com/microsoft/onnxruntime-genai.git
    cd onnxruntime-genai
    python build.py --use_openvino
    ```
  </Tab>
</Tabs>

## Basic Configuration

### Python API

```python theme={null}
import onnxruntime_genai as og

model_path = "path/to/model"

# Create config and set OpenVINO provider
config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")

# Load model
model = og.Model(config)
tokenizer = og.Tokenizer(model)

# Generate
params = og.GeneratorParams(model)
params.set_search_options(max_length=1024)

generator = og.Generator(model, params)
```

### genai\_config.json

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "provider_options": [
          {
            "openvino": {
              "device_type": "CPU"
            }
          }
        ]
      }
    }
  }
}
```

## Device Selection

### CPU Acceleration

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")
config.set_provider_option("openvino", "device_type", "CPU")

model = og.Model(config)
```

### GPU Acceleration

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")
config.set_provider_option("openvino", "device_type", "GPU")

model = og.Model(config)
```

### NPU Acceleration

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")
config.set_provider_option("openvino", "device_type", "NPU")

model = og.Model(config)
```

<Tabs>
  <Tab title="CPU">
    **Best for:**

    * General-purpose inference
    * Development and testing
    * Systems without dedicated GPU/NPU

    **Performance:**

    * Excellent on Intel CPUs
    * Multi-threading support
    * INT8 quantization available
  </Tab>

  <Tab title="GPU">
    **Best for:**

    * Parallel processing
    * Large batch sizes
    * FP16 inference

    **Performance:**

    * 2-4x faster than CPU (model dependent)
    * Lower latency for vision models
  </Tab>

  <Tab title="NPU">
    **Best for:**

    * Energy-efficient inference
    * Mobile and edge devices
    * Low-power scenarios

    **Performance:**

    * Minimal power consumption
    * Offloads CPU/GPU
    * Optimized for INT8
  </Tab>
</Tabs>

## CPU Optimization

### Thread Configuration

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")

# Set CPU threads
config.set_provider_option("openvino", "device_type", "CPU")
config.set_provider_option("openvino", "num_streams", "4")

model = og.Model(config)
```

### Performance Hints

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "provider_options": [
          {
            "openvino": {
              "device_type": "CPU",
              "performance_hint": "THROUGHPUT",
              "num_streams": "AUTO"
            }
          }
        ]
      }
    }
  }
}
```

<Info>
  **Performance Hints:**

  * `LATENCY`: Optimize for single-request latency
  * `THROUGHPUT`: Optimize for maximum throughput
  * `CUMULATIVE_THROUGHPUT`: Balance latency and throughput
</Info>

## Advanced Configuration

### Model Caching

Enable model caching to speed up subsequent loads:

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")

# Enable caching
config.set_provider_option("openvino", "cache_dir", "./ov_cache")

model = og.Model(config)
```

### Load Config

Provide advanced OpenVINO configuration:

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "provider_options": [
          {
            "openvino": {
              "device_type": "CPU",
              "cache_dir": "./ov_cache",
              "load_config": {
                "CPU": {
                  "INFERENCE_PRECISION_HINT": "f32",
                  "PERFORMANCE_HINT": "LATENCY"
                }
              }
            }
          }
        ]
      }
    }
  }
}
```

### Device Filtering

Select specific devices in multi-device systems:

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "provider_options": [
          {
            "openvino": {
              "device_type": "GPU"
            },
            "device_filtering_options": {
              "hardware_device_type": "gpu",
              "hardware_device_id": 0,
              "hardware_vendor_id": 32902
            }
          }
        ]
      }
    }
  }
}
```

## Stateful Models

OpenVINO supports stateful models with internal KV cache management:

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")

# Enable stateful model (CausalLM)
config.set_provider_option("openvino", "enable_causallm", "True")

model = og.Model(config)
```

<Note>
  When `enable_causallm` is set to "True", OpenVINO manages the KV cache internally, reducing memory overhead.
</Note>

## Quantization Support

### INT8 Quantization

OpenVINO provides excellent INT8 performance:

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)  # Use INT8 quantized model
config.clear_providers()
config.append_provider("openvino")
config.set_provider_option("openvino", "device_type", "CPU")

model = og.Model(config)
```

### Precision Hints

```json theme={null}
{
  "openvino": {
    "device_type": "CPU",
    "load_config": {
      "CPU": {
        "INFERENCE_PRECISION_HINT": "i8"
      }
    }
  }
}
```

<Tabs>
  <Tab title="FP32">
    * Full precision
    * Highest accuracy
    * Baseline performance
  </Tab>

  <Tab title="FP16">
    * Half precision
    * 2x memory reduction
    * Faster on GPU
  </Tab>

  <Tab title="INT8">
    * Quantized precision
    * 4x memory reduction
    * 2-4x speedup on CPU
  </Tab>
</Tabs>

## Multi-Device Execution

OpenVINO can distribute inference across multiple devices:

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")

# Use multiple devices
config.set_provider_option("openvino", "device_type", "MULTI:CPU,GPU")

model = og.Model(config)
```

## Troubleshooting

### OpenVINO Not Found

```bash theme={null}
# Install OpenVINO
pip install openvino

# Verify installation
python -c "import openvino; print(openvino.__version__)"
```

### Device Not Available

```python theme={null}
import onnxruntime_genai as og

try:
    config = og.Config(model_path)
    config.clear_providers()
    config.append_provider("openvino")
    config.set_provider_option("openvino", "device_type", "GPU")
    model = og.Model(config)
except Exception as e:
    print(f"GPU not available: {e}")
    print("Falling back to CPU")
    config.set_provider_option("openvino", "device_type", "CPU")
    model = og.Model(config)
```

### Performance Issues

<AccordionGroup>
  <Accordion title="Enable Model Caching">
    ```python theme={null}
    config.set_provider_option("openvino", "cache_dir", "./ov_cache")
    ```

    First load will be slower, but subsequent loads will be much faster.
  </Accordion>

  <Accordion title="Optimize Thread Count">
    ```python theme={null}
    config.set_provider_option("openvino", "num_streams", "AUTO")
    config.set_provider_option("openvino", "performance_hint", "THROUGHPUT")
    ```
  </Accordion>

  <Accordion title="Use Quantized Models">
    INT8 quantized models provide 2-4x speedup on Intel CPUs.
  </Accordion>
</AccordionGroup>

## Benchmarking

```python theme={null}
import time
import onnxruntime_genai as og

# CPU benchmark
config_cpu = og.Config(model_path)
config_cpu.clear_providers()
config_cpu.append_provider("openvino")
config_cpu.set_provider_option("openvino", "device_type", "CPU")
model_cpu = og.Model(config_cpu)

# GPU benchmark (if available)
config_gpu = og.Config(model_path)
config_gpu.clear_providers()
config_gpu.append_provider("openvino")
config_gpu.set_provider_option("openvino", "device_type", "GPU")

try:
    model_gpu = og.Model(config_gpu)
    print("GPU available for benchmarking")
except:
    model_gpu = None
    print("GPU not available")

# Run inference
tokenizer = og.Tokenizer(model_cpu)
prompt = "Tell me about AI"
input_tokens = tokenizer.encode(prompt)

for device, model in [("CPU", model_cpu), ("GPU", model_gpu)]:
    if model is None:
        continue
    
    params = og.GeneratorParams(model)
    params.set_search_options(max_length=100)
    
    start = time.time()
    generator = og.Generator(model, params)
    generator.append_tokens(input_tokens)
    
    while not generator.is_done():
        generator.generate_next_token()
    
    end = time.time()
    print(f"{device} - Time: {end - start:.2f}s, Tokens/sec: {100 / (end - start):.2f}")
```

## Best Practices

<CardGroup cols={2}>
  <Card title="Use Model Caching" icon="database">
    Enable `cache_dir` to reduce model loading time on subsequent runs.
  </Card>

  <Card title="Choose Right Device" icon="microchip">
    Use CPU for latency, GPU for throughput, NPU for power efficiency.
  </Card>

  <Card title="Optimize Precision" icon="gauge">
    Use INT8 models for best CPU performance on Intel hardware.
  </Card>

  <Card title="Performance Hints" icon="lightbulb">
    Set appropriate performance hints based on your use case.
  </Card>
</CardGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Model Optimization" icon="chart-line" href="/optimization/openvino">
    Optimize models for OpenVINO
  </Card>

  <Card title="Quantization Guide" icon="compress" href="/optimization/quantization">
    Learn about INT8 quantization
  </Card>
</CardGroup>