> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hardware Acceleration Overview

> Choose the right execution provider for optimal performance across different hardware platforms

ONNX Runtime GenAI supports multiple hardware acceleration providers to optimize model inference across different platforms. Each execution provider targets specific hardware and offers unique performance characteristics.

## Available Execution Providers

ONNX Runtime GenAI supports the following execution providers:

<CardGroup cols={2}>
  <Card title="CUDA" icon="microchip" href="/acceleration/cuda">
    NVIDIA GPU acceleration with comprehensive memory management
  </Card>

  <Card title="DirectML" icon="windows" href="/acceleration/directml">
    Cross-vendor GPU acceleration on Windows platforms
  </Card>

  <Card title="OpenVINO" icon="cpu" href="/acceleration/openvino">
    Intel hardware optimization for CPU, GPU, and NPU
  </Card>

  <Card title="QNN" icon="mobile" href="/acceleration/qnn">
    Qualcomm NPU acceleration for edge and mobile devices
  </Card>

  <Card title="WebGPU" icon="globe" href="/acceleration/webgpu">
    Browser-based GPU acceleration using WebGPU API
  </Card>
</CardGroup>

## Platform Compatibility Matrix

| Provider     | Windows | Linux | macOS | Android | Browser |
| ------------ | ------- | ----- | ----- | ------- | ------- |
| **CUDA**     | ✅       | ✅     | ❌     | ❌       | ❌       |
| **DirectML** | ✅       | ❌     | ❌     | ❌       | ❌       |
| **OpenVINO** | ✅       | ✅     | ✅     | ❌       | ❌       |
| **QNN**      | ✅       | ✅     | ❌     | ✅       | ❌       |
| **WebGPU**   | ✅       | ✅     | ✅     | ❌       | ✅       |
| **CPU**      | ✅       | ✅     | ✅     | ✅       | ✅       |

## Hardware Type Support

| Provider     | CPU | GPU | NPU | Target Hardware         |
| ------------ | --- | --- | --- | ----------------------- |
| **CUDA**     | ❌   | ✅   | ❌   | NVIDIA GPUs             |
| **DirectML** | ❌   | ✅   | ❌   | All DirectX 12 GPUs     |
| **OpenVINO** | ✅   | ✅   | ✅   | Intel CPUs, iGPUs, NPUs |
| **QNN**      | ❌   | ❌   | ✅   | Qualcomm Hexagon NPUs   |
| **WebGPU**   | ❌   | ✅   | ❌   | Browser-supported GPUs  |

## Performance Considerations

### Memory Management

Each provider handles memory differently:

* **CUDA**: Device memory with host-pinned allocations for efficient transfers
* **DirectML**: D3D12 resource management with upload/readback heaps
* **OpenVINO**: CPU-accessible memory with optional device acceleration
* **QNN**: CPU-accessible NPU memory
* **WebGPU**: GPU buffers with async CPU-GPU synchronization

### Precision Support

<Tabs>
  <Tab title="FP32">
    All providers support full precision (FP32) inference.
  </Tab>

  <Tab title="FP16">
    * **CUDA**: Native FP16 support on modern GPUs
    * **DirectML**: FP16 support on compatible hardware
    * **OpenVINO**: FP16 optimization for Intel GPUs
    * **QNN**: FP16 support on Hexagon DSPs
    * **WebGPU**: Limited FP16 support (browser-dependent)
  </Tab>

  <Tab title="INT8">
    Quantized inference support varies:

    * **CUDA**: INT8 optimization with TensorRT
    * **OpenVINO**: INT8 quantization for Intel hardware
    * **QNN**: INT8 native support on NPUs
  </Tab>
</Tabs>

## Provider Selection Guide

Choose the right provider based on your deployment scenario:

### Server Deployment

```python theme={null}
import onnxruntime_genai as og

# NVIDIA GPU server
config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")
model = og.Model(config)
```

### Windows Desktop

```python theme={null}
import onnxruntime_genai as og

# Cross-vendor GPU support
config = og.Config(model_path)
config.clear_providers()
config.append_provider("dml")
model = og.Model(config)
```

### Edge Devices

```python theme={null}
import onnxruntime_genai as og

# Intel edge hardware
config = og.Config(model_path)
config.clear_providers()
config.append_provider("openvino")
config.set_provider_option("openvino", "device_type", "CPU")
model = og.Model(config)
```

### Mobile Deployment

```python theme={null}
import onnxruntime_genai as og

# Qualcomm Snapdragon devices
config = og.Config(model_path)
config.clear_providers()
config.append_provider("qnn")
model = og.Model(config)
```

## Configuration in genai\_config.json

Providers can be configured directly in your model's `genai_config.json`:

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "provider_options": [
          {
            "cuda": {}
          }
        ]
      }
    }
  }
}
```

<Note>
  The `provider_options` array specifies execution providers in priority order. ONNX Runtime will use the first available provider.
</Note>

## Device Filtering

For multi-device systems, you can filter by hardware type:

```json theme={null}
{
  "provider_options": [
    {
      "openvino": {
        "device_type": "GPU"
      },
      "device_filtering_options": {
        "hardware_device_type": "gpu",
        "hardware_device_id": 0
      }
    }
  ]
}
```

<Warning>
  Provider availability depends on your installation. Install provider-specific packages:

  * CUDA: `onnxruntime-genai-cuda`
  * DirectML: `onnxruntime-genai-directml`
  * Other providers may require building from source.
</Warning>

## Next Steps

<CardGroup cols={2}>
  <Card title="CUDA Setup" icon="microchip" href="/acceleration/cuda">
    Configure NVIDIA GPU acceleration
  </Card>

  <Card title="DirectML Setup" icon="windows" href="/acceleration/directml">
    Enable DirectML on Windows
  </Card>

  <Card title="OpenVINO Setup" icon="cpu" href="/acceleration/openvino">
    Optimize for Intel hardware
  </Card>

  <Card title="QNN Setup" icon="mobile" href="/acceleration/qnn">
    Deploy to Qualcomm devices
  </Card>
</CardGroup>