> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# CUDA Execution Provider

> Accelerate inference with NVIDIA GPUs using the CUDA execution provider

The CUDA execution provider enables high-performance inference on NVIDIA GPUs with optimized kernels for generative AI workloads.

## Requirements

### Hardware

* NVIDIA GPU with Compute Capability 6.0 or higher
* Recommended: RTX 20 series or newer for optimal performance
* Minimum: 4GB GPU memory (varies by model size)

### Software

* CUDA Toolkit 11.8 or 12.x
* cuDNN 8.x or later
* NVIDIA driver 520.61.05 or newer (Linux) / 528.33 or newer (Windows)

<Note>
  The CUDA provider is included in the `onnxruntime-genai-cuda` package.
</Note>

## Installation

<Tabs>
  <Tab title="Python">
    ```bash theme={null}
    pip install onnxruntime-genai-cuda --pre
    ```
  </Tab>

  <Tab title="C++">
    Download the CUDA package from the [releases page](https://github.com/microsoft/onnxruntime-genai/releases).
  </Tab>

  <Tab title="C#">
    ```bash theme={null}
    dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda
    ```
  </Tab>
</Tabs>

## Basic Configuration

### Python API

```python theme={null}
import onnxruntime_genai as og

model_path = "path/to/model"

# Create config and set CUDA provider
config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")

# Load model
model = og.Model(config)
tokenizer = og.Tokenizer(model)

# Generate
params = og.GeneratorParams(model)
params.set_search_options(max_length=1024)

generator = og.Generator(model, params)
```

### genai\_config.json

Configure CUDA in your model configuration:

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "provider_options": [
          {
            "cuda": {}
          }
        ]
      }
    }
  }
}
```

## GPU Memory Management

### Memory Allocation

The CUDA provider uses ONNX Runtime's allocator for GPU memory:

```python theme={null}
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")

# Enable memory arena for better performance
config.set_provider_option("cuda", "enable_cuda_graph", "1")
config.set_provider_option("cuda", "gpu_mem_limit", "4294967296")  # 4GB

model = og.Model(config)
```

<Info>
  **GPU Memory Limit**: Set `gpu_mem_limit` to restrict CUDA memory usage (in bytes). This is useful for multi-tenant scenarios.
</Info>

### KV Cache Management

ONNX Runtime GenAI optimizes key-value cache management:

```python theme={null}
params = og.GeneratorParams(model)

# Configure cache sharing for better memory efficiency
params.set_search_options(
    max_length=2048,
    past_present_share_buffer=True  # Reduces memory usage
)
```

<Warning>
  When using beam search (`num_beams > 1`), `past_present_share_buffer` must be set to `False`. CUDA graph is also incompatible with beam search.
</Warning>

## Performance Tuning

### CUDA Graph Optimization

Enable CUDA graphs to reduce kernel launch overhead:

```python theme={null}
config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")

# Enable CUDA graph (for greedy search only)
config.set_provider_option("cuda", "enable_cuda_graph", "1")

model = og.Model(config)
```

<Note>
  CUDA graphs are only compatible with greedy search (`num_beams=1`). Disable for beam search scenarios.
</Note>

### Multi-Profile Support

Optimize for multiple sequence lengths:

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "enable_cuda_graph": "1",
        "cuda_graph_enable_multi_profile": "1"
      }
    }
  }
}
```

### Stream Configuration

The CUDA provider uses a dedicated stream for async operations. Memory transfers are optimized using pinned host memory:

```cpp theme={null}
// C++ example of stream usage
void* GetStream() { 
  return g_stream.get(); 
}

// Async memory copy
cudaMemcpyAsync(dst, src, size, cudaMemcpyDeviceToHost, GetStream());
cudaStreamSynchronize(GetStream());
```

## Precision Options

### FP16 Inference

The CUDA provider supports FP16 for reduced memory and faster inference:

```python theme={null}
# Use FP16 model variant
model_path = "model-fp16-cuda"

config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")

model = og.Model(config)
```

<Tabs>
  <Tab title="FP32">
    * **Memory**: Higher usage
    * **Speed**: Baseline performance
    * **Precision**: Full precision
    * **Hardware**: All CUDA GPUs
  </Tab>

  <Tab title="FP16">
    * **Memory**: 50% reduction
    * **Speed**: 2-3x faster on Tensor Cores
    * **Precision**: Mixed precision
    * **Hardware**: Compute Capability 7.0+ (Volta, Turing, Ampere, Ada)
  </Tab>
</Tabs>

## Advanced Configuration

### Device Selection

Select a specific GPU in multi-GPU systems:

```python theme={null}
import os

# Set CUDA device before loading model
os.environ["CUDA_VISIBLE_DEVICES"] = "0"  # Use first GPU

config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")

model = og.Model(config)
```

### Session Options

Fine-tune ONNX Runtime session settings:

```json theme={null}
{
  "model": {
    "decoder": {
      "session_options": {
        "enable_mem_pattern": true,
        "enable_cpu_mem_arena": false,
        "intra_op_num_threads": 1,
        "provider_options": [
          {
            "cuda": {
              "gpu_mem_limit": "8589934592",
              "arena_extend_strategy": "kSameAsRequested"
            }
          }
        ]
      }
    }
  }
}
```

## Troubleshooting

### Out of Memory Errors

```python theme={null}
# Reduce memory usage
config.set_provider_option("cuda", "gpu_mem_limit", "4294967296")

# Use smaller batch sizes
params.set_search_options(batch_size=1)

# Reduce max length
params.set_search_options(max_length=512)
```

### Performance Issues

<AccordionGroup>
  <Accordion title="Enable CUDA Graph">
    ```python theme={null}
    config.set_provider_option("cuda", "enable_cuda_graph", "1")
    ```
  </Accordion>

  <Accordion title="Use FP16 Models">
    Switch to FP16 model variants for Tensor Core acceleration.
  </Accordion>

  <Accordion title="Optimize KV Cache">
    ```python theme={null}
    params.set_search_options(past_present_share_buffer=True)
    ```
  </Accordion>
</AccordionGroup>

### Driver Compatibility

Verify CUDA installation:

```bash theme={null}
# Check CUDA version
nvcc --version

# Verify driver
nvidia-smi

# Test CUDA availability
python -c "import onnxruntime_genai as og; print('CUDA available')"
```

## Benchmarking

```python theme={null}
import time
import onnxruntime_genai as og

config = og.Config(model_path)
config.clear_providers()
config.append_provider("cuda")

model = og.Model(config)
tokenizer = og.Tokenizer(model)

prompt = "Tell me about AI"
input_tokens = tokenizer.encode(prompt)

params = og.GeneratorParams(model)
params.set_search_options(max_length=100)

start = time.time()
generator = og.Generator(model, params)
generator.append_tokens(input_tokens)

while not generator.is_done():
    generator.generate_next_token()

end = time.time()
print(f"Generation time: {end - start:.2f}s")
print(f"Tokens/sec: {100 / (end - start):.2f}")
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Model Optimization" icon="gauge-high" href="/optimization/overview">
    Optimize models for CUDA deployment
  </Card>

  <Card title="Memory Management" icon="memory" href="/guides/memory-management">
    Learn advanced memory techniques
  </Card>
</CardGroup>