> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runtime Options

> Configure model behavior dynamically during inference with SetRuntimeOption API

The SetRuntimeOption API allows you to configure model behavior dynamically during inference without restarting the session. This guide covers all available runtime options and their usage.

## Overview

Runtime options enable you to:

* Terminate generation on-demand
* Enable/disable profiling during inference
* Configure session behavior without reloading the model

All runtime options are set using key-value pairs through the `SetRuntimeOption` API.

## Available Options

### Terminate Session

Control whether to terminate the current generation session or recover from a terminated state.

<ParamField path="terminate_session" type="string">
  **Accepted values:** `"0"` or `"1"`

  * `"1"`: Terminate the current session
  * `"0"`: Recover from a terminated state and continue/restart
</ParamField>

#### How It Works

When you enable session termination:

1. The current generation will throw an exception
2. Your code must handle this exception
3. You can recover by setting the option back to `"0"`

#### Python Example

```python theme={null}
import onnxruntime_genai as og

model = og.Model('model_path')
params = og.GeneratorParams(model)
generator = og.Generator(model, params)

try:
    # Start generation
    while not generator.is_done():
        generator.generate_next_token()
        
        # Check some condition to terminate early
        if should_terminate:
            # Terminate the session
            generator.set_runtime_option("terminate_session", "1")
            
except Exception as e:
    print(f"Generation terminated: {e}")
    
    # Recover and restart if needed
    generator.set_runtime_option("terminate_session", "0")
    # Can now start a new generation
```

#### C++ Example

```cpp theme={null}
#include "ort_genai.h"
#include <iostream>

try {
    auto generator = OgaGenerator::Create(*model, *params);
    
    while (!generator->IsDone()) {
        generator->GenerateNextToken();
        
        if (should_terminate) {
            generator->SetRuntimeOption("terminate_session", "1");
        }
    }
} catch (const std::exception& e) {
    std::cout << "Generation terminated: " << e.what() << std::endl;
    
    // Recover
    generator->SetRuntimeOption("terminate_session", "0");
}
```

#### C# Example

```csharp theme={null}
using Microsoft.ML.OnnxRuntimeGenAI;

try
{
    using var generator = new Generator(model, generatorParams);
    
    while (!generator.IsDone())
    {
        generator.GenerateNextToken();
        
        if (shouldTerminate)
        {
            generator.SetRuntimeOption("terminate_session", "1");
        }
    }
}
catch (Exception ex)
{
    Console.WriteLine($"Generation terminated: {ex.Message}");
    
    // Recover
    generator.SetRuntimeOption("terminate_session", "0");
}
```

<Info>
  See `examples/c/src/phi3_terminate.cpp` in the repository for a complete working example.
</Info>

### Enable Profiling

Dynamically enable or disable ONNX Runtime profiling during generation. When enabled, each token generation produces a separate profiling JSON file.

<ParamField path="enable_profiling" type="string">
  **Accepted values:** `"0"`, `"1"`, or a custom prefix string

  * `"0"`: Disable profiling
  * `"1"`: Enable profiling with default prefix `"onnxruntime_run_profile"`
  * `"<custom_prefix>"`: Enable profiling with custom file prefix
</ParamField>

#### How It Works

When profiling is enabled:

* Each `generate_next_token()` call creates a separate profiling file
* Files are named: `{prefix}_{timestamp}.json`
* You can start/stop profiling at any point during generation
* Useful for profiling specific portions of the generation process

#### Python Example

```python theme={null}
import onnxruntime_genai as og

model = og.Model('model_path')
params = og.GeneratorParams(model)
generator = og.Generator(model, params)

# Start generation without profiling
for i in range(10):
    generator.generate_next_token()

# Enable profiling with default prefix
generator.set_runtime_option("enable_profiling", "1")

# Profile the next 5 tokens
for i in range(5):
    generator.generate_next_token()
    # This creates: onnxruntime_run_profile_{timestamp}.json

# Disable profiling
generator.set_runtime_option("enable_profiling", "0")

# Continue generation without profiling
while not generator.is_done():
    generator.generate_next_token()
```

#### Custom Prefix Example

```python theme={null}
# Enable profiling with custom prefix
generator.set_runtime_option("enable_profiling", "my_model_profile")

for i in range(5):
    generator.generate_next_token()
    # This creates: my_model_profile_{timestamp}.json

# Disable profiling
generator.set_runtime_option("enable_profiling", "0")
```

#### C++ Example

```cpp theme={null}
auto generator = OgaGenerator::Create(*model, *params);

// Start profiling
generator->SetRuntimeOption("enable_profiling", "1");

for (int i = 0; i < 5; ++i) {
    generator->GenerateNextToken();
}

// Stop profiling
generator->SetRuntimeOption("enable_profiling", "0");
```

#### C# Example

```csharp theme={null}
using var generator = new Generator(model, generatorParams);

// Enable profiling with custom prefix
generator.SetRuntimeOption("enable_profiling", "inference_profile");

for (int i = 0; i < 5; i++)
{
    generator.GenerateNextToken();
}

// Disable profiling
generator.SetRuntimeOption("enable_profiling", "0");
```

## Profiling vs SessionOptions

<Note>
  **Runtime Option vs Session Option**

  There are two ways to enable profiling in ONNX Runtime GenAI:

  1. **SessionOptions** (`enable_profiling` in `genai_config.json`):
     * Session-level configuration
     * Collects all profiling data from session creation to end
     * Aggregates data into a single JSON file
     * Cannot be started or stopped dynamically
  2. **Runtime Option** (this API):
     * Can be enabled/disabled at any point during generation
     * Each token generation produces its own profiling file
     * Useful for profiling specific portions of generation
     * More flexible for targeted performance analysis
</Note>

## Analyzing Profiling Data

The profiling JSON files can be analyzed using:

<CardGroup cols={2}>
  <Card title="Chrome Tracing" icon="chrome">
    Open `chrome://tracing` in Chrome/Edge and load the JSON file
  </Card>

  <Card title="Perfetto" icon="chart-line">
    Use [Perfetto UI](https://ui.perfetto.dev/) for advanced analysis
  </Card>

  <Card title="Custom Scripts" icon="code">
    Parse the JSON for automated performance analysis
  </Card>

  <Card title="ONNX Runtime Tools" icon="wrench">
    Use ONNX Runtime's profiling analysis tools
  </Card>
</CardGroup>

## Common Patterns

### Profile Specific Generation Stages

```python theme={null}
import onnxruntime_genai as og

model = og.Model('model_path')
params = og.GeneratorParams(model)
generator = og.Generator(model, params)

# Phase 1: Prompt processing (no profiling)
prompt_tokens = tokenizer.encode(prompt)
generator.append_tokens(prompt_tokens)

# Phase 2: First few tokens (with profiling)
generator.set_runtime_option("enable_profiling", "first_tokens")
for i in range(10):
    generator.generate_next_token()

generator.set_runtime_option("enable_profiling", "0")

# Phase 3: Remaining tokens (no profiling)
while not generator.is_done():
    generator.generate_next_token()
```

### Conditional Termination

```python theme={null}
import onnxruntime_genai as og

model = og.Model('model_path')
tokenizer = og.Tokenizer(model)
params = og.GeneratorParams(model)
generator = og.Generator(model, params)

max_time_seconds = 10.0
start_time = time.time()

try:
    while not generator.is_done():
        generator.generate_next_token()
        
        # Terminate if taking too long
        if time.time() - start_time > max_time_seconds:
            print("Generation timeout - terminating")
            generator.set_runtime_option("terminate_session", "1")
            
except Exception as e:
    # Handle graceful termination
    partial_output = tokenizer.decode(generator.get_sequence(0))
    print(f"Partial output: {partial_output}")
```

### Debug Performance Issues

```python theme={null}
import onnxruntime_genai as og

model = og.Model('model_path')
params = og.GeneratorParams(model)
generator = og.Generator(model, params)

# Profile only the slow tokens
token_times = []

for i in range(100):
    start = time.time()
    generator.generate_next_token()
    elapsed = time.time() - start
    token_times.append(elapsed)
    
    # If a token is slow, enable profiling for the next few
    if elapsed > 0.1:  # 100ms threshold
        print(f"Slow token {i} detected: {elapsed:.3f}s")
        generator.set_runtime_option("enable_profiling", f"slow_token_{i}")
        
        # Profile next 5 tokens
        for j in range(5):
            generator.generate_next_token()
        
        generator.set_runtime_option("enable_profiling", "0")
```

## Best Practices

<AccordionGroup>
  <Accordion title="Use Profiling Sparingly">
    Profiling adds overhead to generation. Enable it only when needed for performance analysis, not in production.
  </Accordion>

  <Accordion title="Handle Termination Gracefully">
    Always wrap termination in try-catch blocks and handle partial results appropriately.
  </Accordion>

  <Accordion title="Use Descriptive Prefixes">
    When profiling, use descriptive prefixes that make it easy to identify which portion of code generated each profile.
  </Accordion>

  <Accordion title="Clean Up Profile Files">
    Profile files can accumulate quickly. Implement cleanup logic to remove old profiles.
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Constrained Decoding" icon="code-branch" href="/guides/constrained-decoding">
    Control output format with grammar constraints
  </Card>

  <Card title="Multi-LoRA" icon="layer-group" href="/guides/multi-lora">
    Switch between LoRA adapters dynamically
  </Card>

  <Card title="Python API" icon="python" href="/api-reference/python/generator">
    Explore the Generator API reference
  </Card>

  <Card title="Build from Source" icon="code" href="/guides/build-from-source">
    Build ONNX Runtime GenAI from source
  </Card>
</CardGroup>