> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.
# Architecture Overview
> Understanding the core architecture of ONNX Runtime GenAI
## Introduction
ONNX Runtime GenAI is a library designed to run generative AI models with ONNX Runtime. It implements the complete generative AI loop, including:
* Pre and post processing
* Inference with ONNX Runtime
* Logits processing
* Search and sampling
* KV cache management
* Grammar specification for tool calling
The library provides a high-level API that abstracts away the complexity of running generative models while maintaining flexibility and performance.
## Key Components
The ONNX Runtime GenAI architecture consists of four primary components that work together to execute the generative AI loop:
Manages the ONNX model, session options, and device configuration
Handles text encoding/decoding and token stream processing
Orchestrates the generation loop and manages state
Configures search strategies and generation parameters
### Model
The `Model` class (defined in `src/models/model.h:145`) is responsible for:
* Loading and managing ONNX models from disk or memory
* Creating and configuring ORT sessions with appropriate execution providers
* Managing device allocation (CPU, CUDA, DirectML, etc.)
* Providing tokenizer and processor creation
```cpp theme={null}
struct Model : std::enable_shared_from_this {
std::unique_ptr config_;
DeviceInterface* p_device_; // Primary device
DeviceInterface* p_device_inputs_; // Device for inputs
DeviceInterface* p_device_kvcache_; // Device for KV cache
SessionInfo session_info_;
};
```
The Model supports various execution providers including CPU, CUDA, DirectML, TensorRT, OpenVINO, QNN, and WebGPU.
### Tokenizer
The `Tokenizer` class (defined in `src/models/model.h:82`) handles:
* Text encoding to token IDs
* Token decoding to text
* Batch encoding/decoding
* Chat template application
* Streaming token decode with `TokenizerStream`
```cpp theme={null}
struct Tokenizer {
std::vector Encode(const char* text) const;
std::string Decode(std::span tokens) const;
std::unique_ptr CreateStream() const;
};
```
### Generator
The `Generator` class (defined in `src/generators.h:99`) is the central orchestrator that:
* Manages the generation state
* Executes the generation loop
* Coordinates between search strategy and model inference
* Handles token appending and sequence management
```cpp theme={null}
struct Generator {
std::shared_ptr model_;
std::unique_ptr state_; // Model state and inference
std::unique_ptr search_; // Search strategy (greedy/beam)
void AppendTokens(cpu_span input_ids);
void GenerateNextToken();
bool IsDone();
DeviceSpan GetSequence(size_t index) const;
};
```
### GeneratorParams
The `GeneratorParams` class (defined in `src/generators.h:71`) configures:
* Search parameters (beam size, max length, temperature, etc.)
* Sampling options (top-k, top-p, temperature)
* Device configuration
* Guidance for constrained decoding
```cpp theme={null}
struct GeneratorParams {
Config::Search search; // Search configuration
int max_batch_size;
bool use_graph_capture;
std::string guidance_type; // e.g., json_schema or regex
std::string guidance_data;
};
```
## The Generative AI Loop
The core generation loop follows this pattern:
Create Model, GeneratorParams, and Generator instances. Encode the input prompt using the Tokenizer.
Feed the encoded prompt tokens to the Generator using `AppendTokens()` or `AppendTokenSequences()`.
Repeatedly call `GenerateNextToken()` until `IsDone()` returns true:
1. **Run Inference**: The State executes the model with current tokens
2. **Get Logits**: Extract output logits from the model
3. **Apply Constraints**: Process logits (min length, repetition penalty, guidance)
4. **Search Strategy**: Select next token(s) based on search method
5. **Update State**: Append selected token(s) and update KV cache
6. **Check Termination**: Test for EOS tokens or max length
Get the generated sequence(s) using `GetSequence()` and decode with the Tokenizer.
### Example Flow Diagram
```mermaid theme={null}
graph TD
A[Input Text] --> B[Tokenizer.Encode]
B --> C[Input Token IDs]
C --> D[Generator.AppendTokens]
D --> E{Generator.IsDone?}
E -->|No| F[Generator.GenerateNextToken]
F --> G[State.Run - Model Inference]
G --> H[Get Logits]
H --> I[Search.SelectTop / Sample]
I --> J[Append Next Token]
J --> K[Update KV Cache]
K --> E
E -->|Yes| L[Generator.GetSequence]
L --> M[Tokenizer.Decode]
M --> N[Output Text]
```
## Component Relationships
The components interact in a hierarchical manner:
* **Model** creates **State** instances that manage model execution
* **GeneratorParams** configures both **Generator** and **Search** behavior
* **Generator** owns **State** (model execution) and **Search** (token selection)
* **Search** manages **Sequences** (token history for all beams/batches)
* **State** manages **KeyValueCache** for efficient autoregressive generation
```cpp theme={null}
// From src/generators.cpp - Generator creation
Generator::Generator(const Model& model, const GeneratorParams& params)
: model_{model.shared_from_this()},
state_{model.CreateState(sequence_lengths, params)},
search_{CreateSearch(params)} {
// ...
}
```
## Code Example
Here's a complete example showing the component interaction:
```python Python theme={null}
import onnxruntime_genai as og
# Create Model
model = og.Model('model_path')
# Create Tokenizer
tokenizer = og.Tokenizer(model)
# Encode input
prompt = "What is the capital of France?"
input_tokens = tokenizer.encode(prompt)
# Configure generation
params = og.GeneratorParams(model)
params.set_search_options(max_length=100, top_k=50, temperature=0.7)
# Create Generator
generator = og.Generator(model, params)
# Run generation loop
generator.append_tokens(input_tokens)
while not generator.is_done():
generator.generate_next_token()
# Optionally stream tokens
new_token = generator.get_next_tokens()[0]
# Get result
output_tokens = generator.get_sequence(0)
output_text = tokenizer.decode(output_tokens)
print(output_text)
```
```cpp C++ theme={null}
#include "ort_genai.h"
// Create Model
auto model = OgaModel::Create("model_path");
// Create Tokenizer
auto tokenizer = OgaTokenizer::Create(*model);
// Encode input
auto sequences = OgaSequences::Create();
tokenizer->Encode("What is the capital of France?", *sequences);
// Configure generation
auto params = OgaGeneratorParams::Create(*model);
params->SetSearchOption("max_length", 100);
params->SetSearchOption("top_k", 50);
params->SetSearchOption("temperature", 0.7);
// Create Generator and run
auto generator = OgaGenerator::Create(*model, *params);
generator->AppendTokenSequences(*sequences);
while (!generator->IsDone()) {
generator->GenerateNextToken();
}
// Get result
auto output_sequence = generator->GetSequence(0);
auto output_string = tokenizer->Decode(output_sequence);
std::cout << output_string << std::endl;
```
## Device Management
ONNX Runtime GenAI supports multiple device types for different components:
* **p\_device**: Primary computation device (CPU, CUDA, DirectML, etc.)
* **p\_device\_inputs**: Device for model inputs (may differ from primary for some EPs)
* **p\_device\_kvcache**: Device for KV cache storage (typically matches primary device)
The library automatically manages memory allocation and transfers between devices based on the execution provider configuration.
## Next Steps
Learn about supported model architectures and configuration
Explore search strategies and generation parameters
Understand KV cache management and optimization
Browse the complete API documentation