Tokenizer

Encode and decode text for ONNX Runtime GenAI models

The Tokenizer class handles text encoding and decoding for language models. It converts text into token sequences that models can process and decodes token sequences back into readable text.

Constructor

Tokenizer(Model model)

Creates a tokenizer instance from a loaded model.

model · Model · required

The model to create a tokenizer for

Tokenizer.cs:14-17

using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);

Encoding Methods

Encode(string str)

Encodes a single string into a token sequence.

str · string · required

The text to encode

Returns: Sequences - A sequences object containing the encoded tokens

Tokenizer.cs:74-87

string text = "Hello, how are you?";
using Sequences sequences = tokenizer.Encode(text);

EncodeBatch(string[] strings)

Encodes multiple strings into token sequences.

strings · string[] · required

Array of strings to encode

Returns: Sequences - A sequences object containing all encoded sequences

Tokenizer.cs:19-36

string[] texts = new string[] { 
    "First prompt",
    "Second prompt"
};
using Sequences sequences = tokenizer.EncodeBatch(texts);

Decoding Methods

Decode(ReadOnlySpan<int> sequence)

Decodes a token sequence into a string.

sequence · ReadOnlySpan<int> · required

The token sequence to decode

Returns: string - The decoded text

Tokenizer.cs:89-107

ReadOnlySpan<int> tokens = generator.GetSequence(0);
string decodedText = tokenizer.Decode(tokens);
Console.WriteLine(decodedText);

DecodeBatch(Sequences sequences)

Decodes multiple token sequences into strings.

sequences · Sequences · required

The sequences to decode

Returns: string[] - Array of decoded strings

Tokenizer.cs:38-47

string[] decodedTexts = tokenizer.DecodeBatch(sequences);
foreach (string text in decodedTexts)
{
    Console.WriteLine(text);
}

Chat Template Methods

ApplyChatTemplate(string template_str, string messages, string tools, bool add_generation_prompt)

Applies a chat template to format messages for the model.

template_str · string · required

The Jinja template string (can be empty to use model’s default)

messages · string · required

JSON-serialized array of chat messages

tools · string · required

JSON-serialized array of tools (can be empty)

add_generation_prompt · bool · required

Whether to add tokens indicating the start of the AI’s response

Returns: string - The formatted prompt

Tokenizer.cs:109-121

string messages = "[{\"role\":\"user\",\"content\":\"Hello!\"}]";
string prompt = tokenizer.ApplyChatTemplate(
    template_str: "",
    messages: messages,
    tools: "",
    add_generation_prompt: true
);

Token ID Methods

GetBosTokenId()

Returns the beginning-of-sequence token ID.

Returns: int - The BOS token ID

Tokenizer.cs:123-127

int bosTokenId = tokenizer.GetBosTokenId();

GetEosTokenIds()

Returns the end-of-sequence token IDs.

Returns: ReadOnlySpan<int> - Span containing EOS token IDs

Tokenizer.cs:129-136

ReadOnlySpan<int> eosTokenIds = tokenizer.GetEosTokenIds();

GetPadTokenId()

Returns the padding token ID.

Returns: int - The padding token ID

Tokenizer.cs:138-142

int padTokenId = tokenizer.GetPadTokenId();

Advanced Methods

UpdateOptions(Dictionary<string, string> options)

Updates tokenizer options.

options · Dictionary<string, string> · required

Dictionary of option names and values

Tokenizer.cs:49-72

var options = new Dictionary<string, string>
{
    { "add_special_tokens", "true" }
};
tokenizer.UpdateOptions(options);

CreateStream()

Creates a tokenizer stream for incremental decoding (useful for streaming generation).

Returns: TokenizerStream - A tokenizer stream instance

Tokenizer.cs:144-149

using TokenizerStream tokenizerStream = tokenizer.CreateStream();

Complete Example

Here’s a complete example showing encoding, generation, and decoding:

examples/csharp/ModelChat/Program.cs:44-84

using Microsoft.ML.OnnxRuntimeGenAI;
using System.Text.Json;

string modelPath = "/path/to/model";

// Load model and create tokenizer
using Model model = new Model(modelPath);
using Tokenizer tokenizer = new Tokenizer(model);

// Prepare chat messages
string systemPrompt = "You are a helpful AI assistant.";
string userPrompt = "What is the capital of France?";

string messages = $"[{{\"role\":\"system\",\"content\":\"{systemPrompt}\"}}," +
                  $"{{\"role\":\"user\",\"content\":\"{userPrompt}\"}}]";

// Apply chat template and encode
string prompt = tokenizer.ApplyChatTemplate(
    template_str: "",
    messages: messages,
    tools: "",
    add_generation_prompt: true
);

using var sequences = tokenizer.Encode(prompt);
Console.WriteLine($"Encoded {sequences.NumSequences} sequences");

// Create generator and generate response
using GeneratorParams generatorParams = new GeneratorParams(model);
using Generator generator = new Generator(model, generatorParams);

generator.AppendTokenSequences(sequences);

while (!generator.IsDone())
{
    generator.GenerateNextToken();
}

// Decode the output
var outputSequence = generator.GetSequence(0);
string outputString = tokenizer.Decode(outputSequence);

Console.WriteLine("Output:");
Console.WriteLine(outputString);

Streaming Example

examples/csharp/ModelChat/Program.cs:200-213

using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);
using TokenizerStream tokenizerStream = tokenizer.CreateStream();

// ... setup generator and start generation ...

Console.Write("Output: ");
while (!generator.IsDone())
{
    generator.GenerateNextToken();
    // Decode and print each token as it's generated
    Console.Write(tokenizerStream.Decode(generator.GetNextTokens()[0]));
}
Console.WriteLine();

See Also

• Model - Load models
• Generator - Generate text
• GeneratorParams - Configure generation