> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenizer

> Encode and decode text for ONNX Runtime GenAI models

The `Tokenizer` class handles text encoding and decoding for language models. It converts text into token sequences that models can process and decodes token sequences back into readable text.

## Constructor

### Tokenizer(Model model)

Creates a tokenizer instance from a loaded model.

<ParamField path="model" type="Model" required>
  The model to create a tokenizer for
</ParamField>

```csharp Tokenizer.cs:14-17 theme={null}
using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);
```

## Encoding Methods

### Encode(string str)

Encodes a single string into a token sequence.

<ParamField path="str" type="string" required>
  The text to encode
</ParamField>

**Returns:** `Sequences` - A sequences object containing the encoded tokens

```csharp Tokenizer.cs:74-87 theme={null}
string text = "Hello, how are you?";
using Sequences sequences = tokenizer.Encode(text);
```

### EncodeBatch(string\[] strings)

Encodes multiple strings into token sequences.

<ParamField path="strings" type="string[]" required>
  Array of strings to encode
</ParamField>

**Returns:** `Sequences` - A sequences object containing all encoded sequences

```csharp Tokenizer.cs:19-36 theme={null}
string[] texts = new string[] { 
    "First prompt",
    "Second prompt"
};
using Sequences sequences = tokenizer.EncodeBatch(texts);
```

## Decoding Methods

### Decode(ReadOnlySpan\<int> sequence)

Decodes a token sequence into a string.

<ParamField path="sequence" type="ReadOnlySpan<int>" required>
  The token sequence to decode
</ParamField>

**Returns:** `string` - The decoded text

```csharp Tokenizer.cs:89-107 theme={null}
ReadOnlySpan<int> tokens = generator.GetSequence(0);
string decodedText = tokenizer.Decode(tokens);
Console.WriteLine(decodedText);
```

### DecodeBatch(Sequences sequences)

Decodes multiple token sequences into strings.

<ParamField path="sequences" type="Sequences" required>
  The sequences to decode
</ParamField>

**Returns:** `string[]` - Array of decoded strings

```csharp Tokenizer.cs:38-47 theme={null}
string[] decodedTexts = tokenizer.DecodeBatch(sequences);
foreach (string text in decodedTexts)
{
    Console.WriteLine(text);
}
```

## Chat Template Methods

### ApplyChatTemplate(string template\_str, string messages, string tools, bool add\_generation\_prompt)

Applies a chat template to format messages for the model.

<ParamField path="template_str" type="string" required>
  The Jinja template string (can be empty to use model's default)
</ParamField>

<ParamField path="messages" type="string" required>
  JSON-serialized array of chat messages
</ParamField>

<ParamField path="tools" type="string" required>
  JSON-serialized array of tools (can be empty)
</ParamField>

<ParamField path="add_generation_prompt" type="bool" required>
  Whether to add tokens indicating the start of the AI's response
</ParamField>

**Returns:** `string` - The formatted prompt

```csharp Tokenizer.cs:109-121 theme={null}
string messages = "[{\"role\":\"user\",\"content\":\"Hello!\"}]";
string prompt = tokenizer.ApplyChatTemplate(
    template_str: "",
    messages: messages,
    tools: "",
    add_generation_prompt: true
);
```

## Token ID Methods

### GetBosTokenId()

Returns the beginning-of-sequence token ID.

**Returns:** `int` - The BOS token ID

```csharp Tokenizer.cs:123-127 theme={null}
int bosTokenId = tokenizer.GetBosTokenId();
```

### GetEosTokenIds()

Returns the end-of-sequence token IDs.

**Returns:** `ReadOnlySpan<int>` - Span containing EOS token IDs

```csharp Tokenizer.cs:129-136 theme={null}
ReadOnlySpan<int> eosTokenIds = tokenizer.GetEosTokenIds();
```

### GetPadTokenId()

Returns the padding token ID.

**Returns:** `int` - The padding token ID

```csharp Tokenizer.cs:138-142 theme={null}
int padTokenId = tokenizer.GetPadTokenId();
```

## Advanced Methods

### UpdateOptions(Dictionary\<string, string> options)

Updates tokenizer options.

<ParamField path="options" type="Dictionary<string, string>" required>
  Dictionary of option names and values
</ParamField>

```csharp Tokenizer.cs:49-72 theme={null}
var options = new Dictionary<string, string>
{
    { "add_special_tokens", "true" }
};
tokenizer.UpdateOptions(options);
```

### CreateStream()

Creates a tokenizer stream for incremental decoding (useful for streaming generation).

**Returns:** `TokenizerStream` - A tokenizer stream instance

```csharp Tokenizer.cs:144-149 theme={null}
using TokenizerStream tokenizerStream = tokenizer.CreateStream();
```

## Complete Example

Here's a complete example showing encoding, generation, and decoding:

```csharp examples/csharp/ModelChat/Program.cs:44-84 theme={null}
using Microsoft.ML.OnnxRuntimeGenAI;
using System.Text.Json;

string modelPath = "/path/to/model";

// Load model and create tokenizer
using Model model = new Model(modelPath);
using Tokenizer tokenizer = new Tokenizer(model);

// Prepare chat messages
string systemPrompt = "You are a helpful AI assistant.";
string userPrompt = "What is the capital of France?";

string messages = $"[{{\"role\":\"system\",\"content\":\"{systemPrompt}\"}}," +
                  $"{{\"role\":\"user\",\"content\":\"{userPrompt}\"}}]";

// Apply chat template and encode
string prompt = tokenizer.ApplyChatTemplate(
    template_str: "",
    messages: messages,
    tools: "",
    add_generation_prompt: true
);

using var sequences = tokenizer.Encode(prompt);
Console.WriteLine($"Encoded {sequences.NumSequences} sequences");

// Create generator and generate response
using GeneratorParams generatorParams = new GeneratorParams(model);
using Generator generator = new Generator(model, generatorParams);

generator.AppendTokenSequences(sequences);

while (!generator.IsDone())
{
    generator.GenerateNextToken();
}

// Decode the output
var outputSequence = generator.GetSequence(0);
string outputString = tokenizer.Decode(outputSequence);

Console.WriteLine("Output:");
Console.WriteLine(outputString);
```

## Streaming Example

```csharp examples/csharp/ModelChat/Program.cs:200-213 theme={null}
using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using Tokenizer tokenizer = new Tokenizer(model);
using TokenizerStream tokenizerStream = tokenizer.CreateStream();

// ... setup generator and start generation ...

Console.Write("Output: ");
while (!generator.IsDone())
{
    generator.GenerateNextToken();
    // Decode and print each token as it's generated
    Console.Write(tokenizerStream.Decode(generator.GetNextTokens()[0]));
}
Console.WriteLine();
```

## See Also

* [Model](/api/csharp/model) - Load models
* [Generator](/api/csharp/generator) - Generate text
* [GeneratorParams](/api/csharp/params) - Configure generation