> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# OgaTokenizer

> Encode and decode text for model input and output

The `OgaTokenizer` class provides methods to encode text into token sequences and decode token sequences back into text. It's essential for preparing input for the model and interpreting the generated output.

## Class Definition

```cpp theme={null}
struct OgaTokenizer : OgaAbstract {
  static std::unique_ptr<OgaTokenizer> Create(const OgaModel& model);
  
  void Encode(const char* str, OgaSequences& sequences) const;
  OgaString Decode(const int32_t* tokens_data, size_t tokens_length) const;
  
  int32_t GetBosTokenId() const;
  std::span<const int32_t> GetEosTokenIds() const;
  int32_t GetPadTokenId() const;
  int32_t ToTokenId(const char* str) const;
  
  std::unique_ptr<OgaTensor> EncodeBatch(const char** strings, size_t count) const;
  std::unique_ptr<OgaStringArray> DecodeBatch(const OgaTensor& tensor) const;
  
  OgaString ApplyChatTemplate(const char* template_str, const char* messages, 
                              const char* tools, bool add_generation_prompt) const;
  void UpdateOptions(const char* const* keys, const char* const* values, size_t num_options);
};
```

Defined in: `~/workspace/source/src/ort_genai.h:301`

## Methods

### Create()

Create a tokenizer instance from a model.

```cpp theme={null}
static std::unique_ptr<OgaTokenizer> Create(const OgaModel& model)
```

<ParamField path="model" type="const OgaModel&" required>
  The model to create the tokenizer from
</ParamField>

**Returns:** `std::unique_ptr<OgaTokenizer>` - A unique pointer to the created tokenizer

#### Example

```cpp theme={null}
auto model = OgaModel::Create("phi-2");
auto tokenizer = OgaTokenizer::Create(*model);
```

### Encode()

Encode a string into token sequences.

```cpp theme={null}
void Encode(const char* str, OgaSequences& sequences) const
```

<ParamField path="str" type="const char*" required>
  The input text to encode
</ParamField>

<ParamField path="sequences" type="OgaSequences&" required>
  The sequences object to store the encoded tokens
</ParamField>

**Throws:** `std::runtime_error` if encoding fails

#### Example

```cpp theme={null}
auto sequences = OgaSequences::Create();
tokenizer->Encode("A great recipe for Kung Pao chicken is ", *sequences);

// Access the encoded tokens
const int32_t* tokens = sequences->SequenceData(0);
size_t token_count = sequences->SequenceCount(0);
```

### Decode()

Decode token sequences back into text.

```cpp theme={null}
OgaString Decode(const int32_t* tokens_data, size_t tokens_length) const
```

<ParamField path="tokens_data" type="const int32_t*" required>
  Pointer to the token data array
</ParamField>

<ParamField path="tokens_length" type="size_t" required>
  Number of tokens in the array
</ParamField>

**Returns:** `OgaString` - The decoded text string

#### Example

```cpp theme={null}
// Decode tokens from generator output
auto output_sequence = generator->GetSequenceData(0);
size_t output_length = generator->GetSequenceCount(0);
auto output_string = tokenizer->Decode(output_sequence, output_length);

std::cout << "Output: " << output_string << std::endl;
```

#### Example with std::span (C++20)

```cpp theme={null}
#if OGA_USE_SPAN
std::span<const int32_t> tokens = generator->GetSequence(0);
auto output_string = tokenizer->Decode(tokens);
#endif
```

### GetBosTokenId()

Get the beginning-of-sequence token ID.

```cpp theme={null}
int32_t GetBosTokenId() const
```

**Returns:** `int32_t` - The BOS token ID

#### Example

```cpp theme={null}
int32_t bos_id = tokenizer->GetBosTokenId();
std::cout << "BOS token ID: " << bos_id << std::endl;
```

### GetEosTokenIds()

Get the end-of-sequence token IDs.

```cpp theme={null}
std::span<const int32_t> GetEosTokenIds() const  // C++20
std::vector<int32_t> GetEosTokenIds() const      // Pre-C++20
```

**Returns:** A span or vector of EOS token IDs

#### Example

```cpp theme={null}
auto eos_ids = tokenizer->GetEosTokenIds();
for (int32_t eos_id : eos_ids) {
  std::cout << "EOS token ID: " << eos_id << std::endl;
}
```

### GetPadTokenId()

Get the padding token ID.

```cpp theme={null}
int32_t GetPadTokenId() const
```

**Returns:** `int32_t` - The padding token ID

### ToTokenId()

Convert a string to its corresponding token ID.

```cpp theme={null}
int32_t ToTokenId(const char* str) const
```

<ParamField path="str" type="const char*" required>
  The string to convert
</ParamField>

**Returns:** `int32_t` - The token ID for the string

#### Example

```cpp theme={null}
int32_t token_id = tokenizer->ToTokenId("Hello");
std::cout << "Token ID for 'Hello': " << token_id << std::endl;
```

### EncodeBatch()

Encode multiple strings in a batch.

```cpp theme={null}
std::unique_ptr<OgaTensor> EncodeBatch(const char** strings, size_t count) const
```

<ParamField path="strings" type="const char**" required>
  Array of strings to encode
</ParamField>

<ParamField path="count" type="size_t" required>
  Number of strings in the array
</ParamField>

**Returns:** `std::unique_ptr<OgaTensor>` - Tensor containing the encoded batch

### DecodeBatch()

Decode a batch of token sequences.

```cpp theme={null}
std::unique_ptr<OgaStringArray> DecodeBatch(const OgaTensor& tensor) const
```

<ParamField path="tensor" type="const OgaTensor&" required>
  Tensor containing the token sequences to decode
</ParamField>

**Returns:** `std::unique_ptr<OgaStringArray>` - Array of decoded strings

### ApplyChatTemplate()

Apply a chat template to format messages.

```cpp theme={null}
OgaString ApplyChatTemplate(const char* template_str, const char* messages, 
                           const char* tools, bool add_generation_prompt) const
```

<ParamField path="template_str" type="const char*" required>
  The chat template string (can be nullptr to use default)
</ParamField>

<ParamField path="messages" type="const char*" required>
  JSON string containing the chat messages
</ParamField>

<ParamField path="tools" type="const char*">
  JSON string containing tool definitions (optional)
</ParamField>

<ParamField path="add_generation_prompt" type="bool" required>
  Whether to add the generation prompt
</ParamField>

**Returns:** `OgaString` - The formatted prompt with chat template applied

#### Example

```cpp theme={null}
const char* messages = R"([
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "What is the weather?"}
])";

auto prompt = tokenizer->ApplyChatTemplate(nullptr, messages, nullptr, true);
std::cout << "Formatted prompt: " << prompt << std::endl;
```

### UpdateOptions()

Update tokenizer options.

```cpp theme={null}
void UpdateOptions(const char* const* keys, const char* const* values, size_t num_options)
```

<ParamField path="keys" type="const char* const*" required>
  Array of option keys
</ParamField>

<ParamField path="values" type="const char* const*" required>
  Array of option values
</ParamField>

<ParamField path="num_options" type="size_t" required>
  Number of options
</ParamField>

## Streaming Tokenization

For streaming output, use `OgaTokenizerStream` to decode tokens one at a time:

### OgaTokenizerStream

```cpp theme={null}
struct OgaTokenizerStream : OgaAbstract {
  static std::unique_ptr<OgaTokenizerStream> Create(const OgaTokenizer& tokenizer);
  const char* Decode(int32_t token);
};
```

Defined in: `~/workspace/source/src/ort_genai.h:385`

#### Example

```cpp theme={null}
// Create a streaming tokenizer
auto stream = OgaTokenizerStream::Create(*tokenizer);

// Decode tokens one at a time during generation
while (!generator->IsDone()) {
  generator->GenerateNextToken();
  
  const auto new_token = generator->GetNextTokens()[0];
  const char* chunk = stream->Decode(new_token);
  std::cout << chunk << std::flush;
}
```

## Complete Example

From `~/workspace/source/examples/c/src/model_qa.cpp:126`:

```cpp theme={null}
#include "ort_genai.h"
#include <iostream>

int main() {
  OgaHandle handle;
  
  try {
    // Create model and tokenizer
    auto model = OgaModel::Create("phi-2");
    auto tokenizer = OgaTokenizer::Create(*model);
    auto stream = OgaTokenizerStream::Create(*tokenizer);
    
    // Encode input text
    auto sequences = OgaSequences::Create();
    tokenizer->Encode("A great recipe for Kung Pao chicken is ", *sequences);
    
    // Create generator and append tokens
    auto params = OgaGeneratorParams::Create(*model);
    params->SetSearchOption("max_length", 200);
    
    auto generator = OgaGenerator::Create(*model, *params);
    generator->AppendTokenSequences(*sequences);
    
    // Generate and decode output
    std::cout << "Output: ";
    while (!generator->IsDone()) {
      generator->GenerateNextToken();
      const auto new_token = generator->GetNextTokens()[0];
      std::cout << stream->Decode(new_token) << std::flush;
    }
    std::cout << std::endl;
    
    // Or decode the full sequence at once
    auto output_sequence = generator->GetSequenceData(0);
    auto output_length = generator->GetSequenceCount(0);
    auto output_string = tokenizer->Decode(output_sequence, output_length);
    
  } catch (const std::exception& e) {
    std::cerr << "Error: " << e.what() << std::endl;
    return -1;
  }
  
  return 0;
}
```

## See Also

* [OgaModel](/api/cpp/model) - Create models
* [OgaGenerator](/api/cpp/generator) - Generate text
* [OgaSequences](~/workspace/source/src/ort_genai.h:259) - Manage token sequences