> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenizer

> Encode text to tokens and decode tokens to text

The `Tokenizer` class handles text encoding and decoding for language models.

## Constructor

Create a tokenizer from a model.

```python theme={null}
import onnxruntime_genai as og

model = og.Model("/path/to/model")
tokenizer = og.Tokenizer(model)
```

<ParamField path="model" type="Model" required>
  The Model object to create the tokenizer from
</ParamField>

## Properties

### bos\_token\_id

The beginning-of-sequence token ID.

```python theme={null}
bos_id = tokenizer.bos_token_id
```

<ResponseField name="bos_token_id" type="int">
  Token ID for the start of a sequence
</ResponseField>

### eos\_token\_ids

Array of end-of-sequence token IDs.

```python theme={null}
eos_ids = tokenizer.eos_token_ids
```

<ResponseField name="eos_token_ids" type="numpy.ndarray">
  Array of token IDs that mark the end of generation
</ResponseField>

### pad\_token\_id

The padding token ID.

```python theme={null}
pad_id = tokenizer.pad_token_id
```

<ResponseField name="pad_token_id" type="int">
  Token ID used for padding sequences
</ResponseField>

## Methods

### encode()

Encode a string into token IDs.

```python theme={null}
tokens = tokenizer.encode("Hello, world!")
```

<ParamField path="text" type="str" required>
  The text to encode
</ParamField>

<ResponseField name="tokens" type="numpy.ndarray">
  Array of int32 token IDs
</ResponseField>

### decode()

Decode token IDs back into text.

```python theme={null}
text = tokenizer.decode(tokens)
```

<ParamField path="tokens" type="numpy.ndarray" required>
  Array of int32 token IDs to decode
</ParamField>

<ResponseField name="text" type="str">
  The decoded text string
</ResponseField>

### encode\_batch()

Encode multiple strings at once.

```python theme={null}
prompts = ["First prompt", "Second prompt", "Third prompt"]
input_tokens = tokenizer.encode_batch(prompts)
```

<ParamField path="strings" type="list[str]" required>
  List of text strings to encode
</ParamField>

<ResponseField name="tokens" type="OgaTensor">
  Tensor containing all encoded sequences
</ResponseField>

### decode\_batch()

Decode multiple token sequences at once.

```python theme={null}
strings = tokenizer.decode_batch(tokens)
```

<ParamField path="tokens" type="OgaTensor" required>
  Tensor containing token sequences to decode
</ParamField>

<ResponseField name="strings" type="list[str]">
  List of decoded text strings
</ResponseField>

### to\_token\_id()

Convert a token string to its ID.

```python theme={null}
token_id = tokenizer.to_token_id("hello")
```

<ParamField path="token" type="str" required>
  The token string to convert
</ParamField>

<ResponseField name="id" type="int">
  The token ID
</ResponseField>

### apply\_chat\_template()

Apply a chat template to format messages.

```python theme={null}
import json

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What color is the sky?"}
]

prompt = tokenizer.apply_chat_template(
    messages=json.dumps(messages),
    add_generation_prompt=True
)
```

<ParamField path="messages" type="str" required>
  JSON-serialized list of message dictionaries with "role" and "content" fields
</ParamField>

<ParamField path="template_str" type="str" default="None">
  Custom Jinja template string (uses model's default if not provided)
</ParamField>

<ParamField path="tools" type="str" default="None">
  JSON-serialized list of tool definitions
</ParamField>

<ParamField path="add_generation_prompt" type="bool" default="True">
  Whether to add tokens indicating the assistant's turn to respond
</ParamField>

<ResponseField name="prompt" type="str">
  The formatted prompt ready for encoding
</ResponseField>

### create\_stream()

Create a streaming tokenizer for incremental decoding.

```python theme={null}
stream = tokenizer.create_stream()
```

<ResponseField name="stream" type="TokenizerStream">
  A TokenizerStream object for streaming decoding
</ResponseField>

### update\_options()

Update tokenizer options dynamically.

```python theme={null}
tokenizer.update_options(add_special_tokens="false", padding="max_length")
```

<ParamField path="**kwargs" type="dict">
  Key-value pairs of tokenizer options to update
</ParamField>

## TokenizerStream

Streaming decoder for incremental token-by-token decoding.

### decode()

Decode a single token and return the corresponding text chunk.

```python theme={null}
stream = tokenizer.create_stream()

while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(stream.decode(new_token), end="", flush=True)
```

<ParamField path="token" type="int" required>
  The token ID to decode
</ParamField>

<ResponseField name="text" type="str">
  The text fragment for this token (may be empty for partial multi-byte characters)
</ResponseField>

## Example Usage

Basic encoding and decoding:

```python theme={null}
import onnxruntime_genai as og

model = og.Model("/models/phi-3-mini")
tokenizer = og.Tokenizer(model)

# Encode
text = "The first 4 digits of pi are"
tokens = tokenizer.encode(text)
print(f"Encoded {len(tokens)} tokens")

# Decode
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")
```

Streaming generation:

```python theme={null}
import onnxruntime_genai as og

model = og.Model("/models/phi-3-mini")
tokenizer = og.Tokenizer(model)
stream = tokenizer.create_stream()

params = og.GeneratorParams(model)
params.set_search_options(max_length=200)

generator = og.Generator(model, params)
input_tokens = tokenizer.encode("Tell me a story")
generator.append_tokens(input_tokens)

print("Output: ", end="", flush=True)
while not generator.is_done():
    generator.generate_next_token()
    new_token = generator.get_next_tokens()[0]
    print(stream.decode(new_token), end="", flush=True)
print()
```

Batch encoding:

```python theme={null}
import onnxruntime_genai as og

model = og.Model("/models/phi-3-mini")
tokenizer = og.Tokenizer(model)

prompts = [
    "The first 4 digits of pi are",
    "The square root of 2 is",
    "The capital of France is"
]

# Encode batch
input_tokens = tokenizer.encode_batch(prompts)
print(f"Encoded {len(prompts)} prompts")

# Generate for all prompts
params = og.GeneratorParams(model)
params.set_search_options(batch_size=len(prompts), max_length=100)

generator = og.Generator(model, params)
generator.append_tokens(input_tokens)

while not generator.is_done():
    generator.generate_next_token()

# Decode each sequence
for i in range(len(prompts)):
    output = tokenizer.decode(generator.get_sequence(i))
    print(f"Prompt {i}: {output}")
    print()
```

Chat template:

```python theme={null}
import onnxruntime_genai as og
import json

model = og.Model("/models/phi-3-mini")
tokenizer = og.Tokenizer(model)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "What color is the sky?"}
]

prompt = tokenizer.apply_chat_template(
    messages=json.dumps(messages),
    add_generation_prompt=True
)

print(f"Formatted prompt: {prompt}")

input_tokens = tokenizer.encode(prompt)
# Continue with generation...
```