> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/microsoft/onnxruntime-genai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

> Run your first generative AI model with ONNX Runtime GenAI

# Quickstart Guide

This guide will walk you through running your first generative AI model using ONNX Runtime GenAI. We'll use the Phi-3 model as an example, which is optimized for on-device AI scenarios.

<Info>
  This quickstart uses Python. For C# or C++ examples, see the [examples directory](https://github.com/microsoft/onnxruntime-genai/tree/main/examples).
</Info>

## Prerequisites

Before starting, ensure you have:

* Python 3.8 or later installed
* ONNX Runtime GenAI installed (see [Installation](/installation))
* At least 4GB of free disk space for the model
* 8GB+ RAM recommended

## Step 1: Download the Model

First, download a pre-optimized ONNX model. We'll use the Phi-3 Mini model optimized for CPU.

<Steps>
  <Step title="Install Hugging Face CLI">
    ```bash theme={null}
    pip install huggingface-hub[cli]
    ```
  </Step>

  <Step title="Download Phi-3 Model">
    Download the CPU-optimized INT4 quantized model:

    ```bash theme={null}
    huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx \
      --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* \
      --local-dir .
    ```

    This downloads the model to `./cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/`

    <Tip>
      For GPU acceleration, download a GPU-optimized variant:

      ```bash theme={null}
      huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx \
        --include cuda/cuda-int4-rtn-block-32/* \
        --local-dir .
      ```
    </Tip>
  </Step>
</Steps>

### Alternative: Download via Foundry Local

You can also use Foundry Local to download models:

```bash theme={null}
# Install Foundry Local from https://github.com/microsoft/Foundry-Local/releases
foundry model list
foundry model download Phi-4-generic-cpu
foundry cache location
```

## Step 2: Install Required Packages

Ensure you have the necessary Python packages:

```bash theme={null}
pip install numpy
pip install --pre onnxruntime-genai
```

## Step 3: Run Your First Model

Create a Python script to run inference with streaming output:

<CodeGroup>
  ```python simple_chat.py theme={null}
  import onnxruntime_genai as og

  # Load the model
  model = og.Model('cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4')
  tokenizer = og.Tokenizer(model)
  stream = tokenizer.create_stream()

  # Set generation parameters
  search_options = {
      'max_length': 2048,
      'batch_size': 1
  }

  # Define chat template
  chat_template = '<|user|>\n{input} <|end|>\n<|assistant|>'

  # Get user input
  text = input("Input: ")
  if not text:
      print("Error, input cannot be empty")
      exit()

  # Format prompt with chat template
  prompt = f'{chat_template.format(input=text)}'

  # Encode the prompt
  input_tokens = tokenizer.encode(prompt)

  # Create generator
  params = og.GeneratorParams(model)
  params.set_search_options(**search_options)
  generator = og.Generator(model, params)

  # Generate tokens
  print("Output: ", end='', flush=True)

  try:
      generator.append_tokens(input_tokens)
      while not generator.is_done():
          generator.generate_next_token()
          new_token = generator.get_next_tokens()[0]
          print(stream.decode(new_token), end='', flush=True)
  except KeyboardInterrupt:
      print("  --control+c pressed, aborting generation--")

  print()
  del generator
  ```

  ```python batch_generation.py theme={null}
  import onnxruntime_genai as og
  import time

  # Load model
  model = og.Model('cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4')
  tokenizer = og.Tokenizer(model)

  # Multiple prompts for batch processing
  prompts = [
      "The first 4 digits of pi are",
      "The square root of 2 is",
      "The first 6 numbers of the Fibonacci sequence are",
  ]

  # Configure search options
  search_options = {
      'max_length': 200,
      'batch_size': len(prompts)
  }

  # Encode all prompts
  input_tokens = tokenizer.encode_batch(prompts)

  # Create generator with parameters
  params = og.GeneratorParams(model)
  params.set_search_options(**search_options)
  generator = og.Generator(model, params)

  # Append tokens and generate
  generator.append_tokens(input_tokens)

  print("Running generation...\n")
  start_time = time.time()

  while not generator.is_done():
      generator.generate_next_token()

  run_time = time.time() - start_time

  # Print results
  for i in range(len(prompts)):
      print(f"Prompt #{i}: {prompts[i]}")
      print()
      print(tokenizer.decode(generator.get_sequence(i)))
      print()

  total_tokens = sum(len(generator.get_sequence(i)) for i in range(len(prompts)))
  print(f"\nTokens: {total_tokens}, Time: {run_time:.2f}s, Tokens/sec: {total_tokens/run_time:.2f}")
  ```
</CodeGroup>

## Step 4: Run the Script

Execute your script:

```bash theme={null}
python simple_chat.py
```

### Expected Output

You should see output similar to:

```
Input: What is the capital of France?
Output: The capital of France is Paris. It is located in the north-central 
part of the country and is known for its rich history, culture, and iconic 
landmarks such as the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral.
```

## Understanding the Code

Let's break down the key components:

<Steps>
  <Step title="Load Model and Tokenizer">
    ```python theme={null}
    model = og.Model('cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4')
    tokenizer = og.Tokenizer(model)
    stream = tokenizer.create_stream()
    ```

    * `Model`: Loads the ONNX model from the specified directory
    * `Tokenizer`: Handles text encoding/decoding using the model's vocabulary
    * `TokenizerStream`: Enables streaming token decoding for real-time output
  </Step>

  <Step title="Configure Generation Parameters">
    ```python theme={null}
    search_options = {
        'max_length': 2048,
        'batch_size': 1
    }
    params = og.GeneratorParams(model)
    params.set_search_options(**search_options)
    ```

    * `max_length`: Maximum number of tokens to generate
    * `batch_size`: Number of sequences to generate simultaneously
    * Additional options: `top_k`, `top_p`, `temperature`, `num_beams`
  </Step>

  <Step title="Encode Input and Generate">
    ```python theme={null}
    input_tokens = tokenizer.encode(prompt)
    generator = og.Generator(model, params)
    generator.append_tokens(input_tokens)

    while not generator.is_done():
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        print(stream.decode(new_token), end='', flush=True)
    ```

    * Encode text to tokens
    * Create generator with model and parameters
    * Generate tokens one at a time in a loop
    * Decode and print each token for streaming output
  </Step>
</Steps>

## Advanced Examples

### Continuous Chat with History

For a chat application that maintains conversation history:

<CodeGroup>
  ```python Python theme={null}
  import onnxruntime_genai as og

  model = og.Model('cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4')
  tokenizer = og.Tokenizer(model)
  stream = tokenizer.create_stream()

  params = og.GeneratorParams(model)
  params.set_search_options(max_length=2048)

  # System prompt
  system_prompt = '<|system|>\nYou are a helpful AI assistant.<|end|>\n'
  system_tokens = tokenizer.encode(system_prompt)

  generator = og.Generator(model, params)
  generator.append_tokens(system_tokens)
  system_prompt_length = len(system_tokens)

  while True:
      text = input("Prompt (use quit() to exit): ")
      if text == "quit()":
          break
      
      # Format user message
      user_prompt = f'<|user|>\n{text}<|end|>\n<|assistant|>'
      user_tokens = tokenizer.encode(user_prompt)
      generator.append_tokens(user_tokens)
      
      print("\nOutput: ", end='', flush=True)
      
      while not generator.is_done():
          generator.generate_next_token()
          new_token = generator.get_next_tokens()[0]
          print(stream.decode(new_token), end='', flush=True)
      
      print("\n")
  ```

  ```csharp C# theme={null}
  using Microsoft.ML.OnnxRuntimeGenAI;

  using var model = new Model("cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4");
  using var tokenizer = new Tokenizer(model);
  using var stream = tokenizer.CreateStream();

  using var generatorParams = new GeneratorParams(model);
  generatorParams.SetSearchOption("max_length", 2048);

  using var generator = new Generator(model, generatorParams);

  // System prompt
  var systemPrompt = "<|system|>\nYou are a helpful AI assistant.<|end|>\n";
  var systemTokens = tokenizer.Encode(systemPrompt);
  generator.AppendTokenSequences(systemTokens);

  while (true)
  {
      Console.Write("Prompt: ");
      var text = Console.ReadLine();
      if (text == "quit()") break;
      
      var userPrompt = $"<|user|>\n{text}<|end|>\n<|assistant|>";
      var userTokens = tokenizer.Encode(userPrompt);
      generator.AppendTokenSequences(userTokens);
      
      Console.Write("\nOutput: ");
      
      while (!generator.IsDone())
      {
          generator.GenerateNextToken();
          var newToken = generator.GetNextTokens()[0];
          Console.Write(stream.Decode(newToken));
      }
      
      Console.WriteLine("\n");
  }
  ```

  ```cpp C++ theme={null}
  #include <onnxruntime_genai.h>
  #include <iostream>
  #include <string>

  int main() {
      auto model = OgaModel::Create("cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4");
      auto tokenizer = OgaTokenizer::Create(*model);
      auto stream = OgaTokenizerStream::Create(*tokenizer);
      
      auto params = OgaGeneratorParams::Create(*model);
      params->SetSearchOption("max_length", 2048);
      
      auto generator = OgaGenerator::Create(*model, *params);
      
      // System prompt
      std::string system_prompt = "<|system|>\nYou are a helpful AI assistant.<|end|>\n";
      auto sequences = OgaSequences::Create();
      tokenizer->Encode(system_prompt.c_str(), *sequences);
      generator->AppendTokenSequences(*sequences);
      
      while (true) {
          std::cout << "Prompt: ";
          std::string text;
          std::getline(std::cin, text);
          
          if (text == "quit()") break;
          
          std::string user_prompt = "<|user|>\n" + text + "<|end|>\n<|assistant|>";
          sequences = OgaSequences::Create();
          tokenizer->Encode(user_prompt.c_str(), *sequences);
          generator->AppendTokenSequences(*sequences);
          
          std::cout << "\nOutput: ";
          
          while (!generator->IsDone()) {
              generator->GenerateNextToken();
              auto new_token = generator->GetNextTokens()[0];
              std::cout << stream->Decode(new_token) << std::flush;
          }
          
          std::cout << "\n\n";
      }
      
      return 0;
  }
  ```
</CodeGroup>

## Performance Tips

<CardGroup cols={2}>
  <Card title="Choose the Right Quantization" icon="sliders">
    * INT4: Best for CPU, smallest model size
    * FP16: Recommended for GPUs
    * FP32: Highest accuracy, larger size
  </Card>

  <Card title="Use Appropriate Hardware" icon="microchip">
    * CPU: Good for testing and small models
    * CUDA: Best for NVIDIA GPUs
    * DirectML: Windows GPU acceleration
    * TensorRT: Optimized NVIDIA inference
  </Card>

  <Card title="Batch Processing" icon="layer-group">
    Process multiple prompts together to improve throughput:

    ```python theme={null}
    prompts = ["prompt1", "prompt2", "prompt3"]
    input_tokens = tokenizer.encode_batch(prompts)
    ```
  </Card>

  <Card title="Adjust Generation Parameters" icon="gauge-high">
    * Lower `max_length` for faster responses
    * Adjust `temperature` for creativity (0.0-1.0)
    * Use `top_k` and `top_p` for quality/speed tradeoff
  </Card>
</CardGroup>

## Common Issues and Solutions

<AccordionGroup>
  <Accordion title="Slow Generation Speed">
    * Use GPU acceleration if available
    * Download INT4 quantized models for CPU
    * Reduce `max_length` parameter
    * Close other applications to free up RAM
  </Accordion>

  <Accordion title="Out of Memory Errors">
    * Use smaller batch sizes
    * Download a more quantized model (INT4 vs FP16)
    * Reduce `max_length` parameter
    * Ensure enough RAM/VRAM for the model
  </Accordion>

  <Accordion title="Model Not Found Error">
    Verify the model path is correct:

    ```python theme={null}
    import os
    model_path = 'cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4'
    print(f"Model exists: {os.path.exists(model_path)}")
    print(f"Contents: {os.listdir(model_path)}")
    ```

    The directory should contain:

    * `genai_config.json`
    * `*.onnx` files
    * Tokenizer files
  </Accordion>

  <Accordion title="Empty or Incorrect Output">
    * Verify chat template matches your model
    * Check that input prompt is not empty
    * Ensure `max_length` is sufficient
    * Try adjusting temperature and sampling parameters
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Explore More Models" icon="brain" href="https://huggingface.co/models?library=onnx&other=generative-ai">
    Browse ONNX models on Hugging Face for different use cases
  </Card>

  <Card title="Advanced Features" icon="wand-magic-sparkles">
    Learn about:

    * Multi-LoRA support
    * Constrained decoding for JSON output
    * Vision and audio models
    * Custom model optimization
  </Card>

  <Card title="API Reference" icon="code">
    Detailed documentation of all classes and methods in the ONNX Runtime GenAI API
  </Card>

  <Card title="Examples Repository" icon="folder-open" href="https://github.com/microsoft/onnxruntime-genai/tree/main/examples">
    Complete examples for Python, C#, C++, and more advanced scenarios
  </Card>
</CardGroup>

## Download Models

For a comprehensive guide on downloading and preparing models, see:

<Card title="Download Models Guide" icon="download" href="/download-models">
  Learn how to download models via Foundry Local, Hugging Face, or build your own
</Card>