GeneratorParams

Configure text generation parameters for ONNX Runtime GenAI

The GeneratorParams class controls the behavior of text generation, including search strategies, sampling parameters, and output constraints.

Constructor

GeneratorParams(Model model)

Creates a new generator parameters instance.

model · Model · required

The model to create parameters for

GeneratorParams.cs:15-18

using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using GeneratorParams generatorParams = new GeneratorParams(model);

Search Option Methods

SetSearchOption(string searchOption, double value)

Sets a numeric search option (e.g., temperature, top_p, max_length).

searchOption · string · required

The name of the search option to set

value · double · required

The numeric value for the option

GeneratorParams.cs:22-25

generatorParams.SetSearchOption("max_length", 1024);
generatorParams.SetSearchOption("temperature", 0.7);
generatorParams.SetSearchOption("top_p", 0.9);
generatorParams.SetSearchOption("top_k", 50);

SetSearchOption(string searchOption, bool value)

Sets a boolean search option (e.g., do_sample).

searchOption · string · required

The name of the search option to set

value · bool · required

The boolean value for the option

GeneratorParams.cs:27-30

generatorParams.SetSearchOption("do_sample", true);

GetSearchNumber(string searchOption)

Retrieves the value of a numeric search option.

searchOption · string · required

The name of the search option to retrieve

Returns: double - The current value of the option

GeneratorParams.cs:37-41

double temperature = generatorParams.GetSearchNumber("temperature");
Console.WriteLine($"Temperature: {temperature}");

GetSearchBool(string searchOption)

Retrieves the value of a boolean search option.

searchOption · string · required

The name of the search option to retrieve

Returns: bool - The current value of the option

GeneratorParams.cs:43-47

bool doSample = generatorParams.GetSearchBool("do_sample");
Console.WriteLine($"Do sample: {doSample}");

Guidance Methods

SetGuidance(string type, string data, bool enableFFTokens = false)

Sets structured output guidance (JSON schema or grammar) to constrain the model’s output.

type · string · required

The type of guidance: “json_schema” or “lark_grammar”

data · string · required

The guidance specification (JSON schema or LARK grammar string)

enableFFTokens · bool · default: "false"

Whether to enable fast-forward tokens

GeneratorParams.cs:32-35

string jsonSchema = @"
{
  ""type"": ""object"",
  ""properties"": {
    ""name"": { ""type"": ""string"" },
    ""age"": { ""type"": ""number"" }
  },
  ""required"": [""name"", ""age""]
}
";

generatorParams.SetGuidance("json_schema", jsonSchema);

Common Search Options

Here are the most commonly used search options:

Generation Length

min_length · int

Minimum number of tokens to generate (including prompt)

max_length · int

Maximum number of tokens to generate (including prompt)

generatorParams.SetSearchOption("min_length", 10);
generatorParams.SetSearchOption("max_length", 1024);

Sampling Parameters

do_sample · bool

Enable random sampling. When false, uses greedy or beam search

temperature · double

Controls randomness. Higher values (e.g., 1.0) make output more random, lower values (e.g., 0.1) make it more deterministic

top_p · double

Nucleus sampling: only sample from tokens with cumulative probability mass of top_p

top_k · int

Only sample from the top K most likely tokens

generatorParams.SetSearchOption("do_sample", true);
generatorParams.SetSearchOption("temperature", 0.7);
generatorParams.SetSearchOption("top_p", 0.9);
generatorParams.SetSearchOption("top_k", 50);

Penalty Parameters

repetition_penalty · double

Penalty for repeating tokens. Values > 1.0 discourage repetition

generatorParams.SetSearchOption("repetition_penalty", 1.1);

Beam Search Parameters

num_beams · int

Number of beams for beam search. 1 means greedy search

num_return_sequences · int

Number of sequences to return (must be ≤ num_beams)

generatorParams.SetSearchOption("num_beams", 4);
generatorParams.SetSearchOption("num_return_sequences", 1);

Batch Parameters

batch_size · int

Batch size for processing multiple prompts

generatorParams.SetSearchOption("batch_size", 1);

Complete Examples

Basic Generation Configuration

using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using GeneratorParams generatorParams = new GeneratorParams(model);

// Set basic parameters
generatorParams.SetSearchOption("max_length", 1024);
generatorParams.SetSearchOption("temperature", 0.7);
generatorParams.SetSearchOption("top_p", 0.9);
generatorParams.SetSearchOption("do_sample", true);

// Use with generator
using Generator generator = new Generator(model, generatorParams);

Creative vs Deterministic Generation

// Creative generation (higher temperature, sampling enabled)
using GeneratorParams creativeParams = new GeneratorParams(model);
creativeParams.SetSearchOption("do_sample", true);
creativeParams.SetSearchOption("temperature", 1.0);
creativeParams.SetSearchOption("top_p", 0.95);
creativeParams.SetSearchOption("top_k", 50);

// Deterministic generation (low temperature or greedy)
using GeneratorParams deterministicParams = new GeneratorParams(model);
deterministicParams.SetSearchOption("do_sample", false); // Use greedy search
// Or use very low temperature:
// deterministicParams.SetSearchOption("do_sample", true);
// deterministicParams.SetSearchOption("temperature", 0.1);

Beam Search Configuration

using GeneratorParams beamParams = new GeneratorParams(model);
beamParams.SetSearchOption("num_beams", 4);
beamParams.SetSearchOption("num_return_sequences", 1);
beamParams.SetSearchOption("max_length", 512);

Structured Output with JSON Schema

examples/csharp/ModelChat/Program.cs:166-170

using Microsoft.ML.OnnxRuntimeGenAI;

using Model model = new Model("/path/to/model");
using GeneratorParams generatorParams = new GeneratorParams(model);

// Define JSON schema for structured output
string jsonSchema = @"
{
  ""type"": ""object"",
  ""properties"": {
    ""name"": { 
      ""type"": ""string",
      ""description"": ""Person's full name""
    },
    ""age"": { 
      ""type"": ""number",
      ""description"": ""Person's age in years""
    },
    ""email"": { 
      ""type"": ""string",
      ""description"": ""Email address""
    }
  },
  ""required"": [""name"", ""age""]
}
";

generatorParams.SetGuidance("json_schema", jsonSchema);

// The model will now generate valid JSON matching this schema
using Generator generator = new Generator(model, generatorParams);

Complete Example from Source

examples/csharp/Common/Common.cs:134-159

using Microsoft.ML.OnnxRuntimeGenAI;

public static void SetSearchOptions(
    GeneratorParams generatorParams,
    GeneratorParamsArgs args,
    bool verbose)
{
    var options = new List<string>();
    
    // Set do_sample if provided
    if (args.do_sample.HasValue)
    {
        bool val = args.do_sample.Value;
        options.Add($"do_sample: {val}");
        generatorParams.SetSearchOption("do_sample", val);
    }
    
    // Set numeric options
    if (args.min_length.HasValue)
    {
        double val = Convert.ToDouble(args.min_length.Value);
        options.Add($"min_length: {val}");
        generatorParams.SetSearchOption("min_length", val);
    }
    
    if (args.max_length.HasValue)
    {
        double val = Convert.ToDouble(args.max_length.Value);
        options.Add($"max_length: {val}");
        generatorParams.SetSearchOption("max_length", val);
    }
    
    if (args.temperature.HasValue)
    {
        double val = args.temperature.Value;
        options.Add($"temperature: {val}");
        generatorParams.SetSearchOption("temperature", val);
    }
    
    if (args.top_p.HasValue)
    {
        double val = args.top_p.Value;
        options.Add($"top_p: {val}");
        generatorParams.SetSearchOption("top_p", val);
    }
    
    if (args.top_k.HasValue)
    {
        double val = Convert.ToDouble(args.top_k.Value);
        options.Add($"top_k: {val}");
        generatorParams.SetSearchOption("top_k", val);
    }
    
    if (args.repetition_penalty.HasValue)
    {
        double val = args.repetition_penalty.Value;
        options.Add($"repetition_penalty: {val}");
        generatorParams.SetSearchOption("repetition_penalty", val);
    }
    
    if (verbose)
    {
        Console.WriteLine("GeneratorParams: {" + string.Join(", ", options) + "}");
    }
}

See Also

• Generator - Use parameters with generators
• Model - Load models
• Tokenizer - Encode and decode text