GenAI-Perf

GenAI-Perf is a command line tool for measuring the throughput and latency of generative AI models as served through an inference server. For large language models (LLMs), GenAI-Perf provides metrics such as output token throughput, time to first token, time to second token, inter token latency, and request throughput. For a full list of metrics please see the Metrics section.

Users specify a model name, an inference server URL, the type of inputs to use (synthetic or from a dataset defined via a file), and the type of load to generate (number of concurrent requests, request rate).

GenAI-Perf generates the specified load, measures the performance of the inference server and reports the metrics in a simple table as console output. The tool also logs all results in a csv and json file that can be used to derive additional metrics and visualizations. The inference server must already be running when GenAI-Perf is run.

You can use GenAI-Perf to run performance benchmarks on

• Large Language Models
• Multi-Modal Language Models
• Embedding Models
• Ranking Models
• Multiple LoRA Adapters

You can also use GenAI-Perf to run benchmarks on your custom APIs using either customizable frontends or customizable payloads. Customizable frontends provide more customizability, while customizable payloads allow you to specify specific payload schemas using a Jinja2 template.

> [!Note] > GenAI-Perf is currently in early release and under rapid development. While we > will try to remain consistent, command line options and functionality are > subject to change as the tool matures.

Installation

The easiest way to install GenAI-Perf is through pip.

Install GenAI-Perf (Ubuntu 24.04, Python 3.10+)

pip install genai-perf

NOTE: you must already have CUDA 12 installed

export RELEASE="25.01"

docker run -it --net=host --gpus=all  nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk

# Validate the genai-perf command works inside the container:
genai-perf --help

Quick Start

In this quick start, we will use GenAI-Perf to run performance benchmarking on the GPT-2 model running on Triton Inference Server with a TensorRT-LLM engine.

Serve GPT-2 TensorRT-LLM model using Triton CLI

You can follow the quickstart guide in the Triton CLI Github repository to serve GPT-2 on the Triton server with the TensorRT-LLM backend. NOTE: pip dependency error messages can be safely ignored.

The full instructions are copied below for convenience:

# This container comes with all of the dependencies for building TRT-LLM engines
# and serving the engine with Triton Inference Server.
docker run -ti \
    --gpus all \
    --network=host \
    --shm-size=1g --ulimit memlock=-1 \
    -v /tmp:/tmp \
    -v ${HOME}/.cache/huggingface:/root/.cache/huggingface \
    nvcr.io/nvidia/tritonserver:25.01-trtllm-python-py3

# Install the Triton CLI
pip install git+https://github.com/triton-inference-server/triton_cli.git@0.1.2

# Build TRT LLM engine and generate a Triton model repository pointing at it
triton remove -m all
triton import -m gpt2 --backend tensorrtllm

# Start Triton pointing at the default model repository
triton start

Running GenAI-Perf

Now we can run GenAI-Perf inside the Triton Inference Server SDK container:

genai-perf profile \
  -m gpt2 \
  --backend tensorrtllm \
  --streaming

Example output:

                              NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃                         Statistic ┃    avg ┃    min ┃    max ┃    p99 ┃    p90 ┃    p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│          Time to first token (ms) │  16.26 │  12.39 │  17.25 │  17.09 │  16.68 │  16.56 │
│          Inter token latency (ms) │   1.85 │   1.55 │   2.04 │   2.02 │   1.97 │   1.92 │
│              Request latency (ms) │ 499.20 │ 451.01 │ 554.61 │ 548.69 │ 526.13 │ 514.19 │
│            Output sequence length │ 261.90 │ 256.00 │ 298.00 │ 296.60 │ 270.00 │ 265.00 │
│             Input sequence length │ 550.06 │ 550.00 │ 553.00 │ 551.60 │ 550.00 │ 550.00 │
│ Output token throughput (per sec) │ 520.87 │    N/A │    N/A │    N/A │    N/A │    N/A │
│      Request throughput (per sec) │   1.99 │    N/A │    N/A │    N/A │    N/A │    N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

See Tutorial for additional examples.

Configuration File

In addition to setting options via the command-line, GenAI-Perf supports the passing in of a config file (in YAML format). The command is:

genai-perf config -f 

Creating a Template Config File

In order to make it easier for you to use config files, we have added a new subcommand that generates a template config file containing all possible options, pre-populated to their default settings. The command to create this is:

genai-perf create-template

By default the config file is named genai_perf_config.yaml, but you can change that by passing in a custom name using the -f option.
For less experienced users, you can include -v/--verbose and the config file will also contain descriptions for each option (similar to what you would see using -h from the command line).

Here is a sample section of what the template config file looks like:

  endpoint:
    model_selection_strategy: round_robin
    backend: tensorrtllm
    custom:
    type: kserve
    streaming: False
    server_metrics_urls: http://localhost:8002/metrics
    url: localhost:8001
    grpc_method:

and with --verbose:

 endpoint:
    # When multiple model are specified, this is how a specific model should be assigned to a prompt.
    # round_robin: nth prompt in the list gets assigned to n-mod len(models).
    # random: assignment is uniformly random
    model_selection_strategy: round_robin

    # When benchmarking Triton, this is the backend of the model.
    # For the TENSORRT-LLM backend,you currently must set 'exclude_input_in_output' to true
    # in the model config to not echo the input tokens
    backend: tensorrtllm

    # Set a custom endpoint that differs from the OpenAI defaults.
    custom:

    # The type to send requests to on the server.
    type: kserve

    # An option to enable the use of the streaming API.
    streaming: False

    # The list of server metrics URLs.
    # These are used for Telemetry metric reporting.
    server_metrics_urls: http://localhost:9400/metrics

    # URL of the endpoint to target for benchmarking.
    url: localhost:8001

    # A fully-qualified gRPC method name in './' format.
    # The option is only supported by dynamic gRPC service kind and is
    # required to identify the RPC to use when sending requests to the server.
    grpc_method:

Overriding Config Options

Once you have setup your config file to your liking, there could be times where you might want to re-profile with just a few options changed.
Rather than editing your config you can include the --override-config option on the CLI along with the options you want to change. For example:

genai-perf config -f genai_perf_config.yaml --override-config --warmup-request-count 100 --concurrency 32

Analyze

GenAI-Perf can be used to sweep through PA or GenAI-Perf stimulus allowing the user to profile multiple scenarios with a single command. See Analyze for details on how this subcommand can be utilized.

Visualization

GenAI-Perf can also generate various plots that visualize the performance of the current profile run. This is disabled by default but users can easily enable it by passing the --generate-plots option when running the benchmark:

genai-perf profile \
  -m gpt2 \
  --backend tensorrtllm \
  --streaming \
  --concurrency 1 \
  --generate-plots

This will generate a set of default plots such as:

• Time to first token (TTFT) analysis
• Request latency analysis
• TTFT vs Input sequence lengths
• Inter token latencies vs Token positions
• Input sequence lengths vs Output sequence lengths

Process Export Files

GenAI-Perf can be used to process multiple profile export files from distributed runs and generate outputs with aggregated metrics. See Process Export Files for details on how this subcommand can be utilized.

Model Inputs

GenAI-Perf supports model input prompts from either synthetically generated inputs, or from a dataset defined via a file.

When the dataset is synthetic, you can specify the following options:

• --num-dataset-entries : The number of unique payloads to sample from. These will be reused until benchmarking is complete.
• --synthetic-input-tokens-mean : The mean of number of tokens in the generated prompts when using synthetic data, >= 1.
• --synthetic-input-tokens-stddev : The standard deviation of number of tokens in the generated prompts when using synthetic data, >= 0.
• --random-seed : The seed used to generate random values, >= 0.
• --request-count : The number of requests to benchmark
• --warmup-request-count : The number of requests to send before benchmarking

When the dataset is coming from a file, you can specify the following options:

• `--input-file