NVIDIA NeMo Evaluator

The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.

Quick start guide

NVIDIA NeMo Evaluator provide you with evaluation clients, that are specifically built to evaluate model endpoints using our Standard API.

Launching an evaluation for an LLM

1. Install the package

   pip install nvidia-eval-factory-garak
   

2. (Optional) Set a token to your API endpoint if it's protected

   export MY_API_KEY="your_api_key_here"
   

3. List the available evaluations:

   $ nemo-evaluator ls
   Available tasks:
   * garak (in garak)
   ...
   
   

4. Run the evaluation:

   nemo-evaluator run_eval \
       --eval_type garak \
       --model_id microsoft/phi-4-mini-instruct \
       --model_url https://integrate.api.nvidia.com/v1/chat/completions \
       --model_type chat \
       --api_key_name MY_API_KEY \
       --output_dir /workspace/results
   

5. Gather the results

   cat /workspace/results/results.yml
   

Command-Line Tool

Each package comes pre-installed with a set of command-line tools, designed to simplify the execution of evaluation tasks. Below are the available commands and their usage for the garak:

Commands

1. List Evaluation Types

nemo-evaluator ls

Displays the evaluation types available within the harness.

2. Run an evaluation

The nemo-evaluator run_eval command executes the evaluation process. Below are the flags and their descriptions:

Required flags

• --eval_type The type of evaluation to perform
• --model_id The name or identifier of the model to evaluate.
• --model_url The API endpoint where the model is accessible.
• --model_type The type of the model to evaluate, currently either "chat", "completions", or "vlm".
• --output_dir The directory to use as the working directory for the evaluation. The results, including the results.yml output file, will be saved here. Make sure to use the absolute path.

Optional flags

• --api_key_name The name of the environment variable that stores the Bearer token for the API, if authentication is required.
• `--run_config