A Run is a unified entry point for running assessments that defines what evaluation to run, as well as, all the dependencies (i.e., datasets, models, etc.).

To work with the run entry point using the CLI, use the lf run command.

Run Config Overview

Properties

secrets object

Secrets which can be used to reference secret values in designated places.

Default: None

model_adapters array[SDKModelAdapter]

List of all model adapters to be created/updated before running the evaluation.

Default: []

models array[SDKModel]

List of all models to be created/updated before running the evaluation. Can be either a full model definition or a string representing a third-party provider and model key in the format $provider: <provider>/<model_key>.

Default: []

dataset_generators array[SDKDatasetGenerator]

List of all dataset generators to be created/updated before running the evaluation.

Default: []

datasets array[SDKDataset]

List of all datasets to be created/updated before running the evaluation.

Default: []

tasks array[SDKTask]

List of all tasks to be created/updated before running the evaluation.

Default: []

evaluation SDKEvaluation

The evaluation to be run.

Default: None

policies array[Policy]

List of all policies to apply to the AI app.

Default: []

datasets:
  - $ref: "./datasets/hp_trivia.yaml"
models:
  - $provider: "openai/gpt-4.1-nano"
tasks:
  - $ref: "./tasks/hp_trivia.yaml"

evaluation:
  key: "hp-trivia-evaluation"
  display_name: "Harry Potter Trivia evaluation"
  task_specifications:
    - key: "hp-trivia-gpt-4-1-nano"
      task_key: "hp-trivia"
      display_name: "Harry Potter Trivia OpenAI GPT-4.1 Nano"
      task_config:
        config_key: []
      model_key: "openai$gpt-4-1-nano"

models:
  - $ref: "./models/model.yaml"
datasets:
  - $ref: "./datasets/conversation_memory_questions.yaml"
tasks:
  - $ref: "./tasks/conversation_memory.yaml"
evaluation:
  key: "conversation_memory"
  display_name: "Conversation Memory"
  tags: ["General Purpose AI", "Chat Completion"]
  task_specifications:
    - key: "conversation_memory_gpt_4-1_nano"
      display_name: "Conversation Memory"
      task_key: "conversation_memory"
      model_key: "openai-gpt-4-1-nano"