A config specification allows defining allowed configuration options for a given task or a dataset generator.

Definitions

`IntParameterSpec`

Properties

type Literal "int" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

min integer

The minimum value of the parameter.

Default: None

max integer

The maximum value of the parameter.

Default: None

default_value integer

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

...
config_spec:
  - type: "int"
    key: "max_words"
    display_name: "Max Words"
    description: "Maximum number of words the model should use in its answer."
    default_value: 1
    min: 1
    max: 5
  - type: "boolean"
    key: "include_context"
    display_name: "Include Context"
    description: "Whether to include supporting context in the user message."

`BooleanParameterSpec`

Properties

type Literal "boolean" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

default_value boolean

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

...
config_spec:
  - type: "int"
    key: "max_words"
    display_name: "Max Words"
    description: "Maximum number of words the model should use in its answer."
    default_value: 1
    min: 1
    max: 5
  - type: "boolean"
    key: "include_context"
    display_name: "Include Context"
    description: "Whether to include supporting context in the user message."

`FloatParameterSpec`

Properties

type Literal "float" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

min number

The minimum value of the parameter.

Default: None

max number

The maximum value of the parameter.

Default: None

default_value number

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

`StringParameterSpec`

Properties

type Literal "string" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

default_value string

The default value of the parameter.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

string_kind enum StringKind

Default: freeform

Possible StringKind values

Specifies the kind of string parameter.

Allowed Values:

freeform
python
jinja

examples null

Examples for the string parameter.

Default: None

...
config_spec:
  - type: "string"
    key: "field"
    display_name: "Field"

`ModelParameterSpec`

Properties

type Literal "model" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

default_value string

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

...
config_spec:
  - type: "model"
    key: "judge_model"
    display_name: "Judge Model"

`DatasetParameterSpec`

Properties

type Literal "dataset" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

default_value string

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

...
config_spec:
  - type: "dataset"
    key: "qa_dataset"
    display_name: "QA Dataset"
    description: "The dataset to evaluate against."
  - type: "model"
    key: "judge_model"
    display_name: "Judge Model"
  - type: "categorical"
    key: "evaluation_dimension"
    display_name: "Evaluation Dimension"
    description: "The aspect of the model response to evaluate."
    allowed_values: ["groundedness", "relevance"]

`DatasetColumnParameterSpec`

Properties

type Literal "dataset_column" required

The type of the parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

default_value string

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

...
config_spec:
  - type: "dataset_column"
    key: "answer_column"
    display_name: "Answer Column"
    description: "Dataset column containing the correct single-character answer choice."

`ListParameterSpec`

Properties

type Literal "list" required

The type of the parameter.

dtype enum ScalarDtype required

The data type of the elements in the list.

Possible ScalarDtype values

The scalar data type.

Allowed Values:

string
integer
float
boolean

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

default_value array[Any]

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

`CategoricalParameterSpec`

Properties

type Literal "categorical" required

The type of parameter.

key string required

The key of the parameter.

display_name string required

The display name of the parameter.

description string

The description of the parameter.

Default: None

allowed_values array[string] required

Allowed Values

multiple boolean

Whether the parameter can have multiple values.

Default: False

default_value string

The default value to use.

Default: None

nullable boolean

Whether this parameter is nullable.

Default: False

...
config_spec:
  - type: "dataset"
    key: "qa_dataset"
    display_name: "QA Dataset"
    description: "The dataset to evaluate against."
  - type: "model"
    key: "judge_model"
    display_name: "Judge Model"
  - type: "categorical"
    key: "evaluation_dimension"
    display_name: "Evaluation Dimension"
    description: "The aspect of the model response to evaluate."
    allowed_values: ["groundedness", "relevance"]