A Task defines the execution flow of an evaluation of a model or a dataset.

To interact with a task using the CLI, use the lf task command.

Task Overview

<Column>
    ```yaml Single-turn Generic Input Model Task
    display_name: "Single-turn Solver Generic Input"
    key: "singleturn-generic-input"
    description: "Example task that uses a single-turn solver with generic input builder."
    config_spec: []
    definition:
      dataset:
        key: "qa-single-answers"
      solver:
        type: "single_turn_solver"
        input_builder:
          type: generic
          template: >
            {
                "messages": [
                {
                  "role": "system",
                  "content": "You are a helpful assistant. Answer with no punctuation."
                },
                {
                  "role": "user",
                  "content": "{{ sample.question }}"
                }
              ]
            }
      scorers:
        - type: "string_equals"
          ground_truth: "{{ sample.target }}"
    ```

    ```yaml uniqueness.yaml
    display_name: "Uniqueness Task"
    key: "uniqueness-task"
    description: >
      Evaluates the uniqueness rate of values in a field across samples in a dataset.
    tags: ["Data Quality"]
    config_spec:
      - type: "string"
        key: "field"
        display_name: "Field"
    definition:
      type: "benchmark_task"
      evaluated_entity_type: "dataset"
      scorers:
        - type: "python_all_samples"
          compute_scores_snippet: !include "uniqueness_scorer.py"
          metrics:
            - type: "mean"
              field: "is_unique"
              name: "Uniqueness Rate"
    ```
    ```python uniqueness_scorer.py
    from __future__ import annotations

    from collections import Counter
    from typing import Any


    def compute_scores(samples: list[dict[str, Any]]) -> list[dict[str, Any]]:
        field_name = "CONFIG.FIELD"
        values = [
            sample[field_name] if field_name in sample else None for sample in samples
        ]
        counter = Counter(values)

        return [
            {"is_unique": counter[value] == 1 if value is not None else True}
            for value in values
        ]
    ```
</Column>

Definitions

SDKBenchmarkTaskDefinitionTemplate

<Column>
    ```yaml Benchmark Task Definition
    display_name: "Harry Potter Trivia Task"
    key: "hp-trivia"
    description: "Assesses the model knowledge of Harry Potter trivia."
    config_spec: []
    definition:
      evaluated_entity_type: "model"
      dataset:
        key: "hp-trivia-dataset"
      solver:
        type: "single_turn_solver"
        input_builder:
          type: "chat_completion"
          input_messages:
            - role: "system"
              content: "You are a helpful assistant."
            - role: "user"
              content: "Respond to this question concisely. {{ sample.question }}"
      scorers:
        - type: "model_as_a_judge_classifier"
          model_key: "openai$gpt-4-1-nano"
          system_prompt: |
            You are a helpful assistant and rate whether the ground truth answer and the
            candidate answer are semantically the same. Semantically the same means that
            you do not care about small spelling mistakes, capitalization or whether
            additional information is given. This is all still considered correct.

            Respond only with 'correct' or 'incorrect' and nothing else.
          user_prompt: |
            Ground Truth Answer: {{ sample.gt_answer }}
            Candidate Answer: {{ trace.get_last_assistant_text() }}
          correct_labels:
            - "correct"
          incorrect_labels:
            - "incorrect"
          use_structured_outputs: true
    ```
</Column>

SDKSystemTaskDefinitionTemplate

Properties

type Literal "system_task"

Type of the task definition, set to system_task.

Default: system_task

compute_evidence_snippet string required

Python code snippet that computes evidence for the task.

SDKTaskDatasetTemplate

<Column>
    ```yaml Task with a fixed Dataset
    ...
    definition:
      ...
      dataset:
        key: "hp-trivia-dataset"
    ```

    <Callout icon="💡" theme="default">
      Use the CLI command `lf datasets` to list all available datasets.
    </Callout>
</Column>

PythonSolverTemplate

Properties

postprocessor PythonPostprocessorTemplate

An optional postprocessor applied to solver outputs before scoring.

Default: None

type Literal "python" required

The type of the solver.

run_solver_snippet string required

The Python code snippet defining how to run the solver. It must define
a run_solver function with the following API:

async def run_solver(sample, model, trace) -> SolverTrace

where:

• sample is a dictionary representing the current sample.
• model is the model object used for inference (with a predict method).
• trace is a SolverTrace object that can be used to record the conversation and
  should be returned by the function.

PythonScorerTemplate

Properties

key string

Unique identifier assigned to the entity in AI GO!.

Default: None

purpose enum ScorerPurpose

The purpose of this scorer.

• score: The scorer is used to score the solver output or the dataset sample.
• qa: The scorer is used to do QA over the solver output or the dataset sample.

Default: score

metrics array[PythonMetricTemplate, BinaryClassificationMetricTemplate, MulticlassClassificationMetricTemplate, MeanMetricTemplate, MaxMetricTemplate, MinMetricTemplate, StdDevMetricTemplate, FrequencyMetricTemplate, RecallMetricTemplate, PrecisionMetricTemplate, F1ScoreMetricTemplate, WeightedAverageMetricTemplate]

The metrics associated with this scorer, which will produce per-task metrics.

Default: None

display_name string

The display name of the scorer.

Default: None

type Literal "python" required

The type of the scorer.

compute_scores_snippet string, TemplateValue required

The Python code snippet defining how to compute the scores. It must define
a compute_scores function with one the following APIs:

For model tasks:

def compute_scores(sample: dict[str, Any], solver_output: SolverOutput) -> dict[str, Any]:

For dataset tasks:

def compute_scores(sample: dict[str, Any]) -> dict[str, Any]:

Both def and async def are supported.

where

• sample is a dictionary representing the current sample

• solver_output is a SolverTrace object with 2 attributes:

  1. trace: the full solver trace, including all messages, function calls, and function call outputs (ex: solver_output.trace.get_last_assistant_text())
  2. raw_outputs: the list of raw model outputs, one per model call

The function can return scores in one of two formats:

Scores only - return a flat dict of score key/value pairs:

return {"accuracy": 0.95, "is_correct": True}

Scores with metadata - return a structured dict with "scores" and "metadata" keys.
Metadata is stored alongside the scores but is not aggregated into metrics:

return {
    "scores": {"accuracy": 0.95, "is_correct": True},
    "metadata": {"model": "gpt-4", "tokens": 123},
}

AllSamplesPythonScorerTemplate

Properties

key string

Unique identifier assigned to the entity in AI GO!.

Default: None

purpose enum ScorerPurpose

The purpose of this scorer.

• score: The scorer is used to score the solver output or the dataset sample.
• qa: The scorer is used to do QA over the solver output or the dataset sample.

Default: score

metrics array[PythonMetricTemplate, BinaryClassificationMetricTemplate, MulticlassClassificationMetricTemplate, MeanMetricTemplate, MaxMetricTemplate, MinMetricTemplate, StdDevMetricTemplate, FrequencyMetricTemplate, RecallMetricTemplate, PrecisionMetricTemplate, F1ScoreMetricTemplate, WeightedAverageMetricTemplate]

The metrics associated with this scorer, which will produce per-task metrics.

Default: None

display_name string

The display name of the scorer.

Default: None

type Literal "python_all_samples" required

The type of the scorer.

compute_scores_snippet string, TemplateValue required

The Python code snippet defining how to compute the scores. It must define
a compute_scores function with the following API:

def compute_scores(samples: list[dict[str, Any]]) -> list[dict[str, Any]]:

Both def and async def are supported.

where

• samples is the list of samples (in the same order as the dataset)
• the returned list must contain one entry per sample

Each entry can be in one of two formats:

Scores only - a flat dict of score key/value pairs:

[
    {"accuracy": 0.95, "is_correct": True},
    ...
]

Scores with metadata - a structured dict with "scores" and "metadata" keys.
Metadata is stored alongside the scores but is not aggregated into metrics:

[
    {
        "scores": {"accuracy": 0.95, "is_correct": True},
        "metadata": {"model": "gpt-4", "tokens": 123},
    },
    ...
]

ModelAsAJudgeScorerTemplate

Properties

key string

Unique identifier assigned to the entity in AI GO!.

Default: None

purpose enum ScorerPurpose

The purpose of this scorer.

• score: The scorer is used to score the solver output or the dataset sample.
• qa: The scorer is used to do QA over the solver output or the dataset sample.

Default: score

metrics array[PythonMetricTemplate, BinaryClassificationMetricTemplate, MulticlassClassificationMetricTemplate, MeanMetricTemplate, MaxMetricTemplate, MinMetricTemplate, StdDevMetricTemplate, FrequencyMetricTemplate, RecallMetricTemplate, PrecisionMetricTemplate, F1ScoreMetricTemplate, WeightedAverageMetricTemplate]

The metrics associated with this scorer, which will produce per-task metrics.

Default: None

display_name string

The display name of the scorer.

Default: None

type Literal "model_as_a_judge_scorer" required

The type of the scorer.

model_key Key, TemplateValue required

The model to be used as the judge.

system_prompt string, TemplateValue

The system prompt given to the judge model. The prompt can refer
to the following variables dynamically (using {{ }} syntax):

In all scenarios:

1. sample: Sample attributes (ex: {{ sample.answer }})

If the task has a solver:

2. trace: The full solver trace, including all messages, function calls, and function
   call outputs (ex: {{ trace.get_last_assistant_text() }}, {{ trace.function_calls[0].name }})
3. model_outputs: The list of raw model outputs, one per model call (ex: {{ model_outputs[-1] }})

Default: You are a helpful assistant and will be used to judge the output of another model.

user_prompt string, TemplateValue required

The user prompt given to the judge model. The prompt can refer
to the following variables dynamically (using {{ }} syntax):

In all scenarios:

1. sample: Sample attributes (ex: {{ sample.answer }})

If the task has a solver:

2. trace: The full solver trace, including all messages, function calls, and function
   call outputs (ex: {{ trace.get_last_assistant_text() }}, {{ trace.function_calls[0].name }})
3. model_outputs: The list of raw model outputs, one per model call (ex: {{ model_outputs[-1] }})

score_min number, TemplateValue

The minimum score that the judge model can predict.

Default: 0.0

score_max number, TemplateValue

The maximum score that the judge model can predict.

Default: 1.0

use_structured_outputs boolean

Whether to use structured outputs. It is recommended to enable this if the model supports it.

Default: False

LoadTraceTemplate

Loads a conversation trace from a dataset column and injects it into the
multi-turn conversation. The trace items become part of the message history for
subsequent message builders.

Properties

type Literal "load_trace" required

Type

trace_column string, TemplateValue required

The name of the dataset column that contains the trace to load. The value must be deserializable as a Trace object.

LabelerViaModelScorerTemplate

Properties

key string

Unique identifier assigned to the entity in AI GO!.

Default: None

purpose enum ScorerPurpose

The purpose of this scorer.

• score: The scorer is used to score the solver output or the dataset sample.
• qa: The scorer is used to do QA over the solver output or the dataset sample.

Default: score

metrics array[PythonMetricTemplate, BinaryClassificationMetricTemplate, MulticlassClassificationMetricTemplate, MeanMetricTemplate, MaxMetricTemplate, MinMetricTemplate, StdDevMetricTemplate, FrequencyMetricTemplate, RecallMetricTemplate, PrecisionMetricTemplate, F1ScoreMetricTemplate, WeightedAverageMetricTemplate]

The metrics associated with this scorer, which will produce per-task metrics.

Default: None

display_name string

The display name of the scorer.

Default: None

type Literal "labeler_via_model" required

The type of the scorer.

model_key Key, TemplateValue required

The model to be used as the labeler.

system_prompt string, TemplateValue

The system prompt given to the labeler model. The prompt can refer
to the following variables dynamically (using {{ }} syntax):

In all scenarios:

1. sample: Sample attributes (ex: {{ sample.answer }})

If the task has a solver:

2. trace: The full solver trace, including all messages, function calls, and function
   call outputs (ex: {{ trace.get_last_assistant_text() }}, {{ trace.function_calls[0].name }})
3. model_outputs: The list of raw model outputs, one per model call (ex: {{ model_outputs[-1] }})

Default: You are a helpful assistant and will be used to label the output of another model or a dataset sample.

user_prompt string, TemplateValue required

The user prompt given to the labeler model. The prompt can refer
to the following variables dynamically (using {{ }} syntax):

In all scenarios:

1. sample: Sample attributes (ex: {{ sample.answer }})

If the task has a solver:

2. trace: The full solver trace, including all messages, function calls, and function
   call outputs (ex: {{ trace.get_last_assistant_text() }}, {{ trace.function_calls[0].name }})
3. model_outputs: The list of raw model outputs, one per model call (ex: {{ model_outputs[-1] }})

valid_labels array[string, TemplateValue]

The list of valid labels. To allow any label, use an empty list (default).

Default: []

use_structured_outputs boolean

Whether to use structured outputs. It is recommended to enable this if the model supports it.

Default: False

PythonPostprocessorTemplate

Properties

type Literal "python" required

The type of the postprocessor.

postprocess_snippet string required

Python source code defining a postprocess function with the following signature:

def postprocess(
sample: RawSample,
solver_output: SolverOutput,
) -> list[tuple[RawSample, SolverOutput]]

Both def and async def are supported.

SDKTaskTrialsTemplate

Properties

num_trials integer, TemplateValue required

Number of trials to run per sample.

score_aggregators array[ScoreAggregator]

List of score aggregators used by the task.

Default: None

ScoreAggregator

Aggregation configuration for one or more score keys.

Properties

score_name string required

The name of the score that will be aggregated.

aggregator MeanAggregator, MinAggregator, MaxAggregator, PassAtKAggregator, PassPowerKAggregator required

Aggregator

MinAggregator

Aggregates numeric scores by taking the minimum value.

Properties

function Literal "min" required

Function

score_name string

The name to give to the aggregated score.

Default: None

MaxAggregator

Aggregates numeric scores by taking the maximum value.

Properties

function Literal "max" required

Function

score_name string

The name to give to the aggregated score.

Default: None

MeanAggregator

Aggregates numeric scores by computing the mean.

Properties

function Literal "mean" required

Function

score_name string

The name to give to the aggregated score.

Default: None

PassAtKAggregator

Aggregates binary (True/False) scores using the pass@k estimator. Estimates the probability that at least one of k independent attempts will succeed, computed as 1 - (1 - p)^k where p is the empirical pass rate across trials.

Properties

function Literal "pass@k" required

Function

k integer required

The number of independent attempts in the scenario being modelled.

score_name string

The name to give to the aggregated score.

Default: None

PassPowerKAggregator

Aggregates binary (True/False) scores using the pass^k estimator. Estimates the probability that an agent would succeed on all k independent attempts, computed as p^k where p is the empirical pass rate across trials.

Properties

function Literal "pass^k" required

Function

k integer required

The number of independent attempts in the scenario being modelled.

score_name string

The name to give to the aggregated score.

Default: None

ActionRule

Properties

key string required

Key: 1-250 chars, allowed: a-z A-Z 0-9 _ -

Pattern: ^[a-zA-Z0-9_\-]+$

Max Length: 250

action enum ActionRuleAction required

The action to be applied to samples that match the filter.

filter FilterComparison, FilterMembership, FilterUnary required

The filter that determines which samples the action applies to.

FilterUnary

Properties

op enum FilterUnaryOp required

expression string required

An expression encoding what to apply the unary operator to.

Depending on the context, it can refer to different variables:

• When filtering a dataset: it can refer to column values by name (ex: {{ category }}).
• When used within a task action: it can refer to the
  sample, the solver_output or the scores (which is a mapping between scorer
  keys and their corresponding score values dict).

FilterComparison

Properties

op enum FilterComparisonOp required

expression string required

An expression encoding what to compare against the value.

Depending on the context, it can refer to different variables:

• When filtering a dataset: it can refer to the sample and use dot or bracket
  notation to access the columns.
  If filtering a dataset with column names that are illegal under jinja
  substitution rules (e.g. containing spaces), use bracket notation to access
  the column.
• When used within a task action: it can refer to the
  sample, the solver_output or the scores (which is a mapping between scorer
  keys and their corresponding score values dict).

value string, number, integer, boolean required

The value against which the expression is compared.

FilterMembership

Properties

op enum FilterMembershipOp required

expression string required

An expression encoding what to check membership against the values.

Depending on the context, it can refer to different variables:

• When filtering a dataset: it can refer to column values by name (ex: {{ category }}).
• When used within a task action: it can refer to the
  sample, the solver_output or the scores (which is a mapping between scorer
  keys and their corresponding score values dict).

values array[string, number, boolean] required

The set of values to test membership against.