Multi Turn Solver

Runs a multi-turn conversation with the model, where the conversation structure is expressed via a configurable sequence of message builders. Use this for dialogue evaluation, agentic tasks, and any scenario that requires iterative model interaction.

Examples

Example: Multiturn Clarification. A system prompt instructs the model to ask for confirmation before answering questions about European countries. A judge model answers clarifying questions automatically until the evaluated model gives its final answer.

...
definition:
  ...
  solver:
    type: "multi_turn_solver"
    message_builders:
      - type: "chat_message"
        role: "system"
        content: | 
          You are a helpful assistant. 
          
          When asked about a question about countries in Europe, always repeat the
          question back to the user and ask for confirmation that you have understood
          the question.

          When asked a question about non-European countries, answer directly without
          asking for confirmation.

          In both cases, when providing the factual answer, do not include any
          additional explanations or punctuation, but only answer with a single word.
      - type: "chat_message"
        role: "user"
        content: "{{ sample.question }}"
      - type: "generate"
      - type: "generate_message"
        model_key: "<< config.judge_model >>"
        extra_input_messages:
          - type: "chat_message"
            role: "user"
            content: |
              If the previous message is a question, answer it using the information
              in the other messages.
              If the question is asking for confirmation, always answer 'Yes'.

              Otherwise, answer with '<done>'.
        output_role: "user"
        terminate_if:
          includes: "<done>"
          keep_iteration: false
      - type: "generate"

For a sample {"question": "What is the capital city of Austria?", "target": "Vienna"}, the solver builds the following conversation:

Configuration

Properties

type Literal "multi_turn_solver" required

Type

message_format enum TraceFormat

message_builders array[ChatMessageBuilder, Generate, GenerateMessage, GenerateLoop, LoadTrace] required

Message Builders

Related types

ChatMessageBuilder

Adds a chat message to the conversation

Example: Static and Templated Messages. Inserts fixed system and templated user messages into the conversation before the first model call.

type: "chat_message"
role: "system"
content: | 
  You are a helpful assistant. 

  When asked about a question about countries in Europe, always repeat the
  question back to the user and ask for confirmation that you have understood
  the question.

  When asked a question about non-European countries, answer directly without
  asking for confirmation.

  In both cases, when providing the factual answer, do not include any
  additional explanations or punctuation, but only answer with a single word.

Properties

type Literal "chat_message" required

Type

content string required

The chat message content. The prompt can refer to the following variable dynamically (using {{ }} syntax):

In all scenarios:

1. sample: Sample attributes (ex: {{ sample.answer }})
2. messages: The list of messages in the conversation (ex: {{ messages[0].content }})
3. loop_index: (if used in a GenerateLoop) The index of the current iteration (ex: {{ loop_index }})

role enum ChatCompletionRole required

terminate_if TerminationCondition

Generate

Generates a message using the evaluated model.

Example: Generate Step. Calls the evaluated model and appends its response to the conversation as an assistant message.

type: "generate"

Properties

type Literal "generate" required

Type

terminate_if TerminationCondition

GenerateMessage

Generate a message using the specified model (and custom instructions, specified via extra input messages).

Example: Generate Message (Judge). Calls a secondary judge model to produce the next user turn. The conversation continues until the judge outputs <done>, at which point the loop terminates and that turn is discarded.

type: "generate_message"
model_key: "<< config.judge_model >>"
extra_input_messages:
  - type: "chat_message"
    role: "user"
    content: |
      If the previous message is a question, answer it using the information
      in the other messages.
      If the question is asking for confirmation, always answer 'Yes'.

      Otherwise, answer with '<done>'.
output_role: "user"
terminate_if:
  includes: "<done>"
  keep_iteration: false

Given the conversation so far ending with the model asking a clarifying question, the judge receives the full history plus an extra instruction message and replies:

On the next iteration, once the model gives its final answer, the judge outputs <done> and the loop terminates without appending that turn.

Properties

type Literal "generate_message" required

Type

model_id string required

Model Id

extra_input_messages array[ChatMessageBuilder] required

Additional input messages (in addition to the existing messages) given as input to the model.

output_role enum ChatCompletionRole

The role of the output message.

terminate_if TerminationCondition

GenerateLoop

Execute a loop, where at each iteration the sequence of message builders is executed. Each message builder can optionally specify when to terminate the loop. There is a maximum number of iterations.

Example: Loop. Repeats a nested sequence of message builders up to max_iterations times. Here a judge answers confirmation questions until the model produces a final answer or the iteration cap is reached.

type: "loop"
max_iterations: 3
on_max_iterations: "continue"
message_builders:
  - type: "generate_message"
    model_key: "<< config.judge_model >>"
    extra_input_messages:
      - type: "chat_message"
        role: "user"
        content: |
          If the previous message is a question asking for confirmation,
          answer 'Yes'.

          Otherwise, answer with '<done>'.
    output_role: "user"
    terminate_if:
      includes: "<done>"
      keep_iteration: false
  - type: "generate"

For a sample {"question": "What is the capital city of Austria?", "target": "Vienna"}, the loop runs at most 3 iterations:

Properties

type Literal "loop" required

Type

message_builders array[ChatMessageBuilder, Generate, GenerateMessage] required

Message Builders

max_iterations integer

Max Iterations

on_max_iterations enum OnMaxIterations

LoadTrace

Loads a conversation trace from a dataset column and injects it into the multi-turn conversation. The trace items become part of the message history for subsequent message builders.

Example: Load Trace. Injects a pre-recorded conversation from the dataset into the message history before further message builders run. Use this to continue or score an existing conversation without replaying it.

# tasks/task.yaml
...
definition:
  solver:
    type: "multi_turn_solver"
    message_builders:
      # Inject a pre-recorded conversation from the dataset into the message history.
      - type: "load_trace"
        trace_column: "conversation_trace"
      # Continue the conversation from where the loaded trace left off.
      - type: "generate"

Given a dataset row whose conversation_trace column contains a two-turn exchange:

load_trace replays all four turns verbatim; the subsequent generate step then calls the model to continue from turn 4.

Properties

type Literal "load_trace" required

Type

trace_column string required

The name of the dataset column that contains the trace to load. The value must be deserializable as a Trace object.

TerminationCondition

Properties

includes string required

If the messages includes this string, the termination condition is met.

keep_iteration boolean

Within a loop, determines if all messages in the current iteration are kept or not, if the termination condition is met. Outside of a loop, decides if the message is kept or not if the termination condition is met.