Single Turn Solver

Interacts once with the model by sending it input messages and collects one response. Use this for question-answering, summarisation, and any task where no back-and-forth conversation / interactions are needed.

Examples

Example: Harry Potter Trivia. Sends a system prompt and a user question to the model and collects a single response. The dataset has a question column.

...
definition:
  ...
  solver:
    type: "single_turn_solver"
    input_builder:
      type: "chat_completion"
      input_messages:
        - role: "system"
          content: "You are a helpful assistant."
        - role: "user"
          content: "Respond to this question concisely. {{ sample.question }}"

For a sample {"question": "Who is the headmaster of Hogwarts?"}, the model produces a single assistant message, leading to the following solver output:

Configuration

Properties

type Literal "single_turn_solver" required

Type

message_format enum TraceFormat

input_builder GeneralInputBuilder, ChatCompletionInputBuilder required

Input Builder

Related types

GeneralInputBuilder

Example: Generic JSON Input. Builds the model request from a hand-crafted JSON template, useful when the model adapter expects a non-standard input format.

input_builder:
  type: generic
  template: >
    {
        "messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant. Answer with no punctuation."
        },
        {
          "role": "user",
          "content": "{{ sample.question }}"
        }
      ]
    }

For a sample {"question": "What is the capital city of Japan?", "target": "Tokyo"}, the template renders to:

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant. Answer with no punctuation."},
    {"role": "user",   "content": "What is the capital city of Japan?"}
  ]
}

Properties

type Literal "generic" required

The type of input builder.

template string required

Jinja template that takes the sample as input and produces the input in JSON form. Use curly braces {{ sample.attribute }} to denote variables that will be dynamically populated for each sample in the dataset. Full Jinja is supported for the prompt.

ChatCompletionInputBuilder

Example: Chat Completion Input. Builds a standard chat completion request from a list of messages with Jinja2 templates.

input_builder:
  type: "chat_completion"
  input_messages:
    - role: "system"
      content: "You are a helpful assistant."
    - role: "user"
      content: "Respond to this question concisely. {{ sample.question }}"

For a sample {"question": "Who is the headmaster of Hogwarts?"}, the input builder renders to:

Properties

type Literal "chat_completion" required

The kind of input builder.

input_messages array[ChatCompletionMessage] required

Input Messages

ChatCompletionMessage

Example: Chat Completion Message. A single message entry in the input_messages list, combining a fixed role with a Jinja2 content template.

role: "user"
content: "Respond to this question concisely. {{ sample.question }}"

For a sample {"question": "Who is the headmaster of Hogwarts?"}, the user message renders to:

Respond to this question concisely. Who is the headmaster of Hogwarts?

Properties

role enum ChatCompletionRole required

content string required

The prompt content. The prompt can refer to dynamic variables using curly braces (ex. {{ sample.attribute }} to refer to an attribute for the current sample in the dataset). Full Jinja is supported for the prompt.