Python Solver

Runs a custom Python function as the solver. Use this when the built-in declarative solvers cannot express the interaction patterns with the model you need.

Examples

Example: Geography QA. A Python function builds the conversation trace manually - appending a system message, the user question from the dataset, calling the model, and recording the response.

...
definition:
  ...
  solver:
    type: "python"
    run_solver_snippet: |
      async def run_solver(sample, model, trace):
          trace.append_system_message(
              "You are a helpful assistant. Answer with a single word or short phrase."
          )
          trace.append_user_message(sample["question"])
          response = await model.predict(trace.items)
          trace.add_model_response(response)
          return trace

For a sample {"question": "What is the capital of Japan?", "answer": "Tokyo"}, the solver builds and returns the following trace:

Configuration

Properties

type Literal "python" required

The type of the solver.

run_solver_snippet string required

The Python code snippet defining how to run the solver. It must define a run_solver function with the following API:

async def run_solver(sample, model, trace) -> SolverTrace

where:

• sample is a dictionary representing the current sample.
• model is the model object used for inference (with a predict method).
• trace is a SolverTrace object that can be used to record the conversation and should be returned by the function.