Assessing Your AI System

The LatticeFlow platform supports assessing a wide range of use cases -- from base LLM models or RAG systems, to AI agents and custom models (i.e., KYC, models working with structured or unstructured data).

To assess your AI System, we connect to an inference endpoint served either by an external model providers (e.g. OpenAI, Anthropic, etc.) or by an inference engine deployed in your own infrastructure.

Below we summarize the default output formats natively supported by LatticeFlow for different tasks. This is useful to at a glance understand what type of information is used by the analysis.

Type	Description
Chat completion	Generate a model response from a list of messages comprising a a single-turn or multi-turn conversation.
Retrieval-Augmented Generation (RAG)	A chat completion task where the output of the LLM is enhanced by additional information from the knowledge base.
Agents	Generate a model response containing a list of messages, tool calls, inter-agent communication, etc.
Custom AI Systems	A custom model interface defining task specific inputs and outputs.

Important: When integrating your models, you can define custom adapters or transformation in case your model does not fully conform to the expected format.

Chat Completion

Chat completion model expects OpenAI compatible chat completion format.

# e.g., https://api.openai.com/v1/chat/completions
curl <endpoint_url> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "model": "VAR_chat_model_id",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Hi, how can I help you?"
      }
    }
  ]
}

Retrieval-Augmented Generation (RAG)

A RAG based system behaves similarly to a chat completion model, but additionally returns the references retrieved from the knowledge base to support the answer. This information is critical for computing various RAG based metrics.

curl <endpoint_url> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of Switzerland?"
      }
    ]
  }'

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Bern"
      },
      "references": [
         {
           "content": "Switzerland is a federal republic composed of 26 cantons, with Bern serving as the federal city and the seat of the national government."
           "source": "...",
         },
         ...
      ]
    }
  ]
}

Agents

An agentic system generalizes chat completion and RAG systems, by generating traces. A Trace represents a conversation between a user and an agent in the Open Responses format. It stores a sequence of items (messages, function calls, and function call outputs)

curl <endpoint_url> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of Switzerland?"
      }
    ]
  }'

{
  "items": [
    {
      "type": "function_call",
      "id": "...",
      "call_id": "...", 
      "name": "search", 
      "arguments": "{}", 
      "status": "completed"
    },
    {
      "type": "function_call_output", 
      "id": "...", 
      "call_id": "...", 
      "output": "...", 
      "status": "completed"
    },
    {
      "type": "message", 
      "id": "...", 
      "status": "completed", 
      "role": "assistant", 
      "content": [{"type": "output_text", "text": "...", "annotations": []}]
    }
  ]
}

Custom AI Systems

To integrate AI System for custom use cases, a trace format with custom input and output messages is used. The user has the flexibility to integrate as a trace with one item, or split the output input multiple trace items with different semantics.

curl <endpoint_url> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    # arbitrary json payload
  }'

{
  "items": [
    {
      "type": "custom_task_output_message", 
      "content": # arbitrary object as returned by the endpoint
    }
  ]
}