Appearance
Agent Eval
Score a draft version of an agent. Only drafts are evaluated — published agents are immutable. Each evaluation scores a version 1-5 on nine axes: three read the prompt itself, six read a generated response.
Evaluate a single Q&A case against a draft version
POST
/v1/agent-eval/chat-agent/evaluate-case
Generates a response for one test input and scores it on the response axes. Supply expected in the case to also score accuracy.
Authorizations
HTTPBearer
Type
HTTP (bearer)
Request Body
application/json
Evaluate a single Q&A pair — generates a response and scores response axes.
The test input (and optional expected answer).
Reference answer. When present, unlocks the accuracy axis for this case.
A user message to run the prompt against.
Min Length
1Format
"uuid"Any of
Format
"uuid"Responses
Successful Response
application/json
JSON "generated_response": "string", "has_reference": true, "input": "string", "response_average": 0, "response_scores": [ { "axis": "string", "label": "string", "rationale": "string", "score": 0, "severity": "string", "weakness": "string" } ]
{
}
Samples
Evaluate only the prompt axes of a draft version
POST
/v1/agent-eval/chat-agent/evaluate-prompt
Scores the 3 prompt axes (clarity, linguistic quality, fairness) without generating any responses. Use /evaluate-case to score generated responses.
Authorizations
HTTPBearer
Type
HTTP (bearer)
Request Body
application/json
Evaluate only the prompt axes — no generation, no response scoring.
Format
"uuid"Any of
Format
"uuid"Responses
Successful Response
application/json
JSON "agent_id": "string", "cases": [ { "generated_response": "string", "has_reference": true, "input": "string", "response_average": 0, "response_scores": [ { "axis": "string", "label": "string", "rationale": "string", "score": 0, "severity": "string", "weakness": "string" } ] } ], "evaluator_model": "string", "generation_model": "string", "overall_average": 0, "prompt_average": 0, "prompt_scores": [ { "axis": "string", "label": "string", "rationale": "string", "score": 0, "severity": "string", "weakness": "string" } ], "response_average": 0
{
}
Samples
Evaluate prompt axes from raw input (no saved version needed)
POST
/v1/agent-eval/chat-agent/evaluate-prompt-adhoc
Accepts the prompt directly instead of loading from a draft version. Returns scores without persisting anything. Used for previewing evals on unsaved editor changes.
Authorizations
HTTPBearer
Type
HTTP (bearer)
Request Body
application/json
Evaluate prompt axes from raw input — no saved draft/version required.
Used by the editor to preview eval scores on unsaved changes.
Agent name for context.
Default
""The system prompt to evaluate.
Min Length
1Any of
Agent tags for context.
Any of
Responses
Successful Response
application/json
JSON "agent_id": "string", "cases": [ { "generated_response": "string", "has_reference": true, "input": "string", "response_average": 0, "response_scores": [ { "axis": "string", "label": "string", "rationale": "string", "score": 0, "severity": "string", "weakness": "string" } ] } ], "evaluator_model": "string", "generation_model": "string", "overall_average": 0, "prompt_average": 0, "prompt_scores": [ { "axis": "string", "label": "string", "rationale": "string", "score": 0, "severity": "string", "weakness": "string" } ], "response_average": 0
{
}