AI Agent Evaluation & LLM Evals for Production Workflows
Evaluate completed agent and LLM traces with task-aware findings: pass/fail status, claim-level evidence, RAG grounding checks, and the evaluator path that produced each result.
Trace → Evaluate → Explain → Control
An evaluation is only as useful as the evidence behind it and the action it informs. Observyze treats evaluation as four connected stages rather than a standalone scorer.
Capture the evidence
Supported prompt, tool, retrieval, and model-response boundaries are recorded as spans so evaluation has inputs and outputs to compare.
Route by task type
Each completed trace is scored on a path chosen for its detected or supplied task type: rag, code, instruction, or chat.
Return findings, not just a number
Results include per-claim verdicts, locations, evidence, and an evaluator path so you can see why a score landed where it did.
Feed findings into runtime state
Findings support review and alerting, and can update persisted cost or safety circuit state that affects subsequent proxy requests.
The Same Output Doesn’t Mean the Same Evaluation
A short answer, a code snippet, and an instruction-following response fail in different ways. Observyze classifies a trace as rag, code, instruction, or chat and applies a path suited to that shape. Task type is inferred from spans by default and can be supplied explicitly per request or batch.
Retrieval-augmented grounding
task_type: ragThe output is checked against supplied grounding context. Claims are scored supported, contradicted, or unverifiable, with evidence retained per claim.
Requires grounding context in the span.
Code generation
task_type: codeDeterministic checks such as syntax and import validation run before slower model-based scoring, so implementation details absent from the prompt are not treated as fabrications.
Applies to code-shaped outputs.
Instruction following
task_type: instructionThe response is judged for fulfillment of the instruction rather than strict entailment, so reasonable elaboration does not read as hallucination.
Applies to instruction-shaped spans.
General prose and chat
task_type: chatThe default path for conversational and prose outputs without structured grounding context. It reports a scoped judgement with lower evidence density than a grounded path.
Default when no specific path applies.
Coverage is not universal. Each path has prerequisites — grounded evaluation needs grounding context, and deterministic code checks cover the languages and rules they implement. Where prerequisites are missing, the trace is evaluated on a fallback path and the reason code records that.
A Result You Can Trace Back to a Sentence
A pass/fail status and a numeric score are the summary, not the whole result. Every non-supported claim is returned with its verdict, its hallucination type, its location in the output, the evidence that supports or contradicts it, and a short explanation.
{
"trace_id": "tr_9f2c...",
"task_type": "rag", // rag | code | instruction | chat
"status": "fail",
"score": 0.72,
"reason_code": "nli_fast_path", // which evaluator path produced this result
"evaluator_model": "nli-deberta-v3",
"confidence": 0.88,
"evidence_insufficient": false,
"verdict_counts": { "supported": 4, "contradicted": 1, "unverifiable": 1 },
"hallucination_report": [
{
"claim": "Returns are accepted within 90 days.",
"verdict": "contradicted",
"hallucination_type": "factual_contradiction",
"location": { "field": "output", "sentence": 3 },
"evidence": "Policy document: 'Returns are accepted within 30 days.'",
"explanation": "The generated policy window conflicts with the retrieved source."
},
{
"claim": "Free shipping applies to all orders.",
"verdict": "unverifiable",
"location": { "field": "output", "sentence": 5 },
"evidence": null,
"explanation": "No retrieved chunk covers shipping terms."
}
]
}Which Path Produced This Score?
Two traces can show the same score for very different reasons. Observyze records the evaluator path in a reason code, along with the evaluator model, version, confidence, and duration, so a fallback result is never mistaken for a fast-path result.
nli_fast_pathcode_validationscoped_judgeraw_fallbackno_grounding_contextno_claimsmodel_unloadedexception// Optionally pin the evaluation path instead of relying on auto-detection
POST https://api.observyze.com/api/v1/evaluations
{
"trace_id": "tr_9f2c...",
"organization_id": "org_123",
"task_type": "rag", // rag | code | instruction | chat
"fast_only": false, // true = deterministic + NLI paths only
"spans": [ /* captured trace spans */ ]
}A Threshold Is Only as Good as Its Labels
An evaluator that is never checked against human judgement drifts. Observyze accepts corrected scores, corrected pass/fail status, and reviewer notes, and can compute threshold adjustments from that labeled feedback.
Human corrections
Reviewers submit a corrected score or status plus notes. The correction becomes labeled data for that evaluation type.
Threshold auto-tune
Adjustments are computed from labeled feedback and applied only when the recommended change clears a minimum delta, avoiding thrash on small shifts.
Skip when data is thin
Calibration requires at least three human-corrected evaluations per evaluation type per organization within a 30-day window. Below that, it is skipped rather than guessed.
Evaluation timing, stated plainly
Evaluation runs asynchronously. It explains a completed response and can update persisted circuit or budget state that affects subsequent proxy requests. It does not retract an output that has already reached a caller, and it is not a guarantee of correctness — it is evidence to review.
Read: why an evaluation score isn’t enoughFrequently Asked Questions
Technical details, integration patterns, and operational controls.
Observyze evaluates completed traces for factual grounding and safety, and it routes each evaluation by task type. The pipeline routes retrieval-augmented, code, instruction-following, and general chat/prose traces to different evaluation paths, and auto-detects the task type from spans when it is not supplied explicitly.
Explore Related Solutions
Runtime Architecture & Tooling
AI Agent Observability
Trace multi-turn agent executions, tool calls, and costs before evaluation has evidence to work with.
RAG Observability
Isolate retrieval failures from generation errors and inspect injected context chunks.
Why an LLM Evaluation Score Isn’t Enough
A technical guide to claim attribution, evaluator paths, and calibration thresholds.
Want to test without installing anything?
Explore 6 pre-configured production scenarios in our interactive Demo Mode.