Why an LLM Evaluation Score Isn't Enough

Numeric evaluation is popular because it is easy to store, chart, and alert on. You can set a dashboard line at 0.7, route every crossing to Slack, and feel like quality is being monitored. The trouble starts the moment a score crosses that line and someone asks a reasonable question: what do we change? A scalar collapses several unrelated failure modes into one axis. A retrieval miss, a prompt contradiction, a fabricated citation, a formatting violation, and an evaluator misread can all land at very similar scores — and they have completely different fixes. This is not an argument against scores. Thresholds are useful for triage: they tell you which traces deserve attention. The argument is that a score should be the entry point to a finding, not the finding itself.

When an evaluation result reaches an engineer, it has to survive six questions before it can change anything in production: 1. **What failed?** A score of 0.72 is not a description. The finding needs to point at a specific claim, field, or sentence. 2. **Which instruction was violated?** A response can be factually grounded and still fail the task — wrong format, ignored constraint, wrong length, wrong tone. Grounding scores do not measure instruction compliance, and vice versa. 3. **What evidence supports the finding?** An evaluator that asserts a contradiction without quoting the source is asking for trust it has not earned. The finding should carry the span of context it compared against. 4. **Did retrieval or generation cause it?** If the retrieved context never contained the fact, the model is not hallucinating — the pipeline is starving it. If the context contained the fact and the model contradicted it, that is a generation problem. The same output score can mean either. 5. **Which evaluator path produced the result?** A deterministic check, a local natural-language-inference pass, a scoped model judge, and a fallback path have different error profiles. A number that does not disclose its provenance cannot be compared across traces. 6. **Is the threshold even calibrated?** If nobody has corrected the evaluator against human judgement, the pass/fail boundary is an arbitrary constant. Calibration turns “0.72 failed” into “0.72 is above the boundary we validated on labeled examples.”

The single highest-value property of a finding is attribution, and it is the one a scalar score destroys first. Consider a support assistant that answers “Returns are accepted within 90 days.” Three very different worlds produce that sentence: - **Retrieval miss:** no chunk in the retrieved set covered return windows, so the model filled the gap from prior training. Fix the chunking, the embeddings, or the top-k settings. - **Contradiction:** the retrieved policy document said 30 days and the model said 90. Fix the prompt, the model choice, or the decoding parameters. - **Unverifiable:** the retrieved documents were silent and no external evidence was reachable. This is neither of the above; the correct action is to collect more context or route to review, not to retrain anything. A truthful evaluator must be able to represent all three. Collapsing “we could not verify this” into “this is false” is the most common source of evaluator disbelief: teams see confident failures on claims that were simply outside the grounding set, and they stop trusting the pipeline entirely. A three-way verdict — supported, contradicted, unverifiable — is the minimum viable vocabulary.

Evaluator pipelines almost always degrade gracefully, and that is exactly the problem. When a fast local model is unavailable or a span lacks grounding context, the system falls back to a slower, more expensive, and often more variable path. If the result does not record that fallback, you will compare outputs produced by different machines as though they were the same measurement. Practical provenance fields are cheap to add and expensive to omit: - A **reason code** naming the path that produced the result (deterministic check, fast inference pass, scoped judge, fallback). - The **evaluator model and version**, so a scoring shift can be tied to a deployment rather than to your product. - A **confidence** value, so downstream consumers can decide whether a borderline result is actionable. - **Wall-clock duration**, which is what turns evaluation cost from a surprise into a budget line. The payoff is comparability. Once every result states how it was produced, you can legitimately ask whether the fallback path is materially worse than the fast path on your own traffic — and answer it.

A threshold is a policy, and like any policy it should be derived from evidence. In practice most teams ship whatever constant looked reasonable on day one and never revisit it. Calibration closes that loop. The mechanics are simple: reviewers correct evaluator output (a corrected score, a corrected pass/fail status, and notes), the corrections are grouped by evaluation type, and the system recomputes where the pass/fail boundary should sit. Two practical rules keep this from becoming noise: 1. **Require a minimum labeled sample.** A handful of corrections is not a calibration set. Requiring a minimum number of human-corrected evaluations per evaluation type over a fixed window prevents a single vocal reviewer from moving a global threshold. 2. **Ignore sub-threshold deltas.** If the recommended change is smaller than a configured minimum delta, leave the threshold alone. Constant micro-adjustments make scores unstable and comparisons meaningless. And if the labeled sample is too thin, the correct output is “insufficient data” — not a new number. A confidently tuned threshold built on three examples is worse than an untuned one, because it hides its own uncertainty.

// The fields that make a finding reviewable instead of opaque.
interface EvaluationFinding {
  traceId: string;
  taskType: 'rag' | 'code' | 'instruction' | 'chat';
  status: 'pass' | 'fail';
  score: number;

  // Provenance — how this result was produced
  reasonCode:
    | 'deterministic_check'
    | 'fast_inference_pass'
    | 'scoped_judge'
    | 'fallback';
  evaluatorModel: string;
  evaluatorVersion: string;
  confidence?: number;
  durationMs?: number;

  // Attribution — enough detail to act on
  evidenceInsufficient?: boolean;
  verdictCounts?: Record<'supported' | 'contradicted' | 'unverifiable', number>;
  findings: Array<{
    claim: string;
    verdict: 'supported' | 'contradicted' | 'unverifiable';
    location: { field: string; sentence: number };
    evidence: string | null;
    explanation: string;
  }>;
}

Observyze applies these ideas as a task-aware evaluation pipeline. Traces are routed by task type (retrieval-augmented, code, instruction-following, or general chat), and results carry pass/fail status, a score, and per-claim findings with verdicts, locations, evidence, and explanations. Each result also records the evaluator path, model, version, confidence, and duration, plus an evidence-insufficient flag that keeps unverifiable claims distinct from contradictions. Calibration follows the same discipline: human-corrected feedback drives threshold adjustments, and calibration is skipped when there is not enough labeled data to justify a change. None of this makes an automated evaluator authoritative. It makes it auditable — which is the only property that lets a team act on a score without guessing.