A test protocol for an AI classifier

Context

Internal product (the AI module of a monitoring platform)

Problem

The classifier outputs a single label but contains two separate decisions (relevance + sentiment). A single “accuracy percentage” conflates them; where the error is, whose it is and whether it will recur next period had no answer.

Approach

I wrote a protocol that measures the two axes separately, treats false positives as costlier than false negatives, forces a reason code from a closed list on every decision, and automatically catches label↔reason contradictions. Determinism (same input, same output) is checked separately; every finding is owned as “prompt error / decision error / data error”.

Impact

Weekly audits became repeatable; findings reach the product team grouped and prioritised; the same error is never opened twice.