Evaluating a third-party AI provider

Context

Retail chain · 2,525 records, 5 channels

Problem

“Is the vendor’s spam analysis good or bad?” needed a numeric answer.

Approach

I built a six-dimension weighted scoring framework (technical noise, off-brand content, same-name noise, duplicates, precision, decision consistency) and compared the vendor output record by record against our own standard.

Impact

Score 63/100; the most serious finding was that identical empty-text reviews were split randomly from day to day (non-deterministic decisions). Feedback to the vendor became concrete, per-channel and reproducible.