Redesigning a spam/relevance prompt from v1 to v2

Context

Large industrial group and its chairman (person + organisation monitoring) · 6 channels

Problem

The existing LLM prompt was 97.8% right when it said “spam”, but caught only 23.2% of irrelevant content. In a 5,696-record period, 3,901 irrelevant records leaked into reports. On top of that, same-name individuals (6 different accounts sharing the name) and confusable legal entities polluted the data.

Approach

I measured the error, found the root cause (a tie-break sentence overriding the platform rule), rewrote the prompt around a hard “phrase gate”, and added same-name and out-of-scope sections, a structural-noise signature list and a reason code on every decision. I built a 137-record, 7-layer regression set.

Impact

Recall rose from 23% to the 88–96% band (target was 85%+) while precision stayed above 95%. Every version change runs against the same set; regressions are visible immediately.