When an AI moderation system underperforms, the instinct is to reach for a better model. Far more often the model was capable, the instruction was poor, and the organisation had no mechanism to tell the difference.
Every team that puts an LLM to work against a rulebook arrives at the same wall. You have a policy, written in careful, hedged language by people who spent months negotiating it with legal, comms and regulators. And you have a model that will do more or less what you tell it, provided you tell it in a form it can act on. Between those two things sits a translation problem that almost nobody owns.
Policy teams write for trained human reviewers, with all the interpretive latitude that implies. Engineering teams write prompts. The gap between them is where quality quietly goes to die, and by quality I mean the two numbers that actually matter, precision and recall. Precision is how often the system is right when it acts. Recall is how much of the violating content it catches at all. Almost every complaint about an AI moderation system resolves into one or the other.
Where AI enforcement actually fails
The failure modes are consistent across organisations, and operations and policy teams can diagnose their own setup against them quite quickly.
Policy language gets imported wholesale. A rule written to guide a trained reviewer, full of "generally", "in context" and "may be considered", goes into the prompt untouched. The model produces inconsistent output because the instruction is genuinely ambiguous. The model is behaving as expected, however, it is being asked an unanswerable question.
Exceptions to policies are buried somewhere in training decks. Most policies state their intent in a preamble and their rules in a list, and the two do not always agree. Reviewers bridge that gap from experience. The prompt cannot, so the exception is never applied and recall on legitimate content collapses.
Nothing is versioned. Precision moves and the cause is unknowable, because three people edited the wording in the same fortnight and none of them recorded what they changed.
False positives are read as a model problem. They are usually an instruction problem.
A four-step framework
To keep this concrete I will carry one policy line and one hard case through all four steps. The worked example runs in italics throughout, so the approach can be read on its own.
The policy line: content depicting graphic violence, including footage of injured people outside a medical setting, is not permitted.
The hard case: a user posts footage of an injured protester being carried away after a demonstration, captioned with the city, the date and an account of what the security forces did.
A model handed the policy text alone does exactly what the text says. It sees graphic injury, matches the prohibition, removes. It is not wrong about the content. It has simply never been told that documenting an event is a different act from celebrating it, because the policy says so in a preamble the enforceable rules never reference. Every such removal is a precision failure the team will spend the next quarter appealing, and a recall failure on the newsworthy content the platform meant to protect.
Step 1. Decompose the policy into decidable units
A policy is not one rule. It is a stack of them, tangled together. The first step is to break the written rule into atomic questions that can each be answered independently, ideally yes or no, with a defined tiebreak for when the honest answer is "unclear".
That tiebreak is a precision decision in disguise. Sending genuine uncertainty to a human queue costs you a little coverage and protects your precision. Forcing the model to guess does the opposite.
Do this exercise honestly and something uncomfortable surfaces: genuine ambiguity in the policy itself. In my own sessions we repeatedly found rules that no two experienced reviewers had ever applied the same way. The model did not create that inconsistency. It made it impossible to keep ignoring. That value arrives before you touch a prompt.
Decomposed, the rule above becomes a short sequence of questions:
- Does the content depict real violence, or its immediate aftermath, involving identifiable people?
- Is there context establishing a documentary or public-interest purpose: a named place, a date, an event, a caption that reports or condemns rather than celebrates?
- Does the framing glorify the violence, mock the victim, or call for more of it?
- Is the depiction gratuitous relative to that purpose: does the content dwell on the injury beyond what documenting it requires?
- And before any of it: is this first-person footage filmed by the perpetrator of an attack? If so, no context exception applies. Remove.
That last question matters as a design principle. Every large platform ring-fences a small set of categories where no public-interest argument can unlock the content. Decomposition is what lets you encode a hard stop that nothing downstream can override. A single blended judgement cannot.
Step 2. Rewrite the rule as model-ready instructions
Models follow procedures far better than they infer from definitions. The most consequential change I made was to stop writing prompts that describe what a violation is and start writing prompts that describe how to decide. Those are different documents. The first is a definition; the second is a sequence of steps.
Then anchor the procedure with near-miss pairs. A near-miss pair is two pieces of content that look almost identical, where one breaks the rule and the other does not, written out side by side so the model can see the single detail that flips the decision. It is the difference between telling someone the rule and showing them where the line sits. A well-chosen pair does more for precision than several paragraphs of qualification, and you do not have to invent them: just roll up your sleeves, your appeals queue is full of them already.
Applied here, the instruction sets the order of operations: identify the violence, then look for documentary context, then read the framing, and only then assign an outcome. Reporting language, a location, a date or condemnation counts as evidence of purpose. Celebratory captions, mockery or calls for repetition override any claimed documentary framing.
And the near-miss pairs:
- Protest footage captioned with where and when it happened and what it shows: allow, behind a warning screen.
- The same footage captioned to celebrate the injury or to wish it on a named group: remove.
- An injured bystander filmed by a witness, with a caption reporting the incident: allow, behind a warning screen.
- The same imagery cropped tight, set to music, with no context at all: restrict.
Decomposition also buys you something a single blended prompt cannot express: graded outcomes. Real enforcement is not binary. Content can be removed, or age-restricted, or placed behind a warning screen, or made ineligible for recommendation while remaining available to anyone who seeks it out. The largest social platforms are steadily moving in that direction. When each unit is decided separately, each can drive a different lever, and grading is often what lets you raise automation coverage without paying for it in precision.
Step 3. Version and evaluate
Every instruction change gets a version number, a named owner, and a measurement against a stable, human-labelled set before it goes near production. Not "we tried it and it felt better", rather precision and recall, on the same data, against the previous version.
Build that set from real traffic rather than invented examples, weight it towards the ambiguous cases where changes actually show up, and use two independent labellers with a third to break ties. Track how often those labellers agree with each other. It is the most useful diagnostic you own, and it sets a hard ceiling on the quality any model can reach: if your own experts agree only eighty per cent of the time, no prompt will score better than that.
The ledger is unglamorous, and it is where the compounding happens:
| Version | Change | Precision | Recall |
|---|---|---|---|
| v1.0 | Baseline instruction | 71% | 63% |
| v1.1 | Added near-miss pairs | 84% | 69% |
| v1.2 | Tightened definition of "gratuitous" | 89% | 64% |
| v1.3 | Separated documentary context from framing | 88% | 79% |
| v1.4 | Added the perpetrator-footage hard stop | 90% | 83% |
| v1.5 | Split outcomes: remove, warning screen, restrict | 91% | 87% |
Take the shape of the trade-off the illustration above is built to show: a change that tightens a definition can lift precision by several points while quietly costing far more in recall, because the system stops acting on cases it should have caught. On a precision-only dashboard, that release looks like a win. The pattern behind that illustration is one I have seen repeatedly in real programmes: near-miss pairs produce the largest genuine jumps, while rewriting definitions to be more precise is often neutral or slightly negative. You only learn that if you are measuring both numbers together, because a change that lifts precision by suppressing action will quietly cost you recall.
Step 4. Close the loop between enforcement and policy
The final step turns a technique into a system. Errors are routed back, classified, and used to revise either the instruction or the underlying policy, with someone explicitly deciding which.
Some errors mean the prompt was vague. Others mean no human could apply the rule consistently either, and no amount of prompt craft will fix it. Conflating the two is how teams end up endlessly tuning wording against a rule that was never coherent. Low agreement between your labellers is the signal that the fix belongs in policy.
In the case above, an instruction rewrite alone would not have been enough. The exception the decision needed, that documenting an event is not endorsing it, had to exist as an enforceable rule before any prompt could apply it. That is a policy fix, and it belongs back at step one.
What it takes to run
Implementing this does not require extra headcount. It requires governance, and the ability to point resources you already hold towards the same goal. The costs that catch teams off-guard are not people, they are inference and localisation.
The machinery consists of four things: a named owner for the instruction layer with the authority to hold a release; an evaluation set per policy area that is maintained rather than built once, meaning two labellers, a tiebreak and someone watching their agreement; somewhere to store versions with timestamps and approvers, which can be a repository rather than a product; and a feedback loop that sorts each error into an instruction fault or a policy fault. Starting out with the top appealed and overturned policy areas, that is a fraction of one role. Across a dozen policy areas in several languages it is a small team, and still cheaper than the appeals volume it removes.
Two costs are routinely underestimated. The first is inference: four sequential questions cost more per item than one blended judgement, and on a real-time surface that lands as latency as well as spend. Route the obvious cases down a cheap path and reserve full decomposition for the ambiguous band, which is where the coverage is won anyway. The second is language. Instructions do not survive translation: near-miss pairs written in English are likely close to useless in German and potentially misleading in Arabic, because irony, insult conventions and the distance between rudeness and dehumanisation do not map across. Decompose once, then write and evaluate the instruction layer natively in every language you enforce.
What this is worth
Applied consistently and rigorously, this discipline helped drive automated coverage in my own operation up to 95% in some high-volume workflows, while precision and recall were held at or above the human review baseline they replaced, measured on a stable, human-labelled evaluation set refreshed quarterly, not on production sampling. It was not the only lever, but was certainly among the most materially impactful, and by some distance the cheapest.
There is a governance dividend too, and I think it is underrated. A versioned, tested, attributable instruction layer is an audit trail. When a regulator or a risk committee asks why the system decided what it decided, "here is the exact instruction version in force at that timestamp, the precision and recall behind it, and who approved it" is an answer. "Here is the model we used" is not. As enforcement moves further into highly regulated territory across different regimes globally, I reckon that will stop being best practice and will become a precondition.
What moderation teams can adopt today
The first decision is the one most teams leave to chance: who leads this. It cannot be delegated to a single function, but it does need a conductor, and that should be operations. They know the policy at the level of its nuances rather than its headlines, they sit closest to the frontline reviewers who generate every edge case and every label, and they are in constant contact with policy, product and engineering. That makes operations the conductor: policy in for the decomposition, reviewers in for the instruction design, product and engineering in for the evaluation, legal in when an error turns out to be a rule problem. Where I have seen this work, operations held the baton throughout.
Start narrow. Take your highest-volume enforcement reason and decompose it on paper in ninety minutes. Pull twenty near-miss pairs out of last month's appeals. Give your current prompt a version number and a named owner today. Score every change on precision and recall together. And where your own reviewers disagree, send it back to policy rather than back to the prompt.
None of this requires a new vendor, a bigger model or a large engineering team. It requires the expertise your organisation already holds, rendered in a form a machine can execute reliably and accountably. The model is a commodity. The translation layer is where the value lives.