Guides
Part of Content moderation policy: the parts worth your attention
Content moderation tools for 2027: five categories, and most teams only need two
Best content moderation policy tools: the five categories, what a small operation should build rather than buy, and the questions that expose a weak vendor claim.
No product recommendation here, for two reasons. The right answer depends on your volume, your categories and your risk, none of which a page can know. And a named recommendation is stale within a year.
The categories and the buying questions are stable, so those are what this covers.
What to take away
- Detection is the category people shop for and the least likely to be the bottleneck. Case management usually is.
- Anything that produces flags produces work. Buying detection without review capacity makes the queue worse.
- The two things worth building yourself, whatever else you buy, are the decision record and the appeal route.
- For most teams, the two categories worth paying for are queue and case management and appeals handling. The other three can start as a spreadsheet, a specialist arrangement or a later purchase.
The five categories
Detection and classification. Models that score material against categories. Useful at volume, useless below it, and every score needs a threshold decision that is really a decision about which error you prefer.
Five Moderation Tool Categories
Detection
- What it does
- Scores material
- Error profile
- Threshold choice
- Small-op need
- Useless below volume
Matching
- What it does
- Compares to known set
- Error profile
- Blind outside set
- Small-op need
- Specialist bodies
Queue
- What it does
- Routes and records
- Error profile
- Determines consistency
- Small-op need
- Most important
Appeals
- What it does
- Handles challenges
- Error profile
- Often bolted on
- Small-op need
- Rota not product
Examples include OpenAI's Moderation API, Google's Perspective API, Amazon Rekognition Moderation, Microsoft Azure AI Content Safety and Hive Moderation. Public or typical pricing runs from free tiers (Perspective API and OpenAI Moderation API) to roughly $0.50 to $5 per 1,000 items scored at low volume, with volume discounts. Treat those as typical ranges, not quotes.
Matching against known material. Comparing an item against a set of previously identified content. A different technology from classification, with a different error profile: high precision on what is in the set, blind to everything else. In some categories this is handled through arrangements with specialist bodies rather than through commercial products.
Examples include Microsoft PhotoDNA and the NCMEC hash sharing program for known child sexual abuse material, plus Audible Magic and YouTube Content ID for media matching. Access is often restricted to qualified organizations; commercial pricing is typically quoted per asset or per minute, with no public list price for most providers.
Queue and case management. Where items wait, get routed, get decided, and get recorded. The unglamorous category, and the one that most often determines whether an operation is consistent.
Examples include Zendesk, Salesforce Service Cloud, Jira Service Management, Hive Moderation and Checkstep. Pricing is typically per agent per month, with common ranges from about $20 to $150 per agent per month for general helpdesk seats, and higher for moderation specific platforms. Treat those as typical ranges, not quotes.
Appeals handling. Frequently an afterthought bolted onto the queue, which is how appeals end up being decided by the same process that produced the original decision.
Examples include Zendesk, Salesforce Service Cloud, Jira Service Management and Checkstep when configured with a separate appeal queue. The rule is workflow, not product: the appeal must route to a different person and a different view. Typical cost is a seat in the same system, often $20 to $150 per agent per month, not a separate license.
Reporting and analytics. What was decided, by whom, how fast, and with what agreement between reviewers. This is the category that tells you whether any of the rest is working.
Examples include Looker Studio, Tableau, Power BI and Metabase. Looker Studio and Metabase have free tiers; Tableau and Power BI typically run from about $12 to $75 per user per month. The figure that matters is reviewer agreement, often tracked as a percentage, not the dashboard license.
Matching a frustration to one of the five before shopping avoids the standard mistake, which is buying detection to solve a problem in case management and receiving a larger queue.
What a small operation should build
At low volume, most of this is a spreadsheet and a discipline. The parts worth building yourself are the ones that encode decisions rather than automate them.
What a Small Operation Should Build
- Decision record with rule, outcome, decider, date
- Appeal route to a named uninvolved person
- Escalation path for risk of harm
- Dated copy of rules at each version
The threshold at which tooling starts paying is roughly the point where one person can no longer hold the queue in their head, and it is usually reached later than vendors suggest. The practical run list for that scale is in the checklist for a small community.
The buying test
| Question | What a weak answer sounds like |
|---|---|
| Which of the five categories is this, judged on one? | All of them |
| What is the false positive rate on our material? | An accuracy figure from someone else's data |
| Can we set thresholds per category? | One global sensitivity control |
| Does it show the reviewer the surrounding context? | The item only |
| Can we export every decision, with reasoning? | Reports, but not the underlying records |
| What happens to items it cannot classify? | Silence, or a default action |
| Can appeals route to a different person? | Appeals return to the same queue |
| Who is accountable for a wrong automated action? | The model is described as advisory while acting automatically |
The last row is the one to press. A system that acts automatically and is described as decision support has an accountability gap in it, and the gap will be discovered during an argument rather than before one.
The Buying Test
Question
- Which category?
- All of them
- False positive rate?
- Someone else's data
- Per-category thresholds?
- One global control
- Shows context?
- The item only
- Export decisions?
- Reports, not records
- Unclassifiable items?
- Silence or default
- Appeals to different person?
- Same queue
- Who is accountable?
- Advisory but automatic
Weak answer
- Which category?
- False positive rate?
- Per-category thresholds?
- Shows context?
- Export decisions?
- Unclassifiable items?
- Appeals to different person?
- Who is accountable?
What not to buy
A detection product without a plan for the flags. Every flag is a human decision, and the arithmetic of that is what determines whether the purchase helps.
What Not to Buy
- Detection without a plan for the flags
- Automatic action where errors are expensive
- Tools whose categories you cannot redefine
- Anything that cannot export your decisions
Anything that acts automatically in a category where a wrong action is expensive to reverse. Automated action is appropriate where the error is cheap and the volume is high, which is a narrow set.
A tool whose categories you cannot redefine. Vendor categories will not match your rules, and mapping between two taxonomies is a permanent tax on every report you produce.
Anything that cannot export. Your decision history has to outlive the subscription, and it is also the material any later review depends on.
The part no tool covers
Tooling does not decide which error you would rather make, does not write a rule that two people apply the same way, and does not tell an affected person anything useful. Those are editorial and operational choices, and a tool that offers to make them is offering to make them badly.
The threshold decision in particular is a judgment about consequences, and it belongs to whoever owns the risk. Its shape is described in why every threshold is a choice about which mistake to make. The drafting problem is separate again, and is set out in what makes a rule testable.
Where obligations change the calculation
Some capabilities exist because a body requires them of services within its scope: record keeping, notice to affected users, internal complaint handling, periodic reporting. Where that is the case, the requirement sets the specification rather than the vendor, and it differs by market.
The European Union's own description of its rules for online services is one example of a regime that speaks to several of these areas, and one country's copyright office describes a separate notice-and-takedown scheme at its overview of that framework. Neither tells you what applies to your operation.
That question runs through the body that regulates you, which is the subject of finding the responsible body first, and it needs counsel rather than a vendor's compliance claim.
A vendor asserting that a product makes you compliant with something is asserting a conclusion about your organization that it is not in a position to reach. Treat it as marketing, and record the underlying obligation separately, in the way described in keeping an entry that does not overstate.
Common questions
At what volume does automated detection become worth it?
When the queue exceeds what your reviewers can decide with the care your categories require, and not before. The honest intermediate step is usually to narrow what enters the queue rather than to automate decisions.
Should appeals use a different tool?
Not necessarily a different tool, but a different person and a different view. If the reviewer sees the original decision before deciding, it is a confirmation step rather than an appeal.
How do we evaluate a classifier fairly during a trial?
On your own material, including the borderline cases you already argued about, and measured against decisions two of your own people made independently. Vendor benchmarks describe someone else's distribution.
