Reviews
Part of Content moderation policy: the parts worth your attention
Content moderation policy case studies: a review method
Content moderation policy case studies: a practical method for sampling decisions, comparing judgments, documenting reasoning, and revisiting cases over time.
Published accounts of moderation decisions are mostly unusable. They describe the outcome, not the reasoning, and the reasoning is the only part that transfers.
The material worth studying is your own. This page explains how to produce content moderation policy case studies: sampling decisions, comparing them honestly, and writing up the ones that taught you something.
What to take away
- Sample randomly. Reviewing only the decisions that generated complaints tells you about complaints, not about decisions.
- The disagreement between two reviewers is the data. The verdict is not.
- Write up the case, not the item. A case without the reasoning is an anecdote.
Why published accounts do not transfer
Three reasons, and they apply to almost everything you will read.
The rule set is different. An account of a decision under one service's guidelines, at a particular version, tells you very little about a decision under yours.
The operating conditions are different. A decision made in eight seconds by a reviewer measured on agreement with an answer key is a different object from a decision made in ten minutes by someone who knows the community.
The outcome is reported and the reasoning is not. Coverage records that something was removed or restored, usually after a public argument, which selects for the cases where an unusual amount of attention was applied.
Read them for the shapes of hard cases. Do not read them for what you should do.
Sampling your own decisions
Pull a random sample from a defined period. Random matters more than size: a hundred random decisions are far more informative than every complaint you received, because complaints select for confident, articulate users and for a narrow set of categories.
Sampling your own decisions
- Pull a random sample from a defined period
- Random matters more than size
- Stratify by outcome to include items left up
- Include the notice that was sent
Stratify by outcome so that the sample includes items left up. Reviews that only look at removals cannot detect the errors in the other direction, which is half the error and the half nobody sees.
Include the notice that was sent. A correct decision explained by an unusable message is a partial failure and should be recorded as one.
The calibration session
Two or more people re-decide the sampled items independently, without seeing the original outcome, then compare.
Running a calibration session
- Two or more people re-decide independently
- Compare without seeing original outcome
- Record agreement, not accuracy
- Name the cause of each disagreement
- Do not resolve by vote
- Take causes to rule or tooling owners
- Run on a fixed cadence and after rule changes
Run one on a fixed cadence and after any rule change. The number worth tracking over time is the disagreement rate per rule, because it identifies which rule is not carrying its weight.
What a write-up contains
| Field | Why |
|---|---|
| The question, in one sentence | Forces the case to be about something |
| The item, described, not reproduced | Reproducing it recreates the harm in some categories |
| The rule cited, with its version | The operative rule is the one in force at the time |
| The decision, and who made it | Accountability |
| The competing reading | The case is only interesting if there was one |
| What settled it | The transferable part |
| What the notice said | Where most of the residual damage happens |
| What changed afterwards | A case with no consequence teaches nothing |
The two fields that make it worth keeping are the competing reading and what settled it. Without them you have recorded an outcome, which the log already had.
What a write-up contains
- The question, in one sentence
- The item, described, not reproduced
- The rule cited, with its version
- The decision, and who made it
- The competing reading
- What settled it
- What the notice said
- What changed afterwards
De-identifying properly
A case study describing a real decision about a real person carries real risk, and the ordinary reflexes are not enough.
De-identifying properly
- Change identifying details, do not just delete
- Describe the category of item, do not quote
- Blur the date to a month or quarter
- For a minor, default to not writing it up
Remove the name and you have often removed nothing: an unusual set of circumstances identifies a person to anyone who was there. Change the identifying details rather than deleting them, describe the category of item rather than quoting it, and blur the date to a month or a quarter.
Where a case involves a minor, the default should be not to write it up at all outside a formal safeguarding record. The separate structure that governs that is described in which document governs which incident.
If a case cannot be de-identified and still make its point, the point is probably about the person rather than about the rule, and it is not the kind of material this exercise produces.
Reading the aggregate numbers you publish
If you publish your own decisions, the same cautions that apply to reading other people's reports apply to writing yours. Give the denominator, and say whether you count items, accounts or actions. Define the categories rather than naming them, and explain any methodology change loudly, not in a footnote.
Publishing aggregate numbers
- Give the denominator
- Say whether counting items, accounts or actions
- Define the categories rather than naming them
- Explain any methodology change loudly
The reasons those things mislead are set out in reading a transparency report.
Two external reference points are worth having when the categories touch legal obligations rather than product choices. One country's statutory notice-and-takedown scheme for copyright material is described by its own copyright office at an overview of that regime. The European Union publishes its own account of its rules for online services.
Both describe one system's requirements for services within its scope, and neither tells you what applies to you. That question runs through the routes by which a rule reaches an operator.
A year later
Re-read the case file annually. Two questions: would the same decision be made today, and if not, was that a rule change or a drift nobody decided. Drift is normal, invisible, and the reason a policy that was consistent in January is not in December.
Where the decisions concern people inside your own organization rather than users, the review has a different character and different constraints, described in what actually goes wrong with workplace rules.
Common questions
How many cases should we write up?
Very few. Most decisions are routine and produce nothing worth reading. A handful a quarter, chosen because they were genuinely contested, is more useful than a large file nobody opens.
Can we share these outside the team?
Internally, usually yes, with de-identification. Externally is a different decision involving people who did not agree to be a teaching example, and the default should be no.
Should the person affected see the write-up?
They should see the decision and the reasoning at the time, which is a separate and more important obligation. The internal case file is a different document with a different purpose, and conflating the two tends to make the file dishonest.
