A working brief on content moderation policy rules

A policy is a set of rules and a rule is a small piece of writing with a hard job: it has to produce the same answer in two different heads, working fast, on material nobody anticipated.

This is about drafting one rule. Not a whole policy, and not the pipeline it runs inside, which is described in how a moderation system is assembled.

When drafting for a specific service, identify the rulebook the new rule must fit. Examples include Meta Community Standards, YouTube Community Guidelines, and TikTok Community Guidelines. Treat them as references to each platform’s own wording and scope, not as substitutes for the law or for your service’s policy.

What to take away

The six parts of a workable rule

The prohibited act, described as conduct. Not character, not effect on the reader, not intent that cannot be observed. What was done, in terms someone could point at.

Scope. Which surface and which account type. Rules that apply everywhere without qualification are usually rules somebody has not thought about applying to advertising, to direct messages, or to a business account.

The exceptions, named. Documentation, commentary, education, satire, quotation. If reviewers have to infer these, they will infer different ones. Exceptions are where consistency dies and naming them is the only mitigation.

Worked examples, including near misses. Three items that clearly breach, three that clearly do not, and three that are close on either side. The last three carry most of the value and are the ones drafting sessions skip.

The outcome. Removal, reduced distribution, an age gate, a label, an account consequence, or an escalation. A prohibition with no stated outcome will be applied with whatever outcome the reviewer's tooling makes easiest.

A version and a date. People are judged against the rule in force at the time, and you cannot demonstrate that without keeping the old versions.

Illustrative rule and boundary cases

Illustrative rule (not a quotation from a platform): On public posts and comments, do not publish a private person’s home address while identifying it as that person’s residence, unless that person has clearly consented. This rule covers original posts and reposts. It does not cover an official business address or a news report that discusses an address without identifying it as a private person’s residence. Remove covered content, send a notice citing the rule, and escalate cases where consent or identity cannot be established. Version: draft; set and record the effective date before publication.

Words that make a rule untestable

A useful discipline: if you cannot describe a test that a reviewer could apply in fifteen seconds, the rule is not ready. That is not an argument for shallow rules. It is an argument for putting the depth in the examples rather than in the adjectives.

The vagueness doctrine is developed in U.S. constitutional law. In Grayned v. City of Rockford, the U.S. Supreme Court applied vagueness principles to a government ordinance. That doctrine is not a direct constitutional requirement for a private platform’s rules, but it illustrates why people need to understand what conduct a rule prohibits.

Private rules face no such requirement, and the practical result is the same anyway. An unclear rule is applied unevenly, and that unevenness is what people experience as unfairness.

For U.S. services, Section 230 of the Communications Decency Act is relevant to intermediary liability and certain good-faith moderation. It does not prescribe a platform’s individual content rules or replace other notice and appeal duties that may apply.

Testing a rule before it ships

Take fifty real items, drawn from ordinary traffic rather than from a list of known bad ones. Give them to at least two people with the draft rule and nothing else. Compare.

Do not look at the error rate. Look at the disagreements, one by one, and what caused each: a missing definition, a missing exception, a needed example, or a genuine judgment call the rule cannot remove.

The first three are fixable in the text. The fourth needs a decision about who makes that call and whether to escalate it instead of deciding in the queue.

Run the same test again after any substantive edit. A rule that was clear before an amendment is frequently not clear after one, because the amendment was written to address a case rather than to fit the structure.

Two decisions that hide inside every rule

Which mistake you would rather make. A rule tuned to catch more of what it targets will also catch more of what it does not. You cannot tune both directions at once. Make the decision explicitly, per rule, because the right answer differs enormously: for material causing severe irreversible harm, over-removal is the cheaper error, and for commentary, documentation and reporting it is extremely expensive.

Who bears the cost of the residual error. If the answer is always the user and never the operation, the appeal route is doing no work. A rule with a known false-positive rate and no reachable appeal is a decision to accept a certain number of wrong outcomes and not hear about them.

Writing the notice at the same time

Draft the message the affected person receives alongside the rule, not afterwards. A notice generated from a decision code rather than from the reasoning is why so many enforcement messages explain nothing, and the fix is upstream: if the rule has a name a person can understand, the notice can use it.

A notice is usable when it names the item and the rule. It must also say what the person can do next and by when. That reasoning follows a general account of notice.

For services covered by the EU Digital Services Act, certain content restrictions require a statement of reasons that identifies the grounds for the decision, relevant facts, and available redress. This is not a universal U.S. notice requirement; identify the duties that apply to the service before setting a notice deadline.

What the recipient can do with the message is set out in what an appeal can and cannot do.

Appeal routes depend on the service, applicable law, and case eligibility. The EU Digital Services Act provides for internal complaint handling and access to certified out-of-court dispute settlement bodies for covered services. Meta’s Oversight Board reviews only eligible cases under its own rules; it is not a general appeal route for every Meta moderation decision. Name the route and eligibility in the notice.

Reviewing rules from the hard cases

Difficult cases are the only useful input for the next version, so collect them as they happen with the decision and the disagreement. Put them before someone with authority to change the text on a fixed cycle.

A rule never revised against the cases it failed on is a rule nobody has actually tested.

When a rule comes from an obligation on the service, not a product decision, the revision cycle runs on an external clock. Put that clock in your policy record, not someone's memory.

For services within its scope, the EU Digital Services Act also requires notice to users of significant changes to terms and conditions and imposes transparency-reporting duties that vary by service category. Record the applicable duty, effective date, notice, and reporting cycle. National Digital Services Coordinators have oversight roles under the Act, and the European Commission directly supervises designated very large online platforms and search engines; confirm which authority and obligations apply rather than treating every external clock as the same.

The general handling is in tracking what moved and when. Which body imposed the rule, and whether it reaches you directly or through a contract, is covered in how regulation arrives.

Common questions

Should rules be published in full, including the examples?

The examples are the most useful thing you have and publishing them helps people comply. The argument against is gaming, which is real for a small number of categories and overstated for most. Decide per rule rather than as a general posture.

How specific is too specific?

When the rule stops generalizing to items you have not seen. A rule written around one incident will be precise, testable, and useless the next time something slightly different happens.

Who should write these?

Someone who has actually made decisions in the queue, with review by whoever owns the risk. Rules written only by people who have never applied one are consistently too abstract, and rules written only by reviewers are consistently too narrow.

What about rules that exist because of a legal obligation?

Those need counsel, in every place you operate, and the obligation and its implementation should be recorded separately. Merging them means you cannot tell later which parts you are free to change.