Design Harmful Content Detection

Every platform that accepts user content has to find the worst of it quickly: violence, harassment, scams, self-harm, exploitation. The problem is hard because the categories are fuzzy, the stakes are very uneven, and people work to evade detection. The earlier chapter on production guardrails covered moderation around an LLM. This chapter is the classic side: a classifier system over posts, images, and videos.

A mailroom handles huge volume. Obvious junk is thrown out automatically. Anything that looks dangerous goes to a specialist. The ambiguous middle gets a human glance. And the sorters spend their limited time on the pieces that could do the most damage. A moderation system has the same layers.

A policy is a long document. The system needs something precise: a set of categories (hate, graphic violence, spam, scams, self-harm), each with a severity and an action. Some categories are removed automatically at high confidence. Others are only reduced in reach or sent for review. Write the taxonomy down first, since the labels, the metrics, and the thresholds all follow from it.

• -Known-bad matching: hashes or fingerprints of content already banned, caught instantly and cheaply.
• -Cheap filters: keyword and pattern rules, and small fast classifiers, to catch the obvious.
• -Category classifiers: one model per category or a multi-label model, over text, images, video frames, and audio. Embeddings from the content help, and a multimodal model can combine signals.
• -Context signals: account age, past violations, report counts, and how fast the post is spreading.
• -Thresholds per category and severity: very high confidence on severe content acts automatically, a middle band goes to human review, and low scores are left alone.
• -Human review and appeals: reviewers decide the hard cases, and their decisions become new labels.

Reviewers disagree, policies change, and context matters. A sentence can be a joke in one conversation and a threat in another. Use clear guidelines, double-label a sample to measure agreement, and keep a gold set labeled by experts. Treat the labels as an estimate of policy, not as the truth.

There are never enough reviewers. Sorting the queue by classifier probability sounds reasonable and wastes effort, because a near-certain violation seen by 40 people matters less than a likely violation seen by 90,000. A better order is expected harm: the chance it violates, times how many people will see it, times how severe the harm is. The severity weights are a policy decision, and the numbers here are invented. Press Run and compare the two orderings when three reviews are available.

Ranking by probability sends reviewers to the insult in a small group, the spam link, and the violence clip, and prevents an expected harm of 505,838, almost all of it from the one clip. Ranking by expected harm picks the violence clip, the misleading health claim, and the self-harm post, and prevents 730,500. That is about 44% more with the same three reviews. Note that the self-harm post has a lower probability, 0.55, than the spam link, but a higher severity and a much larger audience.

• -Precision and recall per category, on a labeled sample drawn at random as well as from flagged content. Recall cannot be measured from flagged content alone, since you only see what you caught.
• -Prevalence: the share of views that land on violating content. Estimate it by sampling content at random and having it labeled, not by counting what was removed.
• -Time to action: how long harmful content stays visible before it is acted on.
• -Fairness: error rates across languages, dialects, and communities, since models often do worse on languages with less training data.

• -Evasion: spelling tricks, coded language, images embedded with text. Retrain on adversarial examples and watch for new patterns.
• -Drift: new slang, new memes, and events change what content looks like.
• -Over-enforcement: wrongly removing legitimate speech has real costs, so measure appeals that succeed.
• -Reviewer harm: reviewers see disturbing content. Limit exposure and blur where possible.