False positive

A false positive is content a moderation system flags even though it breaks no rule, such as a harmless word that looks like a slur. Every automated filter makes some; the goal is to make them cheap. Act above your own line for each flag, send uncertain results to a person instead of blocking them, and measure the rate on a sample of your own content.

False positives and false negatives

A false positive costs the poster a blocked post or a wait, and costs you appeals and trust. A false negative leaves harm in view until a report or a later check finds it. Moving your line trades one for the other.

Why they happen

• Context. A swear word in a joke between friends reads like abuse on its own.
• Quotes and reports. A user who quotes abuse to report it uses the same words. A model reads the message; a person can read the thread (AI for hate speech and harassment).
• Reclaimed words. Some groups use a word about themselves that is a slur from others.
• Words inside words. A listed string inside a harmless word, the Scunthorpe problem, which text in any language meets in everyday words.
• Language. A model is less reliable on text it was not built for. Detectivision's text results are reliable in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi only (text moderation).

Is there a "good" rate?

No universal number. A false positive that sends a comment to a reviewer for an hour costs less than one that bans an account, and a missed self-harm message costs more than a missed swear word. Set targets per flag and per action: strict where a miss does serious harm and a person reviews every flag, looser where the action is light. Industry averages tell you little, because they come from other content, other rules and other models.

Confidence does not settle it either. In Detectivision it is a score from 0 to 1, not a probability you can compare across flags (flags), so draw a separate line for each flag.

Measure it on your own data

1. Take a random sample of your content, flagged and unflagged.
2. Have people label each item against your rules without seeing the API's answer.
3. Count per flag: precision is caught ÷ (caught + false positives); recall is caught ÷ (caught + false negatives).
4. Repeat for each line you consider.

This function scores one flag at one line from the fields every result has: flags with each raised flag and its confidence, and checked with every flag that was looked for (responses).

def precision_recall(results, labels, flag, line):
    """results: Detectivision answers; labels: True where your reviewer says the item breaks the rule."""
    tp = fp = fn = 0
    for result, breaks_rule in zip(results, labels):
        if flag not in result["checked"]:
            continue  # not looked for (unchecked, or not requested): no verdict to score
        # A flag that was not raised has no confidence: findings under 0.5 never come back.
        confidence = next((f["confidence"] for f in result["flags"] if f["flag"] == flag), 0)
        flagged = confidence >= line
        if flagged and breaks_rule:
            tp += 1
        elif flagged:
            fp += 1  # a false positive
        elif breaks_rule:
            fn += 1  # a false negative
    precision = tp / (tp + fp) if tp + fp else None
    recall = tp / (tp + fn) if tp + fn else None
    return precision, recall

A worked example, with illustrative numbers that are not a measurement of Detectivision: 400 labelled comments, 40 of them insults.

We ran the function on a sample built to these numbers and it returned the same results. The 4 insults missed at 0.5 cannot be recovered by any line you set: findings under a flag's threshold never come back (flags). User reports and later checks catch some of them.

Make them cheap

• A review band. Act automatically above your own line and send the band between 0.5 and that line to a person. In the example, blocking at 0.8 removes 2 harmless comments instead of 14, and 20 items, 12 of them harmless, wait for a reviewer instead. See human-in-the-loop moderation.
• Hold instead of block. Show a held item to its author only, or mark it pending, until a person decides.
• Lighter first actions. Ask the user to rephrase, or hide a post from others, before removing it or banning anyone.
• Appeals. Let users contest a decision and record the outcome. Overturned decisions are your false positives, labelled for free.
• Lines per place. A space for children and an adult forum need different lines for the same flag.

How these fit into a full pipeline: content moderation system design.

Why regulators ask about error rates

The EU Digital Services Act (DSA) covers intermediary services, such as hosting services and online platforms, offered to users in the EU. Under Article 15, providers must publish clear reports on their content moderation at least once a year. Providers that qualify as micro or small enterprises are exempt, unless they are very large online platforms (Article 15(2)).

The reports must cover "any use made of automated means for the purpose of content moderation", including "indicators of the accuracy and the possible rate of error" of those means and the safeguards applied (Article 15(1)(e)). A labelled sample, measured as above, can help you prepare those indicators.

What Detectivision does not do

• It publishes no accuracy figures. No false positive, false negative or accuracy rate is claimed for its flags; measure them on your own content.
• It does not return findings under a flag's threshold. You cannot see or recover them.
• It does not label your sample or review your queue. Your people do; Detectivision gives them Logs, your own actions and webhooks (actions).

Read about flags and thresholds