How to detect toxic comments in an app or community

To detect toxic comments, check each one when it is written, before anyone else sees it. Three tools can do the check: a list of words, a classifier you run yourself, or a moderation API that runs one for you. All three give you a signal, not a decision. What you do with the signal, and whether an author can appeal, decides whether people trust your community.

What counts as a toxic comment

"Toxic" has no single definition. Jigsaw, which makes Perspective API, describes its TOXICITY attribute as "a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion." That covers a lot, so tools split it into kinds. Detoxify is an open-source classifier. Its original model learned from the data of Jigsaw's Toxic Comment Classification Challenge and predicts six labels: toxic, severe_toxic, obscene, threat, insult and identity_hate. OpenAI's moderation guide defines harassment as "Content that expresses, incites, or promotes harassing language towards any target." A second category covers harassment that also includes "violence or serious harm towards any target."

These are the kinds worth telling apart:

The hate speech row follows the United Nations: "any kind of communication in speech, writing or behaviour, that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are, in other words, based on their religion, ethnicity, nationality, race, colour, descent, gender or other identity factor." The UN page says this is not a legal definition. Laws differ, and your rules can go further than the law. Detectivision has a page on hate speech detection, and its profanity filter API covers swearing.

One comment can fit several rows. Swearing at someone is profanity and an insult. So write your rules per kind, not for "toxic" as a whole. "Remove toxic comments" cannot be tested. "Hide comments that insult a person, and hold hate speech for a moderator" can.

Three ways to detect toxic comments

Toxic comment detection tools fall into three groups. Each trades control against how much of the comment it understands.

A word list is the simplest to build. It matches letters and does not read the comment. Libraries such as bad-words (MIT licence) and obscenity (MIT licence) are free to use. obscenity's own README says: "As with all swear filters, Obscenity is not perfect (nor will it ever be)." Profanity filter vs AI moderation covers what lists catch and miss.

A classifier you run gives you scores and full control, and the upkeep is yours. Detoxify is one open-source example, under the Apache-2.0 licence. Its README says it is intended for research, for fine-tuning on carefully built datasets, and for helping moderators flag harmful content faster.

A moderation API runs the classifier for you. You pay per check or use a free allowance, and you depend on a network call. Perspective API, from Jigsaw, will not be in service after 2026, its site says. If you use it now, Leaving Perspective API shows where each of its attributes goes. For other APIs side by side, see content moderation APIs compared and free content moderation APIs.

Which one to pick

• Your rules name words. A banned name, a spam link, a word your community never uses. A list does this exactly, for free.
• Your rules name kinds of abuse, or you serve several languages. Use a classifier or an API. A list would need every insult, in every language and every spelling.
• Comments must stay on your servers. Run a classifier yourself, and plan the upkeep.
• You want both. A short list of your own can run next to a classifier or an API. Check the list first, because it is free, then check the rest.

How to detect toxic comments, step by step

1. Write your rules per kind of abuse. Say which kinds you remove, which you hold for a person and which you allow.
2. Collect your own test comments. Take a few hundred real comments, in every language your community writes. Mark each as fine, borderline or against the rules.
3. Check each new comment. Check it before it is shown (pre-moderation) or right after (post-moderation). The next section compares the two.
4. Set a line for each kind. Run the test comments through the check and sort them by confidence, as shown below.
5. Send the middle band to a person. Give moderators one queue, and a way to record what they decided and why.
6. Tell the author, and allow an appeal. A hidden comment with no explanation reads as censorship.
7. Keep what you learn. Store each result with its comment. Add every reversed decision to your test comments, and move a line when the misses show a pattern.

What to do with a toxicity score

A check returns numbers. Your rules turn them into decisions.

Set your lines

Detectivision raises a flag from a confidence of 0.5, so your lines sit above that. The confidence is a score from 0 to 1, not a probability you can compare across flags: 0.8 for hate and 0.8 for insult do not mean the same thing (confidence score). So set one line per flag. With 0.8 as the example line:

To find the line for one flag, list the test comments that raised it, highest confidence first, and read down. Set your hide line above the first harmless comment. If harmless comments sit high in the list, raise the line, or send more of the middle band to a person. Measure precision and recall on your own comments: figures measured on other people's data say little about yours.

Hide, hold or publish first?

Holding a comment keeps harmful content off the page at the price of a wait. Publishing first never makes anyone wait, but others can see a comment until the check ends and your system acts. Choose per place, not once for the whole app. Self-harm talk is the exception to all four rows: plan a caring response, not only removal (self-harm content, self-harm detection).

Appeals

An appeal is how an author gets a wrong decision reversed. The Santa Clara Principles (version 2.0), written by a coalition of organisations, advocates and academic experts, say what a notice and an appeal should contain:

• A notice gives the reason, including the specific clause of the guidelines the content was found to violate. It says how the content was detected, for example by automated detection. It says how to appeal.
• An appeal gets human review by people who were not involved in the first decision. The author can add information. The author receives the outcome and the reasoning.

In practice:

• Tell the author when a comment is hidden. Name the rule, say that a check flagged it, and show how to appeal.
• Send the appeal to a different moderator than the one who decided first.
• Let the author add context in a sentence or two.
• Answer with the outcome and the reason, and record both.
• Learn from reversals. Every reversed decision shows where your lines were wrong. Add it to your test comments.

Some laws set rules for complaints. See appeal in the glossary.

A working example with the Detectivision API

The example checks one comment and returns what your app should do with it. It asks for six text flags: insult, hate, racism, sexism, sexual and self_harm. Your account's own list comes from GET /api/v1/flags (Flags). The line, 0.8, is a starting point: replace it with your own.

Create a key with Write permission under Dashboard → API Keys (Quickstart). Then send the comment. Example answer (format from the API reference; values illustrative):

curl https://api.detectivision.ai/api/v1/moderation \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text_moderation",
    "input": "Nobody here wants to hear from a loser like you.",
    "mode": "detect",
    "flags": [
      "insult",
      "hate",
      "racism",
      "sexism",
      "sexual",
      "self_harm"
    ],
    "metadata": { "commentId": "c_4821" }
  }'

{
  "id": 2417,
  "type": "text_moderation",
  "mode": "detect",
  "status": "completed",
  "flagged": true,
  "flags": [{ "flag": "insult", "confidence": 0.91 }],
  "checked": [
    "hate",
    "insult",
    "racism",
    "self_harm",
    "sexism",
    "sexual"
  ],
  "unchecked": [],
  "tookMs": 95,
  "usage": { "charged": 2, "remaining": 9998 },
  "metadata": { "commentId": "c_4821" }
}

flagged is true when any flag was raised, and flags lists each one with its confidence from 0 to 1. checked lists every flag that was looked for, and unchecked any that could not be. metadata comes back unchanged, so the answer finds its comment. usage.charged is 2 because an answer given at once costs twice the price of one that comes by webhook (Credits and billing).

The function sends the comment and turns the answer into one of five decisions:

import os
import requests

API_URL = "https://api.detectivision.ai/api/v1/moderation"
FLAGS = ["insult", "hate", "racism", "sexism", "sexual", "self_harm"]
HIDE_AT = 0.8  # your own line: the API raises a flag from 0.5


def check_comment(text, comment_id):
    """Return "publish", "review", "hide", "support" or "retry" for one comment."""
    r = requests.post(
        API_URL,
        headers={"Api-Key": os.environ["DETECTIVISION_API_KEY"]},
        json={
            "type": "text_moderation",
            "input": text,
            "mode": "detect",
            "flags": FLAGS,
            "metadata": {"commentId": comment_id},
        },
        timeout=30,
    )
    if r.status_code == 429 or r.status_code >= 500:
        return "retry"  # busy or unavailable: nothing is charged
    r.raise_for_status()  # any other 4xx needs a changed request
    result = r.json()

    if result["unchecked"]:
        return "retry"  # partial: free; "no flags" is not "clean"
    raised = {f["flag"]: f["confidence"] for f in result["flags"]}
    if "self_harm" in raised:
        return "support"  # a person reaches out; no punishment
    if not raised:
        return "publish"
    return "hide" if max(raised.values()) >= HIDE_AT else "review"

// Node.js 18 or later, on your server: never in a browser or an app.
const API_URL = "https://api.detectivision.ai/api/v1/moderation";
const FLAGS = ["insult", "hate", "racism", "sexism", "sexual", "self_harm"];
const HIDE_AT = 0.8; // your own line: the API raises a flag from 0.5

async function checkComment(text, commentId) {
  const res = await fetch(API_URL, {
    method: "POST",
    headers: {
      "Api-Key": process.env.DETECTIVISION_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      type: "text_moderation",
      input: text,
      mode: "detect",
      flags: FLAGS,
      metadata: { commentId },
    }),
  });
  if (res.status === 429 || res.status >= 500) return "retry"; // nothing is charged
  if (!res.ok) throw new Error(`Detectivision: ${res.status} ${await res.text()}`);
  const result = await res.json();

  if (result.unchecked.length > 0) return "retry"; // partial: free; "no flags" is not "clean"
  const raised = Object.fromEntries(result.flags.map((f) => [f.flag, f.confidence]));
  if ("self_harm" in raised) return "support"; // a person reaches out; no punishment
  const top = Math.max(0, ...Object.values(raised));
  if (top === 0) return "publish";
  return top >= HIDE_AT ? "hide" : "review";
}

What it decides, and why:

• A partial answer is not a clean one. If unchecked lists flags, some checks could not run. The answer is free, so ask again (Responses).
• Busy or unavailable means try again. A 429, 500 or 503 costs nothing. A 429 or 503 comes with a Retry-After header: the seconds to wait. Errors has a retry loop. Any other 4xx needs a changed request: an empty comment, a flag your account does not have, or no credits left.
• self_harm goes to a person. The comment is not hidden as a punishment. Someone reaches out.
• The line is yours. From 0.8 up, the comment is hidden. Between 0.5, where the API raises a flag, and 0.8, it waits for a moderator.

Then your app acts on the decision:

• publish: show the comment.
• review: keep it pending, visible only to its author, and queue it for a moderator.
• hide: hide it from others, and show the author a notice with the reason and how to appeal.
• support: keep it pending and pass it to the person or team who reaches out to authors in distress.
• retry: keep it pending and check again in a minute.

Comments that can wait

A comment that can wait costs half, and your server need not hold a request open. Send it with "async": true. The API answers 202 at once with the request's id, checks the comment shortly after, and sends the result to your webhook (Async requests).

curl https://api.detectivision.ai/api/v1/moderation \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text_moderation",
    "input": "Thanks for the write-up, but you clearly have no idea what you are doing.",
    "mode": "detect",
    "flags": [
      "insult",
      "hate",
      "racism",
      "sexism",
      "sexual",
      "self_harm"
    ],
    "async": true,
    "metadata": { "commentId": "c_4822" }
  }'

{
  "id": 2418,
  "type": "text_moderation",
  "status": "queued",
  "metadata": { "commentId": "c_4822" }
}

The result arrives as a moderation.completed webhook. Its data is the same answer an immediate check returns, with your metadata (Webhooks). Apply the same rules to it. A check that could not be done arrives as moderation.failed. Up to 50 of your checks can wait at once.

Moderators, actions and appeals

In the dashboard, open Logs and choose Set up actions to define your own actions, such as approve, reject and restore. A moderator opens a check in Logs, presses an action and can add a note. Your system receives a moderation.action webhook with the check's metadata, and updates the comment. You can take an action through the API too, with a key that has write permission:

curl https://api.detectivision.ai/api/v1/moderation/2417/actions \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "restore", "note": "appeal upheld: a quote in a report"}'

Your system carries out each action. Detectivision tells it which one was chosen, for which check, with your own data (Metadata and actions). Record the reviewer's decision on an appeal the same way, so the check's history in Logs shows it.

Languages and cost

• Languages. English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi. Text in other languages is checked, but the results are not reliable.
• Price. A comment of up to 2,500 characters costs 1 credit when its result comes by webhook, and 2 when you wait for the answer. Errors and partial results are free.
• Plans. Basic costs $49 a month, with 10,000 text credits (pricing). Can you try it first? Yes. Basic is free for 2 months, with 10,000 text, 10,000 image and 300 video credits.

How the code was checked

On 17 October 2026 the Python function ran with Python 3.9.6 and requests 2.32.5, and the Node.js function with Node.js 22.23.2, against a local stand-in that answers in the documented format. Both returned the expected decision for each of these answers: no flags, a flag below the line, a flag on the line, a flag above it, two flags, self_harm and a partial answer. Both returned retry for a 429, a 500 and a 503. The stand-in also checked each request: the endpoint, the Api-Key header, the JSON body and the flag names. The tests ran against the stand-in, not the live API, and the example answers on this page use illustrative values.

Where people still decide

A check judges the comment it is given. These cases need a person, or context the comment does not carry:

• A conversation, not a comment. Harassment or a threat spread over several comments needs the thread. So does a reply that is abusive only next to what it answers.
• Quotes and reports. A comment that quotes abuse to report it can read like the abuse.
• Tone. Friends tease each other, and people joke about themselves. Detoxify's README warns that a comment with swearing or insults in it is likely to be classified as toxic "regardless of the tone or the intent of the author", humorous and self-deprecating ones included.
• New slang and coded words. Words made up to get past filters keep changing. Lists and models need updates to keep up.
• Other languages. English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi. Text in other languages is checked, but the results are not reliable.

For how a review queue works, see human-in-the-loop moderation. For the whole system around the check, see how to design a content moderation system. For hate speech and harassment on their own, see AI for hate speech and harassment.

Start the free trial

Next: see the toxicity detection API, read profanity filter vs AI moderation, or open the text docs.