Toxicity detection

A toxicity detection API that says what kind.

Toxic is not one thing. Detectivision checks a comment or a chat message for insults, hate, racism, sexism, sexual harassment and self-harm, and answers with each flag it raised and how sure it is, in 12 languages.

What it finds

Seven flags instead of one score

Every account has these seven text flags; the Basic plan also has profanity, for swear words. Ask for all of them, or only those your rules need.

  • Insult

    Insults and personal attacks on someone.

  • Hate against a group of people for who they are.

  • Racism

    Racist or ethnic hate speech.

  • Sexism

    Sexist statements.

  • Sexual language, or sexual harassment of someone.

  • Talk of suicide or self-harm, so someone can step in.

  • Bad habits

    Drug use, drinking, smoking or gambling, talked up.

In code

Check a chat message for all seven at once

Request
curl https://api.detectivision.ai/api/v1/moderation \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text_moderation",
    "input": "Shut up, nobody wants you here, loser.",
    "mode": "detect",
    "flags": [
      "bad_habits",
      "hate",
      "insult",
      "racism",
      "self_harm",
      "sexism",
      "sexual"
    ]
  }'
Response
{
  "id": 3310,
  "type": "text_moderation",
  "mode": "detect",
  "status": "completed",
  "flagged": true,
  "flags": [{ "flag": "insult", "confidence": 0.92 }],
  "checked": [
    "bad_habits",
    "hate",
    "insult",
    "racism",
    "self_harm",
    "sexism",
    "sexual"
  ],
  "unchecked": [],
  "tookMs": 710,
  "usage": { "charged": 2, "remaining": 9998 }
}

flagged is the verdict: true when any flag was raised. checked lists every flag that was looked for; one that could not be is in unchecked, and then the check is free.

How it works
  1. Check before it shows

    Send a message or a comment as it was written; nothing is trimmed. Get the answer at once, or later by webhook with "async": true, at half the price.

  2. Act per flag

    Hide hate at once, hold insults for a moderator, and send self-harm to the people who can help. Each flag is its own decision.

  3. Review in Logs

    Moderators take your own actions, such as Approve or Ban, from the dashboard’s Logs; your system gets each one by webhook.

Related
Questions

Frequently asked

Do I get a single toxicity score?

No. You get flagged, true when any flag was raised, and each raised flag with its confidence from 0 to 1. Use flagged as the overall verdict and the flags to decide what to do.

Which languages are supported?

English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian, Hindi, Chinese and Japanese. Text in other languages is checked, but the results are not reliable.

What does a check cost?

A text costs 1 credit per started 2,500 characters when the answer comes by webhook, and twice that when it comes at once. Asking for fewer flags never costs more.

Can moderators review flagged messages?

A moderator can open a check in the dashboard’s Logs and see its flags, your metadata and your actions. You define those actions, such as Approve or Ban; when a moderator takes one, your system gets a moderation.action webhook and carries it out. Detectivision never acts on your platform itself.