Content moderation API

A content moderation API is a web service that checks content you send it, such as a comment or a photo, and answers with the categories it found, like hate speech or sexual content, and how confident it is. Your app then decides what to do: publish, hold for review or block. It supports people instead of making them read every post.

What you send

One HTTPS request per item, from your server. With Detectivision that is POST https://api.detectivision.ai/api/v1/moderation, with your key in the Api-Key header. Never ship the key in a web page or an app, where anyone could read it.

What the flags cover is on text moderation and on detection pages such as NSFW detection.

What comes back

Here a review comment is checked for three flags, with its id attached.

Example response (format from the API reference; values illustrative):

curl https://api.detectivision.ai/api/v1/moderation \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text_moderation",
    "input": "Terrible seller. You are an idiot.",
    "mode": "detect",
    "flags": ["hate", "insult", "self_harm"],
    "metadata": { "commentId": "c_981" }
  }'

{
  "id": 1050,
  "type": "text_moderation",
  "mode": "detect",
  "status": "completed",
  "flagged": true,
  "flags": [{ "flag": "insult", "confidence": 0.92 }],
  "checked": ["hate", "insult", "self_harm"],
  "unchecked": [],
  "tookMs": 640,
  "usage": { "charged": 2, "remaining": 8417 },
  "metadata": { "commentId": "c_981" }
}

Field by field, as the responses reference defines them:

Detectivision's flags judge the whole text or image: a result says what was found, not where.

How the decision is made

AI content moderation combines three kinds of judgement, in proportions that differ by vendor and category:

• Classifiers: models trained on labelled examples that score each category.
• Language models asked to judge: a general model reads the content against a written rule, such as "is this a personal attack?", and returns a score or a label.
• Rules and lists: word lists, patterns, and fingerprint lists of known illegal files.

Inside one Detectivision check:

1. The API reads the request and the flags to check.
2. Each model that feeds one of those flags scores what it finds. Models that feed none of them are not run.
3. A flag is raised when its score passes the flag's threshold, 0.5 by default. Findings below it are dropped and never come back.
4. The answer lists the raised flags, every flag checked, and any that could not be.
5. Your code applies your rules: block, hold for review or publish.

A confidence is a score, not a probability you can compare across flags: 0.8 for hate and 0.8 for insult do not mean the same. Act above your own line, per flag (flags). Software is consistent at volume but can miss context such as sarcasm, quotes and new slang, which is why unclear cases go to people.

What it does not do

• It does not write your policy. You decide which flags block, which go to review and which you ignore.
• It does not act on your platform. Detectivision never removes a post or bans a user; your system does, when a result or a moderation.action webhook arrives (actions).
• It does not verify age. Finding out how old a user is, called age assurance, is a separate job; see age assurance.
• It does not say where. No word positions, boxes or censored copies come back.
• It does not read every language. Text results are reliable in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi only (text).
• It does not check video, audio or live streams yet. They are coming soon.

How moderation APIs charge

Vendors price in different units: per request or per 1,000 requests, per image, per block of characters, or in credits that come with a monthly plan. Some offer free tiers with limits: free content moderation APIs compares them with dated facts.

Detectivision charges credits per type. Text costs one credit per started 2,500 characters async, images are priced per image, and an answer given at once costs twice the async price. Errors and partial results are free (credits). Plans and the free trial are on pricing.

Choosing one

Compare APIs on your own content, against these points:

• Types today: which of your content types it checks now, by its status page, not its roadmap.
• Languages: results on text your users write, including slang and text typed without accents.
• Categories: whether its flags match your rules, and whether you can check only some.
• Results: separate flags or one score; what a partial or failed check looks like.
• Timing and price: the price at once and async, and what is free.
• Delivery: signed webhooks with retries, and a field for your own ids.
• Data: what it keeps, for how long, and where it is processed.

For vendors side by side, see content moderation APIs compared; for the ways to use one, see types of content moderation.

Start the free trial

Or read the quickstart first.