How to design a content moderation system: checks, rules and queues

A content moderation system has four parts: checks that run when content arrives, rules that turn each result into publish, hold or block, a review queue where people decide what the rules are unsure about, and a record of every decision that your app acts on. Check before publishing where harm spreads fast, such as images and direct messages, and right after publishing where speed matters more, such as comments.

This reference architecture builds each part on a moderation API, with the endpoint and event names from Detectivision's docs, and marks what stays with you. Today the API checks text and images; video, audio and live streams are coming soon (API status, read 16 Oct 2026).

The four parts of a moderation system

1. Checks. Each new post, message, photo or profile goes to a classifier, your own or an API's, which answers with the categories it found and a confidence for each. Detectivision calls these categories flags, and your account's list comes from GET /api/v1/flags (Flags).
2. Rules. Your code turns each result into a decision: publish, hold for review or block. The rules are yours: per content type, per flag, and written down so they can change.
3. A review queue. People decide what the rules are unsure about, plus user reports and appeals.
4. A record, and actions. Every decision is stored with who made it, and reaches the part of your app that enforces it as an event.

Detectivision covers the checks and gives you pieces for the rest: confidence scores to write rules against, Logs with your own actions for the queue, webhooks for the events, and your own ids carried through every step in metadata. The rules, the reviewers and the enforcement stay in your system.

The workflow, step by step

Here is the content moderation workflow for one post, with the names the API uses:

Example payloads for steps 2 and 6 (format from the API reference; values illustrative):

{
  "type": "moderation.completed",
  "timestamp": "2026-10-16T09:41:07.512Z",
  "data": {
    "id": 1043,
    "type": "text_moderation",
    "mode": "detect",
    "status": "completed",
    "flagged": true,
    "flags": [{ "flag": "insult", "confidence": 0.62 }],
    "checked": ["hate", "insult", "racism", "self_harm", "sexism", "sexual"],
    "unchecked": [],
    "tookMs": 95,
    "usage": { "charged": 1, "remaining": 8412 },
    "metadata": { "postId": "p_981", "threadId": 42 },
    "source": "api"
  }
}

{
  "type": "moderation.failed",
  "timestamp": "2026-10-16T09:52:30.104Z",
  "data": {
    "id": 1044,
    "type": "image_moderation",
    "metadata": { "postId": "p_982" },
    "error": "invalid_input",
    "message": "the image could not be downloaded (HTTP 404)"
  }
}

{
  "type": "moderation.action",
  "timestamp": "2026-10-16T10:05:12.000Z",
  "data": {
    "action": { "key": "hide", "label": "Hide post" },
    "note": "insult aimed at another member",
    "by": { "type": "user", "name": "Elif" },
    "item": {
      "id": 1043,
      "type": "text_moderation",
      "flagged": true,
      "flags": [{ "flag": "insult", "confidence": 0.62 }],
      "metadata": { "postId": "p_981", "threadId": 42 },
      "createdAt": "2026-10-16T09:41:05.000Z"
    }
  }
}

Before or after publishing?

Pre-moderation holds content until it is checked; post-moderation publishes it and checks right after. Types of content moderation compares them with reactive, proactive, distributed and automated moderation. In one system you choose per content type, by how fast harm spreads and how long a person will wait:

The rows are a starting point, not a rule. Worked examples by industry: online communities and marketplaces.

At once or in the background?

Every check can answer in the same call, which is the default, or later by webhook when you send "async": true. The price table in Credits and billing is the async price; an answer given at once costs twice that. Text is priced per started 2,500 characters, and an image per check. Pay-per-request moderation covers how that pricing behaves as you grow.

An async request for a comment, asking only for the flags your rules use. Example response (format from the API reference; values illustrative):

curl https://api.detectivision.ai/api/v1/moderation \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text_moderation",
    "input": "Nice build, but whoever designed this is an idiot.",
    "mode": "detect",
    "flags": [
      "hate",
      "insult",
      "racism",
      "self_harm",
      "sexism",
      "sexual"
    ],
    "async": true,
    "metadata": { "postId": "p_981", "threadId": 42 }
  }'

{
  "id": 1043,
  "type": "text_moderation",
  "status": "queued",
  "metadata": { "postId": "p_981", "threadId": 42 }
}

Three design rules follow from the docs:

• Choose your events. Subscribe the endpoint to moderation.completed to receive every result, partial ones included. moderation.flagged alone never tells you about a clean result, or a partial one that raised nothing. A flagged check matches both events, so if you take both, key your handling on the request's id (Webhooks).
• Keep inputs reachable. An image is fetched when its check runs, so a signed URL must still be valid then (Image moderation, Async requests).
• Put your own queue in front of the API. A burst lands in your queue, and workers send checks at a steady pace. If the API answers 429 too_many_requests, wait the seconds in Retry-After and send again (Errors).

From scores to decisions

A raised flag comes with a confidence from 0 to 1. The API raises a flag only when its confidence reaches the flag's threshold, 0.5 by default, and findings below it never come back (Flags). So every line you draw sits at 0.5 or above: act above your own line. Confidence is not a probability you can compare across flags, so draw a line per flag, and per surface where the stakes differ.

A practical start from the docs: send results with confidence between 0.5 and 0.7 to human review. Then label a sample of your own content and move each line to where its mistakes cost least.

The decision table below turns each kind of result into a decision, with illustrative lines. Failures have their own table in When checks fail.

The same table as code. The lines are illustrative, and the structure follows the verdict example in Responses:

# Illustrative lines for comments: tune them on a labelled sample of your own content.
# A flag is raised only at or above its threshold (0.5 by default), so no line can sit lower.
BLOCK_AT = {"hate": 0.7, "racism": 0.7, "sexism": 0.7, "sexual": 0.8}
REVIEW_ONLY = {"insult", "self_harm"}
FLAGS = sorted(BLOCK_AT.keys() | REVIEW_ONLY)  # send these as "flags" in each request


def decide(result: dict) -> str:
    """One moderation result -> "block", "review", "retry" or "publish"."""
    raised = {f["flag"]: f["confidence"] for f in result["flags"]}
    if "self_harm" in raised:
        return "review"  # someone may need help: a person decides
    if any(raised.get(flag, 0) >= line for flag, line in BLOCK_AT.items()):
        return "block"
    if raised:
        return "review"  # raised, but below your block line
    if result["unchecked"]:
        return "retry"  # partial result: never treat it as clean
    return "publish"

// Illustrative lines for comments: tune them on a labelled sample of your own content.
// A flag is raised only at or above its threshold (0.5 by default), so no line can sit lower.
const BLOCK_AT = { hate: 0.7, racism: 0.7, sexism: 0.7, sexual: 0.8 };
const REVIEW_ONLY = ["insult", "self_harm"];
const FLAGS = [...Object.keys(BLOCK_AT), ...REVIEW_ONLY].sort(); // send these as "flags"

// One moderation result -> "block", "review", "retry" or "publish".
function decide(result) {
  const raised = Object.fromEntries(result.flags.map((f) => [f.flag, f.confidence]));
  if ("self_harm" in raised) return "review"; // someone may need help: a person decides
  if (Object.entries(BLOCK_AT).some(([flag, line]) => (raised[flag] ?? 0) >= line)) return "block";
  if (Object.keys(raised).length > 0) return "review"; // raised, but below your block line
  if (result.unchecked.length > 0) return "retry"; // partial result: never treat it as clean
  return "publish";
}

Keep the lines in configuration rather than in code, so a policy change needs no deploy. Check them against GET /api/v1/flags when your app starts: a flag your account does not have is refused with 400 invalid_flags before anything is charged.

A model judges the content it is given. A threat spread over several messages, or a quote shared to report abuse, needs a person who can read the thread. AI for hate speech and harassment lists what a model cannot settle alone.

The review queue

Everything the rules hold ends up in front of a person. Human-in-the-loop moderation covers running a queue in depth. For the design, settle these:

• What enters it. Results in your review band, every raised self_harm, partial results, declined content and failed checks you chose not to publish, user reports, and appeals against earlier decisions.
• What a reviewer sees. The content as the user saw it. Detectivision does not keep images, so keep your own copy or a link that stays valid (Image moderation). Add the flags and their confidence, the rule that held the item, the thread or listing around it (found through the ids in metadata), and the user's earlier flags and actions.
• The order. Sort by urgency, not by arrival: possible self-harm first, then by flag and confidence, then by age.
• Where reviewers work. The dashboard's Logs lists your requests; open one and its panel shows your own actions, up to 20, such as Hide or Ban. Or build your own tool and record decisions through POST /api/v1/moderation/{id}/actions. Either way the decision becomes a moderation.action event (Metadata and actions).
• The record. For every decision keep the request id, the flags and their confidence, the rule and its version, the decision, who made it, the note and the time. GET /api/v1/moderation/{id} returns a request's state, pending until the first action and actioned after it, and the actions taken. The record your appeals depend on belongs in your own database.

Acting on decisions

Detectivision never carries out an action itself (Metadata and actions). It cannot hide a post or ban a user on your platform: it tells your system which action was chosen, for which request, with your metadata, and your system does the rest.

1. Define your actions in the dashboard: a key your system receives, such as hide or ban_user, a label and a colour.
2. Subscribe an endpoint to moderation.action.
3. On each message, find the content through data.item.metadata, carry out data.action.key, and store data.note and data.by.
4. Make every step safe to repeat: set a state rather than toggle one.

data.by.type is user for the dashboard or api for a call with an API key, and data.by.name names the person or the key. If your own review tool calls the API, record which person pressed the button yourself.

Your rules' automatic decisions are made in your system, so record them there. You can also record them as actions through the API, so that Logs shows one history. The API takes up to 60 actions a minute, so at higher volumes keep automatic decisions in your own log only. Taking actions and sending their webhooks is free.

Here is a receiver and a worker built from what Webhooks asks: verify first, answer within 10 seconds, skip a webhook-id you have seen, and apply events in the order of their timestamp. verify() is the function on that page; seen, jobs and posts are stores in your own system. Checks run in the dashboard's playground send moderation.completed too, so the worker skips anything that is not one of your posts.

import os

SECRET = os.environ["DETECTIVISION_WEBHOOK_SECRET"]  # whsec_...


def receive(headers: dict, raw_body: bytes) -> int:
    """Your webhook endpoint: answer within 10 seconds, do the work later."""
    try:
        event = verify(SECRET, headers, raw_body)  # verify() from /docs/webhooks
    except (KeyError, ValueError):
        return 401  # not signed by Detectivision, or older than 5 minutes
    msg_id = headers["webhook-id"]
    if seen.has(msg_id):
        return 200  # a retry of a message you already have
    jobs.put(event)  # a durable queue in your system
    seen.add(msg_id)  # only once queued: if put() fails, the retry is not skipped
    return 200


def handle(event: dict) -> None:
    """A worker. posts.set_state() ignores an event no newer than the post's last change."""
    data, at = event["data"], event["timestamp"]
    item = data.get("item", data)  # moderation.action carries the request in "item"
    post_id = (item.get("metadata") or {}).get("postId")
    if post_id is None:
        return  # not one of your posts, for example a playground check
    if event["type"] == "moderation.completed":
        posts.set_state(post_id, decide(data), at, request=data["id"])
    elif event["type"] == "moderation.failed":
        posts.set_state(post_id, "review", at, error=data["error"])
    elif event["type"] == "moderation.action":
        posts.set_state(post_id, data["action"]["key"], at, by=data["by"])

// Your webhook endpoint: answer within 10 seconds, do the work later.
async function receive(headers, rawBody) {
  let event;
  try {
    event = verify(process.env.DETECTIVISION_WEBHOOK_SECRET, headers, rawBody); // verify() from /docs/webhooks
  } catch {
    return 401; // not signed by Detectivision, or older than 5 minutes
  }
  const msgId = headers["webhook-id"];
  if (await seen.has(msgId)) return 200; // a retry of a message you already have
  await jobs.put(event); // a durable queue in your system
  await seen.add(msgId); // only once queued: if put() fails, the retry is not skipped
  return 200;
}

// A worker. posts.setState() ignores an event no newer than the post's last change.
async function handle(event) {
  const { data, timestamp: at } = event;
  const item = data.item ?? data; // moderation.action carries the request in "item"
  const postId = item.metadata?.postId;
  if (postId === undefined) return; // not one of your posts, for example a playground check
  if (event.type === "moderation.completed") {
    await posts.setState(postId, decide(data), at, { request: data.id });
  } else if (event.type === "moderation.failed") {
    await posts.setState(postId, "review", at, { error: data.error });
  } else if (event.type === "moderation.action") {
    await posts.setState(postId, data.action.key, at, { by: data.by });
  }
}

Order matters in two places. The receiver marks a webhook-id as seen only after the event is queued, so a delivery that fails half-way is queued on its retry, and a copy that reaches the worker twice changes nothing. The worker's timestamp check stops a late copy of a check result from overwriting a moderator's decision on the same post.

When checks fail

Decide per content type, before launch, what happens to content that cannot be checked yet. Keep it pending for images and messages: the system fails closed. Or publish it and check it later for comments: the system fails open. Then handle each case:

Sources for the table: Errors, Responses, Credits and billing and Async requests.

There is no idempotency key (API reference). A resent request is a new request, so if the answer to the first was lost, for example the 202 of an async request, both may be checked and charged. Put your own content id in metadata, store the request id from each answer next to your content, and make applying a result safe to repeat, as set_state() above does. Webhook deliveries are de-duplicated the other way: by webhook-id, which stays the same on every retry.

When your endpoint fails. Answer every delivery with a 2xx within 10 seconds. Anything else is retried after 1 minute, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours and 10 hours: 8 attempts over about 28 hours. An endpoint that fails every attempt for 5 days is switched off. Its failed deliveries stay in its list of the last 30 days and can be sent again, but events that happen while it is off are not sent to it: read those requests back with GET /api/v1/moderation/{id}, using the ids you stored (Webhooks).

Watching the service. GET /api/v1/status needs no API key. Use it for dashboards and alerts, not before every request: a 503 or a partial result already tells you what you need (Service status).

Appeals and the law

Two rules shape how decisions are made and recorded.

Who is who.

• Turkey's Presidency of Cyber Security (Siber Güvenlik Başkanlığı) issued the regulation on separate services for children and on age verification by social network providers. It was published in the Resmî Gazete on 10 October 2026 (issue 33396) and applies from 1 November 2026 (Art. 13).
• A social network provider (sosyal ağ sağlayıcı), in that regulation, is a person or company that lets users create, view or share content such as text, images, sound and location for social interaction (Art. 4/e). The regulation covers providers reachable from Turkey, whether based in Turkey or abroad (Art. 2). It treats anyone under 18 as a child (Art. 4/ç) and sets a separate, restricted service for users aged 15 to 17 (Art. 8).
• The Digital Services Act (DSA, Regulation (EU) 2022/2065) applies to intermediary services offered to users in the EU, wherever the provider is established (Art. 2(1)). A hosting service stores information its users provide (Art. 3(g)(iii)); an online platform is a hosting service that also makes that information public at the user's request (Art. 3(i)). Art. 17 applies to every hosting service. Art. 20 applies to online platforms, but not to those that qualify as micro or small enterprises under Commission Recommendation 2003/361/EC, unless they are designated very large online platforms (Art. 19).

The filtering settings that Art. 8/4 of the Turkish regulation requires for accounts of users aged 15 to 17 are part of that separate service, so if your checks do that filtering, an appeal about it goes to a person. The design above can help you meet both rules: every decision record says what found the content, who decided and which rule applied; appeals and complaints go to people; and the queue can report counts and times. The Turkish rules in full: Turkey's under-15 social media rules.

For system-design interviews: the same design as a checklist

Asked to design a content moderation system in an interview, ML-focused or not, you can walk through the same design in this order:

1. Requirements. Content types and surfaces, languages, volume, and how long each kind of content may wait. Sizing, illustrative: 500,000 comments a day is about 6 a second on average, and peaks run higher.
2. Policy. The categories you enforce and what happens for each: hide, warn, ban or escalate.
3. Where checks run. Before or after publishing, per surface.
4. Models. A classifier per category, your own or an API's, that returns a confidence. Your own models add labelled data, offline evaluation per category, a trial run alongside the live system before launch, and monitoring for drift.
5. Paths. At once where a person waits; a queue and workers for the rest, with back-pressure and retries.
6. Decision rules. A line per category at or above the model's threshold, a review band, and overrides such as "possible self-harm always goes to a person".
7. Review queue. What enters, what reviewers see, ordering by urgency, reviewer tools and reviewer wellbeing.
8. Actions. Events to the services that enforce them, safe to repeat and applied in order.
9. Failures. Partial results, outages, failing open or closed per surface, and de-duplication.
10. Record and appeals. Who decided and how, appeals decided by people, and yearly figures where the law asks for them.
11. Metrics. Precision and recall per category on a labelled sample, queue size, time to decision, and the share of decisions reversed on appeal.
12. Feedback. Reviewer decisions become labelled data and move the lines.

What this does not do

• Video, audio and live streams are coming soon. Today the API checks text and images.
• Languages. Text checks are reliable in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi. Text in other languages is checked, but the results are not reliable (Text moderation).
• Positions. Flags judge the whole text or the whole image. Current plans return no word positions, no boxes and no masked or blurred copy, so a decision applies to the whole item.
• Your platform. Detectivision does not hide posts, ban users, write your policy, staff a review team, decide appeals or write statements of reasons. Your system and your people do.
• Images. They are not kept after the check, so your review tool needs its own copy.
• Age checks. Detectivision does not verify ages.
• Legal findings. A flag says what a check found. Whether content is illegal is for you to decide.
• Figures. This guide gives no latency, uptime or accuracy numbers. Measure speed and accuracy on your own content.

Read the docs

Start the free trial