VISIONContent moderation glossary
Content moderation is the work of checking what users post against a platform's rules and the law, and acting on what breaks them. This glossary defines the terms that come up when you build or buy it, in plain words. Where a term needs more than a paragraph, it links to a page of its own.
The entries run from A to Z and cover five areas: how moderation is organised, what gets flagged, how machines decide, rules and law, and the terms of a moderation API. Where a term maps to a field in Detectivision's API, the entry names the field; Responses shows those fields together in one answer. Entries about laws are summaries, not legal advice.
By category, see NSFW, hate speech, toxicity, violence, self-harm and Turkish offensive language detection, and whether AI can moderate hate speech. By API field, see flags, modes, async requests and actions.
What Detectivision does not do. It checks text and images today; video, audio and live streams are coming soon. Its flags judge the whole text or image, and text results are reliable in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi only. It does not verify or estimate age, match hashes or detect CSAM, and it does not act on your platform: your system carries out every decision.
Action
In Detectivision, an action is a decision your team defines, such as Approve, Hide or Ban, and takes on a checked item. You can define up to 20 in the dashboard and take them in Logs or through the API (POST /api/v1/moderation/{id}/actions). Each one sends a moderation.action webhook to the endpoints that subscribe to it, with the action's key, the note, who took it, and the item with your metadata. Your system then carries it out: Detectivision never removes a post or bans a user on your platform itself. Taking actions and sending their webhooks is free.
Age assurance
Age assurance is the umbrella term for finding out how old a user is. It covers age verification, which confirms an age against evidence such as an identity document, and age estimation, which infers an age or an age range, for example from a face. Where the UK's Online Safety Act 2023 requires a service to use either, it must be highly effective at telling whether a user is a child (section 12). It is a different job from content moderation, which checks what users post. Detectivision does not offer age assurance.
Full entryAge estimation
Age estimation infers a user's age or age range instead of confirming it, for example from a face in a selfie or from how an account behaves. The UK's Online Safety Act 2023 defines it as "any measure designed to estimate the age or age-range of users" (section 230). An estimate has a margin of error, so it is most useful when a user is clearly above or below the age that matters. A user simply stating their age is not age estimation under that Act. Detectivision does not estimate age.
Full entryAge verification
Age verification confirms a user's age against evidence: an identity document, a digital ID or a trusted record. The UK's Online Safety Act 2023 defines it as "any measure designed to verify the exact age of users", and says self-declaring an age is neither verification nor estimation (section 230). Turkey's regulation for social network providers (services where users create and share content) reachable from Turkey requires, from 1 November 2026, a check at sign-up with an age token from e-Devlet, the state's online services gateway; the token says only whether the user is 15 or over and, if so, 18 or over (Articles 6 and 7). Detectivision does not verify age.
Full entryAppeal
An appeal asks a platform to reverse a moderation decision, such as a removed post or a suspended account. Under the EU Digital Services Act, online platforms other than micro and small enterprises must offer a free, electronic complaint system for at least six months after a decision, reverse decisions that prove unfounded, and not decide complaints solely by automated means (Articles 19 and 20). A Turkish regulation for social network providers reachable from Turkey says appeals about age checks, account suspensions and closures, and measures on accounts of 15-to-17-year-olds need meaningful human review, not fully automated handling (Article 10/3, from 1 November 2026). Record every appeal and its outcome.
Async check
An async check is a moderation request answered later instead of at once. In Detectivision you add "async": true to the request; the API answers 202 with the request's id, runs the check when there is capacity, usually within seconds, and sends the result to your webhook as moderation.completed, moderation.flagged or moderation.failed. You can also poll GET /api/v1/moderation/{id}. An async check costs the price table; an answer given at once costs twice that. Use it for anything that can wait a moment, such as post-moderation or a backfill.
Automated moderation
Automated moderation lets software make some or all moderation decisions: blocking, holding or allowing content by the flags and scores that models and rules return. It keeps up with volume no team can read and applies the same lines every time, but it makes mistakes in both directions and only knows the categories it was built for. Some laws require a person at certain steps: under the EU Digital Services Act, online platforms other than micro and small enterprises may not decide complaints solely by automated means (Article 20(6)).
Full entryBlocklist
A blocklist is a list of words or phrases that a filter blocks wherever they appear. It is fast, free to run and easy to explain, which makes it a common first filter. Its limits come from matching letters instead of reading meaning: it blocks harmless words that contain a listed one (the Scunthorpe problem), misses misspellings and new slang, cannot tell a quote or a joke from an attack, and needs upkeep in every language. An allowlist of words never to block eases the first problem, not the others.
Classifier
A classifier is a model trained on labelled examples to sort new content into categories, such as hate or not hate, with a score for each. Moderation APIs use classifiers alongside other methods, such as large language models asked to judge content against a written rule, and plain lists. Unlike a word list or hash matching, a classifier can judge content it has never seen, but it is only as good as its training data and its categories. Test one on a sample of your own content before you trust its scores.
Confidence score
A confidence score is a number from 0 to 1 that says how strongly a check found a category. In Detectivision's results it is the confidence field of each raised flag: the strongest finding for that flag. It is a score, not a probability you can compare across flags, so 0.8 for hate and 0.8 for insult do not mean the same thing. Act first on which flags were raised, then tune per flag, for example by sending results between 0.5 and 0.7 to a person.
Content moderation
Content moderation is the work of checking what users post against a platform's rules and the law, and acting on what breaks them: removing or hiding it, limiting who sees it, or warning or suspending the user. It covers text, images, video, audio and live streams, and usually combines software with people. It is not editing, because the platform does not write the content, and it is not age assurance, which checks how old a user is rather than what they post.
Content moderation API
A content moderation API is a web service that checks content you send it, such as a comment or a photo, and answers with the categories it found and how confident it is. Your app then decides what to do: publish, hold for review or block. With Detectivision you POST text, or an image's URL, to /api/v1/moderation with your key in the Api-Key header, and get flags with their confidence back. Text and image checks work today; video, audio and live streams are coming soon.
Full entryCSAM
CSAM stands for child sexual abuse material: images or videos that show the sexual abuse or exploitation of children. It is illegal to make, possess or share, and depending on the law that applies, a platform that finds it may have to report it to the authorities. Known material is found by hash matching against lists of already identified files; new material needs specialised tools and trained staff. Detectivision does not detect CSAM, judge the age of people in images, match hashes or report to authorities.
Distributed moderation
Distributed moderation lets a platform's users make or shape decisions by voting, reporting or reviewing each other's posts: a post with enough down-votes or reports is hidden, or trusted members work through a queue. It grows with the community and reflects its norms, but it is slow on quiet content, can be gamed by organised groups, and exposes volunteers to harmful material, so it needs other types behind it. Detectivision has no voting feature; its checks can run alongside one.
Full entryFalse negative
A false negative is harmful content that a moderation system lets through, such as hate speech written with deliberate misspellings. Raising a threshold cuts false positives and adds false negatives; lowering it does the reverse. In Detectivision, findings under a flag's 0.5 threshold never come back, so you cannot recover them by lowering a line; you can only act above your own, higher line. User reports and proactive checks catch some of what a filter misses.
Full entryFalse positive
A false positive is content a moderation system flags although it breaks no rule: a harmless word that contains a swear word, a slur quoted in a news report, a word a community has reclaimed. Every automated filter makes some. Keep them cheap: hold uncertain items for a person instead of blocking them, let users appeal, and measure the rate on a labelled sample of your own content. There is no universal good rate; it depends on the harm and on what a flag triggers.
Full entryFilter evasion
Filter evasion is writing harmful content so that a filter misses it: swapping letters for digits or symbols (leetspeak), putting spaces or dots inside a word, misspelling on purpose, using lookalike letters from other alphabets, or putting the words inside an image. Word lists are the easiest to evade. Models that read whole sentences can catch more variants, though none catch all, so test the spellings your users actually write before you rely on a filter.
Flag
A flag is a category a moderation check looks for, such as hate, insult or sexual content, and the way a result says it found one. In Detectivision a result lists the raised flags in its flags field, each with its key and confidence; every flag looked for in checked; and any that could not be looked for in unchecked. Flags depend on the plan: GET /api/v1/flags lists yours, and the public status endpoint shows those every account has. Build on flag keys, which stay stable, not on model labels, which can change.
Hash matching
Hash matching turns a file into a short fingerprint, a hash, and compares it with lists of hashes of known illegal material, such as child sexual abuse material (CSAM). Cryptographic hashes match only identical files; perceptual hashes also match copies that were resized, re-encoded or slightly edited. A match points to a copy, or a near copy, of a known item. It says nothing about new content, which needs classifiers, so platforms that must find both use both. Detectivision does not offer hash matching.
Full entryHate speech
Hate speech attacks or demeans people for who they are: their race, ethnicity, religion, gender, sexual orientation, disability or similar traits. It differs from an insult, which targets one person for anything. Laws define it differently from country to country, and a platform's own rules may cover more than the law does. Detectivision's text check has hate, racism and sexism flags, and its image check a hate flag for hateful imagery. A flag is a moderation signal, not a legal finding.
Human-in-the-loop moderation
Human-in-the-loop moderation means software makes the clear calls and sends the uncertain ones to a person, whose decision is final and recorded. Items reach a reviewer when a flag's confidence falls in a middle band, when a user reports them, or when someone appeals. Detectivision supports this with Logs, your own actions and the moderation.action webhook; the reviewers are your people. Some laws require a person at certain steps, such as deciding appeals.
Full entryHybrid moderation
Hybrid moderation combines software and people: automated checks screen everything and act alone on clear cases, while moderators decide uncertain or high-impact ones and handle reports and appeals. With Detectivision, your code applies two lines per flag, for example block above 0.9 and send 0.5 to 0.9 to review (illustrative numbers), and your moderators act on each item in that middle band, in Logs or in your own tool. The split keeps volume off people without leaving the hard calls to software.
Full entryInsult
An insult attacks or belittles a person: name-calling, mocking someone's looks or intelligence, telling someone they are worthless. It needs no swear word, as in "you are an idiot", and a swear word is not always an insult, as when someone curses a bad day. Detectivision's text check has an insult flag, separate from profanity and from hate, which attacks people for who they are. In Turkey, insult (hakaret) is also an offence under Article 125 of the Turkish Penal Code (Law 5237); a flag is a moderation signal, not a court's finding.
Metadata
Metadata is your own data attached to a moderation request, so a result can be matched to what it is about. In Detectivision it is any JSON object up to 16 KB, sent as metadata. It comes back unchanged in the answer, in GET /api/v1/moderation/{id}, in Logs and in every webhook about that request. Put in it what your system needs to act, such as user, post or comment ids. Detectivision does not read it.
Mode
Mode is the request field that sets how much a Detectivision check returns. Use detect: the answer lists the flags raised and their confidence, which is what you decide on. The API reference also describes capture, the default, and censor, which add positions and censored copies for flags that give positions. Detectivision's flags today judge the whole text or image, so no word positions, boxes or censored copies come back in any mode.
Moderation queue
A moderation queue is the list of items waiting for a person to decide. Items enter when software is unsure, such as a confidence in a middle band, when users report them, or when someone appeals, and leave when a moderator takes an action. In Detectivision a moderator opens a request in the dashboard's Logs, or your own tool acts through the API; the request's state is pending until the first action and actioned after it, and each action reaches your system as a moderation.action webhook once an endpoint subscribes to that event.
Full entryNotice and action
Notice and action is the process by which anyone can report illegal content to a service, which must then decide what to do. Under the EU Digital Services Act, every hosting service, online platforms included, must offer an easy electronic way to send such notices, confirm receipt when the sender left contact details, decide in a timely, diligent, non-arbitrary and objective manner, tell the sender the decision and how to seek redress, and say if automated means were used (Article 16).
NSFW
NSFW, short for not safe for work, labels content you would not want on screen in an office or in front of children: mostly nudity and sexual content, and on some platforms gore. It is a broad label rather than a precise category, so APIs split it into narrower ones. Detectivision's image and text checks report it with a sexual flag that judges the whole image or text; no blurred copy comes back.
Partial result
A partial result is an answer in which some flags could not be checked at that moment. In Detectivision its status is partial, the flags that could not be checked are listed in unchecked, the rest are reported as usual, and the check is free: usage.charged is 0. Treat an unchecked flag as unknown, not clean: retry, or hold the item if that flag matters to you. The public status endpoint shows when flags are back.
Post-moderation
Post-moderation publishes content first and checks it right after, removing what breaks the rules. Posters never wait, which suits busy comment sections and chats, but harmful content can be seen until the check ends and your system acts. With Detectivision, send each item with "async": true and its id in metadata, and hide it when a moderation.flagged webhook arrives with a flag above your line. An async check costs half the price of an answer given at once.
Full entryPre-moderation
Pre-moderation checks content before anyone else can see it: nothing is published until it passes. It keeps harmful content off the platform at the price of a wait for the poster and, where people review, a queue that must keep up. With an API, your server either waits for an answer given at once, which costs twice the async price, or keeps the item pending and publishes it when an async result comes back clean. It suits profile photos, listings and spaces for young users.
Full entryPrecision and recall
Precision and recall measure a filter on content whose right answer you know. Precision is the share of flagged items that really broke a rule; recall is the share of rule-breaking items that were flagged. A strict line raises precision and lowers recall; a loose one does the reverse. Measure both per flag on a labelled sample of your own content, because figures measured on someone else's data say little about yours. Detectivision publishes no accuracy figures.
Proactive moderation
Proactive moderation looks for harmful content without waiting for reports: checking posts as they arrive, re-checking an account's older posts after one bad post, or checking the archive again after a rule changes. It finds harm nobody reports, at the cost of checking much that is fine. With Detectivision, send stored items as async checks with their ids in metadata, read the results by webhook, and stay under the API's rate limits.
Full entryProfanity
Profanity is swearing: swear words, obscenities and vulgar slang, whoever they are aimed at. It differs from an insult, which attacks a person and may contain no swear word at all. Platforms draw the line differently: a children's game may block all of it, an adult forum may allow it. Detectivision's text check has a profanity flag that judges the whole text and returns no word positions or masked copy; GET /api/v1/flags shows whether your plan includes it.
Reactive moderation
Reactive moderation acts only on what users report: a report button feeds a queue and a moderator decides. It is cheap and uses context the reporter has, but harm stays up until someone reports it, and quiet places such as small groups may never be reported. Under the EU Digital Services Act, every hosting service must let anyone report illegal content electronically (Article 16), so a platform that serves users in the EU needs a report path whichever other types it runs.
Full entryScam image
A scam image is a picture made to trick people, such as a fake payment confirmation or a fake giveaway. Marketplaces, dating apps and communities meet them in listings, profiles and messages. Detectivision's image check has a scam flag that judges the whole image. It classifies scam-style images; it cannot tell whether a payment really happened, so never treat a clean result as proof of payment.
Scunthorpe problem
The Scunthorpe problem is a filter blocking harmless text because a banned string hides inside it. It is named after the English town of Scunthorpe: in a case reported in April 1996, AOL's sign-up filter would not accept the town's name, which contains a swear word. It comes from matching letters instead of reading words, and it happens in every language: everyday words in Spanish, French, German and Russian contain strings from word lists, and lowercasing text without a language's own rules can create more.
Full entrySelf-harm content
Self-harm content talks about, shows or encourages suicide or hurting oneself. Plan a caring response, not only removal: showing support resources, alerting a trained team, or limiting how the content spreads. Detectivision's text check has a self_harm flag so someone can step in. A flag is a signal, not a diagnosis, and the right response depends on your policy and on the support services where your users live.
Statement of reasons
A statement of reasons explains to a user why a platform restricted their content or account. Under the EU Digital Services Act, hosting services must give one for removals, demotions, limits on visibility, and payment, service or account suspensions, stating the facts relied on, whether automated means were used, the legal or contractual ground, and the ways to seek redress (Article 17). Online platforms other than micro and small enterprises also send them to a public database run by the European Commission (Articles 19 and 24(5)).
Threshold
A threshold is the score at which a check raises a flag. In Detectivision every flag has one, 0.5 by default, and findings below it never come back, so no line of yours can sit below it. What you set is your own line above it: act on a confidence over 0.8, for example, and send 0.5 to 0.8 to a person (illustrative numbers). Set lines per flag, because confidence is not comparable across flags.
Toxicity
Toxicity is a broad label for rude, hostile or abusive language. Some services answer with a single toxicity score, which is quick to use but hides why something scored high. Detectivision returns separate flags instead, such as insult, hate, racism, sexism, sexual and self_harm, so each can get its own rule. If you need one verdict, use the result's flagged field, which is true when any flag was raised.
Trust and safety
Trust and safety (T&S) is the work of keeping a platform's users safe from harm and abuse, and the team that does it: writing the rules, moderating content, fighting fraud and spam, protecting children, handling reports and appeals, and answering regulators. Content moderation is one part of it. A moderation API covers one task inside that part, checking content against categories; the policies, people and decisions stay with the platform.
User-generated content (UGC)
User-generated content is anything a platform's users create and post, rather than the platform itself: comments, reviews, messages, profile photos, usernames and bios, listings, videos and live streams. It is what content moderation checks. Most of the legal duties in this glossary, from notice and action to age checks, attach to services that host it, and app stores set their own rules for apps that show it.
Webhook
A webhook is an HTTPS request that a service sends to a URL of yours when something happens, so your system does not have to keep asking. Detectivision sends events such as moderation.completed, moderation.flagged, moderation.failed and moderation.action, signed under the Standard Webhooks specification. Verify the signature, answer with a 2xx status within 10 seconds, and use the webhook-id header to process each message once: a failed delivery is tried again, up to 8 attempts over about 28 hours.
Age verification checks how old a user is; moderation checks what they post. The terms, the methods UAE, UK, EU and Turkish rules accept, and who does what.
A content moderation API checks the text or images you send and says what it found and how sure it is. How requests, results, prices and decisions work.
A false positive is harmless content a moderation system flags. Why they happen, how to measure them on your own data, and how thresholds and review help.
Hash matching compares an upload's fingerprint with lists of known illegal images such as CSAM. How it works, who offers it, and how classifiers differ.
Human-in-the-loop moderation sends what software is unsure about to people. How a review queue works, what to record, and where the law requires a person.
The Scunthorpe problem is a filter blocking harmless text because a banned word hides inside it. Why it happens in every language, and the fixes.
Six ways to moderate user content, from checking before anything is posted to acting on reports: how each works, what it costs in delay, and when it fits.