Firebase content moderation: Firestore messages and Storage uploads

To moderate content in Firebase, trigger a Cloud Function when a message is written to Firestore or a file is uploaded to Cloud Storage, send it to a moderation API, and write the verdict back. Your app shows only content marked approved, and security rules enforce it. The code below does this for text and images with Detectivision; the same pattern works with any HTTP moderation API.

How it fits together

1. Your app adds a document to messages with status: "pending". The security rules accept no other status.
2. onDocumentCreated starts a function. It sends the text to POST /api/v1/moderation in detect mode and waits for the answer.
3. The function writes approved, review or rejected back to the message, with the flags it got.
4. Readers ask for where("status", "==", "approved"). Authors also see their own pending messages.
5. For images, the app uploads to uploads/{uid}/{uploadId}. onObjectFinalized makes a signed URL that works for 15 minutes, sends it for a check, and keeps the verdict in the Firestore document users/{uid}/uploads/{uploadId}, where the Storage rules read it. A rejected file is deleted.

Content that can wait takes a second path: the function queues the check with async: true, which costs half, and an HTTPS function takes the result by webhook. See Backlogs and bursts.

What changed since the 2017 recipe

Firebase's own recipe is the blog post "Content Moderation with Cloud Functions for Firebase" from 2 June 2017. Its two samples are still in the firebase/functions-samples repository, both last changed in October 2026.

Both approaches work. A word list is free and runs inside your function, and it knows only the words on it; the 2017 post notes that its list held about 400. An API check needs a network call and credits, and it judges the whole message, reliably in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi. This guide holds content back instead of rewriting it, because Detectivision's flags judge the whole text or image: they return no word positions and no blurred copy.

Before you start

• The Blaze plan. Firebase says that "to deploy functions, your project must be on the Blaze pricing plan" (Get started). The emulators run on any plan.
• Node.js 22 with firebase-functions 7.4.0 and firebase-admin 14.5.0, the latest versions on npm on 16 October 2026. Cloud Functions supports Node.js 20 and 22 (Get started), but firebase-admin 14 dropped Node.js 20 (release notes).
• A Detectivision API key with Write permission, from Dashboard → API Keys (Quickstart). It goes into Secret Manager, never into your app: the API refuses calls from browsers, and a key shipped in an app can be read by anyone (Authentication).
• Permission to sign URLs. Inside Cloud Functions, getSignedUrl has no private key to sign with, so it signs through IAM and needs the iam.serviceAccounts.signBlob permission. Enable the IAM Service Account Credentials API and give the functions' service account the Service Account Token Creator role (Google Cloud). Unless you set another one, 2nd gen functions run as the Compute Engine default service account, PROJECT_NUMBER-compute@developer.gserviceaccount.com (Cloud Run).

firebase functions:secrets:set DETECTIVISION_API_KEY

gcloud services enable iamcredentials.googleapis.com
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
  --role="roles/iam.serviceAccountTokenCreator"

Moderate new Firestore messages

Start with one shared file. It holds the secret, the API call and the rule that turns an answer into a verdict.

import { initializeApp } from "firebase-admin/app";
import { getFirestore } from "firebase-admin/firestore";
import { defineSecret } from "firebase-functions/params";
import * as logger from "firebase-functions/logger";

initializeApp();
export const db = getFirestore();

export const DETECTIVISION_API_KEY = defineSecret("DETECTIVISION_API_KEY");
const API_URL = "https://api.detectivision.ai/api/v1/moderation";

// Your own line. The API raises a flag at 0.5 or above. From REJECT_AT up,
// an item is rejected on its own; below it, a person decides.
const REJECT_AT = 0.7;

type Flag = { flag: string; confidence: number };
type Answer = {
  id: number;
  status: "completed" | "partial";
  flagged: boolean;
  flags: Flag[];
  error?: string; // error answers carry error and message instead
};
export type Outcome = {
  verdict: "approved" | "review" | "rejected";
  flags: Flag[];
  requestId: number | null;
  error: string | null;
};

export function outcomeOf(answer: Answer): Outcome {
  if (answer.status === "partial") {
    // Some flags could not be checked, so "no flags" does not mean clean.
    return { verdict: "review", flags: answer.flags, requestId: answer.id, error: "partial" };
  }
  const verdict = !answer.flagged
    ? "approved"
    : answer.flags.some((f) => f.confidence >= REJECT_AT)
      ? "rejected"
      : "review";
  return { verdict, flags: answer.flags, requestId: answer.id, error: null };
}

// One check, answered at once. Throws when it is worth trying again.
export async function check(request: Record<string, unknown>): Promise<Outcome> {
  const res = await fetch(API_URL, {
    method: "POST",
    headers: {
      "Api-Key": DETECTIVISION_API_KEY.value(),
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ mode: "detect", ...request }),
    signal: AbortSignal.timeout(60_000),
  });
  const answer = (await res.json()) as Answer;
  if (res.status === 429 || res.status >= 500) {
    // Busy or briefly unavailable: the trigger runs again (retry: true).
    throw new Error(`Detectivision ${res.status}: ${answer.error}`);
  }
  if (!res.ok) {
    // The input cannot be checked, for example an empty text: a person decides.
    logger.warn("Detectivision refused the check", { answer });
    return { verdict: "review", flags: [], requestId: null, error: answer.error ?? null };
  }
  return outcomeOf(answer);
}

Three decisions sit in that file:

• REJECT_AT is your own line. The API raises a flag only when its confidence reaches 0.5, and findings below that never come back (Flags). So act above your own line: here a flag at 0.7 or more rejects on its own, and anything between 0.5 and 0.7 goes to a person, the review band the docs give as an example.
• A partial result goes to review. It means some flags could not be checked right now. It is free, and "no flags" then does not mean clean (Responses).
• Only busy and unavailable answers are retried. A 429 or 5xx throws, and with retry: true Cloud Functions runs the trigger again; for 2nd gen functions it keeps trying for up to 24 hours (Firebase). Any other error is a 4xx, which needs a changed request: an input that cannot be used, a wrong key, or no credits left. A retry would not help, so the message goes to review with the error code. Errors cost nothing (Errors).

Now the trigger for new messages:

import { FieldValue } from "firebase-admin/firestore";
import { onDocumentCreated } from "firebase-functions/v2/firestore";
import { check, db, DETECTIVISION_API_KEY } from "./moderation";

export const moderateMessage = onDocumentCreated(
  { document: "messages/{messageId}", secrets: [DETECTIVISION_API_KEY], retry: true, timeoutSeconds: 120 },
  async (event) => {
    const snapshot = event.data;
    if (!snapshot || snapshot.get("status") !== "pending") return;

    const outcome = await check({
      type: "text_moderation",
      input: snapshot.get("text"),
      metadata: { path: snapshot.ref.path },
    });

    // An event can arrive more than once: only a message still pending gets a verdict.
    await db.runTransaction(async (tx) => {
      const current = await tx.get(snapshot.ref);
      if (current.get("status") !== "pending") return;
      tx.update(snapshot.ref, {
        status: outcome.verdict,
        moderation: { ...outcome, checkedAt: FieldValue.serverTimestamp() },
      });
    });
  },
);

• No loop. onDocumentCreated fires for new documents only, so the function's own update does not start it again. The 2017 sample listened to every write and needed a sanitized field to stop itself.
• No double verdicts. Firebase delivers each event at least once, and "a single event may result in multiple function invocations" (Firestore triggers). The transaction writes only while the message is still pending.
• Room to wait. The code stops waiting for the API after 60 seconds, so timeoutSeconds: 120 gives the function time to record the outcome. Event-driven functions may run up to 540 seconds (Manage functions).
• A Firestore profanity filter, and more. profanity is one of the eight text flags this check runs, next to insult, hate and the rest. To check only some, add flags: ["profanity", "insult"] to the request (Flags).
• Your ids travel with the check. metadata comes back unchanged in the answer, in webhooks and in the dashboard's Logs (Metadata and actions).

The function sends this request. Example response (format from the API reference; values illustrative):

curl https://api.detectivision.ai/api/v1/moderation \
  -H "Api-Key: $DETECTIVISION_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "text_moderation",
    "input": "You are an idiot",
    "mode": "detect",
    "metadata": { "path": "messages/9vXq2Lr7" }
  }'

{
  "id": 1050,
  "type": "text_moderation",
  "mode": "detect",
  "status": "completed",
  "flagged": true,
  "flags": [{ "flag": "insult", "confidence": 0.87 }],
  "checked": [
    "bad_habits",
    "hate",
    "insult",
    "profanity",
    "racism",
    "self_harm",
    "sexism",
    "sexual"
  ],
  "unchecked": [],
  "tookMs": 95,
  "usage": { "charged": 2, "remaining": 4210 },
  "metadata": { "path": "messages/9vXq2Lr7" }
}

flagged is the verdict, flags lists what was raised, and checked lists every flag that was looked for. With a confidence of 0.87, this message would be rejected. Text checks are reliable in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi; other languages are checked, but the results are not reliable (Text moderation).

Moderate Firebase Storage uploads

Detectivision fetches an image from a URL. From Firebase Storage, send a V4 signed URL that the function makes itself and that expires soon.

import { FieldValue } from "firebase-admin/firestore";
import { getStorage } from "firebase-admin/storage";
import { onObjectFinalized } from "firebase-functions/v2/storage";
import { check, db, DETECTIVISION_API_KEY } from "./moderation";

export const moderateUpload = onObjectFinalized(
  { secrets: [DETECTIVISION_API_KEY], retry: true, timeoutSeconds: 120 },
  async (event) => {
    const { bucket, name, contentType, generation } = event.data;
    // Only images under uploads/{uid}/{uploadId}.
    const match = /^uploads\/([^/]+)\/([^/]+)$/.exec(name);
    if (!match || !contentType?.startsWith("image/")) return;
    const [, uid, uploadId] = match;

    // The verdict lives in Firestore, where the Storage rules read it.
    // One verdict per version of the file; a repeated event is skipped.
    const record = db.doc(`users/${uid}/uploads/${uploadId}`);
    const existing = await record.get();
    if (existing.get("generation") === String(generation) && existing.get("status") !== "pending") return;
    await record.set({ path: name, generation: String(generation), status: "pending" });

    // A V4 signed URL that works for 15 minutes: enough for a check answered at once.
    const file = getStorage().bucket(bucket).file(name);
    const [url] = await file.getSignedUrl({
      version: "v4",
      action: "read",
      expires: Date.now() + 15 * 60 * 1000,
    });

    const outcome = await check({
      type: "image_moderation",
      input: url,
      metadata: { path: name },
    });

    await record.update({
      status: outcome.verdict,
      moderation: { ...outcome, checkedAt: FieldValue.serverTimestamp() },
    });
    if (outcome.verdict === "rejected") await file.delete();
  },
);

• Not a download URL. Firebase's SDK reference calls what getDownloadURL returns "a long lived download URL with a revokable token" that "can be used to share the file with others". In our emulator test, getting that URL went through the security rules, but the URL itself then opened with a plain request and no sign-in. A signed URL from the function expires; this one after 15 minutes, and V4 allows at most 7 days (Signed URLs).
• What the API accepts. An http(s) URL it can download within 10 seconds, of a JPEG, PNG, WebP, BMP or GIF up to 10 MB. It downloads with the user agent DetectivisionBot/1.0 (+https://detectivision.ai) (Image moderation). A signed URL needs no allow-list; a proxy in front of your bucket might.
• Image flags today: hate, scam, sexual and violence (status, 16 October 2026). Each judges the whole image.
• One verdict per file version. onObjectFinalized fires for "a new object (or a new generation of an existing object)" (Storage triggers). The function keeps the generation it checked, so a repeated event is skipped and a new file at the same path is checked again.
• A rejected file is deleted. That also ends every download URL made for it.

Keep pending content hidden

The status field only helps if nobody can set it but the function, and if readers cannot fetch pending content. That is the job of the security rules.

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /messages/{messageId} {
      // Signed-in users see approved messages; authors also see their own.
      allow read: if request.auth != null
        && (resource.data.status == 'approved' || resource.data.authorId == request.auth.uid);
      // A new message starts as pending. Only the function sets a verdict.
      allow create: if request.auth != null
        && request.resource.data.keys().hasOnly(['authorId', 'text', 'status', 'createdAt'])
        && request.resource.data.authorId == request.auth.uid
        && request.resource.data.status == 'pending'
        && request.resource.data.text is string
        && request.resource.data.text.size() > 0
        && request.resource.data.text.size() <= 2000
        && request.resource.data.createdAt == request.time;
      allow update, delete: if false;
    }
    match /users/{uid}/uploads/{uploadId} {
      allow read: if request.auth != null
        && (request.auth.uid == uid || resource.data.status == 'approved');
      allow write: if false;
    }
  }
}

• Rules are not filters. Firebase puts it plainly: "security rules are not filters—queries are all or nothing" (Securely query data). A query for all messages fails for every reader, because it could return someone else's pending message. Ask for approved messages, or for the reader's own, as below.
• Nobody approves their own message. A new message may carry only these four fields, and its status must be pending. The function writes with the Admin SDK, and server client libraries "bypass all Cloud Firestore Security Rules" (Firebase).
• No edits. The function runs on create only, so an edited message would skip the check. If your app needs edits, save the edit as a new pending message, or check it in an onDocumentUpdated function.
• 2,000 characters is this app's choice. The API takes up to 10,000 (Text moderation).

import { addDoc, collection, getDocs, limit, orderBy, query, serverTimestamp, where } from "firebase/firestore";
import { ref, uploadBytes } from "firebase/storage";
// db = getFirestore(app), storage = getStorage(app), user = the signed-in user

// Send a message: it starts as pending.
await addDoc(collection(db, "messages"), {
  authorId: user.uid,
  text,
  status: "pending",
  createdAt: serverTimestamp(),
});

// List approved messages. The query must match what the rules allow.
const approved = await getDocs(
  query(collection(db, "messages"), where("status", "==", "approved"), orderBy("createdAt", "desc"), limit(50)),
);

// Upload a photo into your own folder.
const uploadId = crypto.randomUUID();
await uploadBytes(ref(storage, `uploads/${user.uid}/${uploadId}`), file, { contentType: file.type });

Ordering approved messages by time needs an index on status and createdAt; the error from the first run links to it (Index types).

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /uploads/{uid}/{uploadId} {
      // New files only, into your own folder: images the API accepts, under 10 MB.
      // In Storage rules, create covers every write of file contents, overwrites
      // included; resource == null stops a new image replacing an approved one.
      allow create: if request.auth != null
        && request.auth.uid == uid
        && resource == null
        && request.resource.size < 10 * 1024 * 1024
        && request.resource.contentType.matches('image/(jpeg|png|webp|gif|bmp)');
      // The owner sees a pending upload; others only once the function approved it.
      allow read: if request.auth != null
        && (request.auth.uid == uid
          || firestore.get(/databases/(default)/documents/users/$(uid)/uploads/$(uploadId)).data.status == 'approved');
      // No client deletes either: a freed path could take a new file while the
      // old verdict still says approved. Delete through a function instead.
      allow update, delete: if false;
    }
  }
}

• create covers overwrites. Firebase's rule reference says create "Applies to writes to file contents" and update "Applies to updates to (pre-existing) file metadata". Our first emulator run let a user replace an uploaded image at the same path; resource == null closed that.
• Storage rules read the verdict from Firestore. The first time you save rules that call firestore.get(), Firebase asks you to let the two products connect. Each such read counts toward Firestore billing, and one rule evaluation may read at most two documents (Storage rule conditions).
• Deploy everything together: firebase deploy --only functions,firestore:rules,storage.

Backlogs and bursts

Some content can wait: comments, profile bios, or the messages written before you added moderation. Queue those checks with async: true. The API answers 202 at once with the request's id, runs the check shortly after at the table price, half of an answer at once, and sends the result by webhook (Async requests).

import { onDocumentCreated } from "firebase-functions/v2/firestore";
import { DETECTIVISION_API_KEY } from "./moderation";

// Comments can wait: queue the check at the table price. The verdict comes by webhook.
export const queueComment = onDocumentCreated(
  { document: "comments/{commentId}", secrets: [DETECTIVISION_API_KEY], retry: true },
  async (event) => {
    const snapshot = event.data;
    if (!snapshot || snapshot.get("status") !== "pending") return;

    const res = await fetch("https://api.detectivision.ai/api/v1/moderation", {
      method: "POST",
      headers: {
        "Api-Key": DETECTIVISION_API_KEY.value(),
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        type: "text_moderation",
        input: snapshot.get("text"),
        mode: "detect",
        async: true,
        metadata: { path: snapshot.ref.path },
      }),
    });
    // 202: queued. Anything else throws, and the trigger runs again (retry: true).
    if (res.status !== 202) throw new Error(`Detectivision ${res.status}: ${await res.text()}`);
  },
);

Give comments the same rules as messages. The result arrives at an HTTPS function:

import crypto from "node:crypto";
import { FieldValue } from "firebase-admin/firestore";
import { defineSecret } from "firebase-functions/params";
import { onRequest } from "firebase-functions/v2/https";
import { db, outcomeOf, type Outcome } from "./moderation";

const WEBHOOK_SECRET = defineSecret("DETECTIVISION_WEBHOOK_SECRET");

// The check from /docs/webhooks: HMAC-SHA256 of "{id}.{timestamp}.{raw body}".
function verified(secret: string, id: string, timestamp: string, signatures: string, body: string) {
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false;
  const key = Buffer.from(secret.slice("whsec_".length), "base64");
  const expected = crypto.createHmac("sha256", key).update(`${id}.${timestamp}.${body}`).digest("base64");
  return signatures
    .split(" ")
    .map((s) => s.split(",")[1] ?? "")
    .some((s) => s.length === expected.length && crypto.timingSafeEqual(Buffer.from(s), Buffer.from(expected)));
}

export const moderationWebhook = onRequest({ secrets: [WEBHOOK_SECRET] }, async (req, res) => {
  const body = req.rawBody.toString("utf8");
  const ok = verified(
    WEBHOOK_SECRET.value(),
    req.get("webhook-id") ?? "",
    req.get("webhook-timestamp") ?? "",
    req.get("webhook-signature") ?? "",
    body,
  );
  if (!ok) {
    res.status(401).send("bad signature");
    return;
  }

  const event = JSON.parse(body);
  const path: unknown = event.data?.metadata?.path;
  if (typeof path === "string" && /^(messages|comments)\/[^/]+$/.test(path)) {
    let outcome: Outcome | null = null;
    if (event.type === "moderation.completed") outcome = outcomeOf(event.data);
    if (event.type === "moderation.failed") {
      outcome = { verdict: "review", flags: [], requestId: event.data.id, error: event.data.error };
    }
    if (outcome) {
      const done = outcome;
      const ref = db.doc(path);
      // Every finished check is announced, also the ones a trigger already decided.
      await db.runTransaction(async (tx) => {
        const current = await tx.get(ref);
        if (current.get("status") !== "pending") return;
        tx.update(ref, { status: done.verdict, moderation: { ...done, checkedAt: FieldValue.serverTimestamp() } });
      });
    }
  }
  // Answer within 10 seconds. Cloud Functions does not run code after the answer.
  res.status(204).end();
});

export { moderateMessage } from "./messages";
export { moderateUpload } from "./uploads";
export { queueComment } from "./comments";
export { moderationWebhook } from "./webhook";

To connect it:

1. Deploy, then add the function's URL under Dashboard → Webhooks and subscribe it to moderation.completed and moderation.failed (Webhooks).
2. Copy the endpoint's signing secret, which starts with whsec_, and store it: firebase functions:secrets:set DETECTIVISION_WEBHOOK_SECRET.
3. Leave moderation.flagged off. A flagged check sends moderation.completed too, so you would handle it twice.

What the handler does, and why:

• It verifies the raw body. The signature covers the exact bytes Detectivision sent, so the check runs on req.rawBody, before any parsing, and refuses a message stamped more than 5 minutes from your clock.
• It writes only to documents still pending. Every finished check reaches your endpoint, including the ones a trigger already answered at once. Deliveries also repeat: a failed one is sent again over about 28 hours, with the same webhook-id.
• It writes, then answers. Detectivision waits 10 seconds for a 2xx. Cloud Functions gives no CPU to code after the response; Google's words are "Any code run after graceful termination cannot access the CPU and will not make any progress."
• Async does not lift the limits. Calls with one API key are limited to 1,200 a minute, and a 429 means waiting the seconds in Retry-After (Errors). The trigger's retry: true sends the request again later. For a backlog, queue at a steady pace instead of all at once.

What it costs

• Text costs a unit per started 2,500 characters. An image in detect mode is one unit.
• A unit costs a credit when queued with async: true, and twice that when answered at once, as moderateMessage and moderateUpload do (Credits and billing).
• Free: errors, partial results, and async checks that fail (Credits and billing, Async requests).
• Plans: Basic comes with 10,000 text credits and 10,000 image credits. Pricing has the trial and the other plans.

Firebase bills its side separately on the Blaze plan: the function runs, Secret Manager reads beyond its free tier (Configure your environment), and the Firestore reads your Storage rules make.

Limits, and what this does not do

• It is not an official Firebase Extension. It is your own code, in your project.
• It does not mask words or blur images. Flags judge the whole text or image, so the function holds content back instead of rewriting it.
• It reads text reliably in English, Spanish, Portuguese, French, German, Italian, Turkish, Arabic, Russian and Hindi. Other languages are checked, but the results are not reliable.
• It checks text and still images. Video, audio and live stream moderation are coming soon, so this page leaves them out.
• It gives you no review screen. Items marked review wait for a person. Build an admin view in your app, or use the dashboard's Logs, where a moderator opens a check with your metadata; actions you take there reach your system as moderation.action webhooks. Detectivision never changes anything in your project itself (Metadata and actions).
• It does not check edits, which the rules above forbid.
• It is one of four App Store precautions, not all of them. Apple's Guideline 1.2 asks apps with user content for "A method for filtering objectionable material from being posted to the app", which this can help with. Reporting, blocking and published contact information are features you build (App Store Guideline 1.2).

How this was checked

Every step ran on 16 October 2026. Firebase and Google Cloud facts quoted on this page come from their documentation, read the same day; the links are under Sources.

Not covered: a deployed project (IAM signing, Secret Manager and event delivery in Google Cloud) and real API answers. The answer shown above is an example in the documented format.

Start the free trial

Next: read the docs, or see the same job on Supabase in Supabase content moderation with Edge Functions.