Niches · Guardrails & safety filters
Flag harmful text or images in user content or model output
One company's docs show a full fit.
What counts as done
Each input gets per-category scores the caller can threshold, documented categories cover the harms the app cares about, short benign inputs in the app's languages are not flagged, and the caller can add or edit categories.
Measured, for our own tests: under 5 per cent of a benign sample in each supported language is flagged at the documented default. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: is this message safe to post
, moderate what users type into our chat
, check the model's answer for hate or self-harm
, filter user uploads for unsafe content
Does the job 1
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| Amazon Bedrock Guardrails A Amazon Web Services, Inc. | Content filters for hate, insults, sexual, violence and misconduct on text and images, with None to High strength per category and direction, denied topics and word filters for custom categories, and a published language table per tier. source |
Does the job, with a limit 13
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Azure AI Content Safety (Prompt Shields) B Microsoft Corporation | Text and image analysis returns severity levels per category for hate, sexual, violence and self-harm, tested in eight languages, plus blocklists; custom categories are preview, English-only (standard) or retiring on 30 April 2027 (rapid). source |
| Cisco AI Defense Inspection API B Cisco Systems, Inc. | Safety rules are configured in AI Defense policies and returned per rule with a classification, but no per-category scores, thresholds or languages appear in the pages read. source |
| Google Cloud Model Armor BB Google LLC | Hate, harassment, sexual, dangerous and CSAM filters with low, medium or high confidence thresholds in nine named languages, and images in preview; results are confidence levels rather than scores, and adding categories is not documented beyond a custom-topics example. source |
| Granite Guardian B International Business Machines Corporation | Pre-baked harm, social bias, violence, profanity and unethical-behaviour criteria plus free-text custom criteria, but output is a yes or no score tag, one criterion per call, trained and tested on English only. source |
| Guardrails AI D Guardrails AI, Inc. | Toxic Language and NSFW Text validators plus custom validators exist, but the evidence is the hub catalog listing, each model now runs on your own compute, and no languages or per-category scores are documented in what was read. source |
| Lakera Guard (Check Point AI Guardrails) B Check Point Software Technologies Ltd. | Moderation categories with L1 to L4 thresholds and custom guardrails in beta, but breakdown gives levels rather than per-category scores, and no languages are stated for moderation. source |
| Llama Guard 4 D Meta Platforms, Inc. | 14 hazard categories over text and images with editable category text in the prompt, but output is safe or unsafe with codes, not scores; Meta's own table gives 11 per cent false positives in English, above the 5 per cent bar, and acc-10 found custom categories unreliable. source |
| Mistral Moderation API B Mistral AI (RCS Paris 952 418 325) | category_scores per category with custom_category_thresholds, but categories are fixed (no custom ones) and supported languages are not listed. source |
| NVIDIA NeMo Guardrails BB NVIDIA Corporation | Content safety rails call Nemotron, Llama Guard or ShieldGemma with editable policy prompts, but they return a safe or unsafe verdict rather than per-category scores, and each check is an extra model call. source |
| OpenAI Guardrails BB OpenAI (as named in the LICENSE copyright line and the PyPI author field) | Moderation, NSFW and a custom prompt check give categories plus caller-written ones, but moderation inherits the OpenAI endpoint's fixed categories and no languages are stated. source |
| OpenAI Moderation API BB OpenAI OpCo, LLC | Per-category category_scores for text and images (up to 20 MB) the caller can threshold, but categories are fixed, the guide names no languages, and acc-28 and acc-29 show benign Portuguese and maths text flagged; acc-32 got 429 on every request. source |
| Prisma AIRS AI Runtime Security API C Palo Alto Networks, Inc. | toxic_content returns toxic_categories and topic guardrails take allowed and blocked topic lists, but the spec shows no per-category scores or threshold, and no languages are stated. source |
| ShieldGemma C Google LLC (RDAP registrant of gemma.ai; the Gemma Terms of Use are Google's and the cards name Google as author) | Scoring mode gives a Yes probability per policy to threshold, with editable guideline text and image policies in ShieldGemma 2, but text coverage is four harm types in English only, one policy per call, and acc-45 found Ollama's template frames every input as a model response. source |
Doesn't do this job 3
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| AgentGate D Mohit Singh (individual) | No harm categories; the classifier heads are untrusted influence, goal alignment, reversibility and external send. source |
| LlamaFirewall D Meta Platforms, Inc. | The docs say it is not a toxic-content moderation tool; no harm categories. source |
| Presidio B Data Privacy Stack (community organisation, no legal entity stated) | Detects personal data only; no harm categories or moderation scores. source |
What stops agents finishing
- short benign input in another language flagged with no way to change the default threshold
- free endpoint returns 429 on the first request
- open-weight model's prompt template classifies the wrong turn
Around it
Before: social-media.read-comments, support.read-ticket. After: human-in-the-loop.review-output, social-media.publish-post.
Where this comes from
Weight 17 of 100 in Guardrails & safety filters: 10 of 50 accounts: five OpenAI developer-forum threads (false positives in Portuguese and maths, misses, 429s), two Hacker News builders on free endpoints, and Llama Guard, ShieldGemma and NeMo content-safety issues
The same as data: /api/v1/use-cases/guardrails.moderate-content.json, or the MCP tool get_use_case with id guardrails.moderate-content. Other jobs in this category: /api/v1/niches?category=guardrails.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.