Guardrails and Content Moderation for Generative AI Products
The naive approach to guardrails is a single moderation API call on the model's output, and it catches the obvious cases while missing a lot of the subtler ones — a response that's technically not toxic but is wildly off-brand, or one that leaks a system prompt detail it shouldn't. We now think of guardrails as several independent layers, each catching a different failure mode, rather than one filter doing all the work.
Input-side filtering matters as much as output filtering and gets less attention. For a client-facing generation tool, we added prompt injection detection on user input before it ever reaches the model, because a meaningful share of the abuse attempts we saw in testing were aimed at manipulating the system prompt rather than getting harmful content out directly — jailbreak attempts disguised as ordinary requests.
Output moderation itself needs to be tuned per product, not applied as a generic off-the-shelf threshold. A children's education product and a mature-audience creative writing tool need very different sensitivity settings, and using the same default moderation threshold across both — which we did on an early project before correcting it — either over-blocks legitimate content or under-blocks unacceptable content, sometimes both at once for different categories.
We log every moderation block with enough context to review later, because false positives are a real cost — a legitimate user blocked repeatedly will churn just as reliably as one who saw something harmful. Reviewing blocked-content logs weekly on one project surfaced a pattern where medical terminology was consistently over-flagged, and adjusting the threshold for that category cut false positives by roughly 40% without loosening anything else.
The layer that catches the most in practice isn't the fancy one — it's a hard-coded denylist of topics the business has decided are simply off-limits regardless of what a model or classifier thinks, reviewed and updated by the client's own team. Guardrails work best when the client, not just the engineering team, owns the policy the system enforces.
