OpenAI releases new content moderation tooling

OpenAI released an updated Moderation endpoint, a classifier that scores a piece of text for whether it is sexual, hateful, violent or promotes self-harm — categories drawn from OpenAI’s own content policy. Unlike its other API endpoints, the Moderation endpoint was free to use for developers building on OpenAI’s API and did not count against usage limits, and the company said it had improved the tool’s speed and accuracy over the version it replaced. It also published a technical paper describing the classifier’s methodology and the evaluation dataset used to test it.

The intended use was pre-deployment filtering: an application built on GPT-3 or another OpenAI model could run generated or user-submitted text through the Moderation endpoint before displaying it, catching content that violated policy without requiring every developer to build their own classifier. OpenAI framed the release explicitly as using one AI system to assist human oversight of others — automating a task that would otherwise fall to manual review at a scale manual review could not sustain.

On its own the tool was a piece of developer infrastructure rather than a research result, but it is one of the clearer examples of a lab shipping a safety mechanism as a free, general-purpose product ahead of the wave of generative AI applications that followed later that year, and it set a precedent — a dedicated, free moderation layer — that most large model providers had matched in some form within two years.