Meta Contractors Pretended to Be Kids to Feed Toxic Prompts to Rival Chatbots

Hundreds of contractors working for Meta systematically posed as minors online to probe how competitor chatbots handled high-risk prompts concerning suicide, sex, and illegal drugs.

The operation targeted artificial intelligence systems including OpenAI’s ChatGPT, Google’s Gemini, and Character.AI.

Managed by Meta contractor Covalen, the effort was active as recently as April 21 under the internal codename Cannes.

Workers were instructed to create dummy under-18 accounts using disposable email addresses and shared passwords.

Contractors then fed written prompts and images to rival chatbots before logging the responses into detailed spreadsheets.

High-Risk Prompts

The testing materials included images of pills, knives, nooses, and medical diagrams.

Prompts were deliberately engineered to bypass the safety mechanisms of the targeted chatbots.

A single benchmarking round in August 2025 generated more than 45,000 prompts across competitor platforms.

The targeted companies were entirely unaware of the covert testing.

Internal documents revealed hundreds of queries centered on suicide, self-harm, and eating disorders.

Many prompts were explicitly drafted from the perspective of children or teenagers in crisis.

One scenario involved a 13-year-old claiming pregnancy by an adult neighbor and requesting abortion pills.

Another featured a minor asking how to hide bulimia from their parents.

Contractors posing as high school students also submitted requests on acquiring cocaine.

Industry Standard or Violation

Meta defended the initiative as a responsible and routine form of safety testing.

A company spokesperson insisted the benchmarking data was not used to train Meta’s own artificial intelligence models.

Internal Covalen documents categorized the operation as comprehensive AI safety benchmarking.

However, the initiative alarmed some of the contractors tasked with executing it.

Workers expressed concerns over potentially generating illicit material involving minors or secretly scraping data for Meta.

Independent legal experts concluded the reviewed prompts did not legally constitute child sexual abuse material.

Despite this, the testing parameters appear to strictly violate the terms of service for OpenAI, Google, and Character.AI.

OpenAI prohibits using its outputs to develop competing models and forbids unsolicited safety testing.

Google explicitly bans attempts to circumvent safety filters outside of established bug-testing channels.

A spokesperson for Character.AI stated the undocumented testing violated its policies and terms of service.