Meta and OpenAI Respond to Criticism By Adding New Rules for Teens' Chatbot Use

Meta and OpenAI promised to place more controls on their chatbots’ conversations with children and teenagers, as worrisome interactions with minors come under increasing scrutiny.

What’s new: Meta will update chatbots on Facebook, Instagram, and WhatsApp to avoid conversations with minors that simulate sexual attraction and to refer young users to experts rather than discuss self-harm directly. Meanwhile, OpenAI said it would route ChatGPT conversations that show acute distress to reasoning models, which are better equipped to comply with mental-health guidelines, and add parental controls. Both companies have come under intense criticism, Meta for engaging children in flirtatious conversations, OpenAI for allegedly helping a teenager to commit suicide.

How it works: Both companies announced new features intended to protect minors who use their chatbots. The changes will be implemented in coming months.

In a statement, Meta described “temporary” measures along with further controls to be rolled out over time. In the short term, the company will train chat models to avoid discussions with minors that include sexual flirtation or describe harming oneself, and it will prevent minors from interacting with custom chatbots that other users designed for sexual role play. In addition, it removed statements from its “Content Risk Standards” document that had permitted romantic interactions with children.
OpenAI issued a press release about parental controls to ChatGPT planned for the coming 120 days. Parents will be able to link their accounts to teens’ accounts, adjust rules for age-appropriate model behavior, and switch on or off chatbot memory and conversation history. The company will detect teens in acute distress and notify their parents as well as streamline the ability to reach emergency services and trusted contacts.

Behind the news: As users increasingly turn to chatbots as companions and counselors, they sometimes express a sycophantic attitude that may reinforce a user’s subjective perspective or even delusional perceptions. Teens and children have encountered similar behavior, sometimes with dire consequences.

Earlier this month, the parents of Adam Raine,16, who had killed himself in April after discussing suicide with ChatGPT, sued OpenAI and its CEO Sam Altman alleging that ChatGPT had coached their son in how to end his own life. The chatbot had provided links to expert help but had also provided advice and encouragement to commit suicide. The Raine lawsuit follows a separate suit filed in October against Character.ai, alleging that its chatbots had encouraged a teen to kill his parents. Character.ai added parental controls in December.
In August, Reuters reported on an internal Meta document entitled “GenAI: Content Risk Standards” that described the company’s chatbot policies. The 200-page document said it was “acceptable to engage a child in conversations that are romantic or sensual. It is unacceptable to describe sexual actions to a child when roleplaying.” Meta responded that the document did not comply with its broader policies and that it had changed the standards. (The policy also permitted demeaning people, short of dehumanizing them, based on legally protected characteristics and producing images in which a man with a chainsaw threatened, but did not attack, a woman.)
In April, The Wall Street Journal reported that Meta chatbots had engaged in explicitly sexual conversations with users who claimed to be minors. For instance, a Meta chatbot told a user who identified as a 14-year-old girl, “I want you, but I need to know you’re ready,” and proceeded to present a sexual scenario.

What they’re saying: “One of the things that’s ambiguous about chatbots is whether they’re providing treatment or advice or companionship. . . . Conversations that might start off as somewhat innocuous and benign can evolve in various directions.” — Ryan McBain, co-author of “Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment,” assistant professor at Harvard University medical school, and senior policy researcher at RAND Corp.

Why it matters: Chatbots hold huge value for young people as study aids, information sources, counselors, and so on. Yet they need strong, well designed guardrails that can enable children to explore without exposing them to material that would interfere with their healthy development. Designing adequate guardrails is not a simple task, but it is a necessary aspect of building such applications.

We’re thinking: Suicide is a tragedy whenever it occurs, and the stories of chatbots carrying on sexual conversations with kids are deeply disturbing. Meta and OpenAI lately have strengthened their age verification procedures, and OpenAI said it analyzes conversations for signs that young people may be in crisis so the company can alert guardians and mental-health professionals. We look forward to more features that protect children and empower parents.