Toward Safer (and Sexier) Chatbots: Inside Character AI and OpenAI’s policy changes to protect younger and vulnerable Users

Chatbot providers, facing criticism for engaging troubled users in conversations that deepen their distress, are updating their services to provide wholesome interactions to younger users while allowing adults to pursue erotic conversations.

What’s new: Character.AI, which provides chatbots designed for entertainment and companionship, temporarily barred teen users from parts of its offering and announced plans to offer a service for younger users. Meanwhile OpenAI, which faces a lawsuit on allegations that ChatGPT contributed to a teenager’s suicide, updated ChatGPT to better help users in psychological distress and reaffirmed that it would allow adult users to generate erotic content later this year.

Character.AI limits access: The startup imposed limits on young users after it received "reports and feedback from regulators, safety experts, and parents" that expressed concern about its impact on teen users, BBC News reported.

• Character.AI moved to limit chat time for users under 18, starting with 2 hours daily and tapering to zero by November 25.
• The company will roll out a new in-house age verification model and use third-party technology to prevent younger users from engaging in adult chat.
• In addition, it will establish an independent AI Safety Lab where it plans to collaborate with other organizations to improve safety alignment and other features for AI entertainment.

OpenAI detects distress: Around 0.15 percent of ChatGPT users — roughly 1.2 million out of the service’s 800 million weekly active users — show signs of suicidal intent and/or excessive emotional attachment to the chatbot, OpenAI revealed. The company said it has made its models more responsive to such issues, paving the way to provide interactions geared toward adults who don’t suffer from distress.

• OpenAI updated ChatGPT to avoid encouraging certain groups of users to engage in dangerous and/or self-destructive behavior. The effort targets three vulnerable groups: (i) people with severe mental illnesses like psychosis or mania, (ii) people with depression and suicidal ideation, and (iii) people with excessive emotional attachments to AI.
• In a test of 1,000 mental health-related conversations, the new version of GPT-5 boosted desired responses to mental-health crises from 27 percent to 92 percent, and reduced undesired responses by 65 percent. For suicidal conversations, the rate of desired responses rose from 77 percent to 91 percent, and the rate of undesired responses fell by 65 percent. Conversations that showed signs of excessive attachment saw undesired responses drop by 80 percent and desired responses rise by 50 percent to 97 percent.
• Sam Altman wrote on the social network X that, given the company’s progress in providing psychologically sensitive output and restricting output based on users’ ages, it would provide output geared toward verified adults, including erotica, beginning in December.

Behind the news: Both Character.AI and OpenAI were sued by families of underage users who committed suicide after they conversed with their chatbots. In the U.S., California recently passed a state law that outlaws exposing minors to sexual content and requires supporting users who are suicidal and otherwise at risk psychologically. In August, 44 state attorneys general warned xAI, Meta, and OpenAI to restrict sexually explicit material as much as possible. xAI openly embraced adult interactions in July, when it introduced sexually explicit chatbots.

Why it matters: Chatbot companionship is a growing phenomenon, and companies that offer such services — or simply chat — must be ready to manage emotional relationships between users and their software. Managing sexually charged interactions and conversations about mental illness are linked under the umbrella of building guardrails. Sycophancy also plays a role, since models that are prone to agreeing with users can encourage dangerous behavior. A depressed, underage user and a permissive chatbot make a worrisome combination.

We’re thinking: Mental health is a hard problem, in part because it affects so many people. A recent study shows that 5.3 percent of Americans had suicidal thoughts in 2024 — far higher than ChatGPT users’ 0.15 percent. It’s important that chatbot providers do what they can to help troubled users get help.