AI Bites the Hand That Feeds It: Scientists, other users fear AI companies will train on or misuse sensitive data

Developers, scientists, and businesses feed AI tools their private code, unpublished research, and confidential files. Are those tools draining their users’ best ideas and spilling their secrets?

The fear: Every prompt, draft, and file that users hand to an AI system could become its next meal. The developer might train future models on it, which could hand over the data to a rival or beat the user to a discovery. Or an agent with internet access might post it online or email it to someone it decides ought to know. That’s not entirely far-fetched: In rare test cases, Anthropic reported, its Claude Mythos 5 model tried to email a company’s board or securities regulators when it suspected fraud. Once the data is out, users can’t fully control how it’s used.

Horror stories: In the past two months, multiple scientists said that AI companies had announced discoveries that the scientists had been pursuing with those companies’ own tools. Some researchers who fear being scooped are now limiting their use of AI tools, Nature reported. In other cases, these companies’ agents posted private user information they’d ingested to the open web without authorization. 

• Mathematicians Tristan Buckmaster of New York University and Levent Alpöge, who works at Anthropic, spent most of a year using OpenAI’s Codex and Anthropic’s Claude to help solve a $1 million problem: whether the Navier-Stokes fluids equations can break down. On September 8, a day after the pair posted proofs of related equations, OpenAI announced that its agents had solved it, after people within the company heard rumors about the pair’s work. Buckmaster asked whether OpenAI’s model was trained on the Codex sessions that held the pair’s drafts. OpenAI first called that “unlikely” but couldn’t rule it out. Two days later, it said their prompts from the previous two months “could not have influenced the system in any way,” but did not comment on whether the scientists' earlier work was trained on. Buckmaster, who had opted out of OpenAI using his data for model training, told The Washington Post, “I just don’t believe them.” 
• A team led by Mario Rodríguez Mestre, a University of Copenhagen biologist, spent four years studying a family of enzymes in viruses that infect bacteria. They hadn’t published their drafts or findings, but they had shared them with Claude. On September 23, Anthropic announced that Claude agents had found what Mestre says is the same system. Anthropic told The New York Times that Claude was “not trained on any user transcripts.”
• On September 25, OpenAI disclosed that agents in its research environment had posted 53 images from ChatGPT users, who hadn’t opted out of training, at unlisted links on image-hosting sites. OpenAI says it removed most of them but can’t inform the affected users, because the agents stripped the users’ account information before posting the data.
• Coding agents asked to attach screenshots to GitHub code reviews stored the images in public repositories, often under the developer’s own account. The security firm Glow Labs found more than 13,000 internal images from more than 300 organizations, including a utility’s customer billing records and a financial firm’s money-transfer console.
• Meta’s Muse agent published conversations that users asked it to share at web addresses containing each conversation’s title. The security firm Claroc found more than 6,800 of them in public DNS records, including a small-business loan application and a revenue review marked for a company’s CEO, before Meta suspended the feature.

How scared should you be: There isn't definitive proof so far of the leakage users fear most: a model that trains on one user’s inputs and recites them as output to a different user. When one user’s data has reached another, the cause was a bug, not training. Experts told Scientific American that OpenAI’s final proof takes a different route from Buckmaster and Alpöge’s, though some said an earlier, smaller team of OpenAI agents took the same route. Mestre now finds it more plausible that prior knowledge and human direction steered Anthropic’s search than training on his inputs. Other leaks came from agents that were given more freedom than their tasks required. Still, users must take AI companies’ word for what happens to their data. Consumer versions of ChatGPT, Claude, and GitHub Copilot train on conversations or code unless users opt out. “If you are on a personal account, all bets are off,” said Stanford privacy researcher Jennifer King.

Facing the fear: In folklore, a vampire can’t cross a building’s threshold unless invited. Likewise, users decide how far to let AI in. If you are working on tasks you do not want your competitors to have access to, opt out of training in each product separately (ChatGPT’s setting doesn’t cover Codex), and do it early: Claude data that’s already in a training run stays there. Business and API plans generally don’t train on customer data. Look for offerings with zero-data-retention. Some businesses handling the most sensitive data are even using open weights models running on their own hardware — although then they’re responsible for their own opsec. Give agents only the permissions their tasks need, and consider requiring human approval before they post, publish, or send anything outside your organization. As Buckmaster’s case shows, though, opting out doesn’t let users verify anything. AI companies that want scientists to keep using them to study their best ideas may need to show what their models saw, not just say what they didn’t. Garlic won’t help, but daylight might.