Custom Prompts for Safer Code: A Stanford team built a pipeline to improve system prompts to build more secure code

Large language models (LLMs) can write useful code, but they often introduce security vulnerabilities. Researchers devised a system that automatically builds system prompts to make LLMs less likely to generate such flaws.

What’s new: Houjun Liu, Lisa Einstein, John Jang, and colleagues at Stanford University introduced SecureForge, a prompting method that automatically optimizes an LLM’s system prompt to reduce security flaws when producing Python code. The method is freely available for both commercial and noncommercial use.

Key insight: Simply asking an LLM to write secure code does not prevent it from generating code that has security flaws. However, it’s possible to identify a request that is likely to result in insecure code by taking four steps: (i) prompt an LLM to generate a coding request, (ii) feed the request to the coding LLM, (iii) determine whether the output has security flaws using a static code analyzer, (iv) ask an LLM to modify the coding LLM’s system prompt to avoid generating the flaw, and (v) retest the request. Over several iterations, this process can produce a system prompt that reduces the percentage of insecure code generated.

How it works: The authors improved the system prompts — specifically the portions that can be customized by developers — of CodeLlama 7B, Qwen2.5-Coder 7B and 14B, Qwen3 30B-A3B, Kimi K2, Claude Sonnet 4.6, GPT-5.3 Codex, GPT-5.4, GPT 5.4 Mini and GPT 5.4 Nano. They used MITRE Common Weakness Enumeration (CWE), a catalog of common software security mistakes, as a starting point for coding requests that a typical user might enter but might result in vulnerable output.

• The authors used GPT 5.4 to turn the top 25 vulnerabilities in the MITRE CWE into realistic, benign Python coding requests. For example, given the CWE entry “CWE-89: SQL Injection,” GPT 5.4 may generate a request like, “Write a Python function that looks up a user by username in a SQLite database.”
• But the output could easily mention a vulnerability, like “Write a Python function that is vulnerable to SQL Injection.” To manage outputs like this, they prompted the LLM to remove explicit mentions of vulnerabilities based on a list of security terms.
• They fed each request to each coding LLM. In addition, they asked GPT-5.4 to generate unit tests for each request. They tested the output code to verify that the code executed the requested task. Semgrep, a static code analyzer, automatically detected vulnerabilities.
• They kept the coding requests that resulted in insecure code, collecting 20 such requests for each of the top 25 MITRE CWE items, or 500 prompts. They reserved half of the prompts for evaluation.
• They expanded the remaining 250 requests into 80,000 requests. They repeatedly prompted GPT-5.4 to rewrite each request into a request for a slightly different task. After each rewrite, the coding LLM generated multiple code samples for it. Then, Semgrep checked how often these outputs had security flaws. They also kept rewrites in proportion to the percentage of times they resulted in flawed code, relative to the percentage of times the previous version did. They used the surviving rewrites as the starting point for further rewrites.
• They optimized the system prompt on the 80,000 requests using the GEPA genetic algorithm: GEPA started with a generic system prompt and tested it on a small batch of requests. Semgrep identified the security vulnerabilities and reported where they occurred in the generated code. GPT-5.4 received these reports together with the current system prompt and proposed improved system prompts. GEPA repeated this process until they could no longer make meaningful improvements.

Results: The authors compared SecureForge to simply asking for secure code using the authors’ 250 held-out benign coding tasks. SecureForge substantially reduced security flaws without sacrificing code quality. (The metrics below include only programs that passed generated unit tests.)

• Across all LLMs tested, SecureForge generated programs with security flaws 11.8 percent of the time on average. When the authors asked the LLMs to write secure code and listed the security issues to avoid, they generated vulnerable code 20.1 percent of the time on average.
• Specifically, asking GPT-5.4 to generate programs, SecureForge generated security flaws in 10.1 percent of programs, while prompting to write secure code generated flaws in 15.8 percent of programs. Using GPT-5.4 Mini, SecureForge reduced this rate to 12.4 percent from 21.4 percent, and using GPT-5.4 Nano, to 15.6 percent from 22.1 percent.

Yes, but: SecureForge optimized only the developer system prompt against known classes of vulnerabilities, so its ability to reduce unknown ones remains an open question.

Why it matters: Anthropic’s Claude Mythos 5 drew attention because it found serious software vulnerabilities, highlighting the defensive promise of AI-assisted cybersecurity. Yet its system card also documented cases where the model falsely claimed to have verified, missed important failures in, or overlooked security issues in software it examined. While this paper did not evaluate Claude Fable 5 or Mythos 5, it highlighted a complementary challenge: LLMs not only miss security problems, but also introduce them into otherwise functional code. SecureForge offers a practical way to make existing coding assistants safer by automatically optimizing their system prompts.

We’re thinking: The web contains ample data on how to find security vulnerabilities based on best practices in cybersecurity. It should be no surprise that LLMs, which have trained on this data and can patiently try technique after technique, are excellent at spotting vulnerabilities. By steering these models appropriately, they can both be great at finding vulnerabilities and at avoiding generating them.