We Should Be Encouraged By AI’s Cybersecurity Abilities: Why GLM-5.3’s High ExploitBench score is good news
Anthropic recently released an encouraging analysis of the cyber capabilities of open weight model GLM-5.3.
Dear friends,
Anthropic recently released an encouraging analysis of the cyber capabilities of open weight model GLM-5.3. The study shows that GLM-5.3 approaches the cyber capabilities of Claude's closed weight Mythos. For example, the diagram below shows that, using a comparable number of tokens, GLM-5.3 succeeded in 12% of attempts to exploit vulnerabilities on a subset of ExploitBench tasks, compared to Mythos’ 14%. (Interestingly, the gap reported by Anthropic is smaller in GLM-5.3’s creator's report on the full benchmark: 54.4% success compared to Mythos' 78.0%.) This paves the way for cyberdefenders, including ones that do not have access to Mythos, to use a highly capable model to defend themselves. Given attackers’ access to similar capabilities, the urgency of ramping up defenses grows.
Further, the cost of finding vulnerabilities and using it to create an attack (that is, to create a payload that targets the identified vulnerability) is falling. Anthropic found that spending $20.40 on GLM-5.3-Flash tokens was sufficient to find a recently disclosed flaw in Google Chrome. The results reported used an unmodified version of GLM-5.3. It is also easy to obtain frontier open weight models that have had their guardrails weakened. Such models would be even more effective for cyberdefense (or cyberoffense).
A lot of AI risks, including cyber vulnerabilities and how to contain agents, are engineering problems to be solved. To be clear, these are hard engineering problems! But I find that a lot of the fear-mongering scenarios implicitly assume that we will make little or no progress on them.
For an analogy, consider aviation. According to the National Postal Museum, in 1919, one person died for about every 115,000 miles flown. Today, airlines fly about 6 trillion passenger miles. So, airplanes kill about 6 trillion /115,000 = 52 million persons a year, right? Because we have engineered airplanes to be much safer, we now see one death approximately every 30 billion passenger miles.
Similarly, even though mass media describes the OpenAI-Hugging Face hack as dangerous agents “going rogue,” this incident is leading teams everywhere to engineer better sandboxing and monitoring – important changes that are making AI safer. (On OpenWorker, our agent harness that supports security workflows, Rohit Prsad, Devika Verma and I are building on Nvidia's new safety framework to improve sandboxing.)
In aviation and in software, we improve safety by repeatedly finding problems, fixing them, and further identifying and addressing root causes that reduces the odds of future problems. Frontier models are speeding this process up.
The advanced cyber capabilities of open weight models do give attackers a window to do damage. Long term, defenders have the advantage, because they have more information with which to discover and mitigate vulnerabilities. The key is to carry out this important and difficult engineering work as quickly as we can, so we can all come out the other side with safer, more robust software.
Keep building!
Andrew
