Dear friends,

A key AI engineering skill is using coding agents. Your skill at steering them both to write code and to carry out non-code tasks, such as analyzing data or managing system operations, allows you to get a lot more done.

The rapid pace of evolution for coding agents means this skill, too, is evolving rapidly — faster than other top-level AI engineering skills. Proprietary agents (like Claude Code, Codex, and Cursor) and open agents (like OpenCode and Pi) progress in strides via both harness and model improvements. So keeping up with how to use coding agents requires a continuous process of experimentation, building, and learning.

In interviewing dozens of top AI Engineers and reflecting on our own team’s use of coding agents, we found a consistent high-level workflow for building software with them. The key steps are:

• Planning. This includes (i) brainstorming, which may include research, experimentation, and understanding the existing codebase (if any) and (ii) writing a spec that captures requirements, technical design, and architecture, followed by generating an execution plan. You might also review the plan to interrogate key assumptions and check for security, overengineering, and other gaps.
• Execution, where you build, test, and verify, with the right balance between agent autonomy and human oversight. This involves (i) having the agent build the software, with a calibrated level of agent autonomy and (ii) verifying its output via automated and/or human checks.
• Deployment and monitoring, in which you (i) deploy, perhaps gated with a CI/CD pipeline or additional human gates, and (ii) use agents to watch logs, surface issues, and propose and execute improvements.

This high-level workflow is similar to the one typically used to build software before coding agents. Now, we focus much less on code and instead focus on deciding what to build, designing the architecture, writing the spec, and verifying outputs.

The duration of each step can vary significantly between projects, and steps can be omitted. For example, the spec for a greenfield (meaning built-from-scratch) prototype might be loosely described in a quickly written prompt, whereas the spec for a brownfield (pre-existing) project with many users might require much more effort to write and verify. Further, the workflow is highly iterative, and skilled developers know when feedback from a later step should lead them back to an earlier one. For example, if verification fails, they know how to steer the agent to rebuild and fix errors; or if monitoring surfaces issues, how to have agents update the system and redeploy.

To use coding agents effectively in this workflow, the key skills are:

• Directing the workflow
• Enabling agent autonomy
• Reviewing the work
• Customizing the agent and its environment
• Coding agent foundations

Directing the workflow. You know how to navigate each step of the workflow above. This involves deciding how much human and how much agent effort to spend on each and when to go back to an earlier step to iterate. It requires deeply understanding the tradeoffs of speed, cost, technical risk, and human effort, so you can decide how much to research and plan up front, when to retain human ownership over critical work, how to choose the architecture, how much detail to write into a set of planning artifacts (like a spec), and how to decompose the work into verifiable steps.

Enabling agent autonomy. When applying a coding agent to the steps in the workflow, you choose the autonomy level: Do you watch it and go back-and-forth interactively or delegate a larger chunk of work to it? And when do you set a clear goal and have it loop until it succeeds? Additionally, you have to manage the context carefully for the agent. As the build proceeds through different phases, you will calibrate when to make sure key learnings, user feedback, and assumptions — including assumptions that changed partway through the build — are captured for the agent to use downstream. Additionally, you will decide when to set up many agents to run in parallel on a decomposition of the task — either by having a human or a higher-level agent orchestrate these other agents — and how to manage human attention across concurrent agent sessions. You also know how to run agents safely, setting permissions and gating actions appropriately to let development proceed quickly while limiting the risk of leaks, data loss, or other damage.

Reviewing the work. The output of a coding agent is uncertain. We don’t know in advance what good ideas it might come up with and what bugs it will implement. Reviewing and verifying the output is a key step to ensure you are getting the result you want and to redirect the agent if not. You will design testing and validation that is matched to the task, applying both behavioral and functional verification as needed. You might also test user flows, perhaps having an agent provide screenshots as evidence of success or failure. For qualitative/behavioral evaluation, eval sets, perhaps with LLM-as-a-judge, can be used.

You also need to decide how much of these tests should be automated. Some workflows will have all testing and validation fully automated so the agent can check its work and know when it has succeeded in completing a task. You have to evaluate the tests to ensure they correspond to your aims, and you will evolve them if not. Additionally, you use agentic code review and run AI-enabled security and architecture audits. When AI review isn’t sufficient, you judiciously insert human reviews of the code behavior (and, infrequently, of code as well) while exploring how to automate this review further. Finally, you verify deployment and can operationalize monitoring and incident management with agents.

Customizing the agent and its environment. Your ability to update both the agent and the environment it works in allows your agents to efficiently get the context they need, access tools, and build correctly and efficiently. You know how to integrate agent skills, plugins, and MCP servers. Occasionally you will prune them when they are no longer necessary (such as when a new model obviates an old skill). You can use hooks to automate repeatable parts of the development process, like triggering automated code reviews or CI/CD pipelines. You can also maintain the environment the agent works in: updating the standing context (such as AGENTS.md or CLAUDE.md) with information on the codebase, key architectural assumptions, code style, and data access patterns. You know how to preserve state across multiple sessions and across parallel agents, and accumulate agent learnings over time, perhaps by running post-run retrospectives to capture what did and did not work. You also know how to set up consistent conventions and structure to make your codebase navigable to the agent, and how to occasionally clear out agent-generated debt. When you work in a team, you consider how to coordinate context across different developers’ agents.

Coding agent foundations. Finally, to make good decisions throughout, you have a good understanding of how coding agents work: how they carry out codebase search/retrieval, how they manage their context windows, how different operations (like adding tool calls, MCP servers, etc.) affect context, how agents and subagents interact, and how the agent is built by wrapping a harness around an LLM. This makes the agent less of a black box and helps you to recognize failure modes, such as overengineering a simple solution, losing rigor because the agent lacks an explicit verification process, stopping short of the goal, or agent actions that risk destruction of files or production data. It also helps you reason about the agent’s state and steer it by giving it the right prescription or context. And when monitoring a run, this understanding allows you to better spot when the agent goes off-track and requires your intervention.

I find that social media often gives oversimplified descriptions of how to use coding agents. For example, it is sometimes useful to get agents to run autonomously for hours and burn millions or tens of millions of tokens. But currently the practical utility of very long-horizon tasks — especially relative to their cost — has been amplified beyond reality. Instead, most effective coding agent use is a complex, highly iterative process, and being able to intervene with high-skill judgement gives much better results.

Your skill at using coding agents will make you an effective builder. This positions you to also steer the overall build. I will say more about this in next week’s letter. 

Keep building!

Andrew

A MESSAGE FROM DEEPLEARNING.AI

Most advice about coding agents stops at "give it better context." In Spec-Driven Development, built with JetBrains, Paul Everitt teaches the method: project constitutions, a feature spec for each change, and a plan-implement-verify loop. Join in for free

News

Comparing OpenAI and Anthropic’s Data Retention Policies

Anyone who sends private data to an AI model or service deserves a clear answer to three questions: (i) Where do they store it? (ii) How long do they keep it? and (iii) What, if anything, will they do with it (such as train on it, show to humans, etc.) This week, business customers for two leading companies got two responses: one leaked, then confirmed, and another previewed. Technical details are scarce for both. 

What’s new at Anthropic: Since June, businesses using Claude Fable 5 have been required to let Anthropic keep their conversations for 30 days. Now Anthropic will soften that rule, with a new program called Enterprise Frontier Safeguards (EFS) that will require companies with zero data retention (ZDR) policies to keep the data on their own servers or those of specific cloud providers instead. Until EFS is available sometime this fall, eligible enterprise customers will be able to use Fable 5 and Fable 5.1 without Anthropic retaining any data. 

Meanwhile at OpenAI: One day before Bloomberg broke the Anthropic news, OpenAI published a post promising that for business customers, its most capable models come with ZDR. This means OpenAI never logs a business’s prompts and replies. It’s offered to approved businesses that plug the models into their own software. The post also previewed Private Safety Processing (PSP), a system meant to spot misuse spread across many requests without OpenAI reading the prompts.

How it works: Both companies describe the same method: software watches requests over time and flags patterns. No human review is required. Here is what neither explains: “Our employees can’t see it” is not the same as “our systems can’t see it.” To scan the data, software the company wrote has to unlock and read it, wherever it is stored. Sholto Douglas, a member of Anthropic’s technical staff, described the plan on X as monitoring “done via automated systems we provide to you.” Here’s where each company stands.

• In force today: Anthropic keeps Fable 5 and Mythos 5 conversations for 30 days on every platform, per its help center. Further, content that the company flags according to a process set at its discretion can be kept up to two years under its general policy, and reviewers Anthropic approves can read it through a logged process. OpenAI says its staff can’t see ZDR customers’ conversations, except where required by federal law.
• What’s been promised: Anthropic will require data to be retained for the same 30 days, but stored on the customer’s servers — either their own or through a third-party provider such as Google Cloud, Microsoft Foundry, or Amazon Web Services. (Disclosure: Andrew Ng is a member of Amazon’s board.) The 30 day requirement will stay, but the data will sit on the customer’s designated servers instead of Anthropic’s, and customers have the option to manage their own encryption keys, audit logs, and other security data. OpenAI’s Private Safety Processing will scan data held on the customer’s servers or encrypted on OpenAI’s with keys OpenAI says its staff doesn’t have, and report back only a label for the type of activity, not the content.
• Why both companies say they need this: The primary threats EFS and PSP are designed to address are cyberattacks. Some attacks show up only across many requests. Anthropic’s help center cites spying by governments and “best-of-N” jailbreaking, where an attacker rewords a blocked request hundreds of times until one version gets through.
• Undisclosed: It remains unclear how either company’s software reads data they claim they can’t see. OpenAI has promised a technical paper for PSP in September; Anthropic published one for its June data retention policy but has given few technical details about how EFS would work. Neither company has disclosed the specific criteria that they apply to consider something a “cyberattack” or “unsafe.”

Behind the news: Anthropic’s June rule forced an exception to the ZDR contracts some business customers already had. These customers had to switch retention on to use the new models at all. Consequently, many businesses declined to use Fable 5. As we reported in June, the ARC Prize Foundation also declined to run its verified tests of Fable 5 rather than expose its private test questions. Anthropic has since acknowledged the cost; in an August risk report it wrote that the rule would “be unpopular with customers who have come to expect zero retention” and could hurt its business if rivals didn’t do the same. OpenAI has not; its post reaffirmed zero retention.

Why it matters: Businesses are deciding now whether to trust these models with sensitive data based only on press releases. For regulated industries like law firms, hospitals, and banks, “we won’t train on your data” and “we don’t have your data” are different promises. Businesses also lack clarity on when Anthropic may change its definition of when someone is acting "unsafely" and may have their prompts — which often contain extremely sensitive data — read by Anthropic employees. And any data that can be found on a server can be compromised by an attacker or demanded by a court. Last year, in The New York Times’ copyright suit, a court ordered OpenAI to preserve chat logs it would normally have deleted, even ones users had erased. OpenAI said ZDR customers were unaffected because it never had their data. Whether logs held on a customer’s servers could be reached in a suit against the AI company is a question neither company has addressed. Neither has published an independent audit of its design. 

We’re thinking: In 2024, we described four levels of cloud AI privacy, and the strongest, where the provider cannot access your data at all, is the level that matters most for sensitive work. This is especially true given loopholes that give frontier labs discretion to apply whatever standard they wish to define what is “safe.” We contrast this with the simple privacy policies of many hyperscalers, where rules are much clearer: We expect them not to look at our data unless a warrant, court order, or other more predictable legal process requires them to. Both companies now say they can preserve user privacy while still catching misuse across many requests. Neither has shown how. Until both of them publish their designs, that’s a roadmap, not a guarantee.

Ox Alpha Revealed as GLM-5.3-Flash

For over a week, the name and maker of the most-used model on OpenRouter remained unknown. Last week, it was publicly announced to be a new GLM series model — and in a surprise to many, the company says it served the model’s free, high-volume preview exclusively with Chinese-made chips. Now anyone can download its weights.

What’s new: Z.ai released GLM-5.3-Flash, a vision-language model it had previewed under the name “Ox Alpha.” It’s the company’s first vision model since April’s GLM-5V-Turbo, and the first model in the GLM-5 family whose vision capability was built from the start rather than added to a language model afterward. 

• Input/output: Text, images, and video in (up to 1,048,576 tokens), text out (up to 128,000 tokens, 44.6 tokens per second)
• Architecture: Hybrid mixture-of-experts transformer that combines linear and sparse attention, 320 billion parameters total, 18 billion active per token
• Features: Adjustable reasoning levels (low, high, and max, the default), reasoning cannot be deactivated, streaming output, tool calling, context caching
• Performance: Achieved 57 points on Artificial Analysis’ Intelligence Index; third among all models and best among open weights models on Artificial Analysis’ GDPval-AA v2 (real-world knowledge work tasks)
• Availability/price: Via GLM Coding Plan subscriptions ranging from $18 to $168 per month, API at $0.15/0.03/$0.50 per million input/cached/output tokens
• Weights/license: Free to download under the commercially permissive MIT license
• Undisclosed: Knowledge cutoff, pretraining data sources

How it works: Z.ai trained GLM-5.3-Flash on text, images, and video from the start rather than melding vision and text transformers afterward. Unlike the larger GLM-5.3, Z.ai pretrained this model from scratch and redesigned the attention layers to handle long inputs more efficiently.

• A first in the GLM series, the model mixes two kinds of attention. Linear attention, a memory-efficient variant whose cost grows in proportion to input length, attends to nearby context, while sparse attention attends to the full context. The company says the combination cuts attention computation to roughly one-third of GLM-5.3’s, and less than DeepSeek-V4-Flash and Kimi K3.
• A step called IndexPool averages every four of the model’s lookup vectors into a single vector to reduce memory use as context approaches 1 million tokens. This, plus the hybrid attention mechanism, cut GLM-5.3-Flash’s key-value cache to less than a quarter of GLM-5.3’s, but remains higher than DeepSeek or Kimi’s best numbers
• The company pretrained the model on a 30-trillion-token multimodal corpus. It used Manifold-Constrained Hyper-Connections, a technique developed by DeepSeek that splits the typical layer-to-layer connection into several parallel connections and keeps those paths stable as they combine.The company says this helped the model scale efficiently.
• To generate training data, Z.ai had the model work in environments where it could render a frontend, a game, or a 3D scene, see the result, and revise. The company then trained the model on those attempts. For front-end work, Z.ai says it tried reinforcement learning that scored the model on how the rendered page turned out
• At inference, the model sends each token through 8 of 288 experts and uses around half the network layers (45) as GLM-4.5 (92). A multi-token prediction layer, which drafts several tokens ahead for the main model to check, speeds generation.

Performance: Independent evaluations place GLM-5.3-Flash just below the top open weights models, while costing roughly an eighth per task of the open weights models just above it. Leading proprietary models cost between 10 to 35 times per task. GLM-5.3-Flash leads all other open weights models tested on one evaluation of real-world work and completes long-running coding tasks nearly as well as the larger GLM-5.3, which costs 16 times more per task.

• GLM-5.3-Flash achieved 57 on Artificial Analysis’ Intelligence Index, a composite of nine evaluations of economically useful tasks, at an average of just $0.09 per task. At a fraction of the price, it approached the open weights leaders Kimi K3 and GLM-5.3 (both set to max reasoning, tied at 60, $0.84 and $0.68 per task, respectively). It matched Claude Opus 4.8 set to max reasoning ($2.03 per task), and outperformed Gemini 3.7 Flash (56, $0.40 per task).
• GLM-5.3-Flash (1,765 Elo) ranked third on Artificial Analysis’ GDPval-AA v2, a head-to-head ranking of models on tasks in economically useful fields, behind Claude Opus 5 set to max reasoning (1,824 Elo) or xhigh reasoning (1,797) and ahead of Grok GLM-5.3 set to max reasoning (1,758) and Grok 4.6 set to xhigh reasoning (1,755)
• GLM-5.3-Flash was able to one-shot 63 percent of problems posed by DeepSWE v.1.1, a test of software engineering skills whose coding tasks are written from scratch rather than taken from public repositories. On this benchmark it cost $0.24 per task, against 69 percent at $3.99 for GLM-5.3 and 74 percent at $11.84 for Claude Opus 5 set to max reasoning. DeepSeek V4 Flash, a comparably sized model, solved 53 percent at twice the cost.
• The model is verbose and slow, particularly for a model its size. The model used 150 million tokens to complete the Artificial Analysis’ Intelligence Index, more than the median (110 million). On multiple hardware providers, it generates an average of 45 tokens per second, significantly slower than GLM-5.3 (78 tokens per second) and the typical model tested by Artificial Analysis (69 tokens per second).

Behind the news: For a week before the launch, Z.ai introduced its preview of GLM-5.3-Flash anonymously, available free and exclusively on the coding harness OpenCode and on the model marketplace OpenRouter for a week before launch. This way, it was able to collect feedback from developers unaware whose model they were testing. The company says Ox Alpha became the most popular model on those services that week. Users speculated that the mystery model belonged to the GLM family within days based on its tokenizer outputs. On August 26, Z.ai confirmed the model and released its weights under a standard MIT license. Two days later, the company released weights for its flagship GLM-5.3 under a license similar to MIT but added a clause requiring any business whose revenue surpasses $10 billion to pass a security review by Z.ai before using the weights commercially. 

Why it matters: While GLM-5.3 still outpaces GLM-5.3 Flash (and virtually all open models) on text benchmarks, Z.ai’s cheap model is also the more advanced and versatile one, for now. GLM-5.3 is merely a highly capable fine-tune, while Flash received a new base, architecture, and vision capability. The company says its next flagship model will inherit this multimodal, hybrid attention architecture, while also training on more data and showing greater capabilities. Months ago, GLM-5V-Turbo outpaced Claude Opus 4.6 on vision-language tasks; the next GLM series model may similarly challenge top proprietary multimodal models.

We’re thinking: Ox Alpha’s anonymous preview created buzz and mystery but also allowed users to judge it on its merits (and deficits). Perhaps the biggest mystery revealed was its reliance on chips from China-based manufacturers. This shows that with the right memory optimization methods, companies can serve a cost-effective, high-performing model at scale on economically-priced hardware — albeit a somewhat smaller and slower model than we've come to expect from the cutting edge.

Custom Models for Law, News, and Finance

Thomson Reuters launched a proprietary large language model family called Thomson. It’s built on a Qwen model using a custom data engineering and re-training pipeline to provide more accurate and complete responses in fields such as law and finance than general-purpose LLMs.

What’s New: Thomson Reuters built Thomson as a domain-specific model that specializes in law, business, tax, finance, news, and related knowledge work fields. Thomson will be deployed first inside one of Thomson Reuters’ existing products, CoCounsel Legal, a tool for research, analysis, and drafting documents. It will be extended to other Thomson Reuters products at unspecified dates.

• Input/output: Text in (up to 262,000 tokens), text out
• Architecture: Mixture-of-experts transformer, 397 billion parameters, 17 billion active per token
• Performance: Thomson-1.0-Large narrowly outperformed GPT-5.4 and Claude Sonnet 5 on both completeness and factuality for tax, legal, and news content; rails both models on open web content on factuality by roughly 15 points; Thomson-1.0-Small similarly outperformed Gemma4-31B and Claude Haiku 4.5.
• Availability: Business customers only.
• Weights/license: Thomson-1.0-Large is proprietary, but a 35 billion parameter version, Thomson-1.0-Small, will be released as an open weights model on Hugging Face for academic and non-commercial use
• Undisclosed: Price

How it works: Thomson Reuters built Thomson-1.0-Large on top of Qwen3.5-397B-A17B, an instruction-tuned open weight model. It employed what the research authors call Continual Learning, a mix of full-weight mid-training and fine-tuning, on a selection of the company’s large data corpus. In a press release, the company disclosed that it invested $40 million in total training costs over three months.

• The authors re-aligned Qwen to better reflect the company’s style and values, including journalistic objectivity. They aligned the model to a constitution using direct preference optimization (DPO). The constitution is open source and can be adapted or revised for similar projects.
• For domain expertise, Thomson Reuters and its partner DatologyAI curated a 200 billion token mid-training dataset out of a 19 trillion token candidate pool. The training data is composed of three roughly equal parts: curated proprietary documents (news, regulatory filings, case law, contracts), synthetic data pairs of successful professional tasks, and select general-capability materials.
• The authors fine-tuned the model using DPO for a final round of alignment training and to increase accuracy and efficiency in agentic deep research. They also used group sequence policy optimization (GSPO) to manage context compaction and document caching, to avoid overfilling the context window or triggering redundant API calls.
• Thomson-Small-1 was trained in a similar manner using Qwen3.6-35B as a base.
• The company touts its data-centric pipeline and mix of alignment, mid-training and post-training methods. It says future versions of Thomson may use an alternate base model — possibly Inkling Large — and will train on a larger data set than 10 percent of the Thomson Reuters corpus.

Behind the news: Thomson Reuters has long used AI in its products for legal and financial professionals and government agencies, but the company’s latest AI-powered product is its first LLM. The company is under pressure from both new AI-native companies like Harvey and Legora and general-purpose LLMs and agents from OpenAI, Anthropic, Google, and others that offer alternatives to many of its products.

• Thomson is less narrowly focused than Harvey’s new legal model Tenet and boasts a larger data corpus that includes news, business, tax, and finance. Thomson Reuters also said Thomson is less narrowly focused on performance on specific benchmarks, including Harvey’s popular BigLawBench and LAB (Legal Agent Benchmark).
• CoCounsel Legal is an AI assistant that draws on proprietary information, as well as users' own documents, to perform legal tasks such as researching questions, comparing documents, and creating timelines of events. Thomson will now be the model that powers some features within CoCounsel, while others will rely on proprietary models, including Anthropic’s Claude. The cost of CoCounsel’s various tiers is not publicly available, but the company claims the product has over one million customers.

Why it matters: The Thomson paper authors argue that their approach to domain-specific LLMs represents a special case of sovereign AI, but at the corporate rather than the national level. Instead of yielding data to a larger AI company and being subject to its infrastructure and data retention policies, Thomson Reuters licenses its models to companies that run them on their own hardware, ensuring that sensitive data never leaves the premises. Also, its models can be further trained on a company’s private data using Thomson’s midtraining methods, without the high costs of pretraining or the relatively limited effectiveness of an inexpensive fine-tuning method like LoRA.

We’re thinking: Model customization remains an exciting prospect for a wide range of AI applications, and the growing availability of open weight models creates an important foundation for teams to experiment with them. Still, companies like Thomson Reuters will have to prove to customers that their products are worth the price given the relatively small improvement in performance.

A Simpler Method to Monitor Models

LLM safety monitors that act during generation often analyze a series of safety scores to catch incorrect or harmful outputs. Researchers found that analyzing a single safety score achieved nearly the same performance. 

What’s new: Mona Schirmer, Metod Jazbec and colleagues at University of Amsterdam, University of Wisconsin-Madison and Johns Hopkins University introduced a simple monitoring system called CRC Monitor — CRC stands for conformal risk control. It evaluates an output’s safety by comparing a single safety score against a threshold that is carefully calibrated for the task at hand.

Key insight: Systems such as e-valuator score LLM output after every reasoning step, conversational turn, or tool call and analyze the history of scores to decide whether to stop. A simpler approach is to stop as soon as the latest step’s score falls below a threshold. The challenge is choosing this threshold reliably. A false-alarm rate measured on a limited validation set may be lower than on new data, simply by chance. Therefore, one needs to choose the threshold more conservatively.

How it works: The authors calibrated CRC Monitor’s threshold for two tasks in which a model should be stopped when its generation goes wrong: incorrect mathematical reasoning and harmful subject matter. For each dataset, they used a separate model, called a verifier, to score the output text at every step. They found thresholds that satisfied two criteria: (i) catch as many unsafe outputs as possible and (ii) keep the number of false alarms below a user-defined limit.

• For mathematical reasoning, the authors built a calibration dataset. They generated solutions to the MATH dataset using Anthropic Claude Haiku 4.5 and Mistral-7B-Instruct. A verifier (Alibaba Qwen2.5-Math-PRM-7B) assigned scores by estimating whether the reasoning remained on track to produce a correct solution. They used OpenAI o3-mini to validate each solution, creating the labels for calibration.
• For harmful subject matter, they evaluated on the Anthropic Red Teaming dataset, which contains conversations designed to elicit harmful responses from LLMs, and FineHarm, which contains multi-turn conversations annotated for harmfulness. As verifiers, they used Llama Guard on Anthropic Red Teaming and a Qwen2.5-1.5B model fine-tuned to detect harmfulness on FineHarm.
• The authors tried many thresholds on the validation sets and measured how often each one erroneously stopped safe outputs (a higher threshold meant more outputs were marked unsafe). Since the actual false-alarm rate may be higher on real data than validation data, they added a small padding number to the measured false-alarm rates, then selected the largest threshold such that the modified false alarm rate was below the user-defined limit.
• During inference, when the verifier score dropped below the threshold, the system stopped generating output.

Results: CRC Monitor flagged unsafe responses faster and maintained the user-requested false-alarm rate while matching the performance of e-valuator in most experiments.

• On the MATH dataset, at a user-defined false-alarm rate of 20 percent, the system detected about 80 percent of incorrect solutions generated by Mistral-7B-Instruct, equal to e-valuator’s 80 percent. It raised alarms after about 35 percent of the reasoning process on average compared to e-valuator’s 40 percent.
• Paired with the more mathematically capable Claude Haiku 4.5, CRC Monitor detected about 75 percent of incorrect solutions, nearly matching e-valuator's 76 percent; it raised alarms after about 40 percent of the reasoning process on average versus e-valuator’s 49 percent.
• On the FineHarm dataset, at a user-defined false alarm rate of 20 percent, CRC Monitor detected almost 99.5 percent of harmful outputs, and it raised alarms after about 14 percent of the conversation on average, matching e-valuator on both counts.
• On Anthropic Red Teaming, CRC Monitor detected fewer harmful conversations (32 percent versus e-valuator’s 54 percent), but it raised alarms much earlier, after about 26 percent of the conversation on average instead of 55 percent.

Why it matters: Stopping LLMs from generating undesirable output can be done more simply and less expensively. An effective monitor that catches such output as it’s being generated requires only a safety score for the latest generation, a calibrated threshold, and a predetermined tolerance for errors. This approach detects problems earlier than a competing method, reducing the user’s exposure to mistaken or objectionable output (and some inference cost). The authors suggest that their approach lends itself to a tiered monitoring architecture that generates a cheap signal continuously, stops according to a calibrated rule, and shifts to more-expensive verifiers only when the cheap signal indicates a poor output. 

We’re thinking: Since a simple safety-detection method can work as well as a complicated one, improving the verifier behind it may lead to greater improvement in performance than designing more complex ways to weigh the results.