A Simpler Method to Monitor Models: CRC Monitor can tell if model output is incorrect or unsafe in real-time

LLM safety monitors that act during generation often analyze a series of safety scores to catch incorrect or harmful outputs. Researchers found that analyzing a single safety score achieved nearly the same performance.

What’s new: Mona Schirmer, Metod Jazbec and colleagues at University of Amsterdam, University of Wisconsin-Madison and Johns Hopkins University introduced a simple monitoring system called CRC Monitor — CRC stands for conformal risk control. It evaluates an output’s safety by comparing a single safety score against a threshold that is carefully calibrated for the task at hand.

Key insight: Systems such as e-valuator score LLM output after every reasoning step, conversational turn, or tool call and analyze the history of scores to decide whether to stop. A simpler approach is to stop as soon as the latest step’s score falls below a threshold. The challenge is choosing this threshold reliably. A false-alarm rate measured on a limited validation set may be lower than on new data, simply by chance. Therefore, one needs to choose the threshold more conservatively.

How it works: The authors calibrated CRC Monitor’s threshold for two tasks in which a model should be stopped when its generation goes wrong: incorrect mathematical reasoning and harmful subject matter. For each dataset, they used a separate model, called a verifier, to score the output text at every step. They found thresholds that satisfied two criteria: (i) catch as many unsafe outputs as possible and (ii) keep the number of false alarms below a user-defined limit.

• For mathematical reasoning, the authors built a calibration dataset. They generated solutions to the MATH dataset using Anthropic Claude Haiku 4.5 and Mistral-7B-Instruct. A verifier (Alibaba Qwen2.5-Math-PRM-7B) assigned scores by estimating whether the reasoning remained on track to produce a correct solution. They used OpenAI o3-mini to validate each solution, creating the labels for calibration.
• For harmful subject matter, they evaluated on the Anthropic Red Teaming dataset, which contains conversations designed to elicit harmful responses from LLMs, and FineHarm, which contains multi-turn conversations annotated for harmfulness. As verifiers, they used Llama Guard on Anthropic Red Teaming and a Qwen2.5-1.5B model fine-tuned to detect harmfulness on FineHarm.
• The authors tried many thresholds on the validation sets and measured how often each one erroneously stopped safe outputs (a higher threshold meant more outputs were marked unsafe). Since the actual false-alarm rate may be higher on real data than validation data, they added a small padding number to the measured false-alarm rates, then selected the largest threshold such that the modified false alarm rate was below the user-defined limit.
• During inference, when the verifier score dropped below the threshold, the system stopped generating output.

Results: CRC Monitor flagged unsafe responses faster and maintained the user-requested false-alarm rate while matching the performance of e-valuator in most experiments.

• On the MATH dataset, at a user-defined false-alarm rate of 20 percent, the system detected about 80 percent of incorrect solutions generated by Mistral-7B-Instruct, equal to e-valuator’s 80 percent. It raised alarms after about 35 percent of the reasoning process on average compared to e-valuator’s 40 percent.
• Paired with the more mathematically capable Claude Haiku 4.5, CRC Monitor detected about 75 percent of incorrect solutions, nearly matching e-valuator's 76 percent; it raised alarms after about 40 percent of the reasoning process on average versus e-valuator’s 49 percent.
• On the FineHarm dataset, at a user-defined false alarm rate of 20 percent, CRC Monitor detected almost 99.5 percent of harmful outputs, and it raised alarms after about 14 percent of the conversation on average, matching e-valuator on both counts.
• On Anthropic Red Teaming, CRC Monitor detected fewer harmful conversations (32 percent versus e-valuator’s 54 percent), but it raised alarms much earlier, after about 26 percent of the conversation on average instead of 55 percent.

Why it matters: Stopping LLMs from generating undesirable output can be done more simply and less expensively. An effective monitor that catches such output as it’s being generated requires only a safety score for the latest generation, a calibrated threshold, and a predetermined tolerance for errors. This approach detects problems earlier than a competing method, reducing the user’s exposure to mistaken or objectionable output (and some inference cost). The authors suggest that their approach lends itself to a tiered monitoring architecture that generates a cheap signal continuously, stops according to a calibrated rule, and shifts to more-expensive verifiers only when the cheap signal indicates a poor output.

We’re thinking: Since a simple safety-detection method can work as well as a complicated one, improving the verifier behind it may lead to greater improvement in performance than designing more complex ways to weigh the results.