Dear friends, 

The gulf between LLM performance on verifiable tasks vs. unverifiable tasks that require judgement is larger than most people realize. Verifiable tasks, like coding or solving math problems, have a right answer that can be checked automatically. Judgement tasks — like how to respond to a customer complaint, how to organize data collection for your AI project, what product feature to add, or how to form business strategies — don’t have unambiguous, ground-truth, correct answers. But they do have better and worse answers.

We have much better techniques for training AI systems to do well on verifiable tasks. As a result, progress on performing such tasks has been much faster than for judgement tasks. In the case of LLMs, RL (reinforcement learning) has been a huge driver of progress. We can set up RL gyms, or simulation environments, in which we let an LLM practice different ways to approach a task; then we automatically check which approaches are more successful (as measured by a reward function) and tune the model’s parameters to take the successful approaches more often. 

For supervised learning tasks, too, verifiable tasks are easier to improve performance on. We can obtain unambiguous labels for verifiable tasks, either via automated labeling or by asking human labelers. Defining the right label for a given output in judgment tasks is much harder.

Furthermore, learning theory shows that machine learning algorithms can learn faster for tasks where it is possible to get to 0% error (called realizable tasks in the literature) than ones where the optimal error (called Bayes error) is non-zero.

It’s true that applying supervised fine-tuning to LLMs can help them learn to perform unverifiable tasks, and progress is being made here too, but data collection and eking out incremental progress is harder and slower. Intelligence is jagged, and we do have a recipe for improving AI’s capability for judgement tasks: We can collect training data and train on a specific domain, laboriously craft reward functions that capture our goals, and so on. For the subset of applications where this is valuable, businesses will invest to get this work done faster.

Society also has a long history of looking at AI’s performance on a subset of tasks and overgeneralizing its capabilities. We saw this when AI beat the human world chess champion (Deep Blue), won Jeopardy (Watson), and excelled at Go (AlphaGo). Each of these moments led to hopes about AI’s immediate applicability to other tasks that turned out to be over-optimistic. Further, we can more easily evaluate performance on verifiable tasks (since test sets are easier to create) than unverifiable ones, which creates a more optimistic impression of progress than reality can justify.

To be clear, there is a lot to be optimistic about in AI — including progress on verifiable tasks and judgement tasks! But the tremendous progress on verifiable tasks, coupled with hype around AGI and RSI, is leading people to overgeneralize to AI’s ability to do judgement tasks well. In the long term, I remain bullish about AI’s ability to do this well too, but a lot of work remains ahead.

One interesting challenge for AI Engineers is that, as verifiable tasks get automated, the need for humans to do judgement tasks — which often requires drawing context from a bewilderingly large array of raw data and identifying the pieces that critically influence a decision — remains important. As I’ve written before, I expect people to continue to have a context advantage over LLMs on many tasks for a long time.

Keep building!

Andrew  

A MESSAGE FROM DEEPLEARNING.AI

In a fast-moving field, give your team skills you can trust. We analyzed 10,000+ job roles to find out which AI engineering skills matter most. DeepLearning.AI for Business gives your whole team a structured path to those skills. See how it works

News

Claude Fills Out Its Model Family

Claude Haiku, Anthropic’s fastest and least expensive model gets a long-awaited update. Meanwhile, the new version of Claude Sonnet nearly matches Claude Opus 5.5 on an index of independent benchmarks, but only at a reasoning level that uses more tokens per task than any other model tested. 

What’s new: Claude Haiku 5.5 replaces the year-old Claude Haiku 4.5. Independent tests show marked improvement, even with a steep price. Two weeks earlier, Anthropic launched Claude Sonnet 5.5, which the company says is best suited to routine coding and office work, including fixing bugs and producing documents, slides, and spreadsheets. It replaces Claude Sonnet 5 at the same per-token prices.

• Input/output: Text and images in (up to 1 million tokens), text out (up to 128,000 tokens; 237 tokens per second for Claude Haiku 5.5, 127.5 tokens per second for Claude Sonnet 5.5)
• Features: Adaptive thinking with five reasoning levels (low, medium, high, xhigh, max; defaults to high via API and medium via Claude Code and Anthropic’s apps), tool use, computer use
• Knowledge cutoff: June 2026
• Performance: At launch, Claude Haiku 5.5 scored a 43 on Artificial Analysis’ Intelligence Index, placing it 13th among available models. Likewise, Claude Sonnet 5.5 was first on Artificial Analysis’ Coding Agent Index (68) and second on Artificial Analysis’ Intelligence Index (56) and Vals AI’s Vals Index (67.04 percent) at launch
• Availability/price: Free and paid plans on Claude.ai (web, iOS, Android) and Claude Code; Claude Haiku 5.5 $.0.10/$0.01/$0.50 via API per million input/cached/output tokens (all prices are 5x if they exceed 100k tokens); Claude Sonnet 5.5 via API $2/0.10/$10 per million input/cached/output tokens; 50 percent off for batched requests
• Weights/license: Proprietary
• Undisclosed: Parameter count, architecture, training data and methods

How it works: Anthropic disclosed few details about how it built Claude Haiku 5.5 or Sonnet 5.5 beyond their reasoning controls and safeguards.

• Anthropic trained the models on a mix of public web data, public and private datasets, synthetic data, and user data from feedback, bug reports, and users who permitted training. After training, the company fine-tuned the model to align with values it defined using a constitution.
• Adaptive thinking, in which the models decide whether and how long to reason before it responds, is on by default. Developers using the API can set their reasoning level with a parameter called effort, which governs all output tokens, including reasoning, response text, and tool calls. At lower levels, the models reason less or not at all on simple problems and make fewer tool calls.
• Classifiers screen requests for risks involving cybersecurity, biology, weapons, and the development of frontier AI models. For cybersecurity, three components decide whether to block a conversation: a probe of the model’s internal activations, a lightweight classifier that runs on Claude Sonnet 5.5 itself, and a separate language model trained to classify requests. Anthropic warns that users should expect more refusals than with Claude Sonnet 5. Blocks related to biology and weapons have no fallback.
• Anthropic says Claude Sonnet 5.5 is the first Sonnet model to launch with classifiers that block attempts to extract its reasoning without falling back to another model, an attempt to make distillation (training a model to imitate another model’s outputs) harder. The reasoning blocks it returns work only in the account that generated them, and the API can reject them if earlier parts of the conversation change.

Performance: Claude Haiku 5.5 at max effort outperforms DeepSeek V4.1 Flash and all other “fast” models, but behind the top models. The latest Haiku version is also the fastest model Artificial Analysis has tested by tokens per second. Claude Sonnet 5.5 set to max effort ranks first or second on three leaderboards from two independent testers. But at every effort setting, another model matches or outperforms Sonnet’s Artificial Analysis score at a lower cost per task.

• Claude Haiku 5.5 (43) ranks 13th on Artificial Analysis’ Intelligence Index, a composite of 10 tests across mathematics, science, coding, and reasoning, ahead of all the “fast” models. Claude Sonnet 5.5 set to max reasoning (56) ranks second. Sonnet 5.5 trails Claude Opus 5.5 set to max reasoning and ties Claude Opus 5.5 set to xhigh reasoning. But to get there, Claude Sonnet 5.5 generates about 193,000 output tokens per task, roughly 60 percent more than Opus, and the most that Artificial Analysis has measured. All three Claude 5.5 models (Opus, Sonnet, and Haiku) are the top three on this metric, with Opus and Sonnet first and second in cost per task.
• On Artificial Analysis’ Coding Agent Index, which tests combinations of models and harnesses on three coding benchmarks, Claude Sonnet 5.5 set to max reasoning running in Claude Code ranks first (68), ahead of Claude Opus 5.5 in Claude Code and Gemini 4 Argon in Antigravity.
• On Vals AI’s Vals Index, eight finance, coding, legal, and tax benchmarks weighted by each sector’s share of U.S. gross domestic product, Claude Sonnet 5.5 set to max reasoning ranks second (67.04 percent accuracy), behind Gemini 4 Argon set to high reasoning (68.9 percent) and just ahead of Claude Opus 5.5 set to max reasoning (66.97 percent).
• Both models are high-speed but also high-latency. Claude Haiku 5.5 generates 237 tokens per second, but takes 337 seconds on average to produce its first answer token. Claude Sonnet 5.5 set to max reasoning generates 127 tokens per second, versus 97 for Claude Opus 5.5 set to max reasoning and 58.3 for GPT-6.1 Sol set to max reasoning, but Sonnet’s average time-to-first token is 430 seconds.

Behind the news: Anthropic launched Claude Haiku 5.5 on October 7, Claude Sonnet 5.5 on September 28, and Claude Opus 5.5 on September 22, At launch, Opus 5.5 led Artificial Analysis and Vals AI’s overall indices. Even with the later releases of Gemini 4 Argon and Muse Wonder and Muse Wonder Full, Claude Opus 5.5 and Sonnet 5.5 remain the top models on independent benchmark indices measuring overall intelligence.

Why it matters: Per-token prices alone don’t tell developers what a task costs. On Artificial Analysis’ Intelligence Index tests, Claude Sonnet 5.5’s cost per task varies more than eighteenfold across reasoning levels, and at max, it costs more per task than Claude Opus 5.5 set to max reasoning. On Vals AI’s tests, the cost is about two-thirds as much. Anthropic concedes Sonnet’s cost advantage over Opus is strongest at its lower reasoning levels. Claude Haiku 5.5, too, can be used as a reasoning model, but its cost advantages stand out for simpler tasks like summaries and classifications. Developers should measure each model’s cost at each relevant reasoning level on their own tasks, alongside accuracy and completion time, since each varies significantly.

We’re thinking: A higher reasoning level can change what an agent does, not just how long it reasons, and the change doesn’t always help. On FrontierCode, a test of whether an agent’s code changes are ready to merge without human edits, Anthropic says that Claude Sonnet 5.5 scores lower at max reasoning than at xhigh (46.2 percent versus 52.1 percent). Test your tasks on different effort levels to see what works best.

Mistral’s Sovereign Appeal

Mistral’s latest model is its largest and most capable, and the most powerful yet released by a European company. Its performance on independent benchmarks is comparable to GPT-6 Luna. It trails less-expensive China-based open weights models but performs better than all U.S.-based open models. It leads all models in finding and repairing a class of software vulnerabilities. 

What’s new: Mistral announced API preview access to Mistral Large 4, nicknamed “le Chonk.” Mistral says it will release the model’s weights by the end of this month. Meanwhile, selected companies, partners, and government agencies have access to a version with limited safeguards and extra cybersecurity capabilities.

• Input/output: Text and images in (up to 1 million tokens), text out
• Architecture: Mixture of experts, 1.05 trillion parameters, 52 billion active per token; 1.6 billion-parameter vision encoder
• Features: Reasoning (none or high via API), function calling, tool use, regional inference
• Performance: Highest-scoring model from outside China and the United States on Artificial Analysis’ Intelligence Index (38), first among publicly available models on Artificial Analysis’ CyberGym-E2E-AA, a test of finding and fixing memory-safety bugs (82 percent)
• Availability/price: Public preview via Mistral Studio API, 1.36/0.14/$4.18 per million input/cached/output tokens, half price for two weeks after launch, half price for batched requests
• Weights/license: Not yet released, license for weights not yet announced
• Undisclosed: Knowledge cutoff, output limit, architecture details, specific training data and methods

How it works: Mistral has said little about Mistral Large 4’s architecture but says it will share more when the model’s weights become available. The company says it pretrained the model from scratch and then fine-tuned it using supervised learning followed by reinforcement learning (RL).

• Mistral trained the model on 3,800 GPUs in its own data centers. The training data included publicly available web data, licensed datasets, and inputs and outputs from users of Mistral’s products who opted into data-sharing. The datasets spanned more than 160 languages, including all of the European Union’s official languages.
• A single RL run mixed many different types of tasks, such as single-turn chat, science problems, safety, factual accuracy, and long-running tool use. The tasks ran in environments that shared resources, including code sandboxes, web search, and APIs. Depending on the task, rewards came from unit tests, reward models, language model judges, or tests that inspect code without executing it.
• The RL run that produced the model’s API preview is still in process. Currently, many copies of the model attempt tasks in parallel, and post-training periodically updates the model at the same time rather than waiting until a round of attempts finishes.

Performance: The preview of Mistral Large 4 outscores every other model built outside China and the United States on Artificial Analysis’ popular Intelligence Index. It trails a handful of other open weights models, all from companies in China, but is ahead of all U.S.-based open models. The model preview’s strongest results come in cybersecurity tasks.

• On the Intelligence Index, a weighted average of 10 evaluations of math, science, coding, and reasoning, Mistral Large 4 set to reasoning (38, $1.13 per task) ties GPT-6 Luna (38, $0.07) and trails three cheaper open weights models: MiMo-V2.6-Pro (46, $0.13), GLM-5.3-Flash (42, $0.25), and DeepSeek V4.1 Flash set to max reasoning (39, $0.27).
• On Artificial Analysis’ Cyber Index, which measures models’ abilities to find and fix security vulnerabilities in code repositories, Mistral Large 4 set to reasoning (50 percent, $2.79 per task) ties GLM-5.3-Flash set to reasoning (50, $0.32) and trails only Grok 4.7 set to xhigh reasoning (56, $11.67) and MiMo-V2.6-Pro set to reasoning (56, $0.18) among publicly available models. The Daybreak Blue version of GPT-6 Sol at max reasoning, available only to organizations selected by OpenAI, tops the list with a 69 percent success rate at $1.77 per task.
• On one component test of the Cyber Index, CyberGym-E2E-AA (tasks related to memory-safety bugs in popular open-source C and C++ projects), Mistral Large 4 solved 82 percent of tasks, the highest score among publicly available models, behind only GPT-6 Sol Daybreak Blue at max reasoning (93).

Behind the news: Mistral’s flagship models have alternated between closed and open weights and have been sometimes more and sometimes less competitive with top proprietary models. The company introduced Mistral Large, a proprietary model, in early 2024. Mistral Large 2 followed in July 2024 with weights licensed for research and non-commercial use; at the time, Mistral Large 2 was competitive with OpenAI’s GPT-4 and Anthropic’s Claude Sonnet 3. In December 2025, Mistral released Mistral Large 3 under the permissive Apache 2.0 license, along with smaller Ministral 3 models, which the company built by repeatedly pruning and distilling the larger model.

Why it matters: The most capable open weights models on popular benchmarks are from companies based in China. Developers in the United States and Europe are trying to compete: In the U.S. Nvidia, Reflection AI, and Thinking Machines Lab, among others, have made progress, but Mistral remains the leading European lab. All of these companies pitch their models to organizations that want AI they can better control, whether via further post-training or greater trust they will be able to access the models regardless of geopolitics. Companies and governments all over the world, from South Korea to Saudi Arabia to Mexico, would prefer not to depend on either the United States or China for their access to intelligence. As Mistral CEO Arthur Mensch said earlier this year: “If you treat intelligence as electricity, then you just want to make sure that your access to intelligence cannot be throttled.”

We’re thinking: If you need AI sovereignty, and some industries and nations do, the best kind is running an open weights model on your own hardware. It doesn’t matter where a model was built (the U.S., Europe, China, or elsewhere) if your data and the model’s weights are entirely under your control.

Computers Built for Local AI

Developers have warmed to the economics and security of local models, either as an alternative or a complement to the cloud. Specialized computers from Apple, Microsoft, and Nvidia aim to take a bite out of the AI market with machines capable of running larger and more capable open models on-device. 

What’s new: Apple’s upgraded Mac mini and Mac Studio desktops enable developers and AI researchers to run and fine-tune local models faster than before, offering an alternative to cloud computing for certain tasks. Nvidia’s DGX Spark series and Microsoft’s Surface Ultra laptops target a similar market: customers who want enough power to build with AI on their local machines, but in a user-friendly package.

How it works: By avoiding per-token cloud or monthly LLM subscription costs, a local setup could potentially save developers money over time, particularly [those running always-on agents]. But genuinely capable hardware is expensive: A maxed-out Mac Studio with 256GB of memory, an 80-core GPU, and 16TB of storage costs $18,299. For those with more modest local agent needs, a Mac mini starts at $899.

• Developers and hobbyists have always used local machines to run (and in some cases train) smaller AI models. They’ve repurposed machines with expensive GPUs designed for gaming, video rendering, or cryptocurrency mining. Those are gradually being replaced by computers custom-built for local AI. Others build their own custom computers with server-grade GPUs and memory.
• Apple’s Macs use custom-designed chips that integrate general-purpose CPUs, GPUs, and specialized neural processing units (NPUs) for machine-learning workloads. Notably, they use Apple Silicon rather than Nvidia’s popular GPUs, and they run Mac OS X rather than Windows or Linux.
• Nvidia’s DGX Spark machines use a similar system-on-a-chip design, with Nvidia-designed CPUs, GPUs, and unified memory. Likewise, Microsoft uses Nvidia’s DGX Spark chips for its Surface Ultra laptops. Nvidia’s chips (whether in its own machines or Microsoft’s) max out at 128 GB of memory, with up to a 6144-core GPU. A top-of-the-line Surface Ultra laptop costs $5,899, but has a much smaller hard drive (only 1 TB) than the most expensive Mac Studio.
• Traditionally, memory has been a bottleneck to running AI models locally. Large models require huge amounts of data to be moved between the CPU and GPU. Apple’s and Nvidia’s unified memory architecture aims to address that bottleneck by giving the processors access to the same pool of memory, reducing the need to copy data back and forth.

Performance: Apple Silicon and Nvidia’s RTX Spark chips were recently tested by an independent website, but most scores are self-reported.

• The website Tom’s Hardware compared Microsoft’s Surface Laptop Ultra, a maxed-out MacBook Pro with an M5 Max processor, and AMD’s AI Ryzen Halo mini-PC. On the AgentPerf-Local benchmark, using a quantized version the popular open model Qwen 3.8-27B, the Surface Laptop Ultra was able to solve all problems within 18.9 minutes, faster than either the Mac (24.74) or the AI Ryzen Halo (42.1). The Surface Laptop Ultra with Nvidia hardware likewise outperformed Apple and AMD chips on AgentPerf-Local using Qwen 3.6 35B A3B (8.08 minutes to 10.12 and 13.29, respectively).
• Apple says that the Mac mini with an M6 chip delivers AI performance up to 4x faster and graphics and storage 2x faster than its previous-generated M4 Mac mini. The Mac Studio with an M5 Ultra chip has 4.3x faster AI performance and up to 1.8x faster graphics than the previous Mac studio M3 Ultra.
• Nvidia says its RTA Spark N1 chip gives comparable AI performance to an RTX 5070-class GPU. This gives it a processing power of 1 petaflop (1,000 teraflops) at FP4 precision, a common quantization for local AI models.
• In general, Apple’s CPUs perform better, but Nvidia’s GPUs outperform Apple’s. Nvidia chips also give those developing or fine-tuning their own models access to Nvidia’s CUDA software, which helps AI models run on any Nvidia GPU.

Behind the news: Professionals using local AI inference are still a small market compared to the cloud, but now large enough to have genuine competition for computers designed for them at a wide range of price points. Mac desktops have sold well since the emergence of open-source personal agents, with news outlets describing weeks-long waits for Macs with more than 24GB of RAM as early as February. Nvidia’s DGX Spark laptops and desktops aim to similarly serve Windows users. For Nvidia, entering the desktop chip market ensures it can continue to sell its products to both consumer and enterprise markets, but also to increasingly blur that distinction. For Apple, it’s an alternative way to capitalize on the AI boom rather than building and hosting models, one that plays to its strengths in manufacturing high-end consumer hardware.

Why it matters: On-device AI models have several advantages over cloud-based models besides cost: more customizability, privacy, data sovereignty, potentially lower latency, and the ability to run without an internet connection. Others see cloud models and local models as complementary, designing workflows where a more powerful proprietary model solves high-level problems, then hands more mechanical work to a low-cost local model.

We’re thinking: We argued back in 2023 that strong commercial interests would increase AI usage at the edge. Not every application needs a frontier model, especially if a mid-sized model is fine-tuned to the task. But the main attraction today for local AI are privacy and data sovereignty, which is increasingly important as AI models handle more sensitive workloads. It’s useful for all of us to have a wide range of options.

Data Mixing, Minus the Guesswork

When you’re building a fine-tuning dataset, finding a mix of subject domains that yields a high-performing model can require expensive trial and error, because fine-tuned performance also depends on the quality of the individual examples. A cause-and-effect approach aims to separate the two effects.

What's new: Researchers at Tsinghua University, Ant Group, and Renmin University of China, with Zinan Tang and Yukun Zhang as lead authors, built CausalMix, a method for choosing the share of training examples drawn from various domains regardless of their quality.

Key insight: Datasets can be analyzed according to their data mixture (the proportions of examples that represent different domains) and data quality (variations among examples in difficulty, reasoning complexity, and/or composition). Data mixture and data quality correlate, and finding their correlation in any given dataset makes it possible to differentiate their separate impacts on the performance of a model trained on the dataset. Given a dataset and a consistent training budget, to find the marginal gain of its data mixture — that is, its effect on a fine-tuned model’s performance — we can subtract the effect of data quality from the model’s performance overall. The best data mixture is the one that yields the largest marginal gain. 

How it works: The authors compiled 512 subsets of 100,000 examples each drawn from the Tulu 3 dataset. Each subset contained examples for instruction following, coding, math, knowledge, and safety in different proportions. In addition, the authors measured data quality with respect to difficulty, reasoning complexity, and writing clarity as computed by OpenDataArena-Tool. They fine-tuned a separate Qwen2.5-0.5B model on each subset and recorded each fine-tuned model’s performance on 10 benchmarks. Then they trained three models, all of them ensembles of decision trees: one to estimate the portion of each Qwen2.5-0.5B’s performance attributable to data quality, one to estimate the portion of a data mixture that correlates with the corresponding data quality, and one to represent the effect of the data mixture on fine-tuned performance.

• Given a subset’s data-quality metrics, LightGBM learned to estimate the portion of the fine-tuned Qwen2.5-0.5B’s performance attributable to data quality. The authors subtracted this output from Qwen2.5-0.5B’s performance overall to find the marginal gain attributable to the data mixture.
• The authors approximated the marginal gain in fine-tuned performance due to the data mixture as equal to the product of (i) the portion of the data mixture that doesn’t correlate with data quality and (ii) a variable that represented the data mixture’s effect.
• To estimate (i), they trained a second LightGBM. Specifically, given the data-quality metrics, the second LightGBM learned to estimate the portion of the data mixture that correlated with data quality. The authors subtracted this output from the actual data mixture to find the portion of the data mixture that didn’t correlate with data quality.
• To estimate (ii), they trained CausalForestDML, given a subset’s data quality metrics, to minimize the difference between the marginal gain and the product of the updated mixture and the model’s estimate of the variable.
• During inference, the authors: (i) randomly sampled 100,000 mixtures, (ii) estimated the marginal gain for each, (iii) computed the top 100 mixtures, and (iv) averaged them to produce the final mixture.

Results: The authors compared their method to various baselines, including distributing data across domains in equal proportions and Data Mixing Optimization (DMO), a method that automatically chooses fine-tuning mixtures based on minimizing loss. Fine-tuning on 800,000 examples, CausalMix outperformed every competing method tested on the 10 benchmarks used to select the mixtures.

• Fine-tuning Qwen2.5-0.5B, CausalMix achieved 32.81 percent on average, compared to DMO’s 32.04 percent.
• With the larger Qwen2.5-7B model, CausalMix achieved 62.28 percent on average versus DMO’s 60.35. The authors selected this mixture without repeating the 512 trial runs at 7B scale.
• To see if their method transferred to other datasets and models, the authors applied their method to Qwen3-4B and code and math problems from AM-Thinking-v1-Distilled-Code&Math, and tested across four other benchmarks. CausalMix (66.66 percent on average) outperformed the competing methods including DMO (63.47 percent on average).

Yes, but: It’s not clear how well the authors’ method extends across datasets and to other datasets. On a held-out subset of Tulu 3, CausalMix often underperformed competing methods. For example, fine-tuning Qwen2.5-0.5B on 800,000 examples from the held-out set, DMO (26.49 percent on average) outperformed CausalMix (25.04 percent on average).

Why it matters: The authors' method provides a rational approach to optimizing data mixtures rather than trial and error, and it guarantees that a good mixture will continue to apply as data or model size scales up. This makes it possible to find the best data mixture using smaller datasets and models and maintain the same proportions when scaling up without jeopardizing performance.

We’re thinking: The impact of any given training example on trained performance deepens on other examples. For instance, one example might be too hard until the model learns from another. The impact of data mixture reflects this principle. How well a model follows an instruction may depend on reasoning it learned from contemplating a math problem, and vice versa.