Agents Come to Speech Recognition: AgenticASR incorporates user corrections to edit speech recognition on the fly

Most speech-to-text systems transcribe speech in a single pass, which makes them unable to correct errors in their outputs. Researchers built a system that allows for interactive corrections.

What’s new: Zixuan Jiang, Yanqiao Zhu, Peng Wang, and colleagues at Shanghai Jiao Tong University, Zhejiang University, Fudan University, and Xiaoice, a Microsoft spinoff, devised a workflow that pairs an automatic speech recognition (ASR) engine with an LLM that attempts to detect and correct transcription errors at every conversational turn. They call it Agentic ASR.

Key insight: Automatic speech recognition systems often manage mistakes by pairing a speech-to-text model with an LLM that rewrites its output. But correcting that system simply appends the user’s request for a correction to the existing transcription. Asking an LLM to rewrite a transcript from scratch risks introducing new errors, especially where the words spoken are idiosyncratic (for instance, the name Megan instead of Morgan). The LLM can more successfully revise the transcription if it divides the correction process into three steps: find the error, understand what the user said, and apply a fix. This approach enables the LLM to act more like an editor than a rewriter.

How it works: Agentic ASR treats transcription as a multi-turn process of refinement, in which users can dictate and then confirm or correct the transcription in further turns.

• Given input speech, a speech-to-text engine (Qwen3-ASR-1.7B) generated its best guess at what the user said, and an LLM (Qwen3-32B) refined the guess using the transcribed output so far.
• The LLM classified the input into one of three intents: (i) confirmation (the user accepted the current state), (ii) new input (the user wanted to add to the transcription), or (iii) correction (the user wanted to alter the transcription).
• Confirmation or new input didn’t change the earlier transcription, but correction did. In that case, the LLM identified the span to edit, decided what the user wanted to change (for example “No, I meant Megan, not Morgan”), and applied the edit.

Results: The authors evaluated the workflow on multilingual speech-to-text benchmarks. Such benchmarks don’t include examples of corrections to output transcripts, so the authors used an LLM and text-to-speech engine to simulate a user dictating and then correcting the output text.

• The authors proposed a metric called S²ER that measures the proportion of transcription attempts that fail to preserve the speaker's intended meaning, which Qwen3-32B judged based on a ground-truth transcription. They compared the S²ER at 0 turns (equivalent to comparing against Qwen3-ASR-1.7B directly transcribing the speech) to the S²ER after the simulated user interacted with their system for 10 turns. Multi-turn interaction consistently reduced semantic errors across all benchmarks. Most gains appeared in the first few turns.
• On the GigaSpeech benchmark, the authors’ method improved S²ER from 21.5 percent at turn 0 to 3.5 percent after 10 turns, while word error rate improved modestly from 11.9 percent to 10.4 percent.
• On AISHELL-NER, which includes lots of names, dates, and the like, the authors method dramatically improved S²ER from 19.9 percent to 2.0 percent while cutting the Named-entity Error Rate from 2.4 percent to 1.2 percent.
• On ASRU2019, whose examples mix languages (for instance, English and Mandarin), the authors’ method decreased S²ER from 28.6 percent to 1.4 percent and improved Mixed Error Rate (which combines word error rate for English with character error rate for Mandarin) from 6.6 percent to 3.3 percent.

Why it matters: Many systems that accept voice input need to understand the user’s intent, not every word spoken. Agentic ASR does this very well, and when every word counts, it makes speech-to-text more robust in a natural way. This capability gains importance as UIs that depend on voice input become more widely used.

We’re thinking: The workflow of decomposing corrections into identifying where to edit, understanding what to change, and applying the change looks like a potential blueprint for interactive tasks such as editing documents, reviewing code, and iterating on a design.