Spice Dog

Hello, I'm Spice Dog.

While advancements in generative AI have significantly improved the accuracy of text comprehension, its output is heavily dependent on the quality of the input data. Speech recognition, in particular, exhibits considerable variation in quality compared to other forms of natural language input, due to factors such as speaking style, ambient noise, and recording equipment. How does this input variability affect subsequent tasks?

Against this backdrop, I came across an interesting paper on the topic of "speech recognition in the age of generative AI." First, I'd like to introduce its contents, and then, using its methods as a reference, I'd like to do a brief examination of it here as well.

The title of the paper is

Measuring the Effect of Transcription Noise on Downstream
Language Understanding Tasks

To put it loosely:

We measured the impact of transcription errors on the language understanding of generative AI!

That's the gist of it. The link to the paper is here.

This paper involves a great many experiments. To summarize it quite casually...

• There is no doubt that "errors" in speech recognition will affect downstream tasks involving generative AI.
• The magnitude of this impact varies depending on the type of task and the LLM model.
• On the other hand, the impact on downstream tasks cannot always be judged solely by the WER (Word Error Rate) of speech recognition.
• The impact varies depending on the quality of the speech recognition error.
• In other words, the WER value isn't the only metric for evaluating the quality of speech recognition.
• If the error can be corrected by LLM, then there might not be a problem with the output quality.

Now, I will explain the contents of the paper, but it will get long as it involves detailed numbers, so I will give you an overview of the blog here.

Overall picture of this blog
The paper demonstrates that even with minor errors at the word level, the generative AI can correct them effectively depending on the type of error, resulting in an acceptable quality for the final output. Although the study focused on English speech, similar results were obtained when evaluating Japanese conversational speech. In particular, a speech recognition engine that transcribes proper nouns with reasonable accuracy and does not significantly alter the interpretation seems to be well-suited to correction by generative AI.

1. Interpretation of the paper

1.1 The impact of the error rate (WER)

"Source: ACL 2025, Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks"

First, we will clarify the impact of the speech recognition error rate (WER) on the generative AI.
We will run a task using multiple LLMs and compare the quality of the output results. (From the following paper.)

Each LLM will perform a summarization task (QMSum), a question-answering task (QAConv), and a dialogue classification task (MRDA), and their performance will be evaluated (scored). The correlation between these evaluation values ​​and the Word Error Rate (WER) of speech recognition performed before the task will be examined.
LLMs used in the evaluation

Mistral
Llama3
Llama3.1
Gpt4oMini;

What is a Meeting Summary Task (QMSum)?
The QMSum task evaluation uses pairs of actual conference transcripts and human-created reference summaries as the dataset. The summaries generated by the LLM are evaluated using the ROUGE index and pairwise evaluation with GPT-4o-mini as the judge.
　ROUGE index (-1/2/L): Scores the degree of overlap between reference summaries.
　Pairwise evaluation: The GPT-4o-mini is given to read both the summary and the reference summary and asked, "Which summary is better?"
　The criteria are: (accuracy of content), (comprehensiveness of important information), (presence or absence of unnecessary information), and (conciseness).

What is a Question Answering Task (QAConv)?
The QAConv task evaluation uses a dataset of question-and-reference answer pairs created by human annotators based on actual conversation transcripts. The LLM takes the transcript and question as input, extracts the relevant short text spans, and generates answers. The generated answers are then compared with the reference answers to quantitatively evaluate the model's information extraction capabilities.

What is the Dialogue Behavior Classification Task (MRDA)?
The LLM is given dialogue behavior labels assigned by human annotators as options, and the LLM classifies the dialogue behavior it understands from each utterance in the speech recognition results into a label. There are 12 types of labels to be classified, including Yes/No questions, question word questions, alternative questions, agreement/acknowledgment, and statements.

Results and Discussion

• Meeting Summary Task (QMSum): The reliance on LLM is relatively low. If the error rate (WER) is low (0.2 or less in this case), the impact on summarization tends to be small regardless of LLM.
• Q&A Answer Task (QAConv): The acceptable WER varies significantly depending on the LLM. While GPT-4o-mini is clearly superior in error handling when the error rate is low, it tends to be surpassed by Llama3.1 as noise increases.
• Dialogue Behavior Classification Task (MRDA): Relatively insensitive to WER. Results were low across all models (low accuracy). It was difficult to accurately assess the impact of the error rate, possibly because the task was too difficult.

1.2 What kinds of speech recognition transcription errors should be corrected to improve the score?

The error rate evaluates all words with the same weight. Therefore, the error between "モグラ" and "メキシコ," and the error between "わたし" and "できます," are treated as having the same error rate. However, in post-processing of the generative AI, the impact can vary greatly depending on the word that is misidentified.
Here, we compared the impact on the output quality of the generative AI by changing which words were modified. (From the paper below.)

Policies and methods
We analyze the impact on downstream tasks by correcting specific types of words from transcription errors generated by speech recognition to their correct meaning.

The system compares the erroneous transcript with the correct text and corrects (cleanses) the parts of the discrepancies corresponding to the target word types using the content of the correct text. The target word types are listed below, and corrections are applied independently to each.

Nouns, verbs, adjectives, adverbs, content words (nouns, verbs, adjectives, adverbs), non-content words (particles, conjunctions), proper nouns

The contribution of cleaning was calculated using CES (Cleaning-Effectiveness Score, CES = Improvement in Task Score / √(Decrease in WER)).
The top 3 and worst 1 in cleaning contribution (CES) are

Meeting Summary Task (QMSum):
Proper noun (0.499) > Content word (0.479) > Noun (0.305) >>>>>>>>> Adjective (-0.023)
Question Answering Task (QAConv):
Proper noun (0.311) > Noun (0.211) > Content word (0.202) >>>>>>>>>> Adjective (0.071)
Dialogue Behavior Classification Task (MRDA):
Proper noun (0.735) >> Adjective (0.290) > Non-content word (0.285) >>>>>> Adverb (0.107)
*Content words: A general term for words that have a substantive meaning, such as things, actions, properties, and states.

Results and Discussion

• Correcting proper nouns has a very significant effect.
• Even with the same WER reduction, the final quality effect differs depending on what is being corrected.
• Even with the same WER improvement (WER 0.6 → 0.4 (20% reduction)), the effect of correcting proper nouns is enormous.

1.3 A paradigm shift: Rethinking the evaluation of speech recognition

Based on the verification so far,

In today's world, where LLM is increasingly used in conjunction with speech recognition, the quality of speech recognition is not practical based solely on the error rate of words and characters.

This has become clear. So, from what perspective should we, who are involved in speech recognition, improve quality? The following is from the paper.

ASR developers
Rather than focusing on overall WER reduction, we should concentrate on improving named entity accuracy.
Original goal: Improve overall WER from 10% to 5%.
New goal setting example (compound): WER for proper noun expressions: 5% → 2%

ASR system evaluators and task designers
Prioritize correcting proper nouns and non-content words, and then focus on the quality of the task after it is executed.
Task performance improves even if the overall WER doesn't decrease.

2. Analysis using AmiVoiceAPI

Now, let's move on to the demonstration section.
We will evaluate Japanese speech using the AmiVoiceAPI engine with a method similar to that described in the paper.

2.1 Error Rate (CER^*) Understanding

*CER corresponds to WER in the paper. Unlike alphabetic languages, character-by-character error rate evaluation is often more effective for Japanese, therefore, CER (Character Error Rate) is used in this analysis.

As the type of audio, we measured the CER of medical conversations, which are expected to contain a large number of technical terms. The target engines are Ami's General-purpose engine (hybrid), E2E engine (End-to-End), and Medical engine (hybrid).
The CER value is the average result of over 100 conversations of approximately 300 characters each across 6 medical departments.
Since this is an error rate, a smaller value indicates fewer errors.

Results and Discussion

It seems that each engine has its own distinct characteristics, with a CER of around 0.15 being the dividing line.
What's most interesting is the trend across medical departments. The medical engine, which should be strong in medical terminology, lags behind the E2E engine in internal medicine and rehabilitation medicine. What a surprise!

Presumably, the conversations in internal medicine and rehabilitation medicine involved a lot of general conversation, allowing the "E2E engine," with its superior hearing, to effectively understand the context. On the other hand, in conversations in dentistry, which likely involves more specialized terminology, the "knowledge" of the medical engine demonstrated its superior capabilities.

*The error rate value depends heavily on the characteristics and quality of the audio used. The results presented here are merely tendencies attributable to the audio used.

2.2　Objective evaluation based on research papers

The purpose of today's evaluation is not to measure the error rate. Following the format of research papers, we will objectively evaluate the quality of the summary. From here on, we will have an engine equipped with medical terminology using the word registration function, and four other engines from outside Ami will also participate.
Please note that the abbreviations for guest engine providers are the same as those used in our published document ("Useful Materials: Comparison of Top 7 Speech Recognition APIs - Key Points for Choosing a Service Without Fail - - AmiVoice Cloud Platform").

List of engines evaluated

• AmiVoice (汎用) (General-purpose)

• AmiVoice (汎用) (General-purpose) + 500 Medical Vocabulary

• AmiVoice (E2E)

• AmiVoice (医療) (Medical)

• Company A

• Company B

• Company E

• Company F

Objective evaluation methods

1. Measure the speech recognition CER for two conversations selected from a dental test set.
2. Input the text of the speech recognition result, along with the reference, into the generative AI and have it evaluate it: AI evaluation (raw)
3. After having the speech recognition result text summarized by the generative AI (ClaudeSonnet 4.6), input it into the generative AI along with the reference and have it evaluate it: AI evaluation (summary)

The evaluation criteria for generative AI, similar to the method used in the paper, explicitly instruct the generative AI through prompts to consider (accuracy of content), (comprehensiveness of important information), (presence or absence of unnecessary information), and (conciseness).

The results are finally here!

• This is a ranking table comparing texts based on speech recognition results from two audio recordings. The tables are sorted in ascending order of their AI evaluation (summary) ranking.

• The "AI evaluation (summary)" and "AI evaluation (raw)" rankings are based on evaluations by the generative AI.

• The ranking (error rate) is a ranking based on the percentage of incorrect answers for each character compared to the correct answer. A smaller number in the CER column indicates a higher rank.

• The arrows and numbers in parentheses in `ranking(AI evaluation (summary))` and `ranking(AI evaluation (raw))` indicate the change in rank from the error rate (CER) ranking. A hyphen indicates no change.

Results and Discussion

• The engine with the lowest error rate did not change its ranking in the AI ​​evaluation.
• AI evaluation is correlated with a low error rate, but reversals can occur.
• While reversals tend to occur when there are no errors in proper nouns, the "structure of the text" during summarization also plays a role. A characteristic example of this is when an E2E engine improves its ranking in summarization.

Furthermore, the AI ​​evaluation was designed to output analysis and commentary.
I'm including some of this as it might be helpful.

AmiVoice (医療) (Medical)
Ranked #1 in recognition rate, AI evaluation (raw), and AI evaluation (summary).
The speech recognition accuracy and AI quality evaluation are perfectly aligned, making it the most reliable system.

AmiVoice (E2E)
WER is average, but summary quality is top-notch.
Even with misidentification, it's possible that medically important information is being captured.
It can be interpreted that the error is one that is easy for LLM to correct.
There is a risk of underestimation in actual operation.

Company E
This system can be evaluated as having high resilience during the summarization stage.
(Regarding Dental_f6) Ranked 8th in recognition rate (last place), but 5th in summary.
Even if there are many misrecognitions, the context and structure may still be preserved.

3. In conclusion

The paper I introduced was published in 2025, and we immediately took notice of it as it provides verification results showing the reality of speech recognition in the era of generative AI. Although we couldn't replicate the same audio and evaluation tools, by conducting our own evaluation following the paper's example, I feel we were able to demonstrate that the paper accurately reflects the feelings that many developers and evaluators using generative AI will likely experience in the future.

Furthermore, what I have reaffirmed as a speech recognition specialist is "the importance of faithfully reproducing sound."
The whole system comes together when you trust in the capabilities of the subsequent processes and diligently perform their work.
It's starting to feel like teamwork in my own work.

Even in the age of AI, we will continue to seriously dedicate ourselves to speech recognition.
We hope that this demonstration will be useful in your efforts to create greater value by combining generative AI with AmiVoice.