*Whisper (large-v2)

Consideration

AmiVoice produced the best results.In terms of character error rate (CER), AmiVoice achieved 100% - 96.68% = 3.32%, while Whisper (large) achieved 11.71% and Whisper (large-v2) achieved 11.20%, representing an extremely significant difference of more than three times.

AmiVoice's speech recognition rate of 96.68% is a relatively high level among our in-house experiments. The microphone was set up properly, the speaker spoke clearly to the participants, and the voice was easy for humans to hear, which is probably why it achieved such high accuracy.

What concerns me is that Whisper makes a fair number of false positives. When I checked the false positives, I found that there were two main types of false positives.

• Whisper misrecognition pattern 1: Incorrect recognition of words with similar pronunciation, ignoring context
  
  • 競合→今日後
  • 好調に推移→好調に注意
  • どこも投資をしている→ドグも投資している
  • ◯◯か何かで→◯◯内科で

• Whisper misrecognition pattern 2: Unusual Kanji conversions
  • 加重平均→過充平均
  • 四半期報告→市販機報告
  • 余資運用→吉運用
  • 残存期間→暫存期間
  • 外貨建て→外科だて
  • 一過性→一家性

Pattern 1 may sound a little like "そう聞こえなくもないかな？" to a human listener, but when you consider the context and other factors, it seems like a strange misrecognition. (This pattern also occurs with AmiVoice, but it was more frequent with Whisper in this audio.)

Pattern 2 uses kanji conversion that doesn't come up much in web searches, so it's a mystery why it produced such an output.

Furthermore, Whisper has a conspicuously high number of deletion errors. When I investigated the cause, I was surprised to find that Whisper has a tendency to intelligently delete sentences it deems unnecessary. For example, Whisper may output the following for utterances such as those shown below. *1

• Speaker: "１月、いや２月の１日、あれ２月１日じゃなかったでしたっけすいません、え、いいんでしたっけ、その日の件ですが"
• Whisper: "２月１日の件ですが" *2

While this processing is thought to improve the readability of speech recognition results, the omitted portions are treated as deletion errors in calculations, as these results cannot be considered correct. Deletion errors resulting from this behavior may not be problematic depending on the application, so it might be acceptable to discount them to some extent during evaluation.

Summary

This time, we compared AmiVoice and Whisper using conference audio.

The results showed a significant difference, with AmiVoice having an error rate (CER) of less than one-third.

Whisper seems to have a tendency to output similar-sounding words without considering the context, and it often misrecognizes words by converting them to kanji characters that are not commonly used in Japanese.

Whisper also tends to intelligently delete unnecessary text, which makes it prone to more deletion errors. These errors may not be a problem depending on your use case, so you might consider discounting them to some extent when comparing and evaluating.

Person who wrote this article

• 

  Shogo Ando

  While researching speech recognition, I found a local speech tech firm and decided to join the team, where I continue to work to this day.

  My hobbies include overseas travel, trying great food and visiting saunas.

  : @anpyan