*Whisper (large-v2)

Consideration

AmiVoice had the best results. In terms of error rate (CER), AmiVoice was 100% - 96.68% =3.32%Similarly, Whisper(large)11.71%,Whisper(large-v2)11.20%And,3 times or moreIt was a very big difference.

AmiVoice's speech recognition rate of 96.68% is a relatively high level among our in-house experiments. The microphone was set up properly, the speaker spoke clearly to the participants, and the voice was easy for humans to hear, which is probably why it achieved such high accuracy.

What concerns me is that Whisper makes a fair number of false positives. When I checked the false positives, I found that there were two main types of false positives.

• Whisper misrecognition pattern 1:Mistaking similar-sounding words out of context
  
  • Competition → After today
  • Trending favorably → Beware of favorable conditions
  • Everywhere is investing → Dog is investing too
  • At ◯◯ or something → ◯◯ Internal Medicine

• Whisper misrecognition pattern 2:It uses kanji conversion that I have not seen often.
  • Weighted average → overfilled average
  • Quarterly report → Commercial machine report
  • Surplus funds management → Good management
  • Remaining period → Interim period
  • Foreign currency → surgical fee
  • Transient → familial

Pattern 1 may sound a little like "maybe that's what it sounds like?" to a human listener, but when you consider the context and other factors, it seems like a strange misrecognition. (This pattern also occurs with AmiVoice, but it was more frequent with Whisper in this audio.)

Pattern 2 uses kanji conversion that doesn't come up much in web searches, so it's a mystery why it produced such an output.

Also, Whisper has a particularly high number of deletion errors. I was surprised when I investigated the cause, but WhisperIntelligently delete unnecessary textFor example, in response to the following utterance, Whisper may output something like this:*1

• Speaker: "January, no, the first of February, wasn't it February 1st? Sorry, is that okay? About that day."
• Whisper: "Regarding February 1st..."*2

This process will improve the readability of the speech recognition results, but it cannot be considered the correct answer.The omitted parts are deletion errors.The deletion error caused by this behavior may not be a problem depending on the application, so it may be ok to take it into account when evaluating it.

Summary

This time, we compared AmiVoice and Whisper using conference audio.

The results showed a significant difference, with AmiVoice having an error rate (CER) of less than one-third.

Whisper seems to have a tendency to output similar-sounding words without considering the context, and it often misrecognizes words by converting them to kanji characters that are not commonly used in Japanese.

Whisper also tends to intelligently delete unnecessary text, which makes it prone to more deletion errors. These errors may not be a problem depending on your use case, so you can take them with a pinch of salt when comparing and evaluating.

Person who wrote this article

• 

  Shogo Ando

  While researching speech recognition, I found a speech recognition company nearby and joined the company, where I continue to work to this day.

  My hobbies are traveling abroad, eating delicious food, and saunas.

  : @anpyan