Hello everyone.

This article explains how to measure speech recognition accuracy using a library called jiwer, which runs on Python.

The official jiwer page is here. https://jitsi.github.io/jiwer/

Prerequisites

• Python knowledge and execution environment required.
• The version of jiwer used for this explanation and operation check is 3.03.
• Speech recognition accuracy can be measured on a character-by-character basis or on a word-by-word basis, but in this explanation we will be measuring it on a character-by-character basis.
• Please note we offer no operational guarantee or technical support for the content covered in this article.
  
  

Regarding Speech Recognition Accuracy (Speech Recognition Rate)

This is one of the main quantitative indicators that represents the performance of a speech recognition engine, and indicates the "accuracy" of speech recognition results.

If the correct sentence and the speech recognition result sentence match perfectly, the speech recognition accuracy will be 100%. If there is a difference between the correct sentence and the speech recognition result sentence, the speech recognition accuracy will decrease according to the number of characters.

For more details, please see this article.

Measuring Speech Recognition Accuracy Using jiwer

Installing jiwer

First, install jiwer.
The official documentation (https://jitsi.github.io/jiwer/) describes installation methods for both poetry and pip, so you may install using whichever method you prefer.

$ poetry add jiwer

$ pip install jiwer

Please note that if you use pip, you need Python 3.7 or higher.

Sample code for measuring speech recognition accuracy

Here is the sample code I prepared.

import jiwer

reference = "お世話になっております"
hypothesis = "お世話んなっとりますー"

output = jiwer.process_characters(reference, hypothesis)
print(f"正解文の文字数: {output.hits + output.substitutions + output.deletions}")
print(f"正解した文字数: {output.hits}")
print(f"置換誤り文字数: {output.substitutions}")
print(f"挿入誤り文字数: {output.insertions}")
print(f"削除誤り文字数: {output.deletions}")
print(f"エラー率: {output.cer*100}")
print(f"音声認識精度: {(1-output.cer)*100}")

The reference is the correct sentence, and the hypothesis is the speech recognition result. These two are input to jiwer.process_characters(), and the result is displayed with print.

When executed, the following result was output:

正解文の文字数: 11
正解した文字数: 8
置換誤り文字数: 2
挿入誤り文字数: 1
削除誤り文字数: 1
エラー率: 36.36363636363637
音声認識精度: 63.63636363636363

Here is a brief explanation of the results:

• The correct answer (正解文の文字数) is "お世話になっております" and has 11 characters.
• Number of characters correctly answered (正解した文字数): 8 characters in "お世話んなっとりますー"
• Number of incorrectly substituted characters (置換誤り文字数): The number of characters that appear in the speech recognition results that are different from the correct sentence.
• Number of inserted characters (挿入誤り文字数): The number of characters that appeared in the speech recognition results but were not in the correct sentence.
• Number of characters deleted incorrectly (削除誤り文字数): The number of characters that were in the correct sentence but did not appear in the speech recognition results.
• Error rate (エラー率): The total number of incorrect characters divided by the number of characters in the correct sentence (1 + 1 + 2) ÷ 11 ≒ 36.36%
• Speech recognition accuracy (音声認識精度): 100% minus character error rate. 100% - 36.36% = 63.64%
  
  

Visualization of speech recognition accuracy

You can visualize the speech recognition accuracy by using jiwer's visualize_alignment().
Here is the sample code. Lines 1 to 6 are the same as the sample above, but line 7 has been replaced with visualize_alignment().

import jiwer

reference = "お世話になっております"
hypothesis = "お世話んなっとりますー"

output = jiwer.process_characters(reference, hypothesis)
print(jiwer.visualize_alignment(output))

When executed, the following result was output:

sentence 1
REF: お世話になっております*
HYP: お世話んなっと*りますー
        S  SD   I

number of sentences: 1
substitutions=2 deletions=1 insertions=1 hits=8

cer=36.36%

The above is visualized on lines 2 to 4, but it seems that jiwer does not take full-width Japanese characters into account, so the display is distorted.
I tried replacing all half-width characters with full-width characters, as this displays nicely.

ＲＥＦ：　お世話になっております＊
ＨＹＰ：　お世話んなっと＊りますー
　　　　　　　　Ｓ　　ＳＤ　　　Ｉ

The view is as follows

• There are two "S" (substitution：置換誤り), with "に" replaced by "ん" and "て" replaced by "と".
• There is one "D" (deletion：削除誤り) and the "お" has been deleted.
• There is one "I" (insertion：挿入誤り) and a "ー" has been inserted.
  
  

Multi-line visualization

For long audio, the speech recognition result may be multiple lines. In that case, jiwer.visualize_alignment() can also input an array. For example, like this:

import jiwer

reference = ["こんにちは","いい天気ですね"]
hypothesis = ["こんばんは","良い元気でしたね"]

print(jiwer.visualize_alignment(output))

The results are as follows.
*Half-width characters have been manually replaced with full-width characters in the visualization to make it easier to read.

sentence　1
ＲＥＦ：　こんにちは
ＨＹＰ：　こんばんは
　　　　　　　ＳＳ

sentence　2
ＲＥＦ：　いい天気で＊すね
ＨＹＰ：　良い元気でしたね
　　　　　Ｓ　Ｓ　　ＩＳ

number of sentences: 2
substitutions=5 deletions=0 insertions=1 hits=7

cer=50.00%

The visualization is output for each row, but it seems that only the total value can be obtained for the speech recognition accuracy. If you want to check the speech recognition accuracy or number of incorrect characters for each row, you will probably need to run jiwer.process_characters() for each row.

Points to Note When Measuring Speech Recognition Accuracy with jiwer

When measuring speech recognition accuracy with jiwer, please note the following:

Note 1: Symbols that are not pronounced are also counted as one character.

Punctuation marks, line breaks, and silent symbols such as "!" and "?" are also counted as one character.

Example)
Correct answer (正解): いつもお世話になります.
Recognition (認識): いつも、お世話になります.

In this example, the "," is an incorrect insertion, so the speech recognition accuracy is not 100%. Also, the "." is counted as one character. Since the number of correct characters is 12 and the inserted incorrect character is 1, the speech recognition rate is (12-1)÷12 ≒ 91.67%.
Depending on the point of view of measuring speech recognition accuracy, it may be a good idea to remove characters that do not need to be counted from both the correct answer sentence and the speech recognition result sentence in advance.

Note 2: Variations in spelling are counted as separate characters

Variations in spelling, such as half-width/full-width, uppercase/lowercase, differences in trailing kana, and differences between alphabets and katakana, are considered to be different characters.

Example)
Correct answer (正解): 10km
Recognition (認識): XNUMXkm　* All characters are full-width. There are four incorrectly substituted characters for every four correct characters, so the speech recognition accuracy is (4-4)/4 = 4%
Recognition (認識): 10KM　* KM is capitalized. There are 4 incorrectly substituted characters for every 2 correct characters, so speech recognition accuracy = (4-2)/2≒50%
Recognition (認識): 十km　* Kanji numerals. For every four correct characters, there is one incorrectly replaced character and one incorrectly deleted character, so speech recognition accuracy = (4-1)/1 ≒ 4%
Recognition (認識): 10 kilometers　* Written in katakana. For every four correct characters, two characters are substituted incorrectly and four characters are inserted incorrectly. The speech recognition accuracy rate is (4-2-4)/4 = -2%.

In particular, the last word, "1 kilometer," has more errors than the number of correct characters, resulting in a negative speech recognition accuracy of -50%. Please be aware that if you do not notice variations in spelling like this, the speech recognition accuracy will be measured low.