完成形

Development environment

• Windows 10
• Visual Studio 2019
• WPF application
• .NET 5.0

Implementation

I would like to implement the app using the following steps:

1. Registration for AmiVoice Cloud Platform (ACP)
2. Project launch
3. Adapting the MVVM model
4. Acquires audio data from a connected microphone
5. Speech recognition over WebSocket using ACP
6. Linking the created program and UI

Step 1: Register for AmiVoice Cloud Platform (ACP)

To use streaming voice recognition, you first need to register with ACP.
For information on how to register, please see the article below.

Try AmiVoice Cloud Platform

Step 2: Launching the project

Open Visual Studio 2019, click "Create a new project", search for "WPF", and select "WPF Application".

Enter the project name, click "Next", and select the target framework ".Net 5.0".

Step 3: Adapting the MVVM model

WPF recommends a design pattern called MVVM (ModelView-View-Model), which is a software architecture that serves as a guideline for determining the internal structure of an application.

This timeThis blogWe will create an application with an internal structure that takes the MVVM pattern into consideration, using this as a reference.

1. 1. Delete MainWindow.xaml and MainWindow.xaml.cs and add Views, ViewModels, and Models folders. To add a folder, right-click on the project name and select Add to create a new folder. In the image, the project name is "RecApp" under "Solution."
   2. Add the MainView window class to the Views folder and the MainViewModel class to the ViewModels folder by right-clicking on each folder and adding them. For the MainView window class, select Window (WPF) and "Window1.xaml" and "Window1.xaml.cs" will be added. The file names are difficult to understand as they are, so change the file names to "MainView.xaml" and "MainView.xaml.cs". Also, at this time, add the MainView.xaml In the tag, add "x:Class="RecApp.Views. Window1Change the class name of Window1 and MainView.xaml.cs written as “” to MainView.

1. Delete the StartupUri property written in App.xaml and override the OnStartup() method in App.xaml.cs with the following code.
   App.xaml.cs

   
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Data;
using System.Linq;
using System.Threading.Tasks;
using System.Windows;
using RecApp.Views;
using RecApp.ViewModels;
namespace RecApp
{
///
/// Interaction logic for App.xaml
///
public partial class App : Application
{
protected override void OnStartup(StartupEventArgs e)
{
base. OnStartup(e);
// ウィンドウをインスタンス化
MainView w = new MainView();
// ウィンドウに対する ViewModel をインスタンス化
MainViewModel vm = new MainViewModel();
// 閉じる際のイベントを設定
w.Closing = vm.Closing;
// ウィンドウに対する ViewModel をデータコンテキストに指定
w.DataContext = vm;
// ウィンドウを表示
w.Show();
}
}
}


Step 4: Acquire audio data from a connected microphone

ready

To obtain audio data from a connected microphone in C#, use a library called NAudio.

NAudio is a library that allows you to easily handle audio file-related processes such as audio input/output, device selection, and format conversion. The implementation of NAudio is also available on GitHub, so if you're interested,here .

First, to use NAudio, download the library from NuGet. Go to Tools > NuGet Package Manager > Manage NuGet Packages for Solution > Browse and search for "NAudio". Change the NAudio version to 1.10.0, check the target project, and install it.
*1.10.0 has enough functionality,Sound TouchWe are using 1.10.0 instead of the latest version because it is compatible with a library that allows you to change the playback speed and pitch in real time when playing audio files.
(WPF has a tag called MediaElement that plays video and audio, and you can change the playback speed with this as well, but there may be a period of silence when you change the playback speed. However, when using SoundTouch, there is no silence, so you can change the playback speed without any discomfort.)

Next, to create a class that performs NAudio-related operations, create a new class file in the Models folder (hereafter referred to as the Audio class). All processing using NAudio will be written in the Audio class.

There are two classes for recording audio using NAudio: the WaveIn class and the WaveInEvent class.

• Setting the microphone device to use for recording
• Processing the recorded audio
• Processing at the end of recording

It is a class that can do things like this. The internal processing of the WaveIn class and WaveInEvent class is different, but they can do the same things. So, how do you use the two classes?

WaveIn: GUI application

WaveInEvent: Console applications (can also be used in GUI applications)

The uses are divided as shown above. Also, the WaveIn class uses Windows Messages, but the WaveInEvent class is designed not to use Windows Messages. Since we are creating a GUI application this time, we will use the WaveIn class to write the recording process.

*The methods that can be used with the WaveIn and WaveInEvent classes are the same, so if you are creating the app we are creating this time using the WaveInEvent class, simply change the WaveIn part to WaveInEvent and the recording processing part will work.

Recording Processing

The code for recording is as follows:

Let's look at each step. First, let's look at the constructor.

This specifies the audio format for recording. In this example, only the sampling frequency and number of channels are specified, but you can also set the bit depth.

The process to be performed when recording starts is written in the RecordingStart method.

specifies the recording device and format required for recording. Also, a new WaveIn class is instantiated each time recording is performed. This is because when the same instance of the WaveIn class is used, the "WAVERR_STILLPLAYING" Windows Message is sent by "waveInUnprepareHeader," one of the Win32 APIs used internally, and the recording stop process is executed as soon as recording starts (the recording stop process is executed if any error occurs). To avoid this, a new instance is created each time.

You can write the processing to be performed on the recorded audio data in DataAvailable.

*DataAvailable is an event that fires when the buffer inside WaveIn is filled with queue. By default, this event fires every 100ms.

A try-catch statement is used to stop recording if an error occurs while processing the recorded data.

You can write the processing to be performed when recording stops in RecordingStopped.

RecordingStopped is the last process to be performed, so the instance is released here.

*After using the RecordingStop method, the WaveIn class processes the following in order: flag change (recording stopped) → DataAvailable (unprocessed) → RecordingStopped.

Recording is started with the StartRecording method of the "WaveIn" class, and stopped with the RecordingStop method.

Step 5: Speech recognition over WebSocket using ACP

ready

The WebSocket part of speech recognition via WebSocket using ACP is described in this article, so this timeSample program on the ACP homepageI would like to incorporate the WebSocket part of this into my app as is.

Real-time speech recognition on Apple Watch using WebSocket + AVAudioEngine

AmiVoice Cloud Platform-Tech Blog

First,hereDownload the sample program from here.

Copy the "com" folder in the sample program's "sample_1.1.8/Wrp/cs/src" into the Models folder.

*The "Wrp" file in the "com" folder describes the processing related to WebSocket. Therefore, please refer to it if you are writing your own processing related to WebSocket.

Next, add a new class file to the Models folder that describes the speech recognition results via WebSocket (hereafter referred to as the WrpSimple class). All processing of the speech recognition results returned via WebSocket will be written in this WrpSimple class.

*In this application, we only parse the returned speech recognition results and display the UI, but there are various other events (for example, an event that fires when a speech segment is detected). Use each event according to the application you want to create.

Recognition result processing

Make the WrpSimple class inherit the WrpListener interface that exists in "com.amivoice.wrp".

By inheriting the interface, you must implement the following seven items:

In this case, we only use resultFinalized because we want to display the UI when speech recognition is complete. resultFinalized is an event that fires when the speech recognition result for a specific speech section is confirmed. The speech recognition result is a string in JSON format. Therefore, we need to parse it and extract only the necessary parts.

The "TextParse" method extracts only the recognition results from the JSON-formatted string.

This process is a direct copy and paste of the text_ method in "WrpTester.cs" in the sample program "sample_1.1.8/Wrp/cs".

Step 6: Linking the created program and UI

ready

We will now link the voice recording and voice recognition result processing created in steps 4 and 5, the class ported from the sample program, and the UI.

First, we will create the UI. We will not discuss UI design this time. Therefore, the app will work with just a WPF TextBox and Button. If you do not want to worry about the UI, copy and paste the following XAML code into "MainView.xaml".

*You will need to change all of the RecApp parts in the code to the name of the project you created.

here

1. Binding
2. ScrollToEndBehavior.AutoScrollToEnd
3. Storyboard
4. Converter

I will briefly explain these four points.

・Binding

"Binding ***" binds to the data (property) of ***. By doing this, when you change the value of an element on the ViewModel side, it will be automatically reflected in the UI.^[1]

・ScrollToEndBehavior.AutoScrollToEnd

When creating an app that complies with MVVM, it does not use code-behind (writing code in MainView.xaml.cs), so it is difficult to write processes that are executed when the state of the View changes. As an alternative, there is the Behavior class.^[2]In the app we are creating this time, the recognition results will be added to the end. Therefore, in order to always display the latest recognition results, it is necessary to scroll to the end every time a recognition result is displayed. Therefore, we will use the Behavior class to scroll to the end every time a recognition result is displayed. The implementation of the Behavior class will be described later.

・Storyboard

Storyboard allows you to create animations in WPF. Here, we are creating a blinking animation to prevent the text box from being operated while recording and to let you know that recording is in progress. The content of the animation is determined in the area enclosed by the Storyboard tag. In this case, we are changing the Opacity property to make it blink. The Style tag written below the Storyboard tag sets the trigger that will fire the created animation. In this case, we are making it animate when the element using this Style is visible on the screen.This articleprovides a detailed explanation of Storyboard. Please refer to it if you want to know more about Storyboard.

Converter

A converter literally means something that converts.This articleAs described in the article, the IsChecked property (value is Boolean), which indicates whether a checkbox is checked, can be used with a Converter to represent its state as a string. In this app, we are trying to set the Visibility property to "Visibility" during recording, and "Hidden" or "Collapse" at other times. The value of the Visibility property is not Boolean, but an enumeration type, so it is not possible to control whether or not it is displayed on the screen with a Boolean type. Therefore, we are using a Converter so that when the value is true, it becomes "Visibility" and when it is false, it becomes "Collapse."

*The BooleanToVisibilityConverter class is implemented as standard. Therefore, by using this, you can implement the processing you want without having to create a new Converter class.

About ViewModel

The ViewModel class is responsible for connecting data-bound data and Model with the View. To data-bind to View elements, "INotifyPropertyChanged" and "ICommand" are used to notify the View that a value has changed. However, using these as is can be quite tedious. Therefore, to make it easier to use, install "Prism.Wpf" from NuGet. Prism is an MVVM framework. Using "Prism" makes it possible to write data-binding-related processes using "INotifyPropertyChanged" and "ICommand" in a short and simple manner.

The entire code for "MainViewModel.cs" is as follows:

Follow the code from top to bottom.

To display the data generated by the Model in the View, the data must be passed to the ViewModel. Create an event variable for the event that passes the data received from the Model to the View.

This time, we will only be exchanging recording status and strings, so we will create a class that only has the recording status and strings.

"BindableBase" is a helper class for implementing "INotifyPropertyChanged". By inheriting this, you can easily notify the View side of value changes.

This is the process to disconnect the connection if the app is shut down while still connected to WebSocket. The closing process is added to MainView with "w.Closing += vm.Closing;" in "App.xaml.cs".

*If you close the app while recording, the WebSocket connection will remain active.

Next, regarding the data binding to the Button and TextBox and the trigger that triggers the animation,

It is written like this. When the value is changed by using SetProperty, a notification that the value has been changed is sent to the View. For more information on the processing of SetProparty,GithubThe code has been published, so please refer to it.

When the record button is pressed, the event to start and stop recording is passed to the View using DelegateCommand.

As shown above, pass the processing you want to perform when the button is pressed to DelegateCommand as an argument.

Regarding WebSockets

It will look like this. This time we will not go into details about the server name to connect to, which is described in another article mentioned above. Set the WrpSimple class created in step 5 as the listener for the Wrp class. This will enable the created speech recognition result event to be fired. Also, the WrpSimple class must be modified to pass data to the ViewModel. For this reason, a new event is defined to pass data to the ViewModel. The contents of this definition will be described later.

The Audio class

It will look like this. The Wrp class is required to send audio to the ACP server. Therefore, the Audio class also needs to be modified to connect with the ACP server and ViewModel. The changes will be described later. For now, it is sufficient to understand that "events are defined and processing is added to connect with the ACP server and ViewModel."

About the Behavior class

First, create a file to write the Behavior class. Create a new Behavior folder in the Views folder. Create a class file in it and name it "ScrollToEndBehavior.cs" (hereafter referred to as the ScrollToEndBehavior class). Write a process in the ScrollToEndBehavior class that will automatically move the scroll bar to the bottom when new recognition results are displayed.

This articleThe ScrollToEndBehavior class created with reference to the attached behavior is as follows.

For attached behaviors seeThis articleSince it is explained in detail in, I will only explain the OnTextChanged event that I added.

OnTextChanged is set to scroll to the end if you change the text in the TextBox. However, it does not scroll if the TextBox has keyboard focus. This is because when editing the recognition results, scrolling to the end every time you change a single character makes editing difficult. For this reason, it is set to scroll only while recording.

There are two concepts of focus in WPF:

• Keyboard Focus
• Logical Focus

There are two types of focus. The difference between them is

• Keyboard focus: The element currently receiving keyboard input, of which there is only one across the entire desktop.
• Logical focus: There can be only one within any given focus range. There can be multiple.

Want to know more?Official DocPlease refer to.

Interaction between the recording class and WebSocket processing

To perform speech recognition, the recorded audio data must be passed to the ACP server. To do this, we will modify the Audio class. Since we will modify the Audio class again in conjunction with the UI in the next section, here we will explain the methods used to connect to the ACP server and perform speech recognition.

First, to use the Wrp class, add "com.amivoice.wrp" to the using at the top, and declare a variable to hold an object within the Audio class. Pass a value to the declared object when instantiating the Audio class.

The procedure for sending audio data to the ACP server is as follows:

1. Connect to the server
2. Sends a message to start sending audio data
3. Sends audio data to the server
4. Send a transmission completion message
5. Disconnect from the server

By following these steps, you can send voice data to the server and perform voice recognition. The corresponding Wrp method is

1. connect
2. feedDataResume
3. feedData
4. feedDataPause
5. disconnect

By using these in order, you can perform speech recognition. Since the audio data is sent to the ACP server using the feedData method, this method must be executed within DataAvailable of the WaveIn class.

Interaction between recording class, recognition result processing class and UI

Here's what our final Audio class looks like:

Define an event to check if the WebSocket connection has been established and display the information from the Model in the View.

Create an event to display button text or Blink depending on the recording status.

Then your final WrpSimple.cs should look like this:

WrpSimple.cs also defines an event to display in the View created in Audio.cs. Here, an event is created to display the recognition results.

Summary

This time, I tried to implement microphone recording in a Windows application. I hope you will also try developing a voice recognition application using ACP.

Reference

1. What is WPF data binding? How to manipulate data in C#
2. WPF Behaviors