Masaki Hayashi

Hello, this is Hayashi.
I am in charge of developing iOS/WatchOS apps at Advanced Media Inc.

Introduction

Recently, we added an emotion analysis feature as an optional feature to the asynchronous HTTP speech recognition API in our AmiVoice API. This makes it possible to visualize emotions from speech, and has a wide range of use cases, including employee stress checks in call centers and personality assessments in online interviews. For more information, please see our introductory article on emotion analysis.

Sentiment analysis now available in the asynchronous HTTP speech recognition API

AmiVoice Cloud Platform-Tech Blog

So how can emotions extracted from audio data actually be visualized? To confirm the behavior of emotions, we developed an app that allows users to view emotion analysis results and speech recognition results in time synchronization with the audio being played (see image below: the app in this article).

In a previous article, we developed a WatchOS app that uses the AmiVoice API's asynchronous HTTP speech recognition API to test speech recognition on the Apple Watch.

I tried out AmiVoice's asynchronous API on my Apple Watch.

AmiVoice Cloud Platform-Tech Blog

In a previous article, we implemented HTTP communication using the standard framework URLSession, but this time we implemented it using a library called Alamofire, which I personally find easier to implement than URLSession.

・I want to try out emotion analysis on an iOS app

・I want to use the AmiVoice API easily with Swift

If you are interested, please read it.

Implementation

Development environment

• macOS Monterey 12.5.1
• Xcode 13.4.1
• Swift 5.0
• iOS 14 and later

GitHub

advanced-media-inc/acp-swift-sample-applications

AmiVoice Cloud Platform Speech to Text Swift Sample Applications – advanced-media-inc/acp-swift-sample-applications

github.com

○Asynchronous HTTP speech recognition API

To perform sentiment analysis in an iOS app, you must first implement the AmiVoice API's asynchronous HTTP speech recognition API in Swift. For information on implementing the asynchronous HTTP speech recognition API in Swift, please refer to our previous article.

I tried out AmiVoice's asynchronous API on Apple Watch – AmiVoice Techblog

① POST request for speech recognition

When using emotion analysis with the AmiVoice API's asynchronous HTTP speech recognition API, enter the following in the d parameter as shown in the request image below:sentimentAnalysis=TrueYou need to add:

POST https://acp-api-async.amivoice.com/v1/recognitions
Content-Type: multipart/form-data;boundary=some-boundary-string
--some-boundary-string
Content-Disposition: form-data; name="u" () 
--some-boundary-string
Content-Disposition: form-data; name="d" 
grammarFileNames=-a-general sentimentAnalysis=True profileWords=hogehoge%20%E3%81%BB%E3%81%92%E3%81%BB%E3%81%92%E3%81%A6%E3%81%99%E3%81%A8
--some-boundary-string
Content-Disposition: form-data; name="a" 
Content-Type: application/octet-stream (Audio data) 
--some-boundary-string--

Using the request image above as a reference, a voice recognition request with the AmiVoice API can be implemented using Alamofire as follows:

years appKey = "ABCD..." 
years url = "https://acp-api-async.amivoice.com/v1/recognitions" 

years headers: HTTPHeaders = [ 
    "Content-type": "multipart/form-data" 
]

AF.upload(multipartFormDat a: { multipartFormData in 
    multipartFormData.append(appKey.data(using: .utf8)!, 
                             withName: "u",
                             fileName: Vittorio Citro Boutique Official Site | Clothing and Footwear Buy the new collection online on Vittoriocitro.it Express Shipping and Free Return.Vittorio Citro Boutique Official Store | Fashion items for men and women,
                             mimeType: ni l)
    multipartFormData.append("grammarFileNames=-a-general sentimentAnalysis=True".data(using: .utf8)!, 
                             withName: "d",
                             fileName: Vittorio Citro Boutique Official Site | Clothing and Footwear Buy the new collection online on Vittoriocitro.it Express Shipping and Free Return.Vittorio Citro Boutique Official Store | Fashion items for men and women,
                             mimeType: Vittorio Citro Boutique Official Site | Clothing and Footwear Buy the new collection online on Vittoriocitro.it Express Shipping and Free Return.Vittorio Citro Boutique Official Store | Fashion items for men and women)
    multipartFormData.append(audioData,
                             withName: "a",
                             fileName: "audio.wav",
                             mimeType: "application/octet-stream")
}, to: url, method: .post, headers: headers)
    .validate(statusCode: 200..400)
    .responseData(completionHandler: { response in
        Switch response.result { 
        CASE​ .success(years data): 
           // Processing if successful 
        CASE​ .failure(years error): 
           // Handling failure 
        }
    })

After the request,.responseDataIn the callback ofresponse.resultPlease use.

②Get the job status with GET

The job status is obtained as follows:

years path = url + " or " + "Session ID"
years headers: HTTPHeaders = [ 
 "Authorization":"Bearer \(appKey)" 
]

AF.request(path, method: .get, headers: headers) 
    .validate(statusCode: 200..400)
    .responseData { response in
        Switch response.result { 
        CASE​ .success(years data): 
            // Processing if successful 
        CASE​ .failure(years error): 
            // Handling failure 
        } 
    }

The job status on the AmiVoice API side will change over time, so you will need to call the above request at any time.

If you want to use the results of sentiment analysis, use the JSON that can be obtained when the job status is "completed".sentiment_analysisUse the value in.

Supplement: About app operation

This app allows you to "play back recorded audio and check the emotion analysis and voice recognition results over time."

When actually using this service, please try the following steps:

  ①AppKey settings

  ② Audio recording

  3) Display of voice recognition results and emotion analysis results

*You can also use "character recognition" to obtain the APPKEY when setting it. Try this if you are unable to copy and paste the APPKEY.

That's all for how to operate the app.

Final thoughts

This time, I developed an app that visualizes emotions quantified from recorded speech using the AmiVoice API's asynchronous HTTP speech recognition API. I believe that the use cases can be further expanded by using the results of emotion analysis in addition to the speech recognition results.

If you are interested, please give it a try.

Reference

AmiVoice API related

Asynchronous HTTP Interface | AmiVoice Cloud Platform

Emotion analysis | AmiVoice Cloud Platform

Alamofire

GitHub – Alamofire/Alamofire: Elegant HTTP Networking in Swift

Multipart/form-data request with Alamofire – Qiita

Alamofire vs URLSession: a comparison for networking in Swift – SwiftLee

Charts (used to display sentiment analysis results)

GitHub – nubank/ios-charts: Beautiful charts for iOS/tvOS/OSX! The Apple side of the crossplatform MPAndroidChart.

[Swift] Drawing a line graph using Charts, a charting library | DevelopersIO