News
  • HOME
  • News
  • Processing delays in the Asynchronous HTTP interface (v2) [Resolved]

Processing delays in the Asynchronous HTTP interface (v2) [Resolved]

Regarding the delay in job processing on the Asynchronous HTTP interface (v2) that began around 14:36 on February 3, 2026, we have confirmed that all delays were resolved and normal operation was restored at around 10:40 on February 4, 2026.

We would like to report the details of this matter as follows:

1. Overview of the Disability

  • Occurrence period: February 3, 2026, 14:36:30 – February 4, 2026, 10:40
  • Target service: Asynchronous HTTP interface (v2)
  • Impact: Significant delays and outages occurred in job processing for all requests.

2. Cause

The configuration changes that were supposed to have been made to the validation environment were mistakenly applied to the production environment. Although the deployment system execution failed, some of the configuration settings were unintentionally left untouched, preventing instances that execute jobs using the Asynchronous HTTP interface (v2) from starting correctly.

The validation environment and production environment are separate accounts, and their settings should not be mixed. Furthermore, the settings for each environment are managed as source code, and changes are reviewed by multiple people and their history is also managed. Furthermore, the deployment system normally operates by automatically selecting and deploying the settings defined for each environment from the source code management system, which should have prevented mix-ups like this from occurring. However, after investigating this issue, we found that for some resources in the Asynchronous HTTP interface (v2), the deployer can freely specify the settings and the account to which they should be applied as parameters during deployment. This meant that it was possible for incorrect settings to be applied to production due to human error.

Furthermore, if the speech recognition server failed to start during job execution, the operations team would normally be notified so that the problem could be isolated and repaired, but because the speech recognition server continued to run without processing, no notification of a failure was made.Other notifications were also received in the failure notification channel checked by the operations staff, but these notifications were not monitored by the operations team, and so they could not be used to discover the problem.

3. Countermeasures

The following measures have been taken:

  • Configuration recovery: The production environment configuration was redeployed and returned to a normal state.
  • Restarting instances: We deleted the instances that had been running in an abnormal state and replaced them with healthy instances one by one.
  • Normalization confirmed: Jobs began to be completed one by one from 10:05 on February 4th, and all pending jobs had been completed by 10:40.

4. Measures to prevent recurrence

We will take drastic measures to prevent recurrence.

Deploy system improvements
To prevent misconfigurations due to human error in the deployment system for the Asynchronous HTTP interface v2, we have modified it so that the deployment worker only needs to select the target account, and the settings are selected automatically. We will also review whether there are any other similar issues and notify the development team as design guidelines.

Strengthening detection and operations
As a quick response, we redefined the notifications for the operations team's monitoring targets, updated the operation rules along with the initial isolation procedures, and notified the operations team. We will also review the conditions for detecting failures in the Asynchronous HTTP interface (v2) and gradually implement them in the notification system.

Once again, we would like to apologize for the inconvenience caused over such a long period of time. We will continue to strive to ensure stable operation and quality improvement of our services, so we appreciate your continued support.


(2026-02-04 10:49)
Regarding the delays in job processing that began around 4:00 PM on February 3, 2026, all delays were resolved around 10:40 AM on February 4, and we have confirmed that the service, including new jobs, is now operating normally. We sincerely apologize for the considerable inconvenience and trouble caused over such a long period of time. The service is currently operating stably, but we are continuing to investigate the root cause and take measures to prevent recurrence. We will report on the results of our investigation and permanent solutions once details are known.

(2026-02-04 10:18)
Regarding the delay in job processing that began around 4:00 PM on February 3, 2026, we completed emergency measures at around 10:04 AM. Currently, backlogged jobs are being processed one by one. Please wait a little longer until all jobs are completed and the system is restored to normal. We are continuing to investigate the root cause and work toward a full recovery. We apologize for any inconvenience caused and appreciate your patience.

(2026-02-04 9:45)
Thank you very much for your continued use of our services.

Since around 4:00 PM on February 3, 2026, all job processing using the Asynchronous HTTP interface (v2) has been experiencing delays.

We are currently investigating the cause and working to restore the service. We will provide an update on the progress as soon as new information becomes available. We sincerely apologize to our customers for the inconvenience caused.

Use API for Free