Receiving user's speech

A basic functionality of VoiceAI Connect is passing the user's speech to the bot.

For bots using a textual channel, this is done by first performing speech-to-text of the user's speech using a third-party service, and then passing the recognized text to the bot. For bots using a voice channel, the audio stream of the user is passed to the bot.

The bot usually uses Natural Language Understanding (NLU) for analyzing the input from the user, and respond to it.

In addition to the text or audio that is passed to the bot, VoiceAI Connect passes parameters to the bot which may include information such as the confidence level of the speech-to-text recognition.

How do I use it?

VoiceAI Connect will send the user's speech to the bot by default.

Textual channels

For each speech-to-text recognition, a text-message event is sent to the bot. The exact event structure depends on the bot framework.

The following table lists the additional fields that can be included in the text-message event:

{
  "words": [
    {
      "word": "hello",
      "offset": 1500,
      "duration": 300
    },
    {
      "word": "world",
      "offset": 1900,
      "duration": 800
    }
  ]
}

Text-message events are sent differently for each bot framework:

{
  "type": "message",
  "text": "Hi.",
  "parameters": {
    "confidence": 0.6599681
  }
}

{
  "type": "message",
  "text": "Hi.",
  "channelData": {
    "confidence": 0.6599681
  }
}

Voice channels

If your bot uses a voice channel (e.g., using Dialogflow bots with Live Hub), the audio stream from the user is passed directly to the bot, without using an external speech-to-text service.

Usually, the speech-to-text of the user's audio is done by the bot framework, so the bot logic would still handle textual messages.

As there is no specific event that is sent for each user utterance for voice channels, no additional parameters are passed to the bot by VoiceAI Connect in such case.

Disabling speech input

There are cases when you want to prevent activation of the speech-to-text service.

For example, if the bot wants to send two messages to the user with some time gap between them, and therefore, doesn’t want to allow speech input after the first message playback.

To disable speech input, set the following parameter to false:

Enabling word level timing information

You can enable the received recognition from the speech-to-text provider to include the words field, which displays the duration of each word and its offset relative to the phrase in which spoken.

How to use it?

This feature is controlled by the VoiceAI Connect Administrator, or dynamically by the bot during the conversation (bot overrides VoiceAI Connect configuration):