Configuring multiple speech providers

VoiceAI Connect supports connecting to various speech services and enables configuration of parameters for each service (e.g., your credentials to the service). A combination of a speech service and its parameters is termed a provider. Typically, a bot is associated with one speech-to-text (STT) provider and one text-to-speech (TTS) provider.

However, there are scenarios where greater control over provider usage is required, and VoiceAI Connect offers you this control through the following features:

1. Load balancing between providers: To overcome limitations of a single speech service provider, you can define multiple parameter sets (e.g., each in a different region or with different credentials) and let VoiceAI Connect balance the load between them.

2. Fallback to alternative providers: If there’s a failure to connect to the primary provider, additional providers can be configured as backups to ensure uninterrupted service.

3. Dynamical provider change: The bot can switch speech providers during a call. For instance, different speech-to-text providers may excel in recognizing specific types of information (e.g., dates or street names), and VoiceAI Connect allows the bot to switch to the most suitable provider as needed during the conversation.

---

How do I use it?

Load balancing and fallback speech-to-text providers

For configuring load balancing and/or fallback speech-to-text providers, you should use the sttProviders bot parameter instead of the sttProvider parameter.

The Administrator can use the following bot parameters for controlling this feature:

Parameter	Type	Description
sttProviders	Array of objects	Defines a list of providers to be used for the call. Each element in the array is an object with fields as detailed below. VoiceAI Connect performs load balancing and/or fallback between the providers specified in the array, according to their priority. Note: The array is limited to 10 providers. Using this parameter overrides configuration of the sttProvider and the sttFallbackProviders parameters.
maxSttRetriesPerPriority	Number	The maximum number of alternative providers to fallback to in the same priority before advancing to a higher priority. A value of -1 indicates that all providers within the same priority level should be attempted before moving to a higher priority. A value of 0 indicates that only one provider within each priority level should be tried. Default: -1. Range: -1 – 100.

The following table lists the fields inside the sttProviders array elements:

Field	Type	Description
sttProvider	String	The name of the provider to use for speech-to-text.
sttPriority	Number (Optional)	The priority of the provider. VoiceAI Connect will use the lowest priority providers. Only in case of failure, it will failover to higher priority providers. If multiple providers are configured with the same priority, VoiceAI Connect will randomly choose (i.e., load balance) between them. Default: 0. Range: 0-100.
sttEndpointID	String (Optional)	A value to be set to the sttEndpointID parameter when using this provider. See Speech-to-text models and contexts for more details.
sttContextId	String (Optional)	A value to be set to the sttContextId parameter when using this provider. See Speech-to-text models and contexts for more details.
sttModel	String (Optional)	A value to be set to the sttModel parameter when using this provider. See Speech-to-text models and contexts for more details.

By setting different values to the sttPriority field, you can achieve various load balancing and/or fallback behaviors.

For example, for load balancing between two providers without any fallback, you can specify the providers without any priority (i.e., using the default priority 0), and restrict fallback between them:

{
  "sttProviders": [
    {
      "sttProvider": "first_provider"
    },
    {
      "sttProvider": "second_provider"
    }
  ],
  "maxSttRetriesPerPriority": 0
}

For configuring a simple fallback, without load balancing, you can specific a different priority for each provider:

{
  "sttProviders": [
    {
      "sttProvider": "first_provider",
      "sttPriority": 0
    },
    {
      "sttProvider": "alternative_provider",
      "sttPriority": 1
    }
  ]
}

And for configuring load balancing between two providers, with a third fallback if both fail, you can specify several providers with the same priority:

{
  "sttProviders": [
    {
      "sttProvider": "first_provider",
      "sttPriority": 0
    },
    {
      "sttProvider": "second_provider",
      "sttPriority": 0
    },
    {
      "sttProvider": "alternative_provider",
      "sttPriority": 1
    }
  ]
}

Dynamic speech-to-text provider change

On bot message, where the sttProvider parameter is different than the current speech-to-text provider, VoiceAI Connect stops the current speech-to-text session with the current provider, and the next speech-to-text request will be with the new provider (if the new speech-to-text provider is listed in the additionalSttProviders parameter list value).

To integrate this feature in your bot, see Changing the speech provider.

The Administrator can use the following bot parameter to define a list of speech-to-text providers allowed for dynamic change:

Parameter	Type	Description
additionalSttProviders	String	Defines a list of up to ten provider names that the bot can use as speech-to-text providers. Note: Configuration cannot include both directSTT and additionalSttProviders. The values listed in the additionalSttProviders parameter must also be one of the values configured in sttProvider.

Example

The following shows an example of a bot configuration for two speech-to-text providers:

{
  "name": "MyBot",
  "provider": "my_azure_provider",
  "displayName": "My Bot",
  "credentials": {
    "botSecret": "..."
  },
  "additionalSttProviders": [
    "my_azure_provider",
    "my_google_provider"
  ]
}

Dynamic text-to-speech provider change

On bot message, where the ttsProvider parameter is different than the current text-to-speech provider, VoiceAI Connect stops the current text-to-speech session with the current provider, and the next text-to-speech request will be with the new provider (if the new text-to-speech provider is listed in the additionalTtsProviders parameter list value). To integrate this feature in your bot, see changing-the-speech-provider.htm.

The Administrator can use the following bot parameter is used to define a list of text-to-speech providers:

Parameter	Type	Description
additionalTtsProviders	String	Defines a list of up to five provider names that the bot can use as text-to-speech providers. Note: If voiceName is not configured, then the default voiceName is used. Configuration cannot include both directTTS and additionalTtsProviders. The values listed in the additionalTtsProviders parameter must also be one of the values configured in ttsProvider.

Example

The following shows an example of a bot configuration for two text-to-speech providers:

{
  "name": "MyBot",
  "provider": "my_azure_provider",
  "displayName": "My Bot",
  "credentials": {
    "botSecret": "..."
  },
  "additionalTtsProviders": [
    "my_azure_provider",
    "my_google_provider"
  ]
}