Speaker verification

VoiceAI Connect can integrate with a speaker verification service to verify and authenticate a person's identity (based on speech samples for the bot). The verification is done using a third-party service (currently, Phonexia Voice Verify or Nuance Gatekeeper).

Each speaker recognition system has two phases:

Speaker verification systems fall into two categories:

In a typical bot deployment, VoiceAI Connect receives a phone call and connects it to your bot. The bot requests a speaker ID from the user and either begins the enrollment process if the user's speaker ID is not in the system, or it begins the verification process if the speaker ID is already in the system.

For VoiceAI Connect Enterprise, Speaker Verification is supported only from Version 2.6 and later. For more information on how to configure this feature on VoiceAI Connect Cloud, click here.

How do I use it?

The following sections explain how to integrate your bot with the speaker verification feature.

For an example on how to implement such a bot, see speaker verification bot examples.

Get user's speaker ID status

After a call is initiated and the bot prompts and receives the user's speaker ID, the bot sends a speakerVerificationGetSpeakerStatus API command (with the speaker ID) to VoiceAI Connect.

VoiceAI Connect sends the information to the verification service and returns the speaker ID status (enrolled true/false) to the bot.

Example of a speakerVerificationGetSpeakerStatus event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationGetSpeakerStatus",
  "activityParams": {
    "speakerVerificationSpeakerId": "123456"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationGetSpeakerStatus",
  "channelData": {
    "activityParams": {
      "speakerVerificationSpeakerId": "123456"
    }
  }
}
Dialogflow CX

Add a Custom Payload fulfillment with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationGetSpeakerStatus",
    "activityParams": {
      "speakerVerificationSpeakerId": "123456"
    }
  }]
}
Dialogflow ES

Add a Custom Payload response with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationGetSpeakerStatus",
    "activityParams": {
      "speakerVerificationSpeakerId": "123456"
    }
  }]
}
This event is handled in parallel to the continuation of the conversation. However, the execution of this event will be delayed if it is sent while there is a prompt being played to the user. For this reason, it is recommended to send this event before playing the desired prompt to the user (see example flow).

The speaker ID status is sent to the bot as the speakerVerificationSpeakerStatus event.

Example of a speakerVerificationSpeakerStatus event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationSpeakerStatus",
  "value": {
    "success": true,
    "enrolled": true,
    "rawResult": "{...}"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationSpeakerStatus",
  "value": {
    "success": true,
    "enrolled": true,
    "rawResult": "{...}"
  }
}
Dialogflow CX

The fields are sent inside the event-speakerVerificationSpeakerStatus session parameter, and can be accessed using a syntax such as this:

$session.params.event-speakerVerificationSpeakerStatus.success
Dialogflow ES
{
  "queryInput": {
    "event": {
      "languageCode": "en-US",
      "name": "speakerVerificationSpeakerStatus",
      "parameters": {
        "success": true,
        "enrolled": true,
        "rawResult": "{...}"
      }
    }
  }
}

The following fields will be sent with the event:

Parameter

Type

Description

success

Boolean

Indication whether the operation has succeeded.

enrolled

Boolean

Indication whether the speaker ID is already enrolled in the verification service.

  • true: the speaker ID is enrolled

  • false: the speaker ID is not enrolled

rawResult

Object

The result that was received from the verification service.

Note: The value of the field will depend on the verification service.

reasonText

String

In case of failure, includes a free text explaining the failure.

Call initiation flow example

Enrollment

If the speakerVerificationGetSpeakerStatus command indicates that the user is not enrolled (i.e., user's speaker ID does not exist in the verification system), then the bot can (with user permission) initiate a speaker verification enrollment procedure by sending a speakerVerificationEnroll API command.

The enrollment can also be performed using outbound calls (i.e., actively calling a user to enrolling them).

Example of a speakerVerificationEnroll event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationEnroll",
  "activityParams": {
    "speakerVerificationType": "text-dependent",
    "speakerVerificationSpeakerId": "123456",
    "speakerVerificationPhrase": "My voice is my password"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationEnroll",
  "channelData": {
    "activityParams": {
      "speakerVerificationType": "text-dependent",
      "speakerVerificationSpeakerId": "123456",
      "speakerVerificationPhrase": "My voice is my password"
    }
  }
}
Dialogflow CX

Add a Custom Payload fulfillment with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationEnroll",
    "activityParams": {
      "speakerVerificationType": "text-dependent",
      "speakerVerificationSpeakerId": "123456",
      "speakerVerificationPhrase": "My voice is my password"
    }
  }]
}
Dialogflow ES

Add a Custom Payload response with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationEnroll",
    "activityParams": {
      "speakerVerificationType": "text-dependent",
      "speakerVerificationSpeakerId": "123456",
      "speakerVerificationPhrase": "My voice is my password"
    }
  }]
}

Receiving enrollment progress notifications

When handling the enrollment event, VoiceAI Connect sends the user's audio to the verification service.

If the enrollment requires additional samples, the speakerVerificationEnrollProgress event will be sent to the bot. This event is especially useful for text-dependent verification, as the bot will need to ask the user to say his passphrase again in such case.

Example of a speakerVerificationEnrollProgress event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationEnrollProgress",
  "value": {
    "moreAudioRequired": true,
    "rawResult": "{...}"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationEnrollProgress",
  "value": {
    "moreAudioRequired": true,
    "rawResult": "{...}"
  }
}
Dialogflow CX

The fields are sent inside the event-speakerVerificationEnrollProgress session parameter, and can be accessed using a syntax such as this:

$session.params.event-speakerVerificationEnrollProgress.moreAudioRequired
Dialogflow ES
{
  "queryInput": {
    "event": {
      "languageCode": "en-US",
      "name": "speakerVerificationEnrollProgress",
      "parameters": {
        "moreAudioRequired": true,
        "rawResult": "{...}"
      }
    }
  }
}

The following fields will be sent with the event:

Parameter

Type

Description

moreAudioRequired

Boolean

When set to true, indicates that additional utterances are required from the user to complete the enrollment.

rawResult

Object

The result that was received from the verification service.

Note: The value of the field will depend on the verification service.

Enrollment completion

When the verification service completes the enrollment, VoiceAI Connect sends the speakerVerificationEnrollCompleted event to the bot, indicating the result.

Example of a speakerVerificationEnrollCompleted event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationEnrollCompleted",
  "value": {
    "success": true,
    "rawResult": "{...}"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationEnrollCompleted",
  "value": {
    "success": true,
    "rawResult": "{...}"
  }
}
Dialogflow CX

The fields are sent inside the event-speakerVerificationEnrollCompleted session parameter, and can be accessed using a syntax such as this:

$session.params.event-speakerVerificationEnrollCompleted.success
Dialogflow ES
{
  "queryInput": {
    "event": {
      "languageCode": "en-US",
      "name": "speakerVerificationEnrollCompleted",
      "parameters": {
        "success": true,
        "rawResult": "{...}"
      }
    }
  }
}

The following fields will be sent with the event:

Parameter

Type

Description

success

Boolean

Indication whether the enrollment operation succeeded.

rawResult

Object

The result that was received from the verification service.

Note: The value of the field will depend on the verification service.

interimRawResults

Array of objects

The results of the intermediate operations (e.g., of each utterance) prior to the last result.

Note: The value of the field will depend on the verification service.

reasonText

String

In case of failure,this includes free text explaining the failure.

Enrollment flow example

Verification

If the speakerVerificationGetSpeakerStatus command returns a "true" (i.e., user's speaker ID exists in the verification system), then the bot can proceed to initiate a speaker verification procedure by sending a speakerVerificationVerify API command.

Example of a speakerVerificationVerify event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationVerify",
  "activityParams": {
    "speakerVerificationType": "text-dependent",
    "speakerVerificationSpeakerId": "123456",
    "speakerVerificationPhrase": "My voice is my password"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationVerify",
  "channelData": {
    "activityParams": {
      "speakerVerificationType": "text-dependent",
      "speakerVerificationSpeakerId": "123456",
      "speakerVerificationPhrase": "My voice is my password"
    }
  }
}
Dialogflow CX

Add a Custom Payload fulfillment with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationVerify",
    "activityParams": {
      "speakerVerificationType": "text-dependent",
      "speakerVerificationSpeakerId": "123456",
      "speakerVerificationPhrase": "My voice is my password"
    }
  }]
}
Dialogflow ES

Add a Custom Payload response with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationVerify",
    "activityParams": {
      "speakerVerificationType": "text-dependent",
      "speakerVerificationSpeakerId": "123456",
      "speakerVerificationPhrase": "My voice is my password"
    }
  }]
}

VoiceAI Connect starts the verification operation by sending the user's audio to the verification service.

Receiving verification progress notifications

When working in text-independent mode, usually several utterances of the user would be required for the verification progress.

In such case, after processing each intermediate utterance of the user, the speakerVerificationVerifyProgress event will be sent to the bot.

Example of a speakerVerificationVerifyProgress event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationVerifyProgress",
  "value": {
    "moreAudioRequired": true,
    "rawResult": "{...}"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationVerifyProgress",
  "value": {
    "moreAudioRequired": true,
    "rawResult": "{...}"
  }
}
Dialogflow CX

The fields are sent inside the event-speakerVerificationVerifyProgress session parameter, and can be accessed using a syntax such as this:

$session.params.event-speakerVerificationVerifyProgress.moreAudioRequired
Dialogflow ES
{
  "queryInput": {
    "event": {
      "languageCode": "en-US",
      "name": "speakerVerificationVerifyProgress",
      "parameters": {
        "moreAudioRequired": true,
        "rawResult": "{...}"
      }
    }
  }
}

The following fields will be sent with the event:

Parameter

Type

Description

moreAudioRequired

Boolean

When set to true, indicates that additional utterances are required from the user to complete the enrollment.

rawResult

Object

The result that was received from the verification service.

Note: The value of the field will depend on the verification service.

Verification completion

In parallel to performing the verification, the conversation with the bot continues, and the user's audio is also sent to the speech-to-text service.

When the verification service is finished, VoiceAI Connect sends the speakerVerificationVerifyCompleted event to the bot, indicating the result.

If there is not enough audio to match a voice print, the VoiceAI Connect sends the speakerVerificationVerifyCompleted event with a"success" value = false to the bot.

Example of a speakerVerificationVerifyCompleted event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationVerifyCompleted",
  "value": {
    "success": true,
    "verified": "yes",
    "rawResult": "{...}"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationVerifyCompleted",
  "value": {
    "success": true,
    "verified": "yes",
    "rawResult": "{...}"
  }
}
Dialogflow CX

The fields are sent inside the event-speakerVerificationVerifyCompleted session parameter, and can be accessed using a syntax such as this:

$session.params.event-speakerVerificationVerifyCompleted.verified
Dialogflow ES
{
  "queryInput": {
    "event": {
      "languageCode": "en-US",
      "name": "speakerVerificationVerifyCompleted",
      "parameters": {
        "success": true,
        "verified": "yes",
        "rawResult": "{...}"
      }
    }
  }
}

The following fields will be sent with the event:

Parameter

Type

Description

success

Boolean

Indication whether the verification operation succeeded.

verified

String

Indicates the result of the verification.

Possible values:

  • yes: there was a match

  • no: there was no match

  • unknown: the result is inconclusive

This field is only sent if the operation succeeded.

rawResult

Object

The result that was received from the verification service.

Note: The value of the field will depend on the verification service.

interimRawResults

Array of objects

The results of the intermediate operations (e.g., of each utterance) prior to the last result.

Note: The value of the field will depend on the verification service.

reasonText

String

In case of failure, includes a free text explaining the failure.

Verification flow example

Unenrollment

There are cases where you want to remove a speaker from the verification service (e.g., the speaker needs to be re-enrolled, or the speaker no longer consents to have their voice print in the system).

To remove a speaker from the service, the bot sends the speakerVerificationDeleteSpeaker event, indicating the user's speaker ID in the speakerVerificationSpeakerId parameter.

Example of a speakerVerificationDeleteSpeaker event:

AudioCodes Bot API
{
  "type": "event",
  "name": "speakerVerificationDeleteSpeaker",
  "activityParams": {
    "speakerVerificationType": "text-dependent",
    "speakerVerificationSpeakerId": "123456",
    "speakerVerificationPhrase": "My voice is my password"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "speakerVerificationDeleteSpeaker",
  "channelData": {
    "activityParams": {
      "speakerVerificationSpeakerId": "123456"
    }
  }
}
Dialogflow CX

Add a Custom Payload fulfillment with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationDeleteSpeaker",
    "activityParams": {
      "speakerVerificationSpeakerId": "123456"
    }
  }]
}
Dialogflow ES

Add a Custom Payload response with the following content:

{
  "activities": [{
    "type": "event",
    "name": "speakerVerificationDeleteSpeaker",
    "activityParams": {
      "speakerVerificationSpeakerId": "123456"
    }
  }]
}

When handling the event, VoiceAI Connect will contact the verification service to delete the specified speaker ID.

Upon completion of the operation, Voice AI Connect sends the SpeakerVerificationActionResult event to the bot.

Example of a SpeakerVerificationActionResult event:

AudioCodes Bot API
{
  "type": "event",
  "name": "SpeakerVerificationActionResult",
  "value": {
    "success": true,
    "rawResult": "{...}"
  }
}
Microsoft Bot Framework
{
  "type": "event",
  "name": "SpeakerVerificationActionResult",
  "value": {
    "success": true,
    "rawResult": "{...}"
  }
}
Dialogflow CX

The fields are sent inside the event-SpeakerVerificationActionResult session parameter, and can be accessed using a syntax such as this:

$session.params.event-SpeakerVerificationActionResult.success
Dialogflow ES
{
  "queryInput": {
    "event": {
      "languageCode": "en-US",
      "name": "SpeakerVerificationActionResult",
      "parameters": {
        "success": true,
        "rawResult": "{...}"
      }
    }
  }
}

The following fields will be sent with the event:

Parameter

Type

Description

success

Boolean

Indication whether the operation has succeeded.

rawResult

Object

The result that was received from the verification service.

Note: The value of the field will depend on the verification service.

reasonText

String

In case of failure, includes a free text explaining the failure.

Configuration

Administrative configuration

The following bot configuration parameters are configured by the VoiceAI Connect Administrator:

Parameter

Type

Description

speakerVerificationProvider

String

References a service provider used to perform the speaker verification.

The value of this parameter should match the name parameter of the provider.

speakerVerificationSpeakerPrefix

String

(optional)

Defines a string that is prefixed to the speakerVerificationSpeakerId value, when used with the verification service to ensure ID is unique.

This parameter can be used if the same verification service instance is used for distinct customers, whose speakers should be differentiated.

The following provider configuration parameters are configured by the VoiceAI Connect Administrator:

Parameter

Type

Description

speakerVerificationUrl

String

Defines the URL of the verification service.

The default value for Nuance Gatekeeper:

gatekeeper.api.nuance.com

oauthTokenUrl

String

The URL of the authentication service.

The default value for Nuance Gatekeeper:

https://auth.crt.nuance.com/oauth2/token

speakerVerificationTenantScope

String

The name of the scope given by Nuance for the tenant.

Note: This parameter is only applicable to Nuance.

The following parameters are required for the "credentials" section of the provider (for Nuance Gatekeeper):

Parameter

Type

Description

oauthClientId

String

Defines the username for authentication with the verification service.

oauthClientSecret

String

Defines the password for authentication with the verification service.

The following parameters are required for the "credentials" section of the provider (for Phonexia):

Parameter

Type

Description

speakerVerificationUsername

String

Defines the username for authentication with the verification service.

speakerVerificationPassword

String

Defines the password for authentication with the verification service.

Example of Nuance Gatekeeper provider configuration:

{
  "name": "my verify provider",
  "type": "nuance-grpc",

  "credentials": {
    "oauthClientId": "my ClientId",
    "oauthClientSecret": "my ClientSecret"
  }
}

Example of Nuance Gatekeeper bot configuration:

{
  "name": "my bot",
  "displayName": "My Bot",
  "provider": "bot provider",
  "speakerVerificationProvider": "my verify provider",
  "speakerVerificationTenantScope": "my scope name",
  "speakerVerificationConfigSet": "text tependent configset",
  "speakerVerificationType": "text-dependent",
  
  "sendEventsToBot": [
        "speakerVerificationSpeakerStatus",
        "speakerVerificationActionResult",
        "speakerVerificationEnrollProgress",
        "speakerVerificationVerifyProgress",
        "speakerVerificationEnrollCompleted",
        "speakerVerificationVerifyCompleted"
      ]
}

Configuring your bot

The following configuration parameters can be configured by the VoiceAI Connect Administrator, or dynamically by the bot during the conversation (bot overrides VoiceAI Connect configuration):

Parameter

Type

Description

speakerVerificationType

String

One of "text-dependent" or "text-independent".

speakerVerificationConfigSet

String

Defines the name of the "configuration set" used for verification by the speaker verification provider.

Note: This parameter is only applicable to Nuance and should correspond to the speaker verification type.

speakerVerificationSpeakerId

String

The Speaker ID.

Can be set using placeholders.

speakerVerificationPhrase

String

(optional) For text-dependent operation type, the phrase used for the voice signature (if required by the verification service).

speakerVerificationEnrollMaxUtterances

Number

The maximum number of utterances to send to verification service for an enroll operation.

If the operation is not complete and the number of utterances exceeds this value, the operation is canceled.

Valid range: 1-100.

Default for text-dependent: 5

Default for text-independent: 20

speakerVerificationVerifyMaxUtterances

Number

The maximum number of utterances to send to verification service for a verify operation.

If the operation is not complete and the number of utterances exceeds this value, the operation is canceled.

Valid range: 1-100.

Default for text-dependent: 1

Default for text-independent: 20

sendEventsToBot

Array of strings

For receiving the notification events, the events names should be specified in this parameter.

The following values can be specified:

  • speakerVerificationSpeakerStatus

  • speakerVerificationActionResult

  • speakerVerificationEnrollProgress

  • speakerVerificationVerifyProgress

  • speakerVerificationEnrollCompleted

  • speakerVerificationVerifyCompleted

Limitations

This feature uses the speech-to-text service for detection of end-of-speech.

For this reason, there are two limitations during the enrollment and verification process:

  1. Speech-to-text must be enabled.

  2. Barge-in must remain disabled.

.