NLP Settings

Natural Language Processing (NLP) is a component of Artificial Intelligence (AI) that helps computers analyze and understand human language. In the Luma Virtual Agent platform, NLP helps the bot understand a user's phrase and match it to a skill or helps parse user input to populate an attribute using synonyms. During the process of matching the user phrase, the NLP module calculates the probability or confidence score of each Skill and Attribute value.

The following configurations are available in the NLP Settings tab to tailor the NLP engine for your bot and how to respond when the user's input does not strongly match one or any skills.

• Fallback
• Disambiguation

Fallback

The Fallback process occurs when the bot is not able to identify high confidence matching skills based on the user's phrase. The Fallback process is triggered based on the Fallback Threshold parameter defined by the Administrator. This parameter defines the minimum confidence score required that must be returned before a skill is executed. If all matching skills fall below this threshold then the Fallback process is triggered and the user is encouraged to rephrase their request. To learn how the Fallback process works in practice, see the Examples section below.

Using the Fallback section, an Administrator can define the bot responses that inform the user when the bot is not able to understand their request.  

• Define the Fallback Threshold: If all identified skills have a confidence score that is less than the defined Fallback Threshold, one of the Bot Responses is displayed. The Fallback Threshold must be in the range from 0.0 to 1.0.
  

• Define the Smalltalk Threshold: Small Talk Threshold aids in avoiding the Virtual agent to match the custom-defined service requests with Small Talk in the bot. This ensures that the Small talk skills are matched only when no custom skills are identified and the confidence score is above the Small Talk threshold set. The Smalltalk Threshold must be in the range from 0.0 to 1.0. It is recommended to set the Small Talk threshold value high.

• Add new responses and edit existing responses.
  
  
• Hover over the row and click the Delete icon to delete a row.
• Click Save to save the Bot Response.

The following image shows the conversation based on the Fallback behavior configuration. For the user's phrase, "reset password in oracle", the fallback process triggered and one of the Bot Responses that was configured from the list, "Sorry, could you say that again?" is shown to the user.  

Disambiguation

Disambiguation is the group of settings that defines what happens when all returned skills are above the Fallback Threshold but there isn't a single high confidence score. Using the NLP Settings, the administrator can define what the bot should do in this case. You may want your bot to take the best result and execute the skill or prompt the user to let them select a result.

There are two main settings under Disambiguation:

1. Proximity Range - Defines the confidence score gap between skills above the Fallback Threshold. If two or more skills are within this range, they can be presented to the user allowing them to choose the skill they are looking for.
2. Confirmation Threshold - Defines the minimum confidence score to execute a skill without prompting the user. If a single skill is above the fallback threshold, below the confirmation threshold, and has no skills within the proximity range, the user will be prompted confirming that they want to execute this specific skill.

The reason why the Disambiguation process exists is the NLP engine may match the user's phrase to a skill which is not a highly confident match and directly execute it. If this is a wrong skill, it may create an unsatisfactory user experience. Hence, prompting the user for confirmation is always a viable option. However, it's important to adjust these settings during user acceptance testing as you may not want users to be prompted for every phrase they enter.

The following image depicts the Disambiguation section in NLP Settings. 

Proximity Range: Define the required confidence gap between skills. A value from 1 to 10 is ideal but the field supports any integer from 0 to 100. 

Suggestion Settings: In this section, configure the settings when a phrase results in multiple skills are within the defined proximity range.

• Number of Skills to Show: Specify the maximum number of skills to show to the user during the disambiguation process.
• Message: Type the message that will be displayed to the user.
• Card: Specify the details such as Skill Name and Description to be displayed to the user in the form of a Card.

Confirmation Settings: Configure the settings to reaffirm the user decision when there only one matching skill with the confidence score that is below the Confirmation threshold:   

• Confirmation Threshold: When the given user phrase matched with skill with the confidence score that is is very low, executing that skill may not be ideal. To help this situation, the Confirmation Threshold allows the admin to define the required confidence score that needs to be met before executing the skill. When all skills have confidence scores below that threshold, the user is prompted with a message to confirm that they want to execute the top matching skill.
• Automate Training With User Responses:  Turn Automated Training Active or Inactive for the tenant. When Automated training is enabled, the user's positive responses to the skill confirmation message will associate the user phrase to the selected Skill. This improves the accuracy in matching user phrases to skills in your Tenant. Refer Training Luma for more details on Automated Training
• Message: Enter the message that will be displayed to the user to confirm the user's decision.
• Quick Replies: Specify options for the user upon selecting Yes or No.

•  Rephrase: The Rephrase text messages are shown when the user doesn't want to proceed with the skills presented as part of Proximity skills and Confirmation Threshold configurations.
• Message: Type the message that must be displayed to the user. For example, "Sorry, I am unable to help you with your request. Could you rephrase that?".

Save: Click Save to apply the configuration settings.

In the below example, for the user phrase, "update ticket", since the confidence is below the Affirmation Threshold, 0.4, the configured message is shown to the user asking for the affirmation, Yes and No. If the affirmation is set to Yes, the bot continues to execute the identified skill. If the affirmation is set to No, the Rephrase Text configured is shown to the user.

During the initial rollout of a new bot, it is recommended to have the Proximity Range set high (20%-30%). This ensures that users are presented with the right skill, however, this hampers the user experience and makes the bot look unintelligent because the user has to confirm everything.  As the training data evolves over time, reduce the Proximity Range to a lower value (5-10%) to reduce the number of confirmations required by users. 

Examples of NLP Settings in Practice

Let's consider the following scenarios and configuration to illustrate how the Disambiguation settings work. The configured Fallback Threshold is 0.3, the Confirmation Threshold is 0.5, and the Proximity Range is 5%. 

Best Skill Identified

• A user enters the phrase “password reset”
• NLP identifies 2 skills:
  • Active Directory Password Reset with a Confidence Score of 0.90
  • Reset Network Printer with a Confidence Score of 0.60
• Active Directory Password Reset skill is immediately executed
  • It's Confidence Score of 0.90 is above the confirmation threshold and the other matching skill has a score of 0.60 which is beyond the 5% proximity range.

Multiple Strong Matching Skills

• A user enters the phrase “View Ticket”
• NLP identifies 2 skills:
  • View ISM Ticket  with a Confidence Score of 0.90
  • View ServiceNow Ticket with a Confidence Score of 0.86
• User is prompted with quick reply buttons to select either of the skills or a button for "None of These"
  • Both skills are above the confirmation threshold and within the proximity range of 5%, this triggers the Disambiguation Process.

Weak Single Matching Skill

• A user enters the phrase “Leave request”

• NLP identifies 1 skill:
  • Employee Vacation Policy with a Confidence Score of 0.45
• The user is prompted asking them to confirm that they want the Employee Vacation Policy Skill
  • The Skill is below the confirmation threshold so the user is prompted to make sure this is the right skill

Weak Multiple Matching Skills

• A user enters the phrase “Cannot connect to Network Printer”

• NLP identifies 3 skill:
  • Network Printer with Confidence Score of 0.49
  • Network Error with Confidence Score of 0.48
  • Wifi Network with Confidence Score of 0.47 
• User is prompted with quick reply buttons to select any of the three skills or a button for "None of These"
  • The Disambiguation process is triggered because the three skills are below the confirmation threshold and within the proximity range of 5%.

No Matches

• A user enters a phrase such as “Create an account in Salesforce”

• NLP identifies 2 skills:
  • AD Account Reset with a Confidence Score of 0.25
  • Employee Onboarding with a Confidence Score of 0.20
• User is prompted to rephrase their request
  • All skills identified are below the Fallback Threshold so the Fallback process is triggered.