In this article, let us discuss the following:
...
Metric | Details |
|---|---|
Total Messages | The total number of messages between the bot and end users across all channels during the selected period. You can compare the total number of interactions between the bot and your users across different channels over different periods and determine the most popular channel of interaction. This also helps you to analyze how effectively your bots are being leveraged by users for various skills. |
Conversation Transactions | This metric shows the currently used, owned and available conversation transactions. It helps the administrator to ensure compliance with license agreements and purchase more transactions if necessary. Conversation transactions relate to triggering any fulfilment type (text, web service, automation, email) on a custom or ITSM skill. To learn about what are conversation transactions, refer Subscription Information#ConversationTransactionRules. |
| Automated Workflow Transactions | This metric shows the currently used, owned and available automated workflow transactions. It helps the administrator to ensure compliance with license agreements and purchase more transactions if necessary. Automation transactions are in addition to conversation transactions and are used when an automation workflow is triggered. To learn about what are automated workflow, refer Subscription Information#AutomatedWorkflowTransactionRules. |
| Agent Transfers | The number of times users triggered the Transfer to Agent skill during the selected period. Analyzing this metric helps you to understand the skills which need to be built, or modify the existing skills to avoid the agent transfer process. |
View Bot User Details
Users include the New Users, Self-service Users and Analyst users’ metrics. This metrics helps you to understand how effectively the bot is being leveraged by different type of users over a selected period.
...
Training Opportunities include the Unmatched Phrases, Aborted Skills, Average Accuracy of the Bot, Total Phrases Trained and Skills Fall Below Required Threshold metrics. Note that clicking the Training metrics navigates you to the Training page, but does not apply the selected filter.
Metric | Details |
|---|---|
Unmatched Phrases | The percentage of phrases entered by users that did not match a skill.Calculated as unmatched phrases divided by total phrases for the selected time period.Analyzing this metric helps you determine whether the user entered any phrases other than the ones defined in Luma for a skill, or whether a new skill needs to be built to cater to the users' request. For well training bots, this percentage should decrease over time. Click the metric to navigate to the Unmatched Phrases tab on the Training page to view the details. |
Aborted Skills | The percentage of skills that were exited early by users (i.e. aborted skills). Calculated as aborted skills divided by total executed and aborted skills during the selected period. Analyzing this metric helps you determine the skills which need to be built or enhanced in order to avoid the users from exiting the chat conversation. Analyzing the user's interaction also helps you to understand why the users aborted the conversations. Click the metrics to navigate to the Aborted Skills tab on the Training page to view the details. |
| Average Accuracy of the Bot | This metric indicates how accurate the bot is at matching user phrases to skills. A lower number indicates that more skills, training and phrases may be required. Calculated as the average NLP score of all entered phrases. Analyzing this metric helps you monitor if accuracy is improving over time. For new bots you should aim to have a score of greater than 70%. Well trained bots should target being above 80%. The average bot accuracy for a given period is to be calculated as follows: Average accuracy = Sum of Accuracy of the skills / number of skills * 100. |
| Total Phrases Trained | The number of phrases added to skills by type of training during the selected time period. User Assisted training is based on phrases added when Automated Training is enabled. Administrator Assisted is the number of phrases that were manually assigned through Training. Clicking this tile navigates you to the Suggested Training tab, where you can assign or un-assign phrases to a suggested skill, existing skill or a new skill. To enable Automated Training see Manage Bot Configuration |
| Skills below the Required Threshold | The percentage of phrases that are not meeting the configured NLP threshold. When a phrase doesn’t meet the threshold a user will be prompted with the Validation message.
|
View Troubleshooting Information
...
Most Popular Skills include the Custom Self-service Skill, ITSM Self-service Skill, Custom Analyst Skill and Skills which Contributed to 80% of Highest Bot Transactions metrics. Analyzing this metrics helps you to understand which skills the users are using effectively. The metrics indicates the most popular skills out of the complete list of skills, and the number of times the skills were executed.
...