Usage Dashboard

Owned by sneha.tripathi
Last updated: Nov 23, 2022
11 min read

The Usage Dashboard provides the Administrators with an insight into how the Virtual Agent is being utilized in the organization. It provides details on License consumption, User Adoption/Engagement, Skills adoption, and Training opportunities. These metrics help the administrator understand and monitor Skills utilization, Unmatched user phrases that help to identify training opportunities for the bot.

To access Usage Chat Dashboard, navigate to Dashboard on the Main Menu, and select Usage Dashboard.

Key metrics related to the following categories are given in the usage Dashboard:

  • License Consumption

  • User Adoption

  • Training Opportunities

  • Skill Adoption

  • Conversations

  • Satisfaction Survey

You may select the Date Range for viewing the Metrics for predefined ranges like Since Inception, Last 24 Hour, Today So Far, Yesterday, Last 7 Days, Last 30 Days, etc., or select Custom to view metric between specific dates. Select “Since Inception“ to view metrics since the deployment of the Virtual Agent inside your organization. By default, the Date range is set to “Last 30 days“. The metrics are recalculated based on selection and are presented on the Dashboard.

Last Refreshed on indicates the time at which the metrics data is calculated and loaded. The dashboard is refreshed automatically every 5 minutes by default. You may click on the refresh icon to view the metrics to reflect the most recent information.

  • License Consumption metrics is not time-bound and is always calculated for the tenant Since Inception.

  • Sessions consumed in the conversations through Test Widget are not considered for calculating the License consumption metrics.

License Consumption

The License Consumption provides the subscription details for the Tenant such as:

  • License Type represents License or Subscription type associated with the Tenant.

  • Total Purchased provides the number of licenses purchased for the Tenant, cumulative from the beginning.

  • Consumed provides the total number of licenses consumed by the Tenant to date.

  • Available provides the total number of licenses that are available for use to date.

User Adoption

This section provides details on how users are interacting with the Virtual Agent.

  • Unique Users indicates the total number of Unique Registered Users who have interacted with the Virtual Agent during the selected time frame.

  • New Registered Users indicates the total number of New Registered Users who have interacted with the Virtual Agent for the first time during the selected time frame.

  • Average Registered Users Chat Time indicates the average time spent per conversation by the users with the Virtual Agent.

Here, Users represent the Registered users in the system and do not include Anonymous users.

Training Opportunities

The metrics in this section help you to understand the training needs of the virtual agent.

  • Unmatched Phrases indicate the total number of user requests or phrases that did not match any skill during the selected time frame.  

  • Average NLP confidence score: It is calculated as the average NLP score of the executed skills for the user requests/phrases during the selected time frame. A lower confidence score indicates the need for further training for the Virtual Agent.

  • Ambiguity Occurrences indicate the total number of ambiguity occurrences identified by the Virtual Agent during the selected time frame.

Skill Adoption

This section provides a list of the most popular/executed skills of the Virtual Agent for the selected time based on the number of skill executions by the users. By default, only 5 Custom skills are displayed. You can include System Skills in the list and configure the number of Skills to be displayed per page. Only Custom skills are shown in the list.

  • Name represents the name of the skill.

  • Number of Executions shows the number of skill executions during the selected timeframe.

  • Include System Skills: If you wish to include System skills in the Skill Adoption report, you can select the checkbox.

  • Small Talk skills are not considered for Skill Adoption calculation.

  • When the “Include System Skills“ is selected, all the System Skills are considered for the Skill Adoption list on the Dashboard and the downloaded report. Specific System Skills cannot be selected.

Conversations

This section provides information on messages sent to and by the Luma Virtual Agent. Here you can see the count of messages sent by the End-users, Virtual Agent, and Support Agents during the selected time frame. The metric includes the conversations from all Chat channels configured for your tenant (except the Test Widget).

Satisfaction Survey

This section provides information on Customer Satisfaction Surveys triggered to end-user by the Luma Virtual Agent. Here you can view your Customer Satisfaction rating, the number of surveys triggered, responses received, and the response rate during the selected time frame. The metric includes the conversations from all Chat channels configured for your tenant (except the Test Widget).

Following details are available for the Skill execution and Support Agent Conversations during the selected time period:

  • Total Surveys: Total number of surveys triggered to the end-user.

  • Total Responses: Total number of responses to the skill survey.

  • Average Rating: This is the Customer Satisfaction Rating. The Average is calculated as the sum of all survey responses received for skill execution divided by the number of responses in the selected time period.

  • Response Rate: This is the rate of Survey responses received. The Response Rate is determined by the total responses divided by the number of surveys triggered to end-users in the selected time period.

Download Reports

You can view and download the metrics information generated for the various segments on the Usage Dashboard. This allows you to get in-depth details on how the metrics were calculated. You can download the report by clicking the download button on the respective section.

Click on the download button on the segment to download the report.

On the confirmation pop-up, click OK to proceed with downloading the report.

The information will be downloaded into a .xlsx file, which can be saved in your local system.

The following are the reports that can be downloaded from the dashboard:

  • License Consumption: Report on the subscription usage of the tenant since the deployment of the bot. Following details are available on the report:

    • User ID represents the userid in Luma Virtual Agent.

    • Username represents the username in Luma Virtual Agent.

    • Email represents the username in Luma Virtual Agent.

    • First Name represents the user’s Firstname as per the user’s profile in Luma Virtual Agent.

    • Last Name represents the user’s Lastname as per the user’s profile in Luma Virtual Agent.

    • Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • Status indicates the user’s current status in Luma Virtual Agent.

    • Created On is the date and time when the user was created in Luma Virtual Agent.

    • Conversations indicate the number of conversations between the user and Luma Virtual Agent to date.

  • Skill Adoption: Report on the Custom and System Skills with the details of the number of times of their execution. Following details are available on the report:

    • Skill Name represents the name of the skill.

    • Number of Executions indicates the number of skill executions during the selected timeframe.

    • Executed By Unique Users represents the number of unique users who executed the Skill during the selected timeframe.

    • Category indicates if the Skill is a Custom skill or System Skill.

    • Last Executed On is the last execution date of the Skill.

  • Conversations: Report on all the messages sent to and by the Luma Virtual Agent. Following details are available on the report:

    • Date is the date and time when the message was sent by the End-User, Virtual Agent, or Support Agent.

    • Username represents the username in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • Conversation ID represents the conversation id as part of which the message is sent.

    • Session ID represents the session id as part of which the message is sent.

    • Channel represents the Luma chat channel through which the message is sent.

    • Skill Identifier: The field contains the skill identifier. If the message is sent as part of a Skill execution, the Skill identifier is populated in the column.

    • Skill Name: If the message is sent as part of a Skill execution, the Skill Name is available in the column.

    • Conversation Initiated By represents if the conversation is initiated by the user or automatically by the Bot.

    • Message Author represents the author (sender) of the message in the conversation.

    • User Message/Channel Event: If the message is sent by the User, the ‘User Message/Channel Event’ column holds the message phrase sent by the End-User or the event triggered.

    • Bot Message: If the message is sent by the Virtual Agent, the field holds the message phrase sent by the Bot.

    • Support Agent Message: If the message is sent by the Support Agent, the field holds the message phrase sent by the Support Agent.

  • Satisfaction Survey: Report on all the Satisfaction Surveys triggered to the end-user and responses received by the Luma Virtual Agent. Following details are available on the report:

    • Date is the date and time when the survey is triggered to the End-User.

    • Conversation ID represents the conversation id as part of which the survey is sent.

    • Related Skill is the skill for which the Customer Satisfaction Survey is triggered

    • Related Agent Username is the Support Agent for which the Customer Satisfaction Survey is triggered

    • Type is the type of Survey triggered.

    • Responded to Survey? indicates if the survey response is received from the end-user.

    • Survey Rating Value represents the survey response received from the end-user.

    • Feedback Text holds the feedback text received for the Survey triggered.

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

  • Unique Registered Users: Report on the Unique Registered Users who have interacted with the Virtual Agent. Following details are available on the report:

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • User Status indicates the user’s current status in Luma Virtual Agent.

    • Total Conversations indicate the number of conversations between the user and the Luma Virtual Agent.

    • Last Conversation ID represents the last conversation between the user and Luma VA

    • Last Conversation Channel is the last channel used by the user to interact with Luma VA

    • Last Conversation Start Date is the conversation initiation date and time

    • Last Conversation End Date is the conversation end date and time

    • Last Session ID is the last session ID in which the user conversation took place.

  • New Registered Users: Reports on New Registered Users who have interacted with the Virtual Agent for the first time. Following details are available on the report:

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • User Status indicates the user’s current status in Luma Virtual Agent.

    • First Conversation ID represents the first conversation between the user and Luma VA

    • First Conversation Channel is the channel used by the user to interact with Luma VA for the very first time

    • First Conversation Start Date is the first conversation initiation date and time

    • First Conversation End Date is the first conversation end date and time

    • First Session ID is the session ID in which the user conversation took place.

  • Average Registered Users Chat Time: Report on the average conversations time i.e. time spent per conversation by the users with the Virtual Agent. Following details are available on the report:

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • User Status indicates the user’s current status in Luma Virtual Agent.

    • Conversation ID represents the conversation between the user and Luma VA

    • Channel is the channel used by the user to interact with Luma VA

    • Conversation Start Date is the conversation initiation time

    • Conversation End Date is the conversation completion time

    • Conversation Duration is the total time spent by the user in the conversation

    • Session ID represents the session in which the user conversation took place.

  • Unmatched Phrases: Reports on all the user requests or phrases that do not match any skill. Following details are available on the report:

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • User Status indicates the user’s current status in Luma Virtual Agent.

    • Phrase is the user request or phrases entered by the user

    • Conversation ID represents the conversation during which the user requested the phrase

    • Channel is the channel used by the user to interact with Luma VA

    • Conversation Start Date is the conversation initiation time

    • Conversation End Date is the conversation completion time

    • Session ID represents the session in which the user conversation took place.

  • Average NLP confidence score: Reports on the average NLP score of the skills executed based on the user requests. Following details are available on the report:

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • User Status indicates the user’s current status in Luma Virtual Agent.

    • Phrase is the user request or phrases entered by the user

    • Skill Name is the Luma skill executed.

    • Skill Identifier is the identifier of the skill executed.

    • NLP Confidence Score is the NLP score at which the skill is executed during the conversation

    • Conversation ID represents the conversation during which the user requested the phrase

    • Channel is the channel used by the user to interact with Luma VA

    • Conversation Start Date is the conversation initiation time

    • Conversation End Date is the conversation completion time

    • Session ID represents the session in which the user conversation took place.

  • Ambiguity Occurrences: Reports on the ambiguity occurrences identified by the Virtual Agent. Following details are available on the report:

    • Username represents the username in Luma Virtual Agent.

    • User Email represents the user’s email id in Luma Virtual Agent.

    • User Role represents the user’s role as per the user’s profile in Luma Virtual Agent.

    • User Group represents the user’s group as per the user’s profile in Luma Virtual Agent.

    • User Status indicates the user’s current status in Luma Virtual Agent.

    • Phrase is the user request or phrases entered by the user

    • Ambiguated Skills: These are the skills identified by the virtual agent after resolving the ambiguity in the user’s request. The skills are comma-separated with the respective confidence score.

    • Proximity Range (%) represents the proximity range used to disambiguate the skill.

    • Confirmation Threshold is the confidence score of the bot at the time of the request

    • Conversation ID represents the conversation during which the user requested the phrase

    • Channel is the channel used by the user to interact with Luma VA

    • Conversation Start Date is the conversation initiation time

    • Conversation End Date is the conversation completion time

    • Session ID represents the session in which the user conversation took place.

© 2019 Serviceaide 1-650-206-8988 http://www.serviceaide.com info@serviceaide.com