Confidence Score is one of the most important factors in Knowledge search. It reflects how confident the system is that the Knowledge Artifact answers the user’s query. The confidence score for an Artifact is calculated based on the metadata matches. It determines the final search results displayed to end-users.
In Luma Knowledge, Artifact search is based on ontology match. the Confidence score is a number ranging from 0 to 1 and indicates the relevancy of an Artifact. Higher the confidence score, the better the Knowledge.
Each Metadata type in Luma Knowledge is allocated a specific Confidence Score Weightage, which is used to calculate the Confidence score for the matching artifact. Confidence Score Weightage is allocated based on the importance of metadata type and the structure of your Knowledge base.
When an end-user searches for Knowledge, the system generates metadata from the user query. The query query’s metadata is matched with the metadata of the Artifacts in Knowledge Base to filter the relevant artifacts. Each Metadata type is assigned a specific confidence score weightage, which is used to calculate the Confidence score for the matching artifact. As matching metadata is found, the respective score is awarded to the artifact. More the number of matching metadata, the better is more the confidence score.Confidence Score and Confidence Score Thresholds are used to classify the matched artifacts in Confidence Bands. The Confidence bands determine the Knowledge artifacts displayed to the end-user as result.
In Luma Knowledge, there are two ways to calculate the Confidence score:
| Table of Contents | ||||
|---|---|---|---|---|
|
Static Confidence Weightage Allocation
...
For a user query, confidence score weightage is assigned based on the ontology generated from the user phrase. This means, that the more metadata types are identified from the user phrase, the more confidence score is available for allocation to matching Artifacts.
...
If a search query generates multiple Subjects and more than one matching subject is available in the Artifacts, an additional weightage of 3% for the phrase is assigned to the metadata type ‘Subject’. The additional weightage for the 'Action' metadata is reduced and assigned to the Subject. The identified metadata is used to calculate the confidence score.
...
Artifact | Matched Metadata | Confidence Score |
|---|---|---|
ChangePassword-Dropbox | path : Dropbox | 0.73 (40+33) |
How to Sign in to Dropbox | path : Dropbox | 0.75 (40+30+5) |
How to change your password for Dropbox | path : Dropbox | 0.45 (40+5) |
| Info |
|---|
|
Dynamic Confidence Weightage Allocation
In Dynamic confidence score allocation, the confidence score is allocated based on the ontology generated from the user query. The confidence score is always calculated based only on the metadata identified. This ensures that Luma Knowledge can find relevant matching Knowledge Artifacts even if the queries are ambiguous or do not generate all the types of metadata.
In dynamic allocation, the metadata types are not assigned a fixed weightage. It is calculated, dynamically based on the metadata identified and the Base weightage points for each metadata type.
There are four steps in calculating the Confidence score in dynamic allocation:
...
Configure Base points
Base points is are the base score scores allocated to each metadata type. Every phrase/word identified as the metadata is assigned the same configured base score. In other words, if the base score of ‘Action’ metadata is 25, each word identified as ‘Action’ from the query is assigned a weightage of 25.
By default, the base weightage is points are configured as below and can be updated as required.
...
| Info |
|---|
|
...
Using the Base points per metadata type, the Total Weightage points for the search is are calculated.
For the ontology generated from the search query, the Weightage points for each metadata type and the Total Weightage points for the search query is are calculated. The Total Weightage points point is a cumulation sum of the weightage points for the identified metadata types. This score is used to derive the Confidence weightage that can be allocated to Artifacts.
...
Base points for Action = 25
Number of words/phrases identified as Motivation Action = 2
Weightage points for Motivation Action = 50 (calculated as 25 x 2 )
...
Weightage points for Motivation = 5
Confidence Weightage for Motivation = 0.02 ( calculate as 5/195)
Weightage for each phrase = 0.02 ( calculated as 0.02/1 = 0.02)
Calculate the Confidence score for Artifact
Using the confidence weightage for the identified metadata, the confidence score for the matching Knowledge artifacts is calculated. Metadata generated from the search query is matched with the Artifact's metadata and the confidence score is assigned accordingly.
...
Metadata | Artifact 1 | Artifact 2 | Artifact 3 | |||
|---|---|---|---|---|---|---|
Matching | Confidence Score | Matching | Confidence Score | Matching | Confidence Score | |
Topic, Subject, Path | 2 | 0.72 | 1 | 0.36 | 2 | 0.72 |
Action | 2 | 0.26 | 1 | 0.13 | 1 | 0.13 |
Motivation | 0 | 0 | 1 | 0.02 | 1 | 0.02 |
Total | 0.98 | 0.51 | 0.87 | |||
Examples
Let us discuss look at the following example examples to understand how the Confidence score is calculated:
...
From the user query 'How to login to dropbox', metadata types Topic, Subject, Action, and Motivation are generated. Based on the identified Metadata and configured base point, the confidence score is allocated to each metadata type.
...
From the user query 'Apply for a Building Permit', metadata types Topic, Subject, and Action are generated. Based on the identified Metadata and configured base point, the confidence score is allocated to each metadata type.
...
Based on the calculated weightage per phrase, the Confidence score for the artifact is calculated.
...