Calculating Confidence score

Confidence Score is one of the most important factors in Knowledge search. It reflects how confident the system is that the Knowledge Artifact answers the user’s query. The confidence score for an Artifact is calculated based on the metadata matches. It determines the final search results displayed to end-users.

In Luma Knowledge, Artifact search is based on ontology match. When an end-user searches for Knowledge, the system generates metadata from the user query. The query metadata is matched with the metadata of the Artifacts in Knowledge Base to filter the relevant artifacts. Each Metadata type is assigned a specific confidence score weightage, which is used to calculate the Confidence score for the matching artifact. As matching metadata is found, the respective score is awarded to the artifact. More the number of matching metadata, the better is the confidence score.

Confidence Score and Confidence Score Thresholds are used to classify the matched artifacts in Confidence Bands. The Confidence bands determine the Knowledge artifacts displayed to the end-user as result.

In Luma Knowledge, there are two ways to calculate the Confidence score:

Static Confidence Weightage Allocation

This is the default confidence weightage allocation method used in Luma Knowledge. In 'Static' allocation, the confidence score weightage assigned to each Metadata Type is fixed and the Artifacts Confidence score is evaluated accordingly. By default, the following weightage is assigned to each metadata type:

For a user query, confidence score weightage is assigned based on the ontology generated from the user phrase. This means, the more metadata types are identified from the user phrase, the more confidence score is available for allocation to matching Artifacts.

Let us discuss the following example to understand how the Confidence score is calculated:

Example 1: “Describe Knowledge Management”- Query generates Metadata Topic, Subject, and Action

From the user query 'Describe Knowledge Management', metadata types Topic, Subject, and Action are generated. Based on the identified Metadata, the confidence score weightage is allocated to each metadata type. Since Motivation is not identified from the user phrase, it is not used for confidence score evaluation.

Luma Knowledge identifies Artifacts that match with the ontology generated from the user query. Based on the metadata type and number of metadata matches, the confidence score for the matching Artifacts is calculated.

Example 2: Multiple Subject match

If a search query generates multiple Subjects and more than one matching subject is available in the Artifacts, an additional weightage of 3% is assigned to the metadata type ‘Subject’. The additional weightage for the 'Action' metadata is reduced and assigned to the Subject. The identified metadata is used to calculate the confidence score.

For user query “How to create a dropbox account“, metadata types Topic, Subject, Motivation, and Action are generated.

Based on the identified metadata, the confidence score for the matching Artifacts is calculated.

Dynamic Confidence Weightage Allocation

In Dynamic confidence score allocation, the confidence score is allocated based on the ontology generated from the user query. The confidence score is always calculated based only on the metadata identified. This ensures that Luma Knowledge can find relevant Knowledge Artifacts even if the queries are ambiguous or do not generate all the types of metadata.

In dynamic allocation, the metadata types are not assigned a fixed weightage. It is calculated, dynamically based on the metadata identified and the Base weightage points for each metadata type.

Configure Base points

Base points is the base score allocated to each metadata type. Every phrase/word identified as the metadata is assigned the same score. In other words, if the base score of ‘Action’ metadata is 25, each word identified as ‘Action’ from the query is assigned a weightage of 25.

By default, the base weightage is configured as below and can be updated as required.

Calculate Total Weightage points for the query and matched phrases

Using the Base points per metadata type, the Total Weightage points for the search is calculated.

For the ontology generated from the search query, the Weightage points for each metadata type and the Total Weightage points for the search query is calculated. The Total Weightage points is a cumulation of the weightage points for the identified metadata types. This score is used to derive the Confidence weightage that can be allocated to Artifacts.

For example, if a search query generates metadata Topic, Subject, Action and Motivation:

Base points for Topic and Subjects = 70
Number of words/phrases identified as Topic and Subjects identified = 2
Weightage points for Topic and Subjects = 140 (calculated as 70 x 2 )

Base points for Action = 25
Number of words/phrases identified as Motivation = 2
Weightage points for Motivation = 50 (calculated as 25 x 2 )

Base points for Motivation = 5
Number of words/phrases identified as Motivation = 1
Weightage points for Motivation = 5 (calculated as 5 x 1 )

Total Weightage points for the search = 195 ( calculated as 140+50+5)

Derive Confidence Weightage

Now using the Weightage points for each metadata type and Total Weightage points, we can derive the Confidence weightage for each metadata type. Based on metadata generated from the user query, the confidence weightage is calculated. The weightage is calculated only for the metadata type identified from the query string.

In the above example,

Total Weightage points for the search = 195

Weightage points for Topic and Subjects =140
Confidence Weightage for Topic and Subjects = 0.72 ( calculate as 140/195)
Weightage for each phrase = 0.36 ( calculated as 0.72/2 = 0.36)

Weightage points for Action = 50
Confidence Weightage for Action = 0.26 ( calculate as 50/195)
Weightage for each phrase = 0.13 ( calculated as 0.26/2 = 0.13)

Weightage points for Motivation = 5
Confidence Weightage for Motivation = 0.02 ( calculate as 5/195)
Weightage for each phrase = 0.02 ( calculated as 0.02/1 = 0.02)

Calculate Confidence score for Artifact

Using the confidence weightage for the identified metadata, confidence score for the matching Knowledge artifacts is calculated. Metadata generated from the search query is matched with the Artifact's metadata and confidence score is assigned accordingly.

For an Artifact with matching metadata, the confidence score is calculated as below:

Examples

Let us discuss the following example to understand how the Confidence score is calculated:

Example 1: “How to login to dropbox“

From the user query 'How to login to dropbox', metadata types Topic, Subject, Action and Motivation are generated. Based on the identified Metadata and configured base point, confidence score is allocated to each metadata type.

Based on the calculated weightage per phrase, Confidence score for the artifact is calculated.

Example 2: “Apply for a Building Permit“

From the user query 'Apply for a Building Permit', metadata types Topic, Subject, and Action are generated. Based on the identified Metadata and configured base point, confidence score is allocated to each metadata type.

Based on the calculated weightage per phrase, Confidence score for the artifact is calculated.

Example 3: “Wifi Router“

From the user query 'Wifi Router', metadata types Topic and Subject are generated. Based on the identified Metadata and configured base point, the confidence score is allocated to each metadata type.

Based on the calculated weightage per phrase, the Confidence score for the artifact is calculated.