Calculating Confidence score

Confidence Score is one of the most important factors in Knowledge search. It reflects how confident the system is that the Knowledge Artifact answers the user’s query. The confidence score for an Artifact is calculated based on the metadata matches. It determines the final search results displayed to end-users.

In Luma Knowledge, the Confidence score is a number ranging from 0 to 1 and indicates the relevancy of an Artifact. Higher the confidence score, the better the Knowledge.

Each Metadata type in Luma Knowledge is allocated a specific Confidence Score Weightage, which is used to calculate the Confidence score for the matching artifact. Confidence Score Weightage is allocated based on the importance of metadata type and the structure of your Knowledge base.

When an end-user searches for Knowledge, the system generates metadata from the user query. The query’s metadata is matched with the metadata of the Artifacts in Knowledge Base to filter the relevant artifacts. As matching metadata is found, the respective score is awarded to the artifact. More the number of matching metadata, the more the confidence score.

In Luma Knowledge, there are two ways to calculate the Confidence score:

Static Confidence Weightage Allocation

This is the default confidence weightage allocation method used in Luma Knowledge. In 'Static' allocation, the confidence score weightage assigned to each Metadata Type is fixed and the Artifacts Confidence score is evaluated accordingly. By default, the following weightage is assigned to each metadata type:

For a user query, confidence score weightage is assigned based on the ontology generated from the user phrase. This means, that the more metadata types are identified from the user phrase, the more confidence score is available for allocation to matching Artifacts.

Let us discuss the following example to understand how the Confidence score is calculated:

Example 1: “Describe Knowledge Management”- Query generates Metadata Topic, Subject, and Action

From the user query 'Describe Knowledge Management', metadata types Topic, Subject, and Action are generated. Based on the identified Metadata, the confidence score weightage is allocated to each metadata type. Since Motivation is not identified from the user phrase, it is not used for confidence score evaluation.

Luma Knowledge identifies Artifacts that match with the ontology generated from the user query. Based on the metadata type and number of metadata matches, the confidence score for the matching Artifacts is calculated.

Example 2: Multiple Subject match

If a search query generates multiple Subjects and more than one matching subject is available in the Artifacts, an additional weightage of 3% for the phrase is assigned to the metadata type ‘Subject’. The additional weightage for the 'Action' metadata is reduced and assigned to the Subject. The identified metadata is used to calculate the confidence score.

For user query “How to create a dropbox account“, metadata types Topic, Subject, Motivation, and Action are generated.

Based on the identified metadata, the confidence score for the matching Artifacts is calculated.

Dynamic Confidence Weightage Allocation

In Dynamic confidence score allocation, the confidence score is allocated based on the ontology generated from the user query. The confidence score is always calculated based only on the metadata identified. This ensures that Luma Knowledge can find matching Knowledge Artifacts even if the queries are ambiguous or do not generate all the types of metadata.

In dynamic allocation, the metadata types are not assigned a fixed weightage. It is calculated, dynamically based on the metadata identified and the Base weightage points for each metadata type.

There are four steps in calculating the Confidence score in dynamic allocation:

Open

Configure Base points

Base points are the base scores allocated to each metadata type. Every phrase/word identified as the metadata is assigned the configured base score. In other words, if the base score of ‘Action’ metadata is 25, each word identified as ‘Action’ from the query is assigned a weightage of 25.

By default, the base points are configured as below and can be updated as required.

Calculate Total Weightage points for the query and matched phrases

Using the Base points per metadata type, the Total Weightage points for the search are calculated.

For the ontology generated from the search query, the Weightage points for each metadata type and the Total Weightage points for the search query are calculated. The Total Weightage point is a sum of the weightage points for the identified metadata types. This score is used to derive the Confidence weightage that can be allocated to Artifacts.

For example, if a search query generates metadata Topic, Subject, Action and Motivation:

Derive Confidence Weightage

Now using the Weightage points for each metadata type and Total Weightage points, we can derive the Confidence weightage for each metadata type. Based on metadata generated from the user query, the confidence weightage is calculated. The weightage is calculated only for the metadata type identified from the query string.

In the above example,

Calculate the Confidence score for Artifact

Using the confidence weightage for the identified metadata, the confidence score for the matching Knowledge artifacts is calculated. Metadata generated from the search query is matched with the Artifact's metadata and the confidence score is assigned accordingly.

For an Artifact with matching metadata, the confidence score is calculated as below:

Examples

Let us look at the following examples to understand how the Confidence score is calculated:

Example 1: “How to login to dropbox“

From the user query 'How to login to dropbox', metadata types Topic, Subject, Action, and Motivation are generated. Based on the identified Metadata and configured base point, the confidence score is allocated to each metadata type.

Based on the calculated weightage per phrase, Confidence score for the artifact is calculated.

Example 2: “Apply for a Building Permit“

From the user query 'Apply for a Building Permit', metadata types Topic, Subject, and Action are generated. Based on the identified Metadata and configured base point, the confidence score is allocated to each metadata type.

Based on the calculated weightage per phrase, the Confidence score for the artifact is calculated.

Example 3: “Wifi Router“

From the user query 'Wifi Router', metadata types Topic and Subject are generated. Based on the identified Metadata and configured base point, the confidence score is allocated to each metadata type.

Based on the calculated weightage per phrase, the Confidence score for the artifact is calculated.