Semantic Chunking for Enhanced Knowledge Curation

Luma now supports Semantic Chunking during knowledge curation. This feature intelligently segments content into meaningful chunks, keeping related information together. As a result, Luma can generate more accurate and contextually relevant answers, since the system can reference complete, logically grouped data.

Note: Semantic Chunking is automatically enabled for new tenants. To enable it for existing tenants, get in touch with the Serviceaide Support Team.

Improvements in Federated service

The following updates improve visibility and navigation within Federated Services:

• Post-import logs now provide more detailed information, making it easier to trace and verify content synchronization activities. For more information, refer to Configuration Tips.

• You can now directly access Data Source and Data Object details from the Sync Pairs screen, simplifying review and configuration workflows.

Other enhancements

Localized Precise Answers: The Precise Answer prompt has been enhanced to automatically return responses in the Bot’s configured language, ensuring a smoother, localized user experience.

Catalog Semantic Search

New Intent-Driven Semantic Catalog Search that allows Luma to understand the user’s intent and retrieve the most relevant catalog item. It enables the system to identify catalogs even if the user’s phrasing is vague, partial, or informal. Note: Please raise a ticket with the support team to migrate your Catalogs to Semantic Search.

More Details - https://serviceaide.atlassian.net/wiki/x/AYCnMgY

Precise answer Improvements

The following are the Precise answer enhancements that are part of this release:

• Prompt updated to generate an answer in the language of the question asked.

• Stronger Guardrails to ensure the precise answer is only generated from the available Knowledge content.

Artifact Retirement Based on Website Content Updates (Webcrawler Enhancements)

The system now supports automatic retirement of artifacts that are no longer present on the source website. During each crawl, the content of the website is compared against the content captured in the previous crawl. If a specific artifact is found to have been removed from the source, it is marked for retirement and subsequently removed from the repository. This ensures that the repository remains aligned with the current state of the source website and does not retain outdated content.

More details - Web Crawler Utility

Other Enhancements

Here are other enhancements from this release:

• Temporal Search Improvement enhances search by prioritizing recent, relevant results. We now support time filters such as Latest, recent, Last, etc.

• Artifact Volume report now also provide Answer and Source article details.

Artifact creation improvements

Following are the Artifact creation enhancements that are part of this release:

• Luma introduces the ‘Document intelligence service’ to effectively create artifacts from various types of file types. Luma now supports a wide variety of document types for knowledge article ingestion. You can now use the Textual/Word, PPT, JPG, PNG, and JPEG to create artifacts.

• Ability to extract information from images uploaded as PDF documents, while creating artifacts. Note that you should contact the support team to enable this service.

• Ability to add Display Image for Knowledge articles.

Knowledge Search improvements

Following are the Knowledge Search updates that are part of this release:

• Enhance the context sent to LLM while generating precise answers. Luma now augments the artifact chunks with article Name and Keywords, providing additional details to LLM to generate better answers.

• Customizable prompts for precise answers tailored to organizational needs.

• Artifact search API now supports multiple article IDs for search

Other enhancements

Here are other enhancements from this release:

• Improved log cleanup processes for federated service synchronization.

• Enhanced Federated Service with delta sync for Azure AD

Search Enhancement

Support for Knowledge Search with Multiple article ids. The Knowledge search API now supports searching for multiple articles with ids.

Web Crawler enhancement

Web Crawler is enhanced to detect the delta in consequent, log error and scheduled execution.

Semantic Knowledge Search

Introducing Semantic Knowledge Search.  This release enhances Knowledge Search in Luma by utilizing vector embedding and full-text search to improve the identification of relevant artifacts, integrating Generative AI for precise answer generation, and structuring the search result set to provide comprehensive and accurate responses. The key benefits of Semantic search are:

• Zero-Touch Curation: No manual keyword extraction or metadata fine-tuning.

• Accuracy Boost: Accuracy improved to over 94%.

• Enhanced User Experience: Natural and precise responses to every user query, leveraging Large Language Models (LLM). Every search results in:

  • Precise Answer from one or more relevant knowledge articles.

  • Source artifacts that are used to generate the precise answer

  • All Relevant artifacts found.

• RAG Implementation: Answers are generated based on the knowledge base, ensuring reliability.

These updates streamline curation, improve accuracy, and enhance user interaction with semantic search.

Note: All new tenants will be provisioned with Semantic search by default. For existing tenants, get in touch with the Operations team to enable Semantic search. This would require migrating the artifact structure to support semantic search i.e. reprocessing of all the Artifacts available in the Tenant. This is a non-reversible process. Once migrated, cannot be reverted.

Updates to Artifact Curation

Following changes to Artifact Curation are part of this release:

• With migration Semantic Search, Artifact curation is not required. So, Luma does not generate Summary, Metadata and FAQs. Topics are generated automatically to main create topic hierarchy in Knowledge graph.

• FAQs and Key phrases can be manually added by the curator to enhance responses.

• Auto-publish configuration parameter: Since there are no dependency on metadata with Semantic search, artifacts can either be kept in Draft or Automatic Published.

• Template: ‘Ontology Type’ is removed from Template fields for Semantic search.

Note: The changes are applicable for only Semantic search. For Metadata based search, the existing behavior should remain as is.

Dashboard Enhancements

The following changes are available in the feedback metrics and the Dashboard reports:

• Luma Knowledge now accepts feedback for the precise answer. The user feedback (positive or negative) is added to each source artifact.

• Views Count: When Luma VA displays a precise answer, the source artifacts are considered viewed. Views are incremented on all the source articles.

• Ineffective best response: The feedback is also rolled over to the artifacts that are part of the search response.

• Artifact Volume is now renamed to Search Volume and divided into two counts: Returned Artifacts and Returned Answers.

Knowledge Curation from Websites (Utility)

Crawl through the Websites and curate knowledge for each web page.

• Knowledge in Luma Knowledge provides Luma access to all the content on the website, making it easier to find the information that users are looking for.

• Generate precise answers to the questions, eliminating the dependency on QnA Maker service.

• Users will be able to find the information they need more quickly, which can improve their overall experience.

• View Artifact to view the original Knowledge source i.e. Webpage, retaining the parent format.

Knowledge Curation from PDFs

Curate Knowledge for each PDF file

• Generate precise answers to the questions, eliminating the dependency on QnA Maker service.

• View Artifact to view the PDF document, retaining the parent format.

• Artifacts in Luma Knowledge ensures that Users find the information they need more quickly, which improves the overall experience.

Wiki Integration

• Configure Federated sync from the Wiki Pages importing information in Luma Knowledge to curate Articles.

• Automatic Sync to pull latest content from your website.

• Generate precise answers to the questions, eliminating the dependency on QnA Maker service.

• View Article to redirect the user to the original source i.e. the wiki page.

Unification of VA and KM

Catalog Management screens to look similar to Luma VA

Unification of VA and KM.

As part of VA-KM unification, following enahncements are done as part of this release:

• Create-Artifact screen enhancement: Revamp Create Artifact screen to look similar to VSA 

• Knowledge Store and Knowledge Graph screen UI changes

• Vocabulary screen , User Management , TenantConfiguration screen and Templates screen UI revamp 

• Remove the dependency on CSDM to correlate tenants & users.

• Support to Provision KM tenant in given VSA Database

• KM Tenant Deprovisioning when VA tenant is provisioned

• Disable User management in KM in unified Tenant

Improvements in Knowledge Search results

Following enhancements are part of the release:

• Artifact Template fields can now be part of Search Results sent to Luma VA. This enables the Skill developers to implement use cases that display the artifact details as part of conversation flows.

• The Artifact Source Name is now available in Artifact Search Results

DEprecating QnA Maker

• Deprecate Q&A Maker and replace FAQ creation via Azure Question Answering Service

• Web URL Parsing: Replace QnA Marker while creating artifacts using Web URL

• Sharepoint Integration without QnA maker

Curation improvements

• Ability to execute all Artifact Curation processes even if one of the processes fails

• Bulk Retry action on Artifact List screen

• Implement Retry Functionality for Artifact information(Open AI and Non-Open AI)

Default Domain

Default Domain for every tenant

Other enhancements

• Updates on Knowledge Store--> Artifact list

• Artifact Details screen updates

Improvements to Knowledge Curation

Following updates have been done to the Knowledge Curation process:

• Luma Knowledge now supports bulk FAQ creation through sync. As a curator you can now upload bulk FAQs using a document and generate metadata.

• FAQ Similarity check is now a tenant configuration that allows the curator to turn off FAQ similarity check for bulk loads.

Knowledge Search Improvements

Enhancing the Knowledge presented to the end-user, the following additional details are now available in the Search results sent to Luma VA:

• Support for Knowledge Source and Artifact Name in the Knowledge Search results. For Knowledge Articles curated using a template with a 'Source' field, the Knowledge source and Artifact name is sent in the search results. The information can be used by the Virtual Agent to display more information to the end user.

• Parent Article link available in the results when a FAQ is returned as result.

UI Enhancements

The following UI enhancements are part of this release:

• Revamped the KM Dashboard to align with the look and feel of the Luma Bot Builder.

• Consistent curation information for articles curated in the system with or without Generative AI.

• Inline status bar on the Artifact Details screen to show the current curation status. Curator can Edit and Save the Knowledge to retrigger curation if any of the stages fail.

Centralized Generative AI Configurations

Centralized Generative AI Configurations for Luma Suite of Products. Empowers Superadmins to effortlessly configure and oversee the Generative AI functionalities for all Luma products, simplifying management and optimizing control through one consolidated platform.

For more information, refer to Configure Generative AI Configurations.

Async Artifact Curation

Knowledge artifacts are now generated asynchronously. Luma Knowledge now executes curation sub-processes such as Summary generation, Metadata creation, and FAQ generation in the background, while the Artifact is available on the Knowledge Store with basic information . Keep track of the curation status on the Artifact details screen. You can restart the sub-processes in case of failures or incorrect information, ensuring a hassle free artifact curation experience.

For more information, refer to Create Artifact using Knowledge Templates.

AI Services Integration

Luma Knowledge is now fully integrated with Luma AI services. Harnessing the power of advanced Prompt engineering, the Generative AI functionalities in Luma Knowledge now uses the AI services. The integration streamlines the metering and management processes for optimal utilization.

Virtual Agent- Luma Knowledge Unification

In process of unifying Luma Knowledge and Luma Virtual Agent, the following updates are done as part of this release:

• Luma Knowledge is now available as a worspace within Luma Bot Builder with implicit login. You can now launch your Luma Knowledge Curation portal directly from Luma admin portal, without the need to log in again.

• Revamped the Side Navigation to align with the look and feel of the Luma Bot Builder.

• Schema changes to support VA and KM Unification.

Other Enhancements

Following are the other updates that are part of this release:

• Tenant language information is now available in the Tenant Details screen.

• You can now sync artifacts from Sharepoint and create artifacts in Luma knowledge using templates.

• Accessing the the Artifact URL from Luma VA now authenticates the user and allows the user to access the Knowledge Artifacts.

• Luma Knowledge now generates FAQs based on Knowledge content. The system analyzes the content and generates FAQs based on the complexity and length of the content.

Text Similarity Service - DE Dup Artifact FAQs

Luma Knowledge now prevents duplicate FAQs during artifact creation. Using in-built Text similarity NLP service, system can now identify and ignore duplicate FAQs with Text similarity of over 90%.
Note: A tenant-level configuration parameter is added to identify the similarity with default value set to 90%. This is backend .

For more information, refer to Create Artifact

Artifact Curation Improvements

Following updates to Artifact curation process are part of this release:

• Every Domain in Knowledge graph now has a default topic, which is automatically assigned to an artifact under the domain if the identified topic is not available in the Knowledge graph. This ensures new Artifacts are always created with a Topic- eliminating the need to add a topic before publishing the artifact.

• Artifact FAQs are now kept up-to date based on the Knowledge. Luma Knowledge now regenerates FAQs whenever the artifact is updated. The existing FAQs are replaced with valid FAQs based on latest Knowledge content.

For more information, refer to Create artifact.

Knowledge API updates

The following Luma Knowledge APIs are updated as part of this release:

• Enhanced Catalog Search API now enables searching for Catalogs in one of more targeted category paths.

• New API to enable OpenAI and relevant features in Luma Knowledge.

UI Improvements

Following are the UI updates that are part of this release:

• Revamp Knowledge Gap screen to quick and easy Knowledge Gap resolutions

• ‘Generate Access Key’ action is now moved to ‘Tenant Details’ screen from the User profile section

• Events for ‘Precise answer’ are now available in the Search Logs. The troubleshoot logs now correctly depicts how is Knowledge presented to the end-users.

• Knowledge graph view on ‘Knowledge Store’ is changed to File explorer view for quick and easy filter and access of Knowledge Articles

• Catgory graph view on ‘Catelog Store’ is changed to File explorer view for quick and easy filter and access of Catalogs available in the system.

Technical Upgrades

Following are the technical upgrades that are part of this release:

• Upgrade MySQL server to MySQL 8.x

• Service bus Improvements: Avoid temp Queues for Request-Reply Operation

Enhancement to Open AI Integrtaion - Support for Non-English languages

Luma Knowledge now support non-English languages when integrating with Open AI. With localization support, Knowledge curator can add Knowledge using content in any language. Luma can easily interprete user intent and respond in the same language.

New Tenant Details screen

View your tenant details on the new Tenant Details screen. The tenant details screen provides details such as Tenant name, external id that are used to integrate Luma Knowledge with other systems.

Improved APIs

The following updated to Luma Knowledge APIs are part of this release:

• The Knowledge Search API now supports Key Phrase-based search. You can now use Keyphrases in addition to query and metadata to find relevant artifacts.

• New API to record feedback and views on FAQs and Artifacts

Other Enhancements

Other enhancements that are part of this release are:

• New Graphical topic and category representation on Knowledge graph and Catelog graph

• Additional Artifact details such as artifact type, Parent artifact details are now available in the excel download for Artifacts Insights.

• Root folder in Knowledge Graph is now renamed as the tenant name.

Integration with Open AI

Leverage Generative AI (Open AI) technology to enhance following functionality within Luma Knowledge.

• Precise answering: Generate precise answers to user’s query from the existing Knowledge.

• Key Phrase and Key word generation: Automatically generate relevant key phrases and keywords for the Knowledge Articles.

• Summary generation: Automatically summarize a content or an article.

• Question and Answer generation: Generate frequently asked questions and corresponding answers based on the content of artifacts.

• Knowledge gap fulfilment: Resolve identified Knowledge Gaps by generating content for the unanswered user queries.

For more information, refer to Integrating Advanced Language Models.

Note: Open AI integration is currently available only for English tenants. To enable Open AI for your tenant, please contact Serviceaide Support Team.

Improved Artifact creation

Enhanced Artifact Creation process. Luma Knowledge now uses Open AI to automatically generate Summary, QnA pairs, Key phrases and Keywords for an Artifact. With Generative AI, system can generate suitable metadata and summary, reducing the time and effort required to review/manage artifacts.
For more information, refer to Create Artifacts using Templates.

Note: To leverage the feature, Open AI/ Language Model should be enabled for your tenant.

Enhanced Knowledge Search with Precise Answers

Get precise answers specific to the question instead of going through the Artifacts. Using Open AI, Luma Knowledge can now interpret user’s request and generate exact answers for user requests. This allows users to access the information quickly. With precise answers, the end-user can view knowledge without the need to go through the whole article.
For more information, refer to Knowledge Search.

Note: To leverage the feature, Open AI/ Language Model should be enabled for your tenant.

Search History

Knowledge Curators and Administrators can now view Search History in Luma Knowledge. The new screen enables curators to revisit search queries and review the corresponding search results. This helps in analyzing the effectiveness and relevance of the search results over time.
For more information, refer to Search History.

Export Artifact Report

Generate a Artifact details report. Curators and Administrators can export the details of Articles in Knowledge Store to a CSV file.
For more information, refer to Knowledge Store | Export Artifact Report

New Tenant Configurations

Following Tenant- level configurations are added as part of the release:

1. Enables Open AI: Activate Open AI for your tenant to leverage advanced language models to manage your knowledge base and conduct more efficient searches effectively.

2. Enable open AI Features: Enable the required Open AI functionalities for your tenant. Ensure that Open AI is already enabled for your tenant.

For more information, refer to Tenant Configurations.

Other Enhancements

Following are the other changes done as part of this release:

• Revamped Artifact Details screen.
  
  

Service Catalogs in Luma Knowledge

You can now sync your access Service Catalogs in your ITSM system through Luma Knowledge. Import your existing Service Catalogs and enable end-users to search for the service catalog. New module 'Catalog Management' in Luma Knowledge allows the Administrator to manage the service catalogs used in their organization. You can also create new catalogs and categorize them. In addition to being searchable through the Knowledge Search Widget, the Catalogs are now available through Luma Virtual Agent.

Similar to Knowledge Articles, the catalogs are available in Catalog Store and Catalog Graph.

For more information, refer to Catalog Management

Explore Catalogs

You can now explore the Catalogs available in Luma Knowledge using the Explore tab on the Search Widget. It enables the end-users to explore the Catalog Graph and view available catalogs. You can browse through the categories and locate the required Catalog.

Catalog KPIs on Dashboard

You can monitor Service Catalog usage (through Luma VA) on Luma Knowledge Dashboard. The Catalog KPIs on the Dashboard provide a view on Total Catalog search request, Catalogs returned as results and Tickets created using the catalog (via Luma VA)

For more information, refer to Luma Knowledge Dashboard.

Enhanced Navigation

The left navigation on Luma Knowledge admin portal is now enhanced for better categorization of modules. Following are the changes:

1. New module 'Catalog Management' is available to allow administrator to mange their Service Catalogs

2. Knowledge related screens like ‘Knowledge Graph’, ‘Knowledge Store’, ‘Semantic analysis’, ‘Templates’ are now available under module 'Knowledge Management'.

3. ‘Import status’ and ‘Vocabulary’ are now part of ‘Tenant’ section

Tenant configuration parameters

New configuration parameter ‘Minimum Catalogs for guided result’ is available under Configuration → Search. The parameter helps Luma disabmiguate the results and determines when user should be guided to locate the service catalog. If the search results in more catalogs than the defined number, user is presented with a guided navigation to browse through the categories and view the most relevant catelog.

Out-of-the-box Catalog Federated Sync

Luma Knowledge enables the administrators to sync their Service Catalogs from their ITSM systems to Luma Knowledge. The Federated syncs with Change Gear, PoB and ISM are available out-of-the-box. The configuration should be updated to connect to your environment.

For more details, you may get in touch with Serviceaide Support Team.

Create Catalogs

In addition to importing catalogs via sync, you can also create the catalogs in Luma Knowledge. System provides the following ways to create catalogs in Luma Knowledge:

• Manual Catalog creation: You can use the ‘Create Catalog’ screen to manually fill the catalog template and create the catalogs.

• Import Catalog using Excel: You can now create multiple Catalogs with a single upload using 'Catalog Import'. Luma Knowledge also supports bulk catalog import using an Excel sheet. Build your excel columns based on the catalog template and add the catalogs as rows in the sheet. Catalog Import in Luma Knowledge creates a catalog for each row and builds the Category Heirarchy based on the information provided.

For more information, refer to Create Catalog.