Semantic Search

Semantic search is an advanced information retrieval technique that enhances search accuracy by understanding the user's intent and the contextual meaning of query terms. Unlike traditional keyword-based searches, semantic search delves into the broader context, considering the relationships between words and the meanings of phrases. Instead of relying on metadata and keyword matching, it uses vector-based search, allowing deeper query comprehension. It uses advanced language models to generate precise and reliable answers.

Semantic search eliminates the need for manual curation or keyword extraction. It uses the user’s intent and generates natural, contextually accurate responses. By leveraging vector embeddings, semantic search can identify and retrieve relevant information more effectively, leading to significant improvements in accuracy. It achieves this by using the Retrieval Augmented Generation (RAG) method. RAG enriches the query with information, ensuring that the responses are based on the knowledge base and improving the overall user experience by providing more relevant and accurate information.

Key Features of Semantic Search

Focus on User Intent: Semantic search prioritizes understanding the user's intent behind the query, leading to more relevant and accurate results. Instead of matching exact keywords, it interprets the meaning and context, offering a more intuitive search experience.

No Manual Curation: Unlike keyword search, Semantic search does not require manual curation or keyword extraction due to no dependency on metadata and summary. The zero-touch approach streamlines the curation process, reducing the need for extensive human intervention.

Contextual Understanding: Luma considers the relationships between words and the meanings of phrases, thereby understanding the broader context of the query. This results in more meaningful and accurate search outcomes, even for complex or ambiguous queries.

High Accuracy: The transition to vector embedding-based search has led to significant improvements in accuracy.

Natural Responses: Semantic search responses are natural and contextually appropriate. Instead of merely fetching answers from articles, it leverages advanced language models to construct precise answers, enhancing the user experience.

Retrieval Augmented Generation (RAG): The RAG concept integrates retrieval mechanisms and large language models to generate accurate responses. It ensures that answers are generated based only on the trusted knowledge base rather than on external sources.

Multiple Knowledge Sources: Precise answers from multiple knowledge articles, ensuring that the most relevant and accurate information is provided. This comprehensive approach improves the reliability of the search results.

Flexible Configuration for Tenants: For new tenants, semantic search is enabled automatically. Existing tenants can transition to the new system through a non-reversible process managed by the operations team, ensuring compatibility with semantic search configurations.

Semantic Search vs. Metadata Search

How Does Semantic Search Work?

Open

Semantic Search involves two main steps:

Data Ingestion: When a user provides documents such as PDFs, web pages, and Word files, the text from these documents is extracted and chunked into manageable pieces. These chunks are processed by an embedding model that converts the text into vectors, which are subsequently stored in a vector database.

Query/Data Retrieval: During the Query/Data Retrieval phase, the user query is processed by the system to identify the relevant chunks from the vector database. These chunks, along with the user’s query, are then used to generate a precise response using a large language model (LLM), which is then delivered back to the user.

This process highlights the integration of text extraction, vectorization, and advanced AI models to enhance the accuracy and relevance of search results.