Agentic AI is great for building your own conversational output that provides customers with the answers they need immediately, powering generative AI applications.
While a basic search engine for your site returns lexical-match results, agentic AI can return semantic results, meaning you can provide answers based on similar words rather than exact matches using semantic search techniques.
The issue for businesses is that answers to customer questions must be grounded in relevant data and aligned with their products and services.
You don't want the entire internet as a source for answers, so you need to define where the answers come from using a retrieval-augmented generation (RAG) system.
A RAG system feeds an LLM its data for customized searches, reducing the risk of hallucinations, irrelevant answers, or inaccurate results by integrating external data with internal sources.
Building a RAG system requires your own corpus, meaning the data that you want to use to customize output from multiple data sources.
Before you start building your own RAG framework, you need:
- An understanding of RAG for unstructured data
- How structured RAG works, especially in your own environment
- The benefits of RAG for your business
- Challenges you might face while building a RAG structure
- Steps to implement your RAG agents
- Examples of structured RAG
- Differences between RAG and knowledge graphs
- Differences between RAG and data lakes
- Some tools to support structured RAG
What is RAG for structured data?
Structured RAG uses data stored in an organized, well-defined database schema.
In any RAG, you need data for your semantic searches, and structured data is one form that can be used. Relational databases like PostgreSQL or MySQL store structured data.
Structured data could also be stored in a file format such as an Excel spreadsheet or a CSV file, both common tabular data formats.
The difference between structured and unstructured data is that structured data is stored in a field of a given type, so you always know what data types to expect from a query.
For example, in a structured database, you have a string field for a user's first name. You must use a string for this field.
In an unstructured database, you would store all user information as documents rather than in specific column types.
How does structured RAG work?
Structured RAG works by retrieving data from structured sources within your organization via a search query, processing it into a vectorized representation with vector embeddings, and then sending the results to an LLM.
Structured RAG agents use metadata to define specific chunks in their vectorized output, further refining the data to improve filtering and retrieval precision.
It's useful when you have large reservoirs of enterprise data and want to build high-performance real-time interactions with your customers.
How structured data is retrieved
Structured data databases use Structured Query Language (SQL) to retrieve subsets of their data through an SQL query.
After the data is indexed, each data chunk is tagged with metadata to describe the type of information it contains. This supports vector search and improves semantic matching.
Structured RAG systems operate on rows of data, and each row can serve as a source for the embedding step in the RAG pipeline.
Businesses can adopt a hybrid approach in which RAG uses exact-match filters alongside semantic searches, which often yields better results.
How data is injected into prompts
A RAG agent needs to inject data into prompts using a specific structure. JSON is common because it's an agnostic format that most API integrations use. Other formats are XML, Markdown, and even plain text.
The structure that you use depends on your business use case, but most applications work with JSON, largely because it supports scalability.
Structure is also important when your application ingests output from an LLM to process it further in downstream workflows. The RAG system instructs the LLM to return answers in a specific structure.
Sticking to a specific structure for both output and input is called schema grounding.
Before you consider your schema structure, remember that the amount of data (context window) you use will affect costs. It's best to limit your context window, which means that you limit the amount of data passed to the LLM without losing the accuracy of your results. This will optimize your AI costs.
What are the benefits of structured RAG?
Structured RAG has several benefits, most of which revolve around the performance of real-time conversational applications and improved decision-making. Result accuracy reduces customer frustration, potential legal issues, and compliance violations.
Here are a few key benefits of structured RAG:
In more detail:
- Reduced AI hallucinations: Using your own data as the basis for LLM output, you reduce the chance of hallucinations and inaccuracies in responses.
- Real-time updates: Real-time access to structured data means that any changes to your policies or products will immediately be reflected in your responses.
- Better control over the output: Businesses have greater control over answers because they use their own data as a source rather than the entire internet.
- Lower costs: Reduced context window saves money on LLM costs.
Benefits aside, however, building a structured RAG also brings challenges.
What challenges does structured RAG face?
Challenges in structured RAG mainly come from the limitations of structured databases and the complexity of building reliable RAG systems.
Structured data has strict constraints on what can be stored in each field. Developers working with structured RAG systems must map out relationships and determine how to build queries and filters to reduce the context window sent to the LLM.
RAG agents work with conversational user input in natural language, making it difficult to translate semantic queries into structured ones. This involves understanding user intent, which can be difficult to map to metadata.
Large amounts of data may be required to process semantic queries, which can slow response times. In addition to query and processing speed, databases must be kept up to date with the latest information to remain fresh and relevant to the user's query.
For example, if you have a new product for sale, you must ensure that the new product information is included in queries, or users of your store might not get accurate answers.
Security is also an issue, and you want to ensure that users cannot query for the prompt, sensitive system information, or use an attack called prompt injection.
What happens if a user asks the agent to respond with information about the underlying server architecture? The agent should refuse to answer such questions.
As you build a RAG structure, consider the solutions to these challenges:
- Map out the structure of your data, because structured data has specific constraints on the type of data that can be stored in each field.
- Process structured data as much as possible to reduce the context window sent to the LLM.
- Considering all semantic user questions is difficult, so the RAG will require several rounds of testing and possible updates when new questions are sent.
- Update the base structured database to keep the corpus fresh.
- Include security during testing to avoid prompt injection or prompt theft.
After you define and resolve these challenges, you can move on to integration.
How is structured RAG implemented?
Implementation of a RAG system has a few common components:
- Ingestion of your corpus, usually from database queries
- Tagging data with your metadata to define query results
- Creating a vector of numbers and storing it in a vector database
- Building a hybrid search with filters and semantic matching
- Connecting your LLM and sending it your data
Instead of manually creating a vector database or a hybrid search described in steps three and four, you can use open-source tools to accelerate development.
Meilisearch indexes structured records and provides filtered semantic search in a single engine. Instead of building a vector database with third-party vendors, you can integrate Meilisearch into your RAG architecture while maintaining the high performance of your data retrievals.
How to connect SQL databases
Regardless of your SQL database engine, you first need to connect to it.
We'll use PostgreSQL as the example database, but the code for other structured databases is similar.
You just need a library that allows you to connect to your database engine.
Here is the Python code to connect to PostgreSQL:
How to enable real-time updates
Instead of using outdated data, you can poll the entire structured database for changes. This keeps your data up-to-date so you can provide fresh answers.
For example, suppose that you recently sold out of an item. You want to reflect this change when users ask questions about it.
In an enterprise environment, you might have several changes to data that you want to capture. To capture every change, RAG agents use a concept called change data capture (CDC).
Let's say that you make changes to a product by inserting a new record. Here is a snippet of code that would pull this change into your RAG system.
First, install the library:
Next, here is the Python code to listen for changes:
What are structured RAG examples?
We've been using general examples to explain why you would want to include structured RAG in your agentic workflows.
Chatbots are the best-known use case, but structured RAG has several other examples where it improves industry workflows:
- BI assistant: A business intelligence (BI) person can ask RAG questions, and, using semantic search, RAG can pull data from corporate data warehouses and return answers. An example question could be 'What was the product that brought in the highest revenue but had the least number of returns?'
- Customer service help: Instead of a customer service representative searching through several systems, an AI assistant could help them find answers to customer questions regarding products and services.
- Fintech analytics: Instead of combing through thousands of documents, RAG lets auditors and financial analysts ask questions about policies, financials, statements, and more to get answers within seconds. The process can save days of labor.
- Internal copilot: HR and employees can use RAG to get answers to company-specific questions without searching through multiple handbooks and other documents.
Some RAG developers work with knowledge graphs, and it's important to know that there are key differences between structured RAG and knowledge graphs.
What is structured RAG vs. knowledge graphs?
The main difference between structured RAG and knowledge graphs lies in how data is interconnected and queried.
A knowledge graph is linked using nodes and interconnected using single phrases or search terms.
A structured RAG understands semantics by using queries, turning data into chunks, and tagging data with metadata to better describe it.
Here is a breakdown of the differences:
What is structured RAG vs. data lakes?
The main difference between structured RAG and a data lake is that a data lake is a large silo of data, whereas a RAG is a retrieval system.
A RAG's source of data could come from a data lake, but a data lake does not necessarily do any of the processing.
Most enterprise environments have a data lake or a large warehouse of information stored in structured and unstructured formats.
Here is a breakdown of the differences:
What tools support structured RAG?
Most enterprise organizations already have tools for working with structured RAG, such as a database, but others may need to be added.
Here are a few tools to consider:
- Vector stores: You need a vector store to save semantic similarity calculations to retrieve answers. A few examples of these stores include Pinecone, Weaviate, and Chroma.
- Databases: Structured RAG uses databases that store structured data, so you can use PostgreSQL, MySQL, or Snowflake. You could also use graph databases like Neo4j, Amazon Neptune, and TigerGraph.
- Hybrid search engines: For better precision, you can use hybrid search engines that combine standard structured queries with semantic searches. An example is Meilisearch, which takes it a step further by offering a single engine for structured filtering and vector searches, simplifying your RAG architecture.
- Orchestration frameworks: Just like any other environment, an orchestration framework ties all tools together to help produce and manage your RAG. Tools include LangChain, LlamaIndex, and Haystack.
With all this information, you can build a fast, future-proof AI-agentic RAG system.
Is RAG for structured data the future of reliable AI?
Enterprise companies need faster, more precise answers to customer queries, and structured data and RAG work well together to deliver them.
If your organization has a large silo of data and needs to offer semantic search, structured RAG can lower the time-to-response for customer or employee queries. It can reduce a workflow that could take days to only a few minutes with the right search.
How tools like Meilisearch simplify RAG for structured data architectures
Building RAG takes effort, but you can simplify your architecture with Meilisearch. We make it faster to generate AI embedders, let you configure a semantic ratio to improve search responses, and integrate with your AI application stack with only a few lines of code.
