Share the article
In 2024, researchers at Stanford published a new standard for chunking and organization called Recursive Abstractive Processing for Tree-Organized Retrieval (RAPTOR).
RAPTOR changes the way retrieval-augmented generation (RAG) works with chunked documents. Instead of creating chunks and storing them randomly, RAPTOR creates a hierarchical tree structure of chunks, categorizes them, and writes summaries for each group.
RAPTOR gives users more precise answers because hierarchical summaries contain high-level information that LLMs can search.
More precise details are contained in each 'leaf' of the tree. To find answers, the LLM searches summaries and then drills down into the data contained in branches and leaves.
This is especially useful for long-document benchmarks such as QASPER, where answers often depend on information spread across multiple sections.
Here's how to effectively use RAPTOR in your RAG systems.
What is RAPTOR RAG?
RAPTOR RAG changes how document chunks are organized for large-language-model processing. Instead of chunking documents into individual sections, RAPTOR RAG stores the chunks as a tree.
Imagine that you have several chunks for a dozen documents. RAPTOR RAG takes the chunks and organizes them by category.
The categories contain detailed information (represented by each category's expanded branches and their leaves, which further describe the information), and the hierarchy of each category includes a detailed summary.
Detailed summaries make it easier for large language models (LLMs) to find information to answer a user's query. An LLM can find information using an abstract summary or drill down into more detailed data, searching the branches and clustered 'leaves' attached to them.
Because chunked data is organized into a tree, using RAPTOR RAG also improves the accuracy of query answers.
An LLM first searches a small number of category summaries and then explores further into the tree.
When chunked data is organized properly, it improves the speed and performance of LLM queries.
The number of summaries in a RAPTOR RAG depends on the number of documents in your data silos.
How does RAPTOR differ from GraphRAG, CRAG, and other advanced RAG methods?
RAPTOR was introduced to solve problems with simple chunking and data indexing. It's one of the many ways chunked data can be organized and used with LLM queries.
Here is a comparison table that highlights the pros and cons of each technique:
What problems does RAPTOR RAG solve?
RAPTOR RAG primarily addresses latency and accuracy of large document chunks.
Chunking data can be done using short phrases or long paragraphs. Each chunk size has its advantages and disadvantages.
Small chunks offer better accuracy, but they add latency.
Larger chunks are better with latency, but the longer the chunks, the higher the probability of introducing hallucinations.
Both of these problems are better solved with the improved organization that RAPTOR RAG offers.
With a RAPTOR RAG strategy, queries no longer search through scattered data or rely only on flat retrieval when semantic meaning is distributed across multiple documents. It's a better strategy overall if your core data spans multiple long documents.
RAPTOR RAG:
- Answers questions where answers span multiple documents
- Helps when you can't decide if large or small chunks are necessary
How does RAPTOR RAG work?
RAPTOR RAG works in five steps: chunking, generating embeddings, clustering similar chunks, creating cluster similarities, and building a recursive tree.
Each iteration builds a more abstract view of the layers below it, but these abstractions improve query performance.
Step #1: Chunking documents
Sending an LLM an entire PDF to parse isn't optimal for token usage, which in turn affects the costs of running AI agents and RAG.
Instead, chunking breaks PDF data into components.
In RAPTOR RAG, chunking can be done in three ways:
- Fixed size: Phrases contain the same number of characters
- Recursive: Chunks are based on words and paragraphs
- Semantics: Cuts up documents based on word meaning, context, and semantic similarity
Here is sample code for chunking:
Step #2: Generating embeddings
Chunks are stored in a vector database, which converts them into numerical representations.
The numerical representation is a score used to assess answer accuracy. A higher score indicates that the chunk is a better response to a specific query.
Chunks are embedded using SBERT (Sentence Bidirectional Encoder Representations from Transformers) forming the leaf nodes of the tree structure.
You can also use OpenAI embeddings if you want hosted embeddings instead of a local sentence-transformers model.
Here is sample code:
Step #3: Clustering similar chunks
Now it's time to build the hierarchy based on the vector database. This relates to hierarchical clustering, but RAPTOR's soft assignments allow chunks to appear in multiple groups rather than forcing each chunk into a single branch.
The chunks are scored, so RAPTOR goes through them, groups similar values, and summarizes them.
RAPTOR uses a soft clustering algorithm via Gaussian Mixture Models (GMMs) which allows a chunk to belong to more than one cluster at once. The overlap avoids accuracy loss from missing words that might add context.
Here is sample code:
Step #4: Creating cluster summaries
Once clustered, similar chunks can be summarized.
Since RAPTOR RAG uses a hierarchy, the cluster summaries at the top end of the tree are very few. The iteration summaries, as RAG works its way up the tree, become increasingly abstract.
The best LLMs for this step are Gemini 2.5 Flash, GPT-4o-mini, and Claude Haiku 4.5.
Here is sample code:
Step #5: Building the recursive tree
Step four builds the layers, but step five organizes the tree into its hierarchy.
Each summary has a parent-child relationship. The parent is the more abstract summarization, and each branch below it is the child node.
A node contains its text, its embedding vector, its layer number, and references to its children.
Here is sample code to illustrate:
How to implement RAPTOR RAG in Python
Implementing a RAPTOR RAG takes several steps, but you can perform each one in Python.
You'll need a document storage location, a vector store, your LLM of choice, and the infrastructure to run the Python code.
The following sections provide the Python code for each step, but the code can be combined to run all steps as a single process.
Step #1: Load and chunk the documents
The chunking step takes documents and segments them into words or characters. These chunks are also known as the leaves in the hierarchy.
Here is the step in Python code:
Step #2: Transform chunks into vectors for semantic comparison
Step two takes the chunks created in step one and creates the vectors.
A vector database converts the chunks and provides a numerical value representing the confidence in their relevance to the query. This value determines if the chunk becomes a part of the query answer.
Here is the Python code:
Step #3: Group similar chunks before summarizing them
Step three takes similar chunks from step two and clusters them. This step is necessary for step four so that the clustered chunks can be summarized.
Here is the Python code:
Step #4: Generate cluster summaries
Here, we use an LLM to condense the clustered chunks from step three and create a summary.
The first summary becomes the parent node to the leaves below.
Here is the Python code:
Step #5: Build the recursive tree
This step recursively moves up the tree to create parent-node summaries and organizes the data into a hierarchy of branches and leaves.
The most important part of this step is linking the nodes together to create relationships between them.
Here is the Python code:
Step #6: Store tree nodes in a vector database
Step six writes every node into a vector database, where each node is stored, along with its text and metadata.
At this point, the tree is ready for queries. Building the tree is expensive, so it's done as few times as possible.
Here is the Python code:
Step #7: Implement hierarchical retrieval
Building the tree starts at the last child node, but retrieval starts at the top of the tree at the first summarization.
As an answer is processed, the retrieval system goes down the hierarchy based on the summaries until it reaches the last leaf nodes.
Here is the Python code:
Step #8: Generate the final response
Step eight takes the nodes retrieved in step seven, inputs the text into a prompt alongside the user's question, and asks an LLM to write the answer.
This is the retrieval step that users see when they type a question.
Here is the Python code:
What are RAPTOR RAG's limitations and known failure modes?
RAPTOR RAG has several advantages, but it has limitations you should learn about before you decide to implement it.
These few disadvantages might change the way you design your system to overcome them:
- Cost: Because building the tree is expensive, it can be a limitation for businesses on a budget. Every summarization is a call to an LLM, so large trees cost more, especially if you need to rebuild the tree often.
- Errors can chain: Each summary builds on the next summary. If the LLM hallucinates in one summary, the errors will propagate to the next parent summary.
- Constant updates to dynamic data: If your source data changes frequently, you must rebuild the tree. This can be time-consuming and cost more than necessary. Businesses might prefer to use RAPTOR RAG on documents that change infrequently or stay static.
- Cluster errors: If chunking produces errors, they will persist in the summaries. As with other chunking procedures, it's important to review chunks to ensure there were no errors. This can be done with quality assurance and sampling chunks before moving on to the next step.
- Too complex for simple queries: RAPTOR RAG is best for answers that require integrating data from multiple documents. Using it on simple queries will unnecessarily waste LLM tokens and AI budgets.
- Document ingestion errors: If your Python script throws an error when ingesting and chunking documents, the RAPTOR RAG system loses context. Without relevant context, the system can hallucinate and return incomplete and inaccurate answers.
How can Meilisearch power the retrieval layer in a RAPTOR pipeline?
RAPTOR's indexing produces a multi-level tree of text nodes.
Meilisearch runs keyword and semantic search together and merges them into a single ranked list.
At search time, it runs both keyword and semantic search in parallel, then merges the results using its ranking rules, which sort matches into buckets by criteria such as typo count, word proximity, and attribute importance.
This is important in a RAPTOR tree because the two node types reward different retrieval styles.
Running both at once means a single query can retrieve the correct node, whether the answer lives in a leaf or in an abstract summary.
Meilisearch's hybrid search capabilities (combining semantic vector search with full-text keyword matching) make it a strong fit for the collapsed-tree retrieval strategy, where all nodes across tree levels are queried simultaneously.
Build RAG systems that can see the whole document
RAPTOR RAG is a good option when your retrieval system needs to answer complex questions across long documents, multiple sections, or scattered pieces of context.
Instead of forcing an LLM to work from isolated chunks, RAPTOR gives it a layered view of the information: broad summaries at the top, detailed chunks at the bottom, and a path between the two.
It is not the right fit for every RAG system, especially if your data changes constantly or your queries are simple.
But when users need detailed answers from large, interconnected knowledge bases, RAPTOR gives you a more organized way to retrieve the right information.
Power RAPTOR retrieval with Meilisearch
Meilisearch can support the retrieval layer in a RAPTOR pipeline by helping you search across both summary nodes and leaf-level chunks. Its hybrid search capabilities combine keyword and semantic search, so users can find the right answer whether their query matches the exact wording of a document or the broader meaning of a summary.
