Understanding vector databases
------------------------------

Vector databases are the go-to for performing searches based on similarity, which plays a key role in **AI-driven applications** like recommending your next favorite movie, identifying someone in a photo, or digging up texts that resonate with your search. At the core of these applications are **vector embeddings**, complex data forms that exceed the storage and retrieval capabilities of traditional databases. 

### The role of vector embeddings

[Vector embeddings](/blog/what-are-vector-embeddings/?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search) are a way to **transform complex non-numeric data**, such as words, sentences, or even images, **into a numerical format** while preserving their semantic **meaning** and relationships. 

Embeddings are **multidimensional objects** generated by machine learning models where each dimension represents a different feature or aspect of the data. To properly capture the data complexity, vectors can range from dozens to thousands of dimensions, depending on the size and nature of the data. 

### Vector databases vs traditional databases

This complexity makes **traditional databases**–designed to store structured data in tables– unfit to handle embeddings. The volume and complexity of these vectors, each potentially containing thousands of dimensions, challenge the row-and-column format. This mismatch necessitates alternative storage and retrieval solutions tailored to the requirements of vector data.

This is where **vector databases** like [Meilisearch](https://www.meilisearch.com/?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search) come into play. They are designed to address the unique demands of vector embeddings, facilitating efficient storage and retrieval of the information they contain. In particular, they enable performing **similarity searches**, also called semantic searches, which are central to leveraging embeddings effectively. 

<Callout>
  Learn more about [how Meilisearch built Arroy in Rust](/blog/spotify-inspired-hybrid-search-and-rust/?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search) an open-source vector store.
</Callout>

In other words, vector databases allow us to interact easily and efficiently with vector embeddings, making them essential for applications that require semantic understanding and similarity matching.

What is similarity search?
--------------------------

If we think of **vector embeddings** as stars in vast cosmic constellations, similarity search, or **vector search**, would be like trying to **find the nearest** stars to your current position in space. In practical terms, this means finding the most relevant documents, images, or products based on your search query.

To do so, you need to **measure the distance** between the **query** **vector** and other **vectors in the database**, typically using methods like [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) or [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance). These are just different techniques for determining how close or distant other data points are from your query, much like gauging the **proximity** of stars in the night sky. 

### The role of machine learning models

However, the **success** of this search isn't just about mathematical calculations; it's highly dependent on the **machine learning model** used to generate and query the vectors. Each vector's meaning is intrinsically tied to the semantic space of the model that created it. Consistency here is crucial, ensuring that all vectors 'speak the same language' and adhere to the same contextual rules, making searches **meaningful and accurate**. That is to say, to achieve relevant search results, it's essential to **use the same model** for generating and querying the embeddings.

**Similarity search** is where vector databases like **Meilisearch** truly shine, as they allow for a wide array of applications such as face recognition, movie recommendations, and personalized content discovery. By allowing users to store vector embeddings alongside their documents, Meilisearch not only facilitates similarity searches but also introduces **hybrid search** capabilities, expanding its potential applications. Through the integration of models from various AI solution providers, Meilisearch enables users to **refine vector embeddings** to better suit their specific needs.

In summary, the ability of these databases to analyze and compare complex data patterns allows for highly relevant and accurate results across diverse fields, enhancing user experiences and operational efficiency.

AI Search is coming to Meilisearch Cloud, join the waitlist:

[Join now](https://share-eu1.hsforms.com/1RN-rGYhHQPu5BNivfknlwAfg3aq?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search)

* * *

[Meilisearch](https://meilisearch.com/?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search) is an open-source search engine that not only provides state-of-the-art experiences for end users but also a simple and intuitive developer experience. 

A long-time actor in keyword search, Meilisearch enables users to address search use cases building upon AI-powered solutions, not only supporting vector search as a vector store but also by providing hybrid search. This hybrid approach blends full-text search with semantic search, enhancing both the accuracy and comprehensiveness of search results.

For more things Meilisearch, you can join the community on [Discord](https://discord.meilisearch.com/) or subscribe to the [newsletter](https://dub.sh/meili-newsletter?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions?utm_source=blog&utm_medium=vector-db-content&utm_campaign=vector-search).

What is a vector database?

The new dashboard incorporates a set of metrics providing insights into your project’s performance. We want to help engineers move faster by making it easier to spot issues during implementation and having better visibility on your Meilisearch database health.

What metrics can you expect?
----------------------------

This first iteration comes with:

*   30 days of data retention
*   Real-time mode

**Search performance metrics**

*   Search latency
*   Maximum search queries per second

**Operations metrics**

*   Bandwidth (in/out)
*   API Calls — Succeeded/Failed

The default view combines the data from the indexes per project, but it’s also possible to filter out the insights for a specific index of your choice. The dashboard also allows you to select a custom range using a date picker or get real-time data insights:

![](https://unable-actionable-car.media.strapiapp.com/monitoring_metrics_custom_gif_gif_382681a817.jpg)

Why use monitoring**?**
-----------------------

With our brand-new dashboard, you can:

*   Ensure your users are getting lightning-fast search responses for every query
*   Monitor how your search functionality holds up under varying loads, not only during peak traffic conditions but also while indexing
*   Identify and rectify any performance bottlenecks or errors
*   Proactively maintain and optimize your project

**Availability and plans**
--------------------------

The new Cloud monitoring metrics are available for all Cloud users across all pricing tiers. Each tier has access to the same metrics and chart functionalities of the monitoring metrics dashboard.

[Try it in Meilisearch Cloud](https://cloud.meilisearch.com/?utm_campaign=monitoring-v1&utm_source=blog)

Learn how to [get started with Meilisearch Cloud](https://www.meilisearch.com/docs/learn/getting_started/cloud_quick_start?utm_campaign=monitoring-v1&utm_source=blog) and read the [Monitoring documentation](https://www.meilisearch.com/docs/learn/cloud/monitoring#requirements?utm_campaign=monitoring-v1&utm_source=blog).

* * *

Stay in the loop by subscribing to our [newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&utm_campaign=monitoring-v1&utm_source=blog). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=monitoring-v1&utm_source=blog) and participate in our [Product Discussions](https://github.com/meilisearch/product/discussions/?utm_campaign=monitoring-v1&utm_source=blog).

For anything else, join our developer community on [Discord](https://discord.meilisearch.com/?utm_campaign=teams-release&utm_source=blog).

Cloud monitoring metrics have arrived!

**🚀 Announcing Meilisearch 1.6**
---------------------------------

[![Meilisearch 1.6 release notes](https://unable-actionable-car.media.strapiapp.com/meilisearch_1_6_1_25b4cb0c21.jpg)](/blog/meilisearch-1-6/?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

Meilisearch 1.6 brings **hybrid search**, significant **improvements in indexing speed**, and a new API to **customize the proximity** ranking rule. Read the release notes to get the full picture.

[Read release notes](/blog/meilisearch-1-6/?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

🔍 Typesense comparison
-----------------------

[![Meilisearch vs Typesense](https://unable-actionable-car.media.strapiapp.com/meilisearch_vs_typesense_cover_1_3b17eec4c2.jpg)](/blog/meilisearch-vs-typesense/?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

We sat down to compare the key features between Meilisearch and open-source search engine **Typesense**. You can read the comparison on our blog.

[Read comparison](/blog/meilisearch-vs-typesense/?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

🤗 New docs home
----------------

![Meilisearch documentation](https://unable-actionable-car.media.strapiapp.com/Screenshot_2024_01_23_at_17_16_39_e046f3c318.png)

We spent some time reworking our **docs home page** to make it more welcoming for developers. We also made **example apps** easier to find. Let us know what you think!

[View documentation](https://www.meilisearch.com/docs?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

🖇️ Links of the month
----------------------

🔒 [Master key and API keys](https://www.meilisearch.com/docs/learn/security/master_api_keys?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)  
Learn to secure your Meilisearch instance and manage public API keys.

⚡ [Meilisearch indexes documents 2x faster](https://twitter.com/StriftCodes/status/1747566185791553639)  
Take a look at the performance boosts in Meilisearch 1.6.

🐬 [MySQL Full-Text Search Limits and New Alternatives Solutions](https://data-sleek.com/blog/the-limits-of-mysql-full-text-search-and-the-quest-for-alternative-solutions/)  
Comparing MySQL full-text search features with other search engines.

🎓 [Dropdown Autocomplete Search Anything in Laravel](https://codecourse.com/watch/dropdown-autocomplete-search-anything-in-laravel)  
Learn to build instant dropdown search with Laravel and Meilisearch.

⚛️ [The Two Reacts](https://overreacted.io/the-two-reacts/)  
Dan Abramov writes on components state, server data, and React.

🥰 Made with Meilisearch
------------------------

**YoungCapital NEXT** introduced their new website with a fresh branding and design. Built with **Next.js** and **Vercel**, their [job portal search](https://next.youngcapital.nl/vacatures) is powered by Meilisearch.

[![YoungCapital NEXT](https://unable-actionable-car.media.strapiapp.com/Screenshot_2024_01_23_at_17_34_37_1_552b1d480f.png)](https://next.youngcapital.nl/vacatures)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.meilisearch.com/) _Discord channel!_

🏡 Life at Meili
----------------

[![Discord](https://unable-actionable-car.media.strapiapp.com/discord_2yf3soo8o238y21k_2d54bc8412.jpg)](/blog/from-slack-to-discord-our-migration/?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

Our team wrote a blog post explaining our reasons for migrating our community to a Discord server, how we went about it, and why we shut down our public Slack.

[Read blog post](/blog/from-slack-to-discord-our-migration/?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

🦸 We're hiring!
----------------

See our open positions:

*   [Head of Developer Marketing](https://www.meilisearch.com/careers?utm_campaign=newsletter&utm_source=january-news-24&utm_medium=blog)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&ref=blog.meilisearch.com).

For more things Meilisearch, join our developers community on [Discord](https://discord.meilisearch.com/). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time.

Meilisearch January Updates

Meilisearch vs Typesense

We’re announcing the release of Meilisearch 1.6. Let’s dive into some of the most important changes. You can also view the [full changelog](https://github.com/meilisearch/meilisearch/releases/tag/v1.6.0) on GitHub.

<Callout>
  Meilisearch 1.6 is also available on [Meilisearch Cloud](https://cloud.meilisearch.com/login?utm_campaign=release-v1-6&utm_source=blog), including all experimental features.
</Callout>

Experimental feature: hybrid search
-----------------------------------

Meilisearch introduces hybrid search. It combines full-text and semantic search to enhance the accuracy and comprehensiveness of search results. Picture a movie app like [where2watch](https://where2watch.meilisearch.com/?utm_campaign=release-v1-6&utm_source=blog). Now, your users will be able to find those movies they can't quite name but remember the story.

Furthermore, Meilisearch now streamlines the creation of vector embeddings. Choose your preferred embedders and Meilisearch will handle all interactions with external tools for you.

### Configuring embedders

You can [configure the embedders](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=release-v1-6&utm_source=blog#generate-vector-embeddings) in your index settings. Select from three types of embedders for your needs:

`openAI`:

*   Uses the [OpenAI](https://openai.com/) API for computing embeddings
*   Requires an OpenAI API key for operation

`huggingFace`:

*   Enables local computation of embeddings by downloading models from the [HuggingFace Hub](https://huggingface.co/models)
*   Operates on your CPU–not your GPU–which may impact indexing performance

`userProvided`:

*   Functions similarly to Meilisearch v1.3, with a key difference: you must define a specific embedder
*   Allows you to add pre-computed embeddings into your documents. You perform searches using vectors instead of text.

To use hybrid search, define at least one embedder in the index settings:

```JSON
{
  "embedders": {
    "default": {
      "source":  "openAi",
      "apiKey": "<your-OpenAI-API-key>",
      "model": "text-embedding-ada-002",
      "documentTemplate": "A movie titled '{{doc.title}}' whose description starts with {{doc.overview|truncatewords: 20}}"
    },
    "image": {
      "source": "userProvided",
      "dimensions": 512
    },
    "translation": {
      "source": "huggingFace",
      "model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
      "documentTemplate": "A movie titled '{{doc.title}}' whose description starts with {{doc.overview|truncatewords: 20}}"
    }
  }
}

```

The `documentTemplate` field acts as a blueprint for creating your document's embedding. It uses the [Liquid template language](https://shopify.dev/docs/api/liquid). While its inclusion is optional, it is highly recommended, especially since embedding models are optimized for concise texts. It keeps only the necessary content, excluding non-essential data like `id`, and helps in adding context to increase relevancy.

### **Hybrid search**

To [perform a hybrid search](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=release-v1-6&utm_source=blog#vector-search-with-auto-embeddings), use the hybrid field within the `POST /index/:index_uid/search` route.

```JSON
{
    "q": "Plumbers and dinosaurs",
    "hybrid": {
        "semanticRatio": 0.9,
        "embedder": "default"
    }
}

```

`embedder`: an embedder from the options configured in your index settings.

`semanticRatio`: a floating value ranging from `0` to `1`; `1` is a  fully semantic search; `0` is an exact, match-focused full-text search; the default is `0.5` which mixes both methods.

Your control over the semantic ratio directly influences how search results are ranked. A higher semantic ratio shifts the focus towards the context and meaning behind your query, ranking results that are more semantically relevant higher. 

On the other hand, a lower semantic ratio increases the weight given to keyword accuracy in the ranking process, bringing results that closely match your specific search terms to the forefront.

### **Breaking changes in the experimental vector search API**

Meilisearch v1.6 introduces some breaking changes in the vector search API.

Previously, you could send vectors without specifying a model. Now, you must define a model in the settings:

```JSON
"embedders": {
    "default": {
      "source": "userProvided",
      "dimensions": 512
    }
}

```

Because Meilisearch now supports multiple embedders, it has updated the vector submission format from arrays to JSON objects.

*   Previews format: `“_vectors”: [[0.0, 0.1]]`
*   New format: `“_vectors”: {“image2text”: [0.0, 0.1, …]}`

For detailed information on these updates, refer to the [documentation](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=release-v1-6&utm_source=blog).

For in-depth technical information, explore the [series of articles](/blog/spotify-inspired-hybrid-search-and-rust/?utm_campaign=release-v1-6&utm_source=blog on [Arroy](https://github.com/meilisearch/arroy), an open-source repository based on [Spotify’s Annoy](https://github.com/spotify/annoy) and developed in Rust. This library, created and maintained by the Meilisearch engine team, specializes in searching for vectors in a space that are near a specified query vector.

**Performance optimization**
----------------------------

### Improved indexing speed

We're thrilled to share a major enhancement in Meilisearch's indexing performance. Our recent tests, including scenarios with frequent and partial document updates, have shown impressive results: **a reduction in indexing time by up to 50%**, and in some cases, **even as much as 75%**. 

Thanks to our latest optimizations, Meilisearch now **stores and pre-computes less data**. Additionally, during document updates, it **re-indexes or deletes only the necessary data**. For instance, in an e-commerce dataset, updating the stock level of a product results in re-indexing just the 'stock' field, rather than the entire product document.

### Disk space usage reduction

Meilisearch reduces internal data storage, leading to a more **compact database size** on your disk. With a dataset of approximately 15Mb, we observed a **40% to 50% reduction in database size**.

This enhancement not only reduces the database size, but also **improves its stability**, making the space savings more evident as the number of documents increases.

**New feature: customize proximity precision**
----------------------------------------------

To further reduce indexing speed, Meilisearch now allows you to tailor the accuracy of the [proximity ranking rule](https://www.meilisearch.com/docs/learn/core_concepts/relevancy?utm_campaign=release-v1-6&utm_source=blog#3-proximity) to your specific needs.

The proximity ranking rule is computationally demanding and may lead to longer indexing times. Reducing its accuracy can greatly enhance performance, and in most scenarios, it will not substantially affect the relevancy of the results.

To adjust its impact [configure the `proximityPrecision` setting](https://www.meilisearch.com/docs/reference/api/settings?utm_campaign=release-v1-6&utm_source=blog#proximity-precision):

```JSON
curl \
  -X PATCH 'http://localhost:7700/indexes/books/settings/proximity-precision' \
  -H 'Content-Type: application/json'  \
  --data-binary '{
    "proximityPrecision": "byAttribute"
  }'
```

The default `proximityPrecision` setting is `byWord`, which calculates proximity based on exact word distances. 

The `byAttribute` setting considers words in the same attribute as proximate, regardless of their exact distance.

Using `byAttribute` can boost the indexing speed, but it might slightly change how relevant the results are. This becomes more noticeable in searches where it's important for words to be close to each other.

For example, when you're looking through song lyrics or long articles, like trying to find 'world war' in a bunch of Wikipedia pages, you might end up with results that contain these words but not necessarily close together or in the desired order. This is also true for [phrase searches](https://www.meilisearch.com/docs/reference/api/search?utm_campaign=release-v1-6&utm_source=blog#phrase-search) and for searches involving [multi-word synonyms](https://www.meilisearch.com/docs/learn/configuration/synonyms?utm_campaign=release-v1-6&utm_source=blog#multi-word-synonyms), where the specific combination of words is crucial.

**New feature: task queue webhook**
-----------------------------------

Meilisearch now offers a webhook feature to notify a custom URL when an asynchronous task finishes (either succeeds, fails, or gets canceled).

This feature is particularly useful for streamlining workflows, saving you from polling the tasks route.

Set up your webhook at launch using these environment variables:

```bash
MEILI_TASK_WEBHOOK_URL=https://mywebsite.com/my-super-webhook?user=1234&number=8

MEILI_TASK_WEBHOOK_AUTHORIZATION_HEADER='Bearer 12340987546wowowlolol'
```

You can also use the respective command-line options.

Once set up, the webhook sends out a payload in [JSON Lines (ndjson) format](https://www.meilisearch.com/docs/learn/core_concepts/documents?utm_campaign=release-v1-6&utm_source=blog#ndjson) to your specified URL, containing the list of finished tasks:

```json
//POST HTTP request to https://myproject.com/mywebhook?common=people

{"uid":4,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"}
{"uid":5,"indexUid":"movie","status":"failed","type":"indexDeletion","canceledBy":null,"details.deletedDocuments":0,"error.message":"Index `movie` not found.","error.code":"index_not_found","error.type":"invalid_request","error.link":"https://docs.meilisearch.com/errors#index_not_found","duration":"PT0.001192S","enqueuedAt":"2022-08-04T12:28:15.159167Z","startedAt":"2022-08-04T12:28:15.161996Z","finishedAt":"2022-08-04T12:28:15.163188Z"}
```

**Experimental feature: limit the number of batched tasks**
-----------------------------------------------------------

To speed up the indexing process, Meilisearch processes similar [tasks](https://www.meilisearch.com/docs/learn/async/asynchronous_operations?utm_campaign=release-v1-6&utm_source=blog#tasks-and-asynchronous-operations) in large batches. However, excessive [queued tasks](https://www.meilisearch.com/docs/learn/async/asynchronous_operations?utm_campaign=release-v1-6&utm_source=blog#task-queue) can occasionally cause crashes or stalls.

To control the number of batched tasks, set the limit at launch using either the command-line argument `--experimental-max-number-of-batched-tasks`, the `MEILI_EXPERIMENTAL_MAX_NUMBER_OF_BATCHED_TASKS` environment variable, or the [configuration file](https://www.meilisearch.com/docs/learn/configuration/instance_options?utm_campaign=release-v1-6&utm_source=blog#configuration-file).

**Contributors shout-out**
--------------------------

We are really grateful for all the community members who participated in this release.We would like to thank [@Karribalu](https://github.com/Karribalu) and [@vivek-26](https://github.com/vivek-26) for their help with [Meilisearch](https://github.com/meilisearch/meilisearch). We also want to send a special shout-out to our SDKs maintainers 🦸

* * *

And that’s it for v1.6! This release post highlights the most significant updates. For an exhaustive listing, read the [changelog on Github](https://github.com/meilisearch/meilisearch/releases/tag/v1.6.0).

Stay in the loop of everything Meilisearch by subscribing to [the newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&ref=blog.meilisearch.com). To learn more about Meilisearch's future and help shape it, take a look at our [roadmap](https://roadmap.meilisearch.com/?utm_campaign=release-v1-6&utm_source=blog) and come participate in our [Product Discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

For anything else, join our developers community on [Discord](https://discord.meilisearch.com/?utm_campaign=release-v1-6&utm_source=blog).

Meilisearch v1.6

⏳ Meilisearch v1.6 release candidate is now available
-----------------------------------------------------

[![](https://unable-actionable-car.media.strapiapp.com/1_6_RC_74d5e70575.jpg)](https://github.com/meilisearch/meilisearch/releases/tag/v1.6.0-rc.0?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

Get a sneak peek at **Meilisearch v1.6**, launching **mid-January**! This update promises **major indexing performance improvements** and introduces **hybrid search** for more comprehensive results.

⚠️ The current **release candidate** is for **testing only** and not recommended for production use. [**Feedback**](https://github.com/orgs/meilisearch/discussions) and [**bug reports**](https://github.com/meilisearch/meilisearch/issues/new/choose) are welcome to refine the final release.

[Read pre-release notes](https://github.com/meilisearch/meilisearch/releases/tag/v1.6.0-rc.0?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

🦀 Spotify-inspired: Elevating Meilisearch with Hybrid Search and Rust
----------------------------------------------------------------------

[![](https://unable-actionable-car.media.strapiapp.com/spotify_inspired_51f4d0331b.jpeg)](/blog/spotify-inspired-hybrid-search-and-rust?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

How we created [**Arroy**](https://github.com/meilisearch/arroy), a **Rust library** building upon the foundations of **Spotify**'s Annoy.

[Read article](/blog/spotify-inspired-hybrid-search-and-rust?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

🌟 We reached 40,000 GitHub stars!
----------------------------------

[![](https://unable-actionable-car.media.strapiapp.com/40_K_a03904ec32.jpg)](https://github.com/meilisearch/meilisearch/)

A huge **thank you** to our **incredible community** for helping us reach this milestone! Your **contributions**, **feedback**, and **enthusiasm** have been pivotal in our journey. Thank you for being a part of our growing family.

**Let's continue to reach new heights together!**

**📰 In the news**
------------------

### **🔍** [**Using vector search**](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

Everything you need to know to get started with this feature.

### 📈 [Introducing Cloud search analytics](/blog/search-analytics-release/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

Get immediate actionable insight to optimize your search.

### 👥 [Teams feature is available on Meilisearch Cloud](/blog/introducing-teams/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

 Host and manage your projects in collaboration with your teammates.

### ✈️ [Migrating to Meilisearch Cloud](https://www.meilisearch.com/docs/learn/update_and_migration/migrating_cloud?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

Migrating from self-hosted? Here’s how to handle the migration.

### 🎓 [Bucket sort guide](/blog/bucket-sort-guide/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

Understand the algorithm behind Meilisearch relevancy.

🥰 Made with Meilisearch
------------------------

[**Bookshop.org**](https://bookshop.org/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog) is an online bookstore based in Brooklyn, New York, with a mission to support local, independent bookstores financially. They have integrated Meilisearch to enhance the search functionality within their extensive catalog.

[![Search results page from Bookshop.org, displaying over 1000 results for "m", and refining as the user keeps typing in the search bar](https://unable-actionable-car.media.strapiapp.com/bookshop_meilisearch_9498082583.gif)](https://bookshop.org/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

_We’re always looking to promote new projects from the community. Post yours in the_ [_#built-with-meilisearch_](https://discord.meilisearch.com/) _Discord channel!_

🏡 Life at Meili
----------------

For us, what you value in work is as important as your technical skills, if not more so. Here’s a recruiter's advice on how to answer interview questions about values.

[![ A spinning top on a wooden surface.](https://unable-actionable-car.media.strapiapp.com/recruiters_advice_0b884ab386.jpeg)](/blog/advice-on-answering-interview-questions-about-values/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

[Read blog post](/blog/advice-on-answering-interview-questions-about-values/?utm_campaign=newsletter&utm_source=december-news-23&utm_medium=blog)

🦸 We're hiring!
----------------

See our open positions:

*   [Head of Developer Marketing](https://jobs.lever.co/meili/6c9637b0-399a-4300-9f03-1e6f4c450efa)

* * *

Get these monthly updates straight to your inbox by subscribing to our [newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&ref=blog.meilisearch.com).

For more things Meilisearch, join our developers community on [Discord](https://discord.gg/meilisearch/?ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/meilisearch/product/discussions?ref=blog.meilisearch.com).

Until next time.

Meilisearch December Updates

_This is part 3 of a series of blog posts_ [_originally published on my personal blog_](https://blog.kerollmops.com/meilisearch-expands-search-power-with-arroy-s-filtered-disk-ann)_._ _Begin the journey with_ [_part 1_](/blog/spotify-inspired-hybrid-search-and-rust/) _and_ [_part 2_](/blog/refining-ann-performance-with-rust/)_._

We can store vectors in [Arroy](https://github.com/meilisearch/arroy) and efficiently compute the search tree nodes, but we still need some features to make it usable in Meilisearch. Our full-text search engine has great support for filtering; selecting subsets of documents is an essential feature to support. For instance, one of our biggest clients requires the ability to filter through over 100 million YouTube video metadata and their associated image embeddings to effectively select videos released within specific time frames, such as a single day or week. This represents the scalability and responsiveness challenges we aim to address with our filtering system, making it a perfect use case to tailor our developments.

Meilisearch supports the following operators on the filterable attributes of your documents: `<`, `<=`, `=`, `!=` `>=`, and `>`. Internally, we extensively [use `RoaringBitmap`s](https://github.com/RoaringBitmap/roaring-rs), which are well-optimized sets of integers that support fast binary operations like unions and intersections. When the engine receives a user request with filters, it first computes the subset of documents from the filter that will be inputted into the search algorithms, the one that ranks documents by quality. This subset is represented by a `RoaringBitmap`.

How was it working before Arroy?
--------------------------------

Ranking only the subset of filtered documents has been working well since 2018, but now that we have a new data structure to search on, we need to look at how to implement it. The engine had already supported the vector store feature for months but was inefficient. We were using an in-memory HNSW and were deserializing the whole data structure in memory, searching for the nearest neighbors of the target vector, which was returning an iterator.

```Rust
fn vector_store(db: Database, subset: &RoaringBitmap, target_vec: &[f32]) -> Vec<DocumentID> {
    // This takes a lot of time and memory.
    let hnsw = db.deserialize_hnsw();

    let mut output = Vec::new();
    for (vec_id, vec, dist) in hnsw.nearest_neighbors(target_vec) {
        let doc_id = db.document_associated_to_vec(vec_id).unwrap();
        if !output.contains(&doc_id) {
          output.push(doc_id);
          if output.len() == 20 { break }
        }
    }

    output
}
```

You may wonder why we are retrieving the document ID associated with the vector ID. Meilisearch supports multiple vectors by document since the beginning of the vector store feature. This is unfortunate because we must look up every vector we are iterating. We need to maintain this lookup table. The iterator can iterate on the whole vector dataset if the subset is small enough, e.g., `document.user_id = 32`. We want the document operations to be atomic and consistent, so we have to store the HNSW on disk and avoid having to maintain synchronization between the LMDB transactions and this vector store data structure. Ho! And the library doesn't support incremental insertion. We must reconstruct the HNSW in memory from scratch every time we insert a single vector.

Integrating Arroy in Meilisearch
--------------------------------

As we worked on updating Meilisearch to include the new vector store, arroy, we tried out mob programming for the first time. This is where we all code together at the same time. It might sound like it would slow us down, but it actually made us super productive! By tackling problems together right as they came up, we got arroy fitted into Meilisearch much faster than if we'd worked alone.

Arroy is different because it doesn’t return an iterator to give back search results. Now, our search engine is smarter and can figure out the exact number of results it needs to give back, even when there are filters to consider. This teamwork improved our search tool and showed us that working together is key when facing big tech challenges.

#### Filtering while searching in Arroy

You can find a description of the arroy internal data structure in [part 1 of this series](https://blog.kerollmops.com/spotify-inspired-elevating-meilisearch-with-hybrid-search-and-rust). Here is a list of the different kinds of nodes you can find:

*   The item nodes. The original vectors that the user provides. The small points on the left.
*   The normal nodes, also called split planes. They represent hyperplanes that split subsets of item nodes in half.
*   The descendant nodes. They are the tree leaves composed of the item IDs you'll find if you follow a certain normal node path.

![split-plane-combined-schema](https://unable-actionable-car.media.strapiapp.com/spli_plane_b53ab895e8.png)

A typical search will load all the normal nodes in a binary heap associated with an infinite distance. Remember that there are a lot of randomly generated trees, and there are, therefore, a lot of entry points. Ascending distances order the binary heap; the shortest nodes found will be popped first.

In the search algorithm, we pop the nearest item from this heap. If it's a normal node, we fetch the left and right sides of the hyperplane:

*   If we find normal nodes again, we associate the distance from the hyperplane with positive and negative distances from the targeted/queried vector.
*   If we find descendant nodes, we do not push them into the queue but rather directly add them to the potential output list as they represent the nearest vectors we have found.

You probably noticed where I want to go, but this is where magic happens. We modified arroy to store the list of descendants in `RoaringBitmap`s instead of a raw list of integers. This is another improvement compared to the original Spotify library, as those lists weigh less. Doing an intersection with the filtered subset is way easier now.

However, there is always an issue: the vector IDs are not the document IDs, and Meilisearch, after executing the filters, only knows about the documents. Iterating on the lookup table, I talked about before, constructing the final bitmap with all the vector IDs corresponding to the filtered documents would not be efficient when many documents are part of this subset, e.g., `document.user_id != 32`. I do not recommend using an `O(n)` algorithm in the search functions.

#### Using multiple indexes for efficient filtering

Fortunately, we developed a fun feature that wasn't meant to be used that way in arroy: the support for multiple indexes in a single LMDB database. We originally developed the multiple indexes feature to be able only to open a single LMDB database to store different vector types. Yes! In Meilisearch v1.6, you can describe the different embedders that live in a single index. You can identify the different vectors with different numbers of dimensions and distance functions you can store in a single document. Defining multiple vectors for a single document associated with the same embedder is also possible.

The indexes are identified by an `u16`. This feature can trick the algorithm into being even more efficient than the previous HNSW solution. Using one store by embedder and vector in a document is interesting because we can now identify the vectors using the document ID. No more lookup database and vector IDs. The vector IDs are reduced to the document IDs. We can use the output of the filters to filter the arroy indexes.

The search algorithm is different on the Meilisearch side. We request the nearest neighbors in every arroy index, sort the results by document ID to be able to deduplicate them and not return the same document multiple times, sort them again by distance, and then return only the top twenty documents. It may seem complex, but we are talking about twenty documents by the number of vectors for a single document. Usually, users will have a single vector.

```Rust
fn vector_search(
    rtxn: &RoTxn,
    database: Database,
    embedder_index: u8,
    limit: usize,
    candidates: &RoaringBitmap,
    target_vector: &[f32],
) -> Vec<(DocumentId, f32)> {
    // The index represents the embedder index shifted and
    // is later combined with the arroy index. There is an arroy
    // index by vector for a single embedded in a document.
    let index = (embedder_index as u16) << 8;
    let readers: Vec<_> = (0..=u8::MAX)
        .map(|k| index | (k as u16))
        .map_while(|index| arroy::Reader::open(rtxn, index, database).unwrap())
        .collect();

    let mut results = Vec::new();
    for reader in &readers {
        let nns = reader.nns_by_vector(rtxn, target_vector, limit, None, Some(candidates)).unwrap();
        results.extend(nns_by_vector);
    }

    // Documents can have multiple vectors. We store the different vectors
    // into different arroy indexes, we must make sure we don't find the nearest neighbors
    // vectors that correspond to the same document.
    results.sort_unstable_by_key(|(doc_id, _)| doc_id);
    results.dedup_by_key(|(doc_id, _)| doc_id);

    // Sort back the documents by distance
    results.sort_unstable_by_key(|(_, distance)| distance);
    results
}
```

The design philosophy behind these improvements
-----------------------------------------------

We're trying to make Meilisearch flexible enough for everyone's needs. Whether your documents are powered by a single embedding, like the kind OpenAI's nifty tools generate, or mixing it up with text and image embeddings, as many e-commerce sites do, we want your search experience to be seamless. We haven't forgotten you for the power users working with multiple embeddings from multi-modal embedders. Our latest improvements ensure that everyone gets to update and fine-tune their search indexes incrementally, making the whole process as smooth as butter.

Big thanks to everyone: Wrapping this up, a huge shout-out to the squad – [@dureuill](https://github.com/dureuill), [@irevoire](https://github.com/irevoire), and [@ManyTheFish](https://github.com/ManyTheFish) – for their genius and grunt work that brought our ideas to life. And watch out for [@irevoire](https://github.com/irevoire)'s upcoming article, where he'll explain how we've implemented incremental indexing in Arroy—meaning you can add new vectors without rebuilding everything from scratch. More on that soon!

You can comment about this article on [Lobste.rs](https://lobste.rs/s/hdokyh/meilisearch_expands_search_power_with), [Hacker News](https://news.ycombinator.com/), [the Rust Subreddit](https://www.reddit.com/r/rust/comments/18pqxdz/meilisearch_expands_search_power_with_arroys), or [X (formerly Twitter)](https://x.com/kerollmops/status/1738837058628551148?s=46&t=0s6OFYJM-Oyl_WD-1jijQA).

* * *

[Meilisearch](https://meilisearch.com/?ref=blog.meilisearch.com) is an open-source search engine that not only provides state-of-the-art experiences for end users but also a simple and intuitive developer experience.

For more things Meilisearch, you can join the community on [Discord](https://discord.gg/meilisearch/?ref=blog.meilisearch.com) or subscribe to the [newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions?ref=blog.meilisearch.com).

Meilisearch expands search power with Arroy's Filtered Disk ANN

_This is part 2 of a series of blog posts_ [_originally published on my personal blog_](https://blog.kerollmops.com/multithreading-and-memory-mapping-refining-ann-performance-with-arroy)_. Begin the journey with_ [_part 1_](/blog/spotify-inspired-hybrid-search-and-rust/)_._

Wouldn't it be great to show you how a single-threaded, memory-mapped key-value store can be more efficient than a hand-written memory-mapped solution? I faced issues while porting the Spotify disk-based Approximate Nearest Neighbors library to Rust and, more specifically, to LMDB. Those issues were primarily due to LMDB and memory safety. Here is the story.

To remind you, [Arroy](https://github.com/meilisearch/arroy) is a library that stores embeddings (vectors of floats) on disk. Some data structures are generated on top of those vectors, which look like trees governed by normals used to recursively split the vectors dataset into subdomains. But you can read more about this in [part 1](/blog/spotify-inspired-hybrid-search-and-rust/ of this series.

### How does Annoy generate the tree nodes?

Annoy, the Spotify library stores the nodes on disk, the item nodes (the ones containing the embeddings), and the other nodes that we will call tree nodes in this article. The advantage of doing this is double:

*   The program's memory usage is low and optimized by the OS as all the nodes live in a memory-mapped file.
*   The concept is simple: Access a node by using its ID. The library will find its offset on disk by using a simple multiplication.

However, there are downsides to doing that. All nodes: items with embeddings along with the tree nodes must have the same size, and If a user wants to identify its embedding using the ID 231, the library will increase the file size to the ID multiplied by the node size. For example, with a vector of 768 dimensions, storing a single item with the ID 231 will generate a file of more than 6.5 TiB.

In generating the final data structure, the tree nodes are all written to disk in the same file as the user item containing the embeddings. Those tree-building processes run in parallel, one running process by tree, and, therefore, requires a lot of synchronization when defining the newly created node ID and the memory area reserved for it, most importantly, when the memory-mapped file is too small to accept more nodes, only a single thread must grow the file, so a mutex is used on top of the mutable memory-map and around a `node_id` variable. One interesting property of tree generation is that the generation process only requires the original user item with embeddings.

### The challenges we encountered when porting it to LMDB

A fun fact is that it is the first time in a long time that I have written C++, and the first time I asked ChatGPT to code for me because I was not confident in doing C++ and feared falling into some segfault. I needed a small program to deserialize embeddings from stdin and give them to Annoy. The chatbot's code was mostly effective, but it omitted a critical [empty vector check](https://gist.github.com/Kerollmops/d0e434016b1a698ec9bdbd86d268885e#file-par_build_tree-cpp-L74), which led to a segmentation fault...

The main obstacle to porting it to LMDB is that writing into this key-value store is single-threaded. It doesn't support concurrent write operations to maintain a correctly balanced BTree. Fun incoming!

#### Reading the user item nodes from different threads

We have used LMDB at Meilisearch since the beginning. It is a well-maintained key-value store used in Firefox and maintained by the OpenLDAP developers. It is memory-mapped and does not maintain a user-end cache of the entries in memory but instead gives you a pointer to the memory-mapped area on disk. The main advantage is that you can keep a pointer to this area for as long as your read transaction lives. Ho yeah! Because it is a transactional key-value store that supports atomic committing, too!

But tree generation doesn't require referring to the generated nodes but only the user items. We previously saw that LMDB gives direct pointers into the memory-mapped file without maintaining any intermediate cache (that could be invalidated). There is another quirk with LMDB: you cannot share a read-only transaction between multiple threads, i.e., `RoTxn: !Sync` and you cannot move the write transaction between threads, i.e., `RwTxn: !Send + !Sync`. Ho! And there is no way to create a read-transaction on uncommitted changes. This is an issue because we want to generate the data-structures trees in the same transaction where we store the items.

But magic is everywhere, starting with the following small and fun data structure. The principle is to keep the pointers to the internal user items with embeddings in a `Vec<*const u8>`. Thanks to Rust, we can ensure, at compile time, that the pointers will live long enough by keeping the lifetime in the struct. Using the `&'t RwTxn` to get the `&'t RoTxn` by using `Deref` also ensures that we cannot modify the database while reading in it by using a `&'t mut RwTxn`. [According to the leading developer of LMDB](https://bugs.openldap.org/show_bug.cgi?id=10138#c9), it is safe to share those pointers between threads and why I implemented `Sync` for this structure.

```Rust
pub struct ImmutableItems<'t, D> {
    item_ids: RoaringBitmap,
    constant_length: Option<usize>,
    offsets: Vec<*const u8>,
    _marker: marker::PhantomData<(&'t (), D)>,
}

impl<'t, D: Distance> ImmutableItems<'t, D> {
    pub fn new(rtxn: &'t RoTxn, database: Database<D>, index: u16) -> heed::Result<Self> {
        let mut item_ids = RoaringBitmap::new();
        let mut constant_length = None;
        let mut offsets = Vec::new();

        for result in database.items()? {
            let (item_id, bytes) = result?;
            assert_eq!(*constant_length.get_or_insert(bytes.len()), bytes.len());
            item_ids.push(item_id);
            offsets.push(bytes.as_ptr());
        }

        Ok(ImmutableLeafs { item_ids, constant_length, offsets, _marker: marker::PhantomData })
    }

    pub fn get(&self, item_id: ItemId) -> heed::Result<Option<Leaf<'t, D>>> {
        let len = match self.constant_length {
            Some(len) => len,
            None => return Ok(None),
        };
        let ptr = match self.item_ids.rank(item_id).checked_sub(1).and_then(|offset| self.offsets.get(offset)) {
            Some(ptr) => *ptr,
            None => return Ok(None),
        };
        let bytes = unsafe { slice::from_raw_parts(ptr, len) };
        ItemCodec::bytes_decode(bytes).map(Some)
    }
}

unsafe impl<D> Sync for ImmutableItems<'_, D> {}
```

You have also probably noticed some other fun optimizations in this simplified version of the data structure. We also know that the user-item nodes have a constant length, so I decided to store it only once, reducing the offsets vector's size by two. Considering that our objective is to store 100M vectors and that this vector is in memory, we shrunk its size from 1526 MiB to 763MiB, which is not much, but better than nothing.

#### Writing the tree nodes in parallel

Ok! Now that we know how to store pointers to items and share them between threads without any user-end synchronization, we need to generate the tree nodes using it. We already know how to deal with LMDB at Meilisearch and decided to implement the same workaround. To write in parallel into LMDB, write into different, independent files and merge everything afterward. This leverages the [Share-Nothing principle](https://en.wikipedia.org/wiki/Shared-nothing_architecture) and isolates the algorithms. This drastically reduces the number of synchronization points [compared to the original C++ code](https://github.com/spotify/annoy/blob/2be37c9e015544be2cf60c431f0cccc076151a2d/src/annoylib.h#L1344-L1445) (look at the `.lock/unlock` calls) to a single line in our code: the atomically increasing global tree node ID.

```Rust
pub fn next_node_id(&self) -> u64 {
    self.0.fetch_add(1, Ordering::Relaxed)
}
```

Our functions that generate normal split nodes based on the user-item nodes are now simply writing the nodes into independent files. The [nodes are appended into a file](https://github.com/meilisearch/arroy/blob/4f193fd534acd357b65bfe9eec4b3fed8ece2007/src/parallel.rs#L20-L71), and the on-disk offsets and bounds are stored into a vector, a single `usize` by node. Using Rayon, we run all tree-generation functions in parallel and, once completed, retrieve the files and boundaries to write the uniquely identified nodes into LMDB sequentially.

### Performances comparison

Our objective at Meilisearch is to support 100M embedding of around 768 dimensions. If we store those as `f32` vectors without any [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction), it would be equivalent to `100M * 768 * 4 = 307B`, in other words, 286 GiB to store the vectors raw, without any internal tree nodes, i.e., no way to search in them efficiently.

If you don't specify the number of trees to generate, the algorithm will continue to create trees while the number of tree nodes is smaller than the number of item vectors. At the end, there must be roughly the same number of tree nodes and items.

#### Discovering the limit of vectors we can index

Arroy and Annoy use memory mapping extensively but in slightly different ways. In a previous article from [@dureuill](https://github.com/dureuill), we saw that [operating systems do not have infinite memory-map space](/blog/dynamic-virtual-address-management/ So, let's dive into performance results.

I noticed something strange when running Arroy with 25M vectors of 768 dimensions. The CPUs were not used, and the program seemed to read the SSD/NVMe a lot, too much 🤔 The tree generation algorithm is splitting the vector space into subdomains with fewer vectors. Still, it must first find the appropriate normal to split the entire domain, and for that, it randomly selects many items. Unfortunately, my machine only has 63 GiB of memory, and indexing those 25M items requires more than 72 Gib. Annoy was also struggling in the same way.

![We cannot index 25M vectors of 768 dimensions with 63 GiB](https://unable-actionable-car.media.strapiapp.com/annoy_arroy_e83f58691c.png)

After investigating, I understood why the whole swap space and memory mapping limits were reached. Not only the item nodes were not only _768 \* 4 bytes_ because we store the norm and other stuff alongside the vectors, but in the case of Arroy, LMDB needs to maintain BTree data structures around the entries, and those are also tacking memory-mapped space. Both programs request random item nodes unavailable in memory, so the OS fetches them from the disk, which takes time. Ho and every single thread is doing that in parallel. CPUs are simply awaiting the disk.

So, after some dichotomy, I found the point where arroy successfully used all of the memory without being bound to the disk speed. It can index 15.625M on a 63 GiB machine. You can see on this htop screenshot that the disk read speed is at zero as all of the vectors fit in RAM, that arroy is writing the tree nodes to disk, and that the CPUs are doing their best. It took less than seven hours to process.

![Successfully indexing 15.6M vector of 768 dimensions with 63 GiB](https://unable-actionable-car.media.strapiapp.com/annoy_arroy2_a873d31d2d.png)

But... Annoy cannot index this same number of documents. It suffers from the same issue we saw before: high disk read and low CPU usage. But why? I needed clarification because the nodes have the same format, the number of item vectors to index is the same, and the algorithm has been ported. So, what is the difference between both solutions?

For those who looked into the C++ codebase and were probably hinted by the memory mapping issue described earlier, you probably noticed this slight difference: Arroy is writing the generated tree nodes into different raw files when Annoy, on the contrary, is [reserving space into the memory-mapped file and directly writing into it](https://github.com/spotify/annoy/blob/2be37c9e015544be2cf60c431f0cccc076151a2d/src/annoylib.h#L1354-L1370). By doing this trick, the OS needs to keep much more space in the memory-mapped area and is forced to invalidate the item nodes from the cache to keep the just-written tree nodes hot in the cache, slowing down the system for no reason, as the tree nodes are not necessary for the algorithm.

So, after even more dichotomy to find the Annoy limits on a 63 GiB machine, I discovered that it could roughly index 10M vectors in five hours.

#### The share nothing principle to the win

So Arroy can index 36% more vectors on the same machine, but how long does it take? Wouldn't it be faster to write into the same file in parallel instead of copying all those tree nodes in a single-threaded way? It will be a much shorter paragraph as I only did some small tests, but Arroy is always faster!

|  | number of vectors | number of threads | building time |
| --- | --- | --- | --- |
| Annoy | 96k | 1 | 5min 38s |
| arroy | 96k | 1 | **3min 45s** |
| Annoy | 96k | 12 | 1min 9s |
| arroy | 96k | 12 | **54s** |
| Annoy | 10M | 12 | 5h 40min |
| arroy | 10M | 12 | **4h 17min** |
| Annoy | 15.625M | 12 | \-- |
| arroy | 15.625M | 12 | **7h 22min** |

Now, you probably tell me that I will need around 400GiB of memory to index 100M vectors, and you are probably right, but [@irevoire](https://github.com/irevoire) will talk about incremental indexing in a future article. I did a linear regression for fun. With those 12 threads and the required memory, I predicted it would take one day, 22 hours, and 43 min 😬. Ho! As we also use this vector store in Meilisearch, we need to provide ways to filter this data structure at search time. This is the next article in this series.

You can comment about this article on [Lobste.rs](https://lobste.rs/s/87aheu/multithreading_memory_mapping_refining), [Hacker News](https://news.ycombinator.com/), [the Rust Subreddit](https://old.reddit.com/r/rust/comments/18jmuoe/multithreading_and_memorymapping_refining_ann), or [X (formerly Twitter)](https://x.com/Kerollmops/status/1735941213516140814).

[Part 3](/blog/arroy-filtered-disk-ann/) of this series is now available, exploring further advancements in our search with Arroy's Filtered Disk ANN.

* * *

[Meilisearch](https://meilisearch.com/?ref=blog.meilisearch.com) is an open-source search engine that not only provides state-of-the-art experiences for end users but also a simple and intuitive developer experience.

For more things Meilisearch, you can join the community on [Discord](https://discord.gg/meilisearch/?ref=blog.meilisearch.com) or subscribe to the [newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions).

Multithreading and Memory-Mapping: Refining ANN performance with Arroy

_This is part 1 of a series of blog posts_ [_originally published on my personal blog_](https://blog.kerollmops.com/spotify-inspired-elevating-meilisearch-with-hybrid-search-and-rust)_._

Uniting keyword and semantic search
-----------------------------------

At Meilisearch [we are working hard on hybrid search](https://github.com/meilisearch/meilisearch/issues/3838), mixing the widespread keyword search algorithm with the not-so-common semantic search. The former is very good at exact matching, while the latter finds stuff you can _describe_ better.

I will explain why we even talk about [Meilisearch/arroy](https://github.com/meilisearch/arroy) and why it is important to us. The semantic search was something new to us. The principle is quite simple: documents are associated with vectors (lists of floats), and the closer they are to each other, the closer they are semantically. When a user queries the engine, you compute the embeddings (vectors) of the query and do an approximate nearest neighbor search to get the nth closest items to the user query. Fun fact: after all this time, Annoy [is still among the top competitors](https://ann-benchmarks.com/gist-960-euclidean_10_euclidean.html), so imagine when we will finally release Arroy.

Seems simple, right? Do you **really** think this series of blog posts will talk about a dead simple subject? Join us on a journey as [@irevoire](https://github.com/irevoire) and I ([@Kerollmops](https://github.com/Kerollmops)), with the assistance of [@dureuill](https://github.com/dureuill), work on porting this cutting-edge library to Rust.

Optimizing storage and search for high-dimension embeddings
-----------------------------------------------------------

The embeddings are something between 768 and 1536 dimensions. We have customers at Meilisearch who want to store over 100M of them. Embeddings that were not modified by any dimension reduction algorithm use 32-bit floats. This means storing this data will take between 286 GiB and 572 GiB, depending on the dimensions.

[Yes, it fits in RAM](https://yourdatafitsinram.net/), but at what cost? Storing on disk is much cheaper. Ho! And it is only the raw vectors. I assure you that doing an approximate neighbors search among all vectors in `O(n)` will be quite slow. So we decided to store them on disk. We have looked at many different solutions before choosing Spotify/Annoy. Initially, we used the HNSW from instant-distance Rust crate, another data structure, to do an ANNs (approximate nearest neighbors) search. However, it is not disk-based; everything lives in memory, which is unpractical.

Spotify's hyperplan trees for efficient ANNs
--------------------------------------------

Spotify worked on a cool C++ library to search in enormous vector sets. The algorithm generates multiple trees that refer to the vectors. The tree nodes are random hyperplanes that split subsets of vectors in half. The root node splits the whole set of vectors in half, and the left and right branches split the subsets of vectors recursively again until the subset is small enough. When we perform an ANNs search, we go through all the root trees and decide whether to go on the hyperplanes' left or right, depending on the haystack provided. The advantage is that every node and vector is directly stored in a memory-mapped file.

![Hyperplanes mapped to binary trees](https://unable-actionable-car.media.strapiapp.com/blog_arroy_1_76462ae24c.png)

The way Annoy and Arroy generate hyperplanes to split the vector space

At the end of the tree, we can see the descendants' nodes. Those define the final list of leaf vectors that fit on this side of the recursive split nodes above. It is a list of unsigned integers the user provides. Annoy represents them as slices of `u32`s, but we decided to go [with RoaringBitmaps](https://roaringbitmap.org/about/) to reduce their size. Annoy cannot compress them because they have a fun constraint: All nodes, user leaf vectors, split nodes, and descendants must be the same size to be accessible using offsets on disk.

The above image shows the way a split plane can be represented. A hyperplane splits a subset of nodes here in two dimensions, but imagine that with 768 to 1536 dimensions. Each hyperplane creates two subsets of points recursively split by another hyperplane. Once the number of points to split is small enough, we create a descendants node containing the item IDs corresponding to these points. Furthermore, the embeddings of the points are never duplicated in memory; we refer to them by their IDs.

Adapting Annoy to LMDB
----------------------

So, if it works so well, why must we port it to Rust? One, because I follow the _Rewrite it in Rust_ dogma 😛, two because this is a C++ library with a lot of horrifying hacks and undefined behaviors, and three because Meilisearch is based on LMDB, an atomic, transaction-based, memory-mapped key-value store. Moreover, [since 2015, they have wanted to use LMDB](https://github.com/spotify/annoy/issues/96), but they have yet to achieve it; it requires a lot of time to change the data structures accordingly.

LMDB uses a BTree to order the entries, and it takes more space for the intermediate data structures than annoy, which directly identifies vectors using the offset in the file. Another advantage of using this key-value store is managing incremental updates. It is easier to insert and remove vectors. Suppose you insert a vector identified by a high 32-bit integer in Annoy. In that case, it will allocate a lot of memory to store it as the dedicated offset, increasing the file size if necessary.

In Meilisearch and Arroy, [we use heed, a typed LMDB Rust wrapper](https://github.com/meilisearch/heed), to avoid undefined behavior, bugs, and stuff related to C/C++ libraries. We, therefore, use `&mut` and `&` extensively to be sure that we are not modifying the key-value store entries while keeping references inside it, and we make sure that the read-write transactions can't be referenced or sent between threads. But that will be for another story.

We first thought using this key-value store would make Arroy slower than Annoy. However, LMDB writes its pages in memory before writing them to disk, which seems much faster than directly writing into mutable memory-mapped areas. On the other hand, LMDB doesn't guarantee the memory alignment of the values that Annoy's file format permits; we'll talk about that later.

Optimizing vector handling with SIMD
------------------------------------

The most CPU-intensive task that Arroy was supposed to do was to try to split clouds of vectors in half. This requires computing a massive number of distances in hot loops. But we read the embeddings from a memory-mapped disk. What could go wrong?

```rust
fn align_floats(unaligned_bytes: &[u8]) -> Vec<f32> {
    let count = unaligned_bytes.len() / mem::size_of::<f32>();
    let mut floats = vec![f32::NAN; count];
    cast_slice_mut(&mut floats).copy_from_slice(unaligned_bytes);
    floats
}
```

We used Instrument on macOS to profile our program and discovered a lot of time was spent moving stuff around, i.e., `_platform_memmove`. The reason is that reading unaligned floats from disk is undefined behavior, so we were copying our floats into an aligned memory area as shown above. Cost: An allocation by reading plus a calling `memmove`.

![Profiling showing a big memmove usage](https://unable-actionable-car.media.strapiapp.com/blog_arroy_2_c34c821b3c.png)

Profiling shows a lot of memory copies to align bytes

While porting the distance function from C++ to Rust, we used the Qdrant SIMD-optimized distance functions without modifying them beforehand. Although, [being aware of the performance cost of reading unaligned memory](https://dwrensha.github.io/capnproto-rust/2020/01/11/unaligned-memory-access), we decided to execute the distance functions on unaligned floats, making sure not to represent it as a `&[f32]` as it would be undefined behavior. The functions take raw slices of bytes and deal with them using pointers and SIMD functions. We unlocked performances. The distances functions were slower, but it compensated the `memmove`s and allocations!

![Profiling showing no memmove usage](https://unable-actionable-car.media.strapiapp.com/blog_arroy_3_519276495b.png)

Directly dealing with unaligned memory removed the copies

Similarly to the `memmove` calls, we can see that the `_platform_memcmp` function is taking a lot of space here. The reason is that LMDB uses this standard function to compare key bytes and decide whether a key is lexicographically located above or below another key in the tree. It is used whenever we read or write in LMDB. [@irevoire](https://github.com/irevoire) drastically reduced the size of the keys and saw an important gain in performance. We further tried using the `MDB_INTEGERKEY`, which makes LMDB compare memory using the computer endianness, but it was complex to use and not showing significant performance gains.

Upcoming challenges
-------------------

While porting this cool algorithm to Rust and LMDB, we were missing one of the most important features: Multithreading the tree building. The main reason for this missing feature is LMDB, which does not support concurrent writes to disk. It is part of the beauty of this library; writing is deterministic. We already know LMDB very well, and we will explain how we leveraged the power of LMDB in part 2 of this series and how we beat the Spotify library.

In addition to equalizing the current Annoy features, we needed more for Meilisearch. We implemented [Microsoft's Filtered-DiskANN feature](https://harsha-simhadri.org/pubs/Filtered-DiskANN23.pdf) in Arroy. By providing the subset of item IDs we want to retrieve, we avoid searching the whole trees to fetch the approximate nearest neighbors. We will talk about this in a soon-to-be-released article.

In Meilisearch v1.6, we've optimized performance for updating only document parts, as with vote counts or view numbers. The described single-threaded version of Arroy constructs tree nodes anew for each vector adjustment. [@irevoire](https://github.com/irevoire) will explore Arroy's incremental indexing in his next article, a capability not offered by Annoy.

You can comment about this article on [Lobste.rs](https://lobste.rs/s/4pw9kf/spotify_inspired_elevating_meilisearch), [Hacker News](https://news.ycombinator.com/), [the Rust Subreddit](https://old.reddit.com/r/rust/comments/18dq493/spotifyinspired_elevating_meilisearch_with_hybrid), or [X (formerly Twitter)](https://twitter.com/Kerollmops/status/1733158652444369286).

[Part 2](/blog/refining-ann-performance-with-rust/ and [part 3](/blog/arroy-filtered-disk-ann/ of this series are now available, exploring further advancements in ANN algorithms with Rust.

* * *

[Meilisearch](https://meilisearch.com/?ref=blog.meilisearch.com) is an open-source search engine that not only provides state-of-the-art experiences for end users but also a simple and intuitive developer experience.

For more things Meilisearch, you can join the community on [Discord](https://discord.gg/meilisearch/?ref=blog.meilisearch.com) or subscribe to the [newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e&ref=blog.meilisearch.com). You can learn more about the product by checking out the [roadmap](https://roadmap.meilisearch.com/?ref=blog.meilisearch.com) and participating in [product discussions](https://github.com/orgs/meilisearch/discussions).

Meilisearch latest news and company updates