v1.43.0
The new settings indexer, a more efficient version of our internal indexer for index settings modifications, now supports filterable, sortable, facet search, and custom (asc/desc) attributes in addition to the already-supported searchable, exact, proximity precision, and embedders.
- For Meilisearch Cloud users, the new settings indexer is disabled by default and can be enabled on a case-by-case basis for scaling purposes.
- For OSS users, the new settings indexer can be disabled by setting the
MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_SETTINGStotrue.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6124
We reduced the time to generate the internal data structures for the facet search to work. We were previously doing a couple of full scans on the internal entries and replaced these with scans on specific entries, only the ones dedicated to the facet searchable fields.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6325
The GET /health route now returns HTTP 500 after a successful task queue compaction, to signal that Meilisearch should be restarted so that tasks can be enqueued again.
For Meilisearch Cloud users, this ensures that compacting the task queue will automatically restart the instance after the compaction.
Fixes a bug where string facet values appearing in <, <=, >, >= and IN filters were not normalized before comparison to facet values.
This would cause some values in documents (e.g. 2026-01-01T00:00:00) to appear to be higher than their filter counterpart, due to being normalized (e.g. like 2026-01-01t00:00:00).
Thanks to @njaard for reporting the issue ❤️
By @dureuill in https://github.com/meilisearch/meilisearch/pull/6340
Fixes the WordDelta::added_or_deleted_words function.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6350.
Thanks to @vladradin99 and @snurfer0 for reporting the issue in https://github.com/meilisearch/meilisearch/issues/6349 and https://github.com/meilisearch/meilisearch/issues/6324 ❤️
- Introduce a meilitool command to export an index words FST by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6332
- Add riscv64 to release binary matrix via cross-rs by @gounthar in https://github.com/meilisearch/meilisearch/pull/6296
- Fix docs related to task cancelation and deletion by @curquiza in https://github.com/meilisearch/meilisearch/pull/6353
- Bump actions/checkout from 5 to 6 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6303
- Bump actions/setup-node from 4 to 6 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/5964
- Bump thin-vec from 0.2.14 to 0.2.16 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6339
- Bump docker/setup-buildx-action from 3 to 4 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6304
- Bump rustls-webpki from 0.103.10 to 0.103.13 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6347
- Bump actix-http from 3.11.2 to 3.12.1 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6348
- Bump sigstore/cosign-installer from 3.10.0 to 4.1.1 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6301
- Bump supercharge/mongodb-github-action from 1.12.0 to 1.12.1 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6302
Thanks again @gounthar, @njaard, @vladradin99 and @snurfer0 for your contributions ❤️
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.42.1...v1.43.0
v1.42.1 🦑
By @dureuill and @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6331
- When using the experimental feature "multimodal", removing a fragment would cause an internal error in subsequent search requests
- When using the experimental feature "multimodal", modifying fragments would sometimes cause an internal error at indexing time
regenerate: falsewould be ignored when modifying embedder settings- When a nested field was declared as searchable, it would not be indexed if its parents were not declared as searchable
- Add new CI run that tests the stable settings indexer by @dureuill in https://github.com/meilisearch/meilisearch/pull/6331
v1.42.0 🦑
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6306
We introduce a new fallback system for the sharding and replication enterprise edition feature, along with a way to determine which remote is available. The engine can avoid machines that are unavailable for a period and resume querying them once they're back online.
The following snippet shows what the /network route looks like now that this PR exposes the remote statuses/availabilities.
"remotes": {
"prod2": {
"url": "http://localhost:7702",
"searchApiKey": "mykey",
"writeApiKey": "mykey",
"status": "available"
},
"prod3": {
"url": "http://localhost:7703",
"searchApiKey": "mykey",
"writeApiKey": "mykey",
"status": "unavailable"
}
}
By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6314
This enhancement extends the Cross-index document hydration introduced in v1.39.0 by allowing the user to filter on the foreign indexes to retrieve the documents.
📓 Note: This implementation doesn't support a remote sharding environment
TheforeignKeys experimental feature must be activated to use the foreign filters:
curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
-H 'Content-Type: application/json' \
--data-binary '{"foreignKeys": true}'
To be able to use the foreign filters, the related field must be set as a foreignKey and as a filterableAttribute in /indexes/{index_uid}/settings:
On the search route, a new _foreign verb has been introduced and should be used as follows:
{
"q": "<query>",
// filters on the movie index:
// genres = action
// AND
// the foreign documents from the actor index match: birthday STARTS WITH \"1958-\" AND popularity >= 3.5
"filter": "genres = action AND _foreign(actors, birthday STARTS WITH \"1958-\" AND popularity >= 3.5)"
}
Note: nesting foreign filters is not supported and will return an error
- Meilisearch running on
127.0.0.1:7700on thedocument-join-hydrationbranch.
curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
-H 'Content-Type: application/json' \
--data-binary '{"foreignKeys": true}'
curl -X POST 'http://127.0.0.1:7700/indexes' \
-H 'Content-Type: application/json' \
--data-binary '{"uid": "actors", "primaryKey": "id"}'
curl -X POST 'http://127.0.0.1:7700/indexes' \
-H 'Content-Type: application/json' \
--data-binary '{"uid": "movies", "primaryKey": "id"}'
curl -X POST 'http://127.0.0.1:7700/indexes/actors/documents' \
-H 'Content-Type: application/json' \
--data-binary '[
{"id": 1, "name": "Tom", "familyName": "Hanks", "birthDate": "1956-07-09"},
{"id": 2, "name": "Meryl", "familyName": "Streep", "birthDate": "1949-06-22"},
{"id": 3, "name": "Leonardo", "familyName": "DiCaprio", "birthDate": "1974-11-11"},
{"id": 4, "name": "Emma", "familyName": "Watson", "birthDate": "1990-04-15"}
]'
curl -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \
-H 'Content-Type: application/json' \
--data-binary '[
{"id": 1, "title": "Forrest Gump", "description": "The presidencies of Kennedy and Johnson, the Vietnam War, the Watergate scandal and other historical events unfold from the perspective of an Alabama man with an IQ of 75.", "actors": [1]},
{"id": 2, "title": "The Devil Wears Prada", "description": "A smart but sensible new graduate lands a job as an assistant to Miranda Priestly, the demanding editor-in-chief of a high fashion magazine.", "actors": [2, 4]},
{"id": 3, "title": "Inception", "description": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.", "actors": [3]},
{"id": 4, "title": "Cast Away", "description": "A FedEx executive undergoes a physical and emotional transformation after crash landing on a deserted island.", "actors": [1]}
]'
curl -X PATCH 'http://127.0.0.1:7700/indexes/movies/settings' \
-H 'Content-Type: application/json' \
--data-binary '{"foreignKeys": [{"fieldName": "actors", "foreignIndexUid": "actors"}], "filterableAttributes": [{"attributePatterns": ["actors"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}'
curl -X PATCH 'http://127.0.0.1:7700/indexes/actors/settings' \
-H 'Content-Type: application/json' \
--data-binary '{"filterableAttributes": [{"attributePatterns": ["birthDate"],"features": {"facetSearch": false,"filter": {"equality": true,"comparison": false}}}]}'
curl -X POST 'http://127.0.0.1:7700/multi-search' \
-H 'Content-Type: application/json' \
--data-binary '{
"queries": [
{
"indexUid": "movies",
"q": "Forrest",
"filter": "_foreign(actors, birthDate = \"1956-07-09\")"
}
],
"federation": {
"limit": 20,
"offset": 0
}
}'
The federated search should return movie documents with the actors array automatically hydrated with full actor objects instead of just IDs:
{
"hits": [
{
"id": 1,
"title": "Forrest Gump",
"description": "...",
"actors": [
{
"id": 1,
"name": "Tom",
"familyName": "Hanks",
"birthDate": "1956-07-09"
}
],
"_federation": {
"indexUid": "movies",
"queriesPosition": 0,
"weightedRankingScore": 0.9848484848484849
}
}
],
"processingTimeMs": 208,
"limit": 20,
"offset": 0,
"estimatedTotalHits": 1
}
-
Fix a race condition when writing network by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6300
We fixed a race condition in network topology changes that could cause errors and prevent documents from being correctly indexed. Additionally, we fixed a bug in the
networkTopologyChangetask batching that was causing it to batch too many task types. We made sure it only batches import tasks, and only those, to avoid out-of-order task processing. -
Throw document template errors when updating the chat settings by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6321
We fixed an issue that prevented the engine from explicitly showing the possible document template errors users could encounter when updating the template in the chat settings. The engine now correctly checks for and throws template errors when they are detected.
-
Fix: Update Index tasks will be properly forwarded to remote nodes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6299
-
Fix action mistake on the chat completions route by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6290
-
Use the latest version of heed with nested rtxns support by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6316
This PR bumps the versions of crates that use heed to the latest version, v0.22.1. This version finally stabilized a long-standing piece of work we were doing with Howard Chu: nested read transactions. We no longer have to rely on unstable pre-releases, but rather on a clean, stable version of LMDB (still a fork, but a better one).
-
Add section to CONTRIBUTING.md to bump mini-dashboard version and testing section to right place by @curquiza in https://github.com/meilisearch/meilisearch/pull/6195
-
Make the no-agent AGENTS.ms more permissive by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6260
-
Remove deleted test commands by @Strift in https://github.com/meilisearch/meilisearch/pull/6283
-
Fix OpenAPI schema generation for chat completions route by @qdequele in https://github.com/meilisearch/meilisearch/pull/6274
-
Rename OpenAPI route names for search rules and compact by @qdequele in https://github.com/meilisearch/meilisearch/pull/6298
-
Update README with new features and demos by @qdequele in https://github.com/meilisearch/meilisearch/pull/6297
-
Prevent shell injection in benchmark workflows by @curquiza & @Kerollmopsin https://github.com/meilisearch/meilisearch/pull/6308 & https://github.com/meilisearch/meilisearch/pull/6318
-
Rename some of the search performance traces by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6323
v1.41.0 🏈
-
Support dynamic search rules with pinning by @YoEight in https://github.com/meilisearch/meilisearch/pull/6182 Introduce the Dynamic Search Rules (DSR) experimental feature, making it easy to promote the right content for the right searches with flexible, condition-based pinning. In this first version, rules can be triggered by query-based conditions such as empty queries or literal substring matches, as well as time windows. Multiple documents can be pinned in a defined order, and pins still work with filtering, pagination, facet distribution, hybrid search, and federated search.
-
Default to
useNetwork: truein sharded instances by @dureuill in https://github.com/meilisearch/meilisearch/pull/6278This release contains breaking changes for users of the
networkexperimental feature.§ Breaking changes
When
network.leaderis set in the instance,useNetworknow defaults totruein search requests when omitted.So, when a network is configured, the following:
Search request without an explicit useNetwork
// POST /indexes/movies/search { "q": "Batman dark knight returns 1", "filter": "genres IN [Action, Adventure]", "facets": ["genres"], "limit": 5 // No `useNetwork` is passed }
is now equivalent to a federated search over multiples remotes so that all shards in the network are covered exactly once.
Equivalent federated search request
// POST /multi-search { "federation": { "limit": 5, "facetsByIndex": { "movies": [ "genres" ] }, "merge": {} }, // assuming 3 remotes 0, 1, 2 and 3 shards a, b, c such that // 0 owns a, b // 1 owns b, c // 2 owns c, a "queries": [ { "indexUid": "movies", "federationOptions": { "remote": "0" }, "q": "Batman dark knight returns 1", "filter": ["genres IN [Action, Adventure]", "_shard = a"] }, { "indexUid": "movies", "federationOptions": { "remote": "0" // useNetwork picked remote 0 for both shard a and b }, "q": "Batman dark knight returns 1", "filter": ["genres IN [Action, Adventure]", "_shard = b"] }, { "indexUid": "movies", "federationOptions": { "remote": "2" }, "q": "Batman dark knight returns 1", "filter": ["genres IN [Action, Adventure]", "_shard = c"] } ] }
Responding with documents from all the shards:
Response
{ "hits": [ { "id": 123025, "title": "Batman: The Dark Knight Returns, Part 1", "overview": "Batman has not been seen for ten years. A new breed of criminal ravages Gotham City, forcing 55-year-old Bruce Wayne back into the cape and cowl. But, does he still have what it takes to fight crime in a new era?", "genres": [ "Action", "Animation", "Mystery" ], "poster": "https://image.tmdb.org/t/p/w500/kkjTbwV1Xnj8wBL52PjOcXzTbnb.jpg", "release_date": 1345507200, "_federation": { "indexUid": "mieli", "queriesPosition": 2, "weightedRankingScore": 0.9894586894586894, "remote": "2" } }, { "id": 142061, "title": "Batman: The Dark Knight Returns, Part 2", "overview": "Batman has stopped the reign of terror that The Mutants had cast upon his city. Now an old foe wants a reunion and the government wants The Man of Steel to put a stop to Batman.", "genres": [ "Action", "Animation", "Mystery" ], "poster": "https://image.tmdb.org/t/p/w500/arEZYd6uMOFTILne9Ux0A8qctMe.jpg", "release_date": 1357171200, "_federation": { "indexUid": "mieli", "queriesPosition": 1, "weightedRankingScore": 0.9894558963186414, "remote": "0" } }, { "id": 16234, "title": "Batman Beyond: Return of the Joker", "overview": "The Joker is back with a vengeance, and Gotham's newest Dark Knight needs answers as he stands alone to face Gotham's most infamous Clown Prince of Crime.", "genres": [ "Animation", "Family", "Action", "Science Fiction" ], "poster": "https://image.tmdb.org/t/p/w500/7RlBs0An83fqAuKfwH5gKMcqgMc.jpg", "release_date": 976579200, "_federation": { "indexUid": "mieli", "queriesPosition": 1, "weightedRankingScore": 0.9427964918160996, "remote": "0" } }, { "id": 155, "title": "The Dark Knight", "overview": "Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker.", "genres": [ "Drama", "Action", "Crime", "Thriller" ], "poster": "https://image.tmdb.org/t/p/w500/qJ2tW6WMUDux911r6m7haRef0WH.jpg", "release_date": 1216166400, "_federation": { "indexUid": "mieli", "queriesPosition": 1, "weightedRankingScore": 0.5784178187403994, "remote": "0" } }, { "id": 49026, "title": "The Dark Knight Rises", "overview": "Following the death of District Attorney Harvey Dent, Batman assumes responsibility for Dent's crimes to protect the late attorney's reputation and is subsequently hunted by the Gotham City Police Department. Eight years later, Batman encounters the mysterious Selina Kyle and the villainous Bane, a new terrorist leader who overwhelms Gotham's finest. The Dark Knight resurfaces to protect a city that has branded him an enemy.", "genres": [ "Action", "Crime", "Drama", "Thriller" ], "poster": "https://image.tmdb.org/t/p/w500/vzvKcPQ4o7TjWeGIn0aGC9FeVNu.jpg", "release_date": 1342396800, "_federation": { "indexUid": "mieli", "queriesPosition": 2, "weightedRankingScore": 0.5772657450076805, "remote": "2" } } ], "query": "Batman dark knight returns 1", "processingTimeMs": 173, "limit": 5, "offset": 0, "estimatedTotalHits": 47, "facetDistribution": { "genres": { "Action": 46, "Adventure": 15, "Animation": 34, "Comedy": 3, "Crime": 14, "Drama": 6, "Family": 15, "Fantasy": 8, "Horror": 1, "Mystery": 4, "Romance": 1, "Science Fiction": 14, "TV Movie": 4, "Thriller": 4, "Western": 1 } }, "facetStats": {}, "requestUid": "019bbcf4-a609-7701-8d82-d370611adfb3", "remoteErrors": {} }This change allows to naturally query all the documents in a sharded context. When
network.leaderis not present (in particular, when nonetworkwas defined), then the behavior is identical to previous versions. -
useNetworkpicks local instance if it owns the shard by @dureuill in https://github.com/meilisearch/meilisearch/pull/6287 To prevent unnecessary network activity, when deciding which remote to ask for a shard in a search over the network, Meilisearch will now always pick the local instance if it owns the shard. -
More efficient FST building by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6279 It improves the construction of the word FST (word dictionary) by removing the need for a full scan of the word docids database. This should drastically improve the database's performance when inserting a large number of documents, even when inserting only a few.
- Handle empty VERSION file as missing by @themavik in https://github.com/meilisearch/meilisearch/pull/6275
- Remap post processing targets for better benchmark readability by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6285
- Update the description of the get task's document payload route by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6284
- @themavik made their first contribution in https://github.com/meilisearch/meilisearch/pull/6275
v1.40.0 🦈
This release introduced support for the distinct attribute in federated search, enabling cross-index distinct attributes with facet distribution support. Additionally, significant performance improvements were delivered, including faster federated search (approximately 100ms faster), optimized JSON document generation for better handling of large documents and a much better memory usage for large workloads.
-
Support
distinctin federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6214The
distinctattribute can now be passed to thefederationobject in federated search to apply a global, cross-index and cross-remote distinct computation to the results.Example of a federated search request with distinct
{ "federation": { "distinct": "genres", // ✨ NEW "facetsByIndex": { // recovering facet distribution is also supported with distinct "comics": [ "genres" ], "movies": [ "genres" ] }, "mergeFacets": {} // merging facet distributions is also supported with distinct }, "queries": [ { "indexUid": "comics", "q": "batman", "attributesToRetrieve": ["title", "genres"], "useNetwork": true // distinct is also supported with network queries }, { "indexUid": "movies", "q": "superman", "attributesToRetrieve": ["title", "genres"], "useNetwork": true } ] }Sample response to a federated search request with distinct
{ "hits": [ { "title": "Batman", "genres": [ "Family", "Adventure", "Comedy", "Science Fiction", "Crime" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 1.0, "remote": "ms2" } }, { "title": "Batman", "genres": [ "Fantasy", "Action" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 1.0, "remote": "ms1" } }, { "title": "Batman & Bill", "genres": [ "Documentary" ], "_federation": { "indexUid": "comics", "queriesPosition": 0, "weightedRankingScore": 0.9848484848484848, "remote": "ms1" } }, { "title": "Superman: Red Son", "genres": [], "_federation": { "indexUid": "movies", "queriesPosition": 1, "weightedRankingScore": 0.9848484848484849, "remote": "ms0" } }, { "title": "Superman, Spider-Man or Batman", "genres": [ "Drama" ], "_federation": { "indexUid": "movies", "queriesPosition": 1, "weightedRankingScore": 0.9848484848484849, "remote": "ms0" } } ], "processingTimeMs": 15, "limit": 5, "offset": 0, "estimatedTotalHits": 11, "facetDistribution": { "genres": { "Action": 1, "Adventure": 1, "Comedy": 3, "Crime": 2, "Documentary": 1, "Drama": 1, "Family": 1, "Fantasy": 1, "Horror": 2, "Romance": 1, "Science Fiction": 1, "Thriller": 1, "Western": 1 } }, "facetStats": {}, "requestUid": "019d05c7-ea65-77a1-8274-22a8ba9e26db", "remoteErrors": {} }Note the following to apply the distinct attribute at the federation level:
- Applying
distinctat the query level at the same time as the federation level is disallowed and will return a HTTP 400 error. - The chosen distinct field will apply to all remotes and indexes, so it must be a filterable attribute for all participating remotes and indexes.
- While Meilisearch attempts to compute the most accurate facet distribution, in distributed contexts this cannot be guaranteed as the distinct algorithm is not applied on all of the remote documents.
- Applying
-
Improve performance of federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6229
Improves performance of federated search: about 100ms faster for all requests. Improves reliability of the HTTP server: the server will no longer be blocked when too many federated search requests are being processed.
-
Optimize the generation of JSON documents by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6257
Addresses performance issues that users might encounter when requesting large documents. Additionally, performance is enhanced when users request only a small subset of fields from large documents.
-
Use the latest version of mimalloc to improve memory usage by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6201
Updates mimalloc from v2 to v3, improving memory sharing between threads and significantly reducing memory usage on large workloads. It also overrides the allocator to use mimalloc at linking time, allowing LMDB, Meilisearch, and other C libraries to share their allocations for better overall memory efficiency. @Kerollmops wrote a blog post about the story behind this improvement.
-
Add
POST /tasks/compactfor task queue compaction by @YoEight in https://github.com/meilisearch/meilisearch/pull/6193Compacts the task queue database and reclaim space so new tasks can keep being enqueued, without deleting existing tasks. This feature is behind the
taskQueueCompactionRouteexperimental feature flag.
[!WARNING]
Once task queue compaction completes, all write operations are blocked until the server is restarted.
- Bump rustls-webpki from 0.103.8 to 0.103.10 in https://github.com/meilisearch/meilisearch/pull/6273
- Bump tar from 0.4.44 to 0.4.45 in https://github.com/meilisearch/meilisearch/pull/6271
- Bump actix-web-lab from 0.24.3 to 0.26.0 in https://github.com/meilisearch/meilisearch/pull/6218
- Bump quinn-proto from 0.11.13 to 0.11.14 in https://github.com/meilisearch/meilisearch/pull/6217
- Check char boundaries when hiding secrets by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6262
- Network fixes by @dureuill in https://github.com/meilisearch/meilisearch/pull/6255
- Override "default" source base url with the config one to support AWS bedrock OpenAI API by @dureuill in https://github.com/meilisearch/meilisearch/pull/5905
- Fix panic on virtiofs filesystem by @itsamine27 in https://github.com/meilisearch/meilisearch/pull/6261
- Add
X-Accel-Buffering: nofor/logs/streamroute by @YoEight in https://github.com/meilisearch/meilisearch/pull/6268 - Add link to documentation on
--experimental-allowed-ip-networksparameter by @paulden in https://github.com/meilisearch/meilisearch/pull/6263 - Update sysinfo version to
0.38.4by @GuillaumeGomez in https://github.com/meilisearch/meilisearch/pull/6211 - Add a no-agent [AGENTS.md](http://agents.md/) file to prevent excessive AI-generated PRs by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6256
- @GuillaumeGomez made their first contribution in https://github.com/meilisearch/meilisearch/pull/6211
- @itsamine27 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6261
v1.39.0 🐳
Add a new ForeignKeys setting allowing to hydrate documents with documents coming from other indexes.
📓 Note: This implementation doesn't support a remote sharding environment
A new experimental feature, foreignKeys, has been added to the /experimental-feature route that must be set to true to activate the hydration.
curl -X PATCH 'http://127.0.0.1:7700/experimental-features' \
-H 'Content-Type: application/json' \
--data-binary '{"foreignKeys": true}'
A new index setting, foreignKeys, has been added to the /indexes/{index_uid}/settings:
// new setting, an array of foreign keys that allows multiple foreign relationships between indexes
"foreignKeys": [
{
// the path in the JSON document containing foreign document ids
"fieldName": "actors",
// the UID of the foreign index containing the documents to fetch during hydration
"foreignIndexUid": "actors"
}
]
With this new feature, a document shaped as follows:
{
"id": 1,
"title": "Forrest Gump",
// Document IDs in foreign index
"actors": [
1
]
}
Will be hydrated as follows in a search response:
{
"id": 1,
"title": "Forrest Gump",
"actors": [
{
"id": 1,
"name": "Tom",
"familyName": "Hanks",
"birthDate": "1956-07-09"
}
]
}
By @ManyTheFish in #6047
Add X-Accel-Buffering: no on POST /chats/{workspace_uid}/chat/completions when the streaming mode is activated. By @YoEight in #6228
-
Fix a memory leak in the indexation pipeline by @Kerollmops in #6212
We fixed an important memory leak caused by an invalid use of bumpalo. If you've seen Meilisearch using more and more memory over time, this issue is no longer visible in the latest version. If you want to read more about the bumpalo-trap we felt in, here is a good summary. It looks like this leak was introduced in v1.12, so approximately a year ago...
-
Avoid losing tasks by applying deltas on the wtxn view by @Kerollmops in #6222
We addressed a small performance regression introduced in v1.38.1 that affected sending updates to the engine during task deletion. We've restored the task deletion performance to match v1.38.0 levels and ensured no race conditions occur while still allowing updates during deletion.
- Ignore flaky test on windows by @dureuill in #6226
- Fix small documentation mistakes by @ManyTheFish in #6205
- Improve messaging when field is missing in the settings indexing pipeline by @Kerollmops in #6213
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.38.0...v1.39.0
v1.38.1 🐍
Meilisearch v1.38.1 fixes a regression we have seen in task processing, which caused the engine to stop processing tasks.
- Revert, optimize, and fix task deletion. By @Mubelotix and @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6219
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.38.0...v1.38.1
v1.38.0 🐍
Meilisearch v1.38 drastically improves indexing performance for embeddings, cleans up the task queue of orphan tasks and batches, ensures they are correctly deleted now, fixes issues users may encounter with OpenAI or VoyageAI when generating embeddings, and improves the tooling for generating routes in the documentation.
- Embeddings: Use the latest version of hannoy with improved performances by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6184 We drastically reduced embedding indexing time by eliminating full database scans. It is more efficient to add a couple of embeddings to a large database now.
- Tasks: Optimize and fix task deletion by @Mubelotix and @kerollmops in https://github.com/meilisearch/meilisearch/pull/5828
- Embedders: Fix the intermittent
connection reset by peererrors when embedding documents or search queries using a remote embedder. By @dureuill in https://github.com/meilisearch/meilisearch/pull/6191
- Enforce that routes appear in the reference documentation by @dureuill and @manythefish in https://github.com/meilisearch/meilisearch/pull/6200 The reference API is now generated from the source code, but it used to miss some actix-declared routes. It is now mandatory to attach a route when using the newly-provided
routes::routesandroutes::pathmacros, which ensures the route is added to the reference. Please refer to the documentation of the individual macros and the many examples throughout the codebase for more details. - Chore: remove verbose debug log on list documents by @StephaneRob in https://github.com/meilisearch/meilisearch/pull/6198
- Add guide to CONTRIBUTING.md to test doc locally by @curquiza in https://github.com/meilisearch/meilisearch/pull/6180
- Docs: capitalize LangChain and MCP in README by @24f1000637 in https://github.com/meilisearch/meilisearch/pull/6183
- Docs: improve documentation of the parameters of the search route by @curquiza in https://github.com/meilisearch/meilisearch/pull/6194
- @StephaneRob made their first contribution in https://github.com/meilisearch/meilisearch/pull/6198
- @24f1000637 made their first contribution in https://github.com/meilisearch/meilisearch/pull/6183
v1.37.0
[!IMPORTANT]
This release contains breaking changes for users of thenetworkexperimental feature.
Meilisearch v1.37 introduces replicated sharding, removes the vectorStoreSetting experimental feature, stabilizes our new vector store for best performance, adds a security fix and miscellaneous improvements.
§ Replicated sharding
[!NOTE] Replicated sharding requires Meilisearch Enterprise Edition (EE).
- Users of Meilisearch Cloud, please contact support if you need replicated sharding.
- Users of the Community Edition, please contact the sales if you want to use replicated sharding in production.
§ Breaking changes
networkobjects sent to thePATCH /networkroute must now contain at least oneshardobject containing at least one remote whenleaderis notnull.
Existing databases will be migrated automatically when upgraded with --experimental-dumpless-upgrade when leader is not null, such that for each remote:
- A shard with the same name as the remote is created
- This shard has exactly one remote in its
remoteslist: the remote with the same name as the shard.
This change will not cause any document to be resharded.
To be able to upgrade without resharding, the migration uses the same name for remotes and for shards. However, in new configurations, we recommend using different names for shards and remotes.
Example of migration
For instance, the following network object:
{
"leader": "ms-00",
"self": "ms-01",
"remotes": {
"ms-00": { /* .. */ },
"ms-01": { /* .. */ }
}
}
is converted to:
{
"leader": "ms-00",
"self": "ms-01",
"remotes": {
"ms-00": { /* .. */ },
"ms-01": { /* .. */ }
},
"shards": { // ✨ NEW
"ms-00": { // shard named like the remote
"remotes": ["ms-00"] // is owned by the remote
},
"ms-01": {
"remotes": ["ms-01"]
}
}
}
The network object for routes PATCH /network and GET /network now contains the new field shards, which is an object whose values are shard objects, and keys the name of each shard.
Each shard object contains a single field remotes, which is an array of strings, each string representing the name of an existing remote.
The shard objects in PATCH /network contain the additional fields addRemotes and removeRemotes meant for convenience:
- pass an array of remote names to
shard.addRemotesto add these remotes to the list of remotes of a shard. - pass an array of remote names to
shard.removeRemotesto remove these remotes from the list of remotes of a shard. - if present and non-
null,shard.remoteswill completely override the existing list of remotes for a shard. - if several of these options are present and non-
null, then the order of application isshard.remotes, thenshard.addRemotes, thenshard.removeShards.
Adding a new shard with some remotes
// PATCH /network
{
// assuming that remotes `ms-0`, `ms-1`, `ms-2` where sent in a previous call to PATCH /network
"shards": {
"s-a": { // new shard
"remotes": ["ms-0", "ms-1"]
}
}
}
Remotes ms-0 and ms-1 own the new shard s-a.
Fully overriding the list of remotes owning a shard
// PATCH /network
{
// assuming remotes `ms-0`, `ms-1`, `ms-2`
// assuming shard `s-a`, owned by `ms-0` and `ms-1`
"shards": {
"s-a": {
"remotes": ["ms-2"]
}
}
}
ms-2 is now the sole owner of s-a, replacing ms-0 and ms-1.
Adding a remote without overriding the list of remotes owning a shard
// PATCH /network
{
// assuming remotes `ms-0`, `ms-1`, `ms-2`
// assuming shard `s-a`, owned by `ms-2`
"shards": {
"s-a": {
"addRemotes": ["ms-0"]
}
}
}
ms-0 and ms-2 are now the owners of s-a.
Removing a remote without overriding the list of remotes owning a shard
// PATCH /network
{
// assuming remotes `ms-0`, `ms-1`, `ms-2`
// assuming shard `s-a`, owned by `ms-0` and `ms-2`
"shards": {
"s-a": {
"removeRemotes": ["ms-2"]
}
}
}
ms-0 is now the sole owner of s-a.
Entirely removing a shard from the list of shards
Set the shard to null:
// PATCH /network
{
"shards": {
"s-a": null
}
}
Or set its remotes list to the empty list:
// PATCH /network
{
"shards": {
"s-a": {
"remotes": []
}
}
}
When network.leader is not null, each shard object in network.shards must:
- Only contain
remotesthat exist in the list ofremotes. - Contain at least one remote.
Additionally, network.shards must contain at least one shard.
Failure to meet any of these conditions will cause the PATCH /network route to respond with 400 invalid_network_shards.
Documents are now sharded according to the list of shards declared in the network rather than the list of remotes. All remotes owning a shard will process the documents that belong to this shard, allowing for replication.
Example of replication
The following configuration defines 3 remotes 0, 1 and 2, and 3 shards A, B, C, such that each remote owns two shards, achieving replication (losing one remote does not lose any document).
{
"leader": "0",
"self": "0",
"remotes": {
"0": { /* .. */ },
"1": { /* .. */ },
"2": { /* .. */ }
},
"shards": {
"A": {
"remotes": ["0", "1"]
},
"B": {
"remotes": ["1", "2"]
},
"C": {
"remotes": ["2", "0"]
}
}
}
- Full replication is supported by having all remotes own all the shards.
- Unbalanced replication is supported by having some remotes own more shards than other remotes.
- "Watcher" remotes are supported by having remotes that own no shards. Watcher remotes are not very useful in this release, and might be upgraded in a future release, so that they keep all documents without indexing them, allowing to "respawn" shards for other remotes.
When useNetwork: true is passed to a search query, it is expanded to multiple queries such that each shard declared in network.shards appears exactly once, associated with a remote that owns that shard.
This ensures that there is no missing or duplicate documents in the results.
When the network experimental feature is enabled, then it becomes possible to filter documents depending on the shard they belong to.
Given s-a and s-b the names of two shards declared in network.shards, then:
_shard = "s-a"in afilterparameter to the search or documents fetch will return the documents that belong tos-a._shard != "s-a"will return the documents that do not belong tos-a_shard IN ["s-a", "s-b"]will return the documents that belong tos-aor tos-b.
You can use these new filters in manual remote federated search to create a partitioning over all shards in the network.
[!IMPORTANT] To avoid duplicate or missing documents in results, for manually crafted remote federated search requests, all shards should appear in exactly one query.
[!TIP] Search requests built with
useNetwork: truealready build a correct partitioning over shards. They should be preferred to manually crafted remote federated search requests in replicated sharding scenarios.
When updating your Meilisearch network using dumpless upgrade, please observe the following guidelines:
- Do not call the
PATCH /networkroute until all remotes of the network are finished updating - If using the search routes with
useNetwork: true, call them on un-updated remotes. Calling it on already updated remotes will cause un-updated remotes to fail the search as they don't know about the_shardfilters.
By @dureuill in https://github.com/meilisearch/meilisearch/pull/6128
§ Remove vectorStoreSetting experimental feature
The new HNSW vector store (hannoy) has been stabilized and is now the only supported vector store in Meilisearch.
As a result, updating to v1.37.0 will migrate all remaining legacy vector store indexes (using arroy) to hannoy, and the vectorStoreSetting experimental feature is no longer available.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6176
We removed a computationally expensive step from vector indexing.
On a DB with 20M documents, this removes 300s per indexing batch of 1100s.
By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6175
§ 🔒 Security
- Bump mini-dashboard (local web interface) which
- now stores API key in RAM instead of in the
localStorage - bumps dependencies with potential security vulnerabilities
- now stores API key in RAM instead of in the
By @Strift and @curquiza in https://github.com/meilisearch/meilisearch/pull/6186 and https://github.com/meilisearch/meilisearch/pull/6172
§ 🔩 Miscellaneous
- Mark Cargo.lock as not linguist-generated by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6181
Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.36.0...v1.37.0
{ // new setting, an array of foreign keys that allows multiple foreign relationships between indexes "foreignKeys": [ { // the path in the JSON document containing foreign document ids "fieldName": "actors", // the UID of the foreign index containing the documents to fetch during hydration "foreignIndexUid": "actors" } ], // the actors field must be filterable on equality "filterableAttributes": [ { "attributePatterns": ["actors"], "features": { "facetSearch": false, "filter": { "equality": true, "comparison": false } } } ] }