by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6587

Control the maximum number of LMDB read transactions at the same time for the task queue.

The following environment variable can be exported to specify the maximum number of simultaneous read transactions:

# Only the environment variable can set this parameter
MEILI_EXPERIMENTAL_TASK_QUEUE_MAX_READERS=100

Product discussion: https://github.com/orgs/meilisearch/discussions/806

• Bump mini-dashboard to v0.4.2 by @Strift in https://github.com/meilisearch/meilisearch/pull/6585

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.53.0...v1.53.1

• Revert the Speed up the search speed a bit more by @Kerollmops introduced in https://github.com/meilisearch/meilisearch/pull/6542

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.52.2...v1.52.3

• Revert the /tasks/stream and /batches/stream SSE routes by @Kerollmops introduced in https://github.com/meilisearch/meilisearch/pull/6533

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.52.1...v1.52.2

• Make the health route checks blocking by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6583
• Revert some search progress traces by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6584

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.52.0...v1.52.1

• Sharding for foreign filters by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6517

  • The foreign filter now supports sharding by retrieving the documents through the network when evaluating the foreign filters and hydrating the documents.

  • The limit of documents that a foreign filter can retrieve has been increased from 100 to 1000.

• Adds indexSize and usedIndexSize returned by GET /indexes/{index_uid}/stats and in GET /stats by @tpayet in https://github.com/meilisearch/meilisearch/pull/6563

• Fix code security issues in GitHub Actions workflows by @curquiza in https://github.com/meilisearch/meilisearch/pull/6565
• Bump quinn-proto from 0.11.14 to 0.11.16 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6538

• Bump peter-evans/repository-dispatch from 3.0.0 to 4.0.1 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6556

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.51.0...v1.53.0

• Introduce the /tasks/stream and /batches/stream SSE routes by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6533 This experimental feature allows establishing a live connection to the engine, avoiding the need to implement long polling and instead letting the engine provide new tasks, batches, and batch progress in a push-based manner. Using this route can drastically reduce network load when retrieving the status of a Meilisearch instance.
• Speed up the search speed a bit more by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6542 We improve the search speed when it comes to formatting the documents by making an O(n) operation, where n is the number of retrieved documents, become an O(1) operation. This improvement can significantly increase search speed when requesting a large (> 20) number of documents.

• Fix duplicate pins in federated search by @dureuill in https://github.com/meilisearch/meilisearch/pull/6543
• Skip the settings when nothing changes by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6544

• Add filter condition to DSRs by @dureuill in https://github.com/meilisearch/meilisearch/pull/6505

  New filter rule activation condition

  Dynamic search rules (DSR) can now declare a new filter condition: it contains a single values key, whose value is a JSON object. The keys of this values JSON object are the facet names (e.g., color, genres, ...), while their values are the values that a filter must resolve to for these facets, so that the rule is active.

  Example of sending a DSR with a filter condition

   // PATCH /dynamic-search-rules/test-filter
   
   {
     "conditions": {
       "filter": {
         "values": {
           "color": "red",
           "category": "shirt"
         }
       }
     },
     "actions": [
       {
         "selector": { "id": "red shirt on sales" },
         "action": { "type": "pin", "position": 0 }
       }
     ]
   }

  Example of search query that enables the DSR above

   {
     "filter": "(category = shirt AND color = red) OR (category = jeans AND color = blue)"
   }

  Note that the filter only needs to have one branch resolving to all the facet values declared in the rule for the rule to be active.

  New lastUpdatedAt field

  Dynamic search rules returned by GET /dynamic-search-rules/{:ruleUid} and POST /dynamic-search-rules contain an additional lastUpdatedAt field.

  This field is automatically updated with the enqueuedAt value of the last task that modified the rule.

  Rules are listed with the most recently updated first

  POST /dynamic-search-rules now lists rules in descending lastUpdatedAt order, meaning that the most recently updated rules will be listed first.

  New environment variables

  New environment variables are defined to control the behavior of the DSR fuel:

  • MEILI_EXPERIMENTAL_DSR_FUEL_FILTER_FUEL (in range 0..4294967296): controls how many filter constraint combinations Meilisearch will attempt to resolve at search time before giving up and applying partial rules.
  • MEILI_EXPERIMENTAL_DSR_FUEL_FILTER_OR_FUEL (in range 0..65536): controls how many filter disjunctions Meilisearch will build when turning a filter to its canonical shape
  • MEILI_EXPERIMENTAL_DSR_FUEL_FILTER_AND_FUEL (in range 0..65536): controls how many filter conjunctions Meilisearch will build in total when turning a filter to its canonical shape
  • MEILI_EXPERIMENTAL_DSR_FUEL_FILTER_DEPTH_FUEL (in range 0..256): controls the maximum recursive depth that Meilisearch allows when turning a filter to its canonical shape

• Speed up search requests by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6528 Improves Meilisearch search speed by avoiding unnecessary internal work. We drastically reduced the number of times we retrieve data from disk to a single time across the whole search pipeline. This improvement will have the greatest effect on datasets with a large number of distinct fields across documents. We have seen search speeds up to 5.4x on a dataset with more than 14k different fields.

• Stabilize dumpless upgrade by @curquiza in https://github.com/meilisearch/meilisearch/pull/6486

  ⚠️ Breaking change following the stabilization of an experimental feature ⚠️

  The flag —-experimental-dumpless-upgrade is renamed —-upgrade-db, keeping the exact same behavior.

• Remove unused experimental features by @curquiza in https://github.com/meilisearch/meilisearch/pull/6489

  ⚠️ Breaking change: removing the following experimental features ⚠️

  • Replication parameters: removing --experimental-replication-parameters because outdated. We now have another way to handle replication.
  • Import dumps with old document indexer fallback: removing --experimental-no-edition-2024-for-dumps. This flag existed to prevent bugs related to our new indexer, which is now stable.
  • No snapshot compression: removing --experimental-no-snapshot-compaction because it's not useful anymore.

• Make sure legacy attribute patterns work by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6531
  • Missing support for the shorthand filterable attributes syntax

• Add missing logs in multisearch by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6529
• Docs: Fix stale pre-v1.0 supported versions statement by @vishnujayvel in https://github.com/meilisearch/meilisearch/pull/6524
• Bump sigstore/cosign-installer from 4.1.1 to 4.1.2 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6427
• Bump actions/github-script from 7 to 9 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6425
• Bump actions/upload-artifact from 4 to 7 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6428

• @vishnujayvel made their first contribution in https://github.com/meilisearch/meilisearch/pull/6524

Meilisearch v1.50 revamps the Dynamic Search Rules, adds support for federated document fetch in sharded configurations, among other improvements

This release introduces breaking changes for users using some experimental features

1. priority has been replaced with precedence, which better reflects the behavior (lower precedence means the rule is applied first)
2. conditions has been modified from an array to an object with two fields: "query" of type QueryCondition and "time" of time TimeCondition
3. New type QueryCondition that contains the fields isEmpty (as previously) and words instead of contains (same type)
4. It is now possible to pass isEmpty: false with words in a QueryCondition. Passing isEmpty:true with words still results in a synchronous error.
5. New type TimeCondition with fields start and end (unchanged semantics from previous type).
6. When specifying the selector of an Action, it is now mandatory to specify an id. Previously, it was optional, but the action would never trigger.
7. When listing rules with POST /dynamic-search-rules, filter.attributePatterns has been replaced with filter.query, an optional string that searches in rule description and conditions.query.words.
8. When calling DELETE /dynamic-search-rules/{:ruleUid} or PATCH /dynamic-search-rules/{:ruleUid} in a sharded configuration, endpoint will not return a HTTP 400 error if called on a follower remote rather than on the leader.

1. PATCH /dynamic-search-rules/{:ruleUid} and DELETE /dynamic-search-rules/{:ruleUid} now register an asynchronous task..
2. The response is modified to return the registered task instead of the modified dynamic search rule.
3. HTTP 404 is no longer returned if the {:ruleUid} portion of the URL refers to a rule that doesn't exist. This is because rules are processed asynchronously, and is consistent with the behavior of DELETE /indexes/{:indexUid}/documents/{:docId} for {:docId}

The default behavior for users using the network experimental feature with sharding configured (leader not null) will change on the following routes:

• GET indexes/:uid/documents
• GET indexes/:uid/documents/:document_id
• POST indexes/:uid/documents/fetch

Meilisearch will now fetch the documents from all the shards and not only on the local machine when processing the request. To keep the same behavior as before, users will have to set useNetwork to false when making their request.

• Dynamic search rules scale up to 75K rules without any impact on the search
• The API of Dynamic Search Rules has been simplified
  • It is harder to send conditions that will result in the rules never activating
• This also unlocks future improvements such as filter activation conditions for search rules

• Add a new DELETE /dynamic-search-rule route that deletes all the DSRs
• Add the concept of "DSR fuel" that determines how much energy is spent resolving DSR during a search. The fuel is initialized with some default variables that can be overridden using environment variables:
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_COUNTED_WORDS: max number of words considered inside of a search query for the purpose of finding conditions.query.words constraints. Defaults to 10, max value is 255
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_ACTIVE_RULES: max number of active rules whose actions are evaluated. Defaults to 1000, max value is 4294967295
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_PIN_ACTIONS: max number of pin actions that are applied. Defaults to 100, max value is 4294967295
  • MEILI_EXPERIMENTAL_DSR_FUEL_WORD_FUEL: max number of constraint combinations that are evaluated for the purpose of finding conditions.query.words constraints. Defaults to 4096, max value is 4294967295

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6484 and https://github.com/meilisearch/meilisearch/pull/6506

• The conditions.query.words behaves differently from query.contains: previously, a rule would match if its conditions query.contains would be substrings of q in the search query in the sense of str::contains. Now, a rule matches if all the words in conditions.query.words appear in q (after normalization). For q = hero super, query.contains = super hero would not match, whereas conditions.query.words = super hero does now match. This behavior is more in line with regular search, and allows improving performance.
• Dynamic search rules are now replicated from the leader to its follower, when in a sharded configuration

GET indexes/:uid/documents, GET indexes/:uid/documents/:document_id and POST indexes/:uid/documents/fetch will now fetch the documents from all the shards in the configured network.

Moreover, a new useNetwork parameter is available to activate or deactivate the usage of the network.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6495

The facets parameter in search and federated search now supports more wildcards. Previously, only the single wildcard "*" was supported, requesting all filterable fields.

Now, patterns containing * are supported with the same matching rules as in filterableAttributes.attributePatterns and localizedAttributes.attributePatterns, such as dogs.*, which will add to the facet distribution all filterable fields that match the pattern (such as dogs.intel, dogs.kefir, etc.).

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6497

Migration via --experimental-dumpless-upgrade would fail in some cases in v1.49, when trying to migrate synonyms that contained no words (empty synonyms, or containing only separator tokens such as &).

Such synonyms are now ignored during migration, avoiding the issue.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6501

In some conditions, the memory consumption of filters would increase quadratically with the length of the filter. This is now resolved for these cases.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6509

Fix a bug where some filters containing escaped characters (such as \) would cause search requests to fail with invalid_search_filter

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6499

Potentially fix an issue when sending a request to AWS S3 to create a new multipart upload, ensuring we resend the request if it fails.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6494

• Add missing route descriptions for documentation by @curquiza in https://github.com/meilisearch/meilisearch/pull/6500
• Make the prototype docs clearer by @curquiza in https://github.com/meilisearch/meilisearch/pull/6503
• Improve maintenability by simplifying partitioning step by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6511
• Fix frequently failing tests on Windows by @dureuill in https://github.com/meilisearch/meilisearch/pull/6520

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.49.0...v1.50.0

• Improve the synonyms storage by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6466 We improve synonyms' performance by changing how we store and retrieve them during query processing. Users may have experienced performance issues when the number of synonyms in an index was high, resulting in a significant impact on search performance. The Meilisearch support team advised moving the settings and synonyms-as-keywords to the dedicated documents. This is no longer an issue; synonyms are loaded lazily, only when a word matches a synonym. You can see performance improvements of up to 13x, depending on the number of synonyms.

• Fix the workflow after broken ranking scores by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6470
• Make OpenAPI rule less strict for routes returning 202 by @curquiza in https://github.com/meilisearch/meilisearch/pull/6488

• Fix a rare S3 snapshots bug by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6472

  We fixed a rare bug that could appear when using the S3 snapshot system. The bug is a race condition that occurs when we try to recycle internal buffers to reduce memory usage, which can cause an internal error and abort the snapshot upload.

• Avoid remote search to return the same document twice by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6473

  When using the remote federated search, Meilisearch was returning the same document twice from different machines. This was due to an internal filter that was not forwarded properly to the remote instances.

• Add missing searchFilterParam to GET chat workspace settings response by @CaroFG in https://github.com/meilisearch/meilisearch/pull/6475
• Split unit tests into separate files (one commit per file) by @0xfandom in https://github.com/meilisearch/meilisearch/pull/6468
• Add missing information from pre-openAPI-migration by @curquiza in https://github.com/meilisearch/meilisearch/pull/6467

Thanks to @0xfandom, who made his first contribution in https://github.com/meilisearch/meilisearch/pull/6468 🎉

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.48.2...v1.48.3