Meilisearch v1.50 revamps the Dynamic Search Rules, adds support for federated document fetch in sharded configurations, among other improvements

This release introduces breaking changes for users using some experimental features

1. priority has been replaced with precedence, which better reflects the behavior (lower precedence means the rule is applied first)
2. conditions has been modified from an array to an object with two fields: "query" of type QueryCondition and "time" of time TimeCondition
3. New type QueryCondition that contains the fields isEmpty (as previously) and words instead of contains (same type)
4. It is now possible to pass isEmpty: false with words in a QueryCondition. Passing isEmpty:true with words still results in a synchronous error.
5. New type TimeCondition with fields start and end (unchanged semantics from previous type).
6. When specifying the selector of an Action, it is now mandatory to specify an id. Previously, it was optional, but the action would never trigger.
7. When listing rules with POST /dynamic-search-rules, filter.attributePatterns has been replaced with filter.query, an optional string that searches in rule description and conditions.query.words.
8. When calling DELETE /dynamic-search-rules/{:ruleUid} or PATCH /dynamic-search-rules/{:ruleUid} in a sharded configuration, endpoint will not return a HTTP 400 error if called on a follower remote rather than on the leader.

1. PATCH /dynamic-search-rules/{:ruleUid} and DELETE /dynamic-search-rules/{:ruleUid} now register an asynchronous task..
2. The response is modified to return the registered task instead of the modified dynamic search rule.
3. HTTP 404 is no longer returned if the {:ruleUid} portion of the URL refers to a rule that doesn't exist. This is because rules are processed asynchronously, and is consistent with the behavior of DELETE /indexes/{:indexUid}/documents/{:docId} for {:docId}

The default behavior for users using the network experimental feature with sharding configured (leader not null) will change on the following routes:

• GET indexes/:uid/documents
• GET indexes/:uid/documents/:document_id
• POST indexes/:uid/documents/fetch

Meilisearch will now fetch the documents from all the shards and not only on the local machine when processing the request. To keep the same behavior as before, users will have to set useNetwork to false when making their request.

• Dynamic search rules scale up to 75K rules without any impact on the search
• The API of Dynamic Search Rules has been simplified
  • It is harder to send conditions that will result in the rules never activating
• This also unlocks future improvements such as filter activation conditions for search rules

• Add a new DELETE /dynamic-search-rule route that deletes all the DSRs
• Add the concept of "DSR fuel" that determines how much energy is spent resolving DSR during a search. The fuel is initialized with some default variables that can be overridden using environment variables:
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_COUNTED_WORDS: max number of words considered inside of a search query for the purpose of finding conditions.query.words constraints. Defaults to 10, max value is 255
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_ACTIVE_RULES: max number of active rules whose actions are evaluated. Defaults to 1000, max value is 4294967295
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_PIN_ACTIONS: max number of pin actions that are applied. Defaults to 100, max value is 4294967295
  • MEILI_EXPERIMENTAL_DSR_FUEL_WORD_FUEL: max number of constraint combinations that are evaluated for the purpose of finding conditions.query.words constraints. Defaults to 4096, max value is 4294967295

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6484 and https://github.com/meilisearch/meilisearch/pull/6506

• The conditions.query.words behaves differently from query.contains: previously, a rule would match if its conditions query.contains would be substrings of q in the search query in the sense of str::contains. Now, a rule matches if all the words in conditions.query.words appear in q (after normalization). For q = hero super, query.contains = super hero would not match, whereas conditions.query.words = super hero does now match. This behavior is more in line with regular search, and allows improving performance.
• Dynamic search rules are now replicated from the leader to its follower, when in a sharded configuration

GET indexes/:uid/documents, GET indexes/:uid/documents/:document_id and POST indexes/:uid/documents/fetch will now fetch the documents from all the shards in the configured network.

Moreover, a new useNetwork parameter is available to activate or deactivate the usage of the network.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6495

The facets parameter in search and federated search now supports more wildcards. Previously, only the single wildcard "*" was supported, requesting all filterable fields.

Now, patterns containing * are supported with the same matching rules as in filterableAttributes.attributePatterns and localizedAttributes.attributePatterns, such as dogs.*, which will add to the facet distribution all filterable fields that match the pattern (such as dogs.intel, dogs.kefir, etc.).

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6497

Migration via --experimental-dumpless-upgrade would fail in some cases in v1.49, when trying to migrate synonyms that contained no words (empty synonyms, or containing only separator tokens such as &).

Such synonyms are now ignored during migration, avoiding the issue.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6501

In some conditions, the memory consumption of filters would increase quadratically with the length of the filter. This is now resolved for these cases.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6509

Fix a bug where some filters containing escaped characters (such as \) would cause search requests to fail with invalid_search_filter

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6499

Potentially fix an issue when sending a request to AWS S3 to create a new multipart upload, ensuring we resend the request if it fails.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6494

• Add missing route descriptions for documentation by @curquiza in https://github.com/meilisearch/meilisearch/pull/6500
• Make the prototype docs clearer by @curquiza in https://github.com/meilisearch/meilisearch/pull/6503
• Improve maintenability by simplifying partitioning step by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6511
• Fix frequently failing tests on Windows by @dureuill in https://github.com/meilisearch/meilisearch/pull/6520

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.49.0...v1.50.0

• Improve the synonyms storage by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6466 We improve synonyms' performance by changing how we store and retrieve them during query processing. Users may have experienced performance issues when the number of synonyms in an index was high, resulting in a significant impact on search performance. The Meilisearch support team advised moving the settings and synonyms-as-keywords to the dedicated documents. This is no longer an issue; synonyms are loaded lazily, only when a word matches a synonym. You can see performance improvements of up to 13x, depending on the number of synonyms.

• Fix the workflow after broken ranking scores by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6470
• Make OpenAPI rule less strict for routes returning 202 by @curquiza in https://github.com/meilisearch/meilisearch/pull/6488

• Fix a rare S3 snapshots bug by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6472

  We fixed a rare bug that could appear when using the S3 snapshot system. The bug is a race condition that occurs when we try to recycle internal buffers to reduce memory usage, which can cause an internal error and abort the snapshot upload.

• Avoid remote search to return the same document twice by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6473

  When using the remote federated search, Meilisearch was returning the same document twice from different machines. This was due to an internal filter that was not forwarded properly to the remote instances.

• Add missing searchFilterParam to GET chat workspace settings response by @CaroFG in https://github.com/meilisearch/meilisearch/pull/6475
• Split unit tests into separate files (one commit per file) by @0xfandom in https://github.com/meilisearch/meilisearch/pull/6468
• Add missing information from pre-openAPI-migration by @curquiza in https://github.com/meilisearch/meilisearch/pull/6467

Thanks to @0xfandom, who made his first contribution in https://github.com/meilisearch/meilisearch/pull/6468 🎉

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.48.2...v1.48.3

Revert #6432 due to a dumpless upgrade bug report.

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.48.0...v1.48.1

by @Mubelotix in https://github.com/meilisearch/meilisearch/pull/5765

Introduces a new POST /render-template route that can be used to render any template or fragment on any input and associated renderRoute experimental feature that gates access to the route.

This route can be used to test document templates and fragments before and after having configured an embedder.

A body payload for the route is of the form:

{
  "template": /* templateTarget object */,
  "input": /* inputTarget object or null */
}

where template describes the template or fragment to render, and input describes what to use to render the template.

Upon calling this route, Meilisearch responds with:

{
  "template": "{{doc.text}}",
  "rendered": "template text after rendering using the input"
}

where template contains the unrendered base text of the document template, or the unrendered base JSON object of a fragment, and rendered contains the result of rendering the template of the chosen input.

If input is null in the request, then rendered is null in the response, and the route can be used solely to retrieve a template or fragment from the settings of an index.

The API of this route is subject to change, so before calling this route, please enable the renderRoute experimental feature:

PATCH /experimental-features --json '{"renderRoute": true}'

1. Rendering a document from an index on a document template from an embedder of that index

// POST /render-template

{
  "template": {
    "kind": "documentTemplate",
    "indexUid": "movies",
    "embedder": "myMoviesEmbedder"
  },
  "input": {
    "kind": "indexDocument",
    "indexUid": "movies",
    "id": "2"
}

{
  "template": "A movie titled {{doc.title}} whose description starts with {{doc.overview|truncatewords:10}}",
  "rendered": "A movie titled Ariel whose description starts with Taisto Kasurinen is a Finnish coal miner whose father has..."
}

2. Rendering an inline document on a fragment from an embedder of an index

// POST /render-template

{
  "template": { 
    "kind": "indexingFragment", 
    "indexUid": "dogs", 
    "embedder": "multi",
    "fragment": "captionedImage" 
  },
  "input": { 
    "kind": "inlineDocument", 
    "inline": { // pass your document inline as a JSON object
      "kind": "dog",
      "name": "iko",
      "breed": "jack russell",
      "mime": "image/png",
      "image": "/9j/4AAQSk..."
    } 
  }
}

{
  "template": {
    "content": [
      {
        "type": "text",
        "text": "A picture of a {{doc.kind}} of breed {{doc.breed}}"
      },
      {
        "type": "image_base64",
        "image_base64": "data:{{doc.mime}};base64,{{doc.image}}"
      }
    ]
  },
  "rendered": {
    "content": [
      {
        "type": "text",
        "text": "A picture of a dog of breed jack russell"
      },
      {
        "type": "image_base64",
        "image_base64": "data:image/png;base64,/9j/4AAQSk..."
      }
    ]
  }
}

3. Rendering a search query on a search fragment from a multimodal embedder of an index

// POST /render-template
{
  "template": { 
    "kind": "searchFragment", 
    "indexUid": "testIndex", 
    "embedder": "testEmbedder",
    "fragment": "justBreed"
  },
  "input": {
    "kind": "inlineSearch",
    "inline": { // pass the search query inline
      "q": "unused",
      "media": {
        "name": "iko",
        "breed": "jack russell"
      },
      "filter": "ignored"
    }
  }
}

    {
      "template": "It's a {{ media.breed }}",
      "rendered": "It's a jack russell"
    }

4. Rendering an inline document on the document template from the chat settings of an index

// POST /render-template

{
  "template": {
    "kind": "chatDocumentTemplate",
    "indexUid": "movies"
	// no embedder to specify since chat document template is global to index
  },
  "input": {
    "kind": "indexDocument",
    "indexUid": "movies",
    "id": "2"
}

{
  "template": "{% for field in fields %}{% if field.is_searchable and field.value != nil %}{{ field.name }}: {{ field.value }}\n{% endif %}{% endfor %}",
  "rendered": "id: 2\ntitle: Ariel\noverview: Taisto Kasurinen is a Finnish coal miner whose father has just committed suicide and who is framed for a crime he did not commit. In jail, he starts to dream about leaving the country and starting a new life. He escapes from prison but things don't go as planned...\ngenres: DramaCrimeComedy\nposter: https://image.tmdb.org/t/p/w500/ojDg0PGvs6R9xYFodRct2kdI6wC.jpg\nrelease_date: 593395200\n"
}

5. Rendering a document from an index on an inline document template

// POST /render-template

{
  "template": {
    "kind": "inlineDocumentTemplate",
    "inline": "You can pass templates inline as well: nice to test them! {{doc.id}}"
  },
  "input": {
    "kind": "indexDocument",
    "indexUid": "movies",
    "id": "2"
}

{
  "template": "You can pass templates inline as well: nice to test them! {{doc.id}}",
  "rendered": "You can pass templates inline as well: nice to test them! 2"
}

6. Rendering an inline document on an inline indexing fragment

// POST /render-template

{
  "template": {
    "kind": "inlineFragment",
    "inline": {
      "json_maps": "supported for fragments",
      "any_string": "is in liquid format: {{doc.test}}"
    }
  },
  "input": {
     "kind": "inlineDocument",
    "inline": {
      "test": true
    }
  }
}

{
  "template": {
    "json_maps": "supported for fragments",
    "any_string": "is in liquid format: {{doc.test}}"
  },
  "rendered": {
    "json_maps": "supported for fragments",
    "any_string": "is in liquid format: true"
  }
}

by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6446

Foreign filters are meant to be used in a retrieval context (search, get document...), but all the actions related to writing or modifying a document could have several unexpected behaviors if foreign filters are accepted. We prefer forbidding the usage of this feature on the writing routes.

The following routes do not support Foreign-filter anymore:

• Edit documents by function: POST /indexes/{index_uid}/documents/edit
• Delete documents by filter: POST /indexes/{index_uid}/documents/delete
• Export to a remote Meilisearch: POST /export

Additional change: we now ensure that the experimental features are checked when parsing a filter

• Support prefix search on words registered in the disableOnAttributes and disableOnNumbers settings by @antcybersec in https://github.com/meilisearch/meilisearch/pull/6432
• Add missing logs in search performance details @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6457
• Ensure the index map budget is a multiplier of the OS page size by @genisis0x in https://github.com/meilisearch/meilisearch/pull/6454

• Reduce risk of vulnerability exploits on GHA by @curquiza in https://github.com/meilisearch/meilisearch/pull/6451

• Replace the queueDocumentsFetch experimental feature with disableDocumentsFetchQueue converting the feature from an opt-in to an opt-out By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6456
• Bump and removes unused dependencies by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6444
• Fix Ollama embeddings changes to fix CI tests by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6450

❤️ Thanks again to @genisis0x and @antcybersec

We now support using the search personalization feature on federated search requests.

Like page/hitPerPage or limit/offset, the personalization option must be specified in the federation attribute to work properly. Otherwise, an error will be returned reminding you to move the attribute in federation.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6414

• We now better support the tokenizer-related settings
• We improved the quality of the new settings indexer to enhance the engine's performance when changing the locales, the dictionary, synonyms, stop words, separator, and non-separator tokens.
• This makes the new settings indexer feature-complete, meaning that, unless you set the MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_SETTINGS environment variable to true, all settings tasks can now be handled by the new settings indexer, bringing a better scaling behavior, much faster cancellation, and a more precise progress view.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6409

We expose more Prometheus metrics to improve observability, specifically to show more metrics on document throughput and ease debugging.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6430

• Putting attributeRank/ wordPosition before words in the rankingRules list will no longer remove hits from the response, by @pjdurden in https://github.com/meilisearch/meilisearch/pull/6437
• Meilisearch will no longer ignore the searchCutoffMs in some conditions when embedding documents, by @dureuill in https://github.com/meilisearch/meilisearch/pull/6447
• Meilisearch will no longer fail to proxy a search request with a filter containing a ' during remote federated search or useNetwork: true search requests, by @dureuill in https://github.com/meilisearch/meilisearch/pull/6445

• CI: Prevent shell injection in GitHub Actions release workflows by @curquiza in https://github.com/meilisearch/meilisearch/pull/6420

Refactor the search pipeline to mutualize the code. The new implementation will always perform a federated search under the hood, and then the output will be transformed into the expected route's output.

Noticeable changes from the user perspective:

• Some error messages have been modified
• Small breaking change: a few error codes change, such as MultiSearch<Error> <--> Search<Error>

• Make it easier and less error-prone to declare a type that is used as a body parameter on a Meilisearch endpoint, by @dureuill in https://github.com/meilisearch/meilisearch/pull/6429
• Remove the now unused vectorStoreBackend setting from the settings, by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6399
• Replace custom hf-hub git dependency by the official one by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6442
• Update Python SDK test CI by @Strift in https://github.com/meilisearch/meilisearch/pull/6439
• Refactor the code to use the MustStopProcessing type everywhere by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6423

• @pjdurden made their first contribution in https://github.com/meilisearch/meilisearch/pull/6437

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.46.1...v1.47.0

Add an experimental feature, queueDocumentsFetch, forcing the routes GET indexes/:uid/documents and POST indexes/:uid/documents/fetch to wait in the search queue if there is no available thread to process them.

This release introduces fixes for a regression in v1.45.0, where we were batching deletions by filter with other deletions or additions. It also enables the new settings indexer to support more parameters, making the engine faster to index documents when those settings are specified.

• Support exact and disable on numbers in the new settings indexer by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6398 Introduce support for exact words and disable-on-words parameters in the new settings indexer, making the engine more efficient when changing these settings.

• Support computing prefixes in the new settings indexer by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6391 Support for the prefix search settings in the new settings indexer, so that changing this parameter makes the engine more efficient.

• Better limit read bytes when creating the S3 multipart part by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6405 This fixes an issue we had with the multipart part size by ensuring we never construct a part larger than the defined multipart part size. With this fix, we always create a multipart with the provided multipart part size, except for the last part. Thanks, @vidit-virmani, for the help investigating the issue.

• Batch of documentDeletionByFilter with documentAdditionOrUpdate by @Kerollmops and @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6415 Correctly implement the support for auto-batching deletion by filter with document replacement and updates. This fixes a regression introduced in v1.45.0.

• Fix a panic with incomplete filters by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6421 Fixes an internal panic when a filter is incomplete by returning an error instead.

• Bump tar from 0.4.45 to 0.4.46 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6418

• OpenAPI CI: disable rule to avoid crash (workaround) by @curquiza in https://github.com/meilisearch/meilisearch/pull/6417
• Remove milli benchmarks by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6419
• Add precision in the index swap documentation by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6408
• Fix broken links in the documentation by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6406