Revert #6432 due to a dumpless upgrade bug report.

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.48.0...v1.48.1

by @Mubelotix in https://github.com/meilisearch/meilisearch/pull/5765

Introduces a new POST /render-template route that can be used to render any template or fragment on any input and associated renderRoute experimental feature that gates access to the route.

This route can be used to test document templates and fragments before and after having configured an embedder.

A body payload for the route is of the form:

{
  "template": /* templateTarget object */,
  "input": /* inputTarget object or null */
}

where template describes the template or fragment to render, and input describes what to use to render the template.

Upon calling this route, Meilisearch responds with:

{
  "template": "{{doc.text}}",
  "rendered": "template text after rendering using the input"
}

where template contains the unrendered base text of the document template, or the unrendered base JSON object of a fragment, and rendered contains the result of rendering the template of the chosen input.

If input is null in the request, then rendered is null in the response, and the route can be used solely to retrieve a template or fragment from the settings of an index.

The API of this route is subject to change, so before calling this route, please enable the renderRoute experimental feature:

PATCH /experimental-features --json '{"renderRoute": true}'

1. Rendering a document from an index on a document template from an embedder of that index

// POST /render-template

{
  "template": {
    "kind": "documentTemplate",
    "indexUid": "movies",
    "embedder": "myMoviesEmbedder"
  },
  "input": {
    "kind": "indexDocument",
    "indexUid": "movies",
    "id": "2"
}

{
  "template": "A movie titled {{doc.title}} whose description starts with {{doc.overview|truncatewords:10}}",
  "rendered": "A movie titled Ariel whose description starts with Taisto Kasurinen is a Finnish coal miner whose father has..."
}

2. Rendering an inline document on a fragment from an embedder of an index

// POST /render-template

{
  "template": { 
    "kind": "indexingFragment", 
    "indexUid": "dogs", 
    "embedder": "multi",
    "fragment": "captionedImage" 
  },
  "input": { 
    "kind": "inlineDocument", 
    "inline": { // pass your document inline as a JSON object
      "kind": "dog",
      "name": "iko",
      "breed": "jack russell",
      "mime": "image/png",
      "image": "/9j/4AAQSk..."
    } 
  }
}

{
  "template": {
    "content": [
      {
        "type": "text",
        "text": "A picture of a {{doc.kind}} of breed {{doc.breed}}"
      },
      {
        "type": "image_base64",
        "image_base64": "data:{{doc.mime}};base64,{{doc.image}}"
      }
    ]
  },
  "rendered": {
    "content": [
      {
        "type": "text",
        "text": "A picture of a dog of breed jack russell"
      },
      {
        "type": "image_base64",
        "image_base64": "data:image/png;base64,/9j/4AAQSk..."
      }
    ]
  }
}

3. Rendering a search query on a search fragment from a multimodal embedder of an index

// POST /render-template
{
  "template": { 
    "kind": "searchFragment", 
    "indexUid": "testIndex", 
    "embedder": "testEmbedder",
    "fragment": "justBreed"
  },
  "input": {
    "kind": "inlineSearch",
    "inline": { // pass the search query inline
      "q": "unused",
      "media": {
        "name": "iko",
        "breed": "jack russell"
      },
      "filter": "ignored"
    }
  }
}

    {
      "template": "It's a {{ media.breed }}",
      "rendered": "It's a jack russell"
    }

4. Rendering an inline document on the document template from the chat settings of an index

// POST /render-template

{
  "template": {
    "kind": "chatDocumentTemplate",
    "indexUid": "movies"
	// no embedder to specify since chat document template is global to index
  },
  "input": {
    "kind": "indexDocument",
    "indexUid": "movies",
    "id": "2"
}

{
  "template": "{% for field in fields %}{% if field.is_searchable and field.value != nil %}{{ field.name }}: {{ field.value }}\n{% endif %}{% endfor %}",
  "rendered": "id: 2\ntitle: Ariel\noverview: Taisto Kasurinen is a Finnish coal miner whose father has just committed suicide and who is framed for a crime he did not commit. In jail, he starts to dream about leaving the country and starting a new life. He escapes from prison but things don't go as planned...\ngenres: DramaCrimeComedy\nposter: https://image.tmdb.org/t/p/w500/ojDg0PGvs6R9xYFodRct2kdI6wC.jpg\nrelease_date: 593395200\n"
}

5. Rendering a document from an index on an inline document template

// POST /render-template

{
  "template": {
    "kind": "inlineDocumentTemplate",
    "inline": "You can pass templates inline as well: nice to test them! {{doc.id}}"
  },
  "input": {
    "kind": "indexDocument",
    "indexUid": "movies",
    "id": "2"
}

{
  "template": "You can pass templates inline as well: nice to test them! {{doc.id}}",
  "rendered": "You can pass templates inline as well: nice to test them! 2"
}

6. Rendering an inline document on an inline indexing fragment

// POST /render-template

{
  "template": {
    "kind": "inlineFragment",
    "inline": {
      "json_maps": "supported for fragments",
      "any_string": "is in liquid format: {{doc.test}}"
    }
  },
  "input": {
     "kind": "inlineDocument",
    "inline": {
      "test": true
    }
  }
}

{
  "template": {
    "json_maps": "supported for fragments",
    "any_string": "is in liquid format: {{doc.test}}"
  },
  "rendered": {
    "json_maps": "supported for fragments",
    "any_string": "is in liquid format: true"
  }
}

by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6446

Foreign filters are meant to be used in a retrieval context (search, get document...), but all the actions related to writing or modifying a document could have several unexpected behaviors if foreign filters are accepted. We prefer forbidding the usage of this feature on the writing routes.

The following routes do not support Foreign-filter anymore:

• Edit documents by function: POST /indexes/{index_uid}/documents/edit
• Delete documents by filter: POST /indexes/{index_uid}/documents/delete
• Export to a remote Meilisearch: POST /export

Additional change: we now ensure that the experimental features are checked when parsing a filter

• Support prefix search on words registered in the disableOnAttributes and disableOnNumbers settings by @antcybersec in https://github.com/meilisearch/meilisearch/pull/6432
• Add missing logs in search performance details @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6457
• Ensure the index map budget is a multiplier of the OS page size by @genisis0x in https://github.com/meilisearch/meilisearch/pull/6454

• Reduce risk of vulnerability exploits on GHA by @curquiza in https://github.com/meilisearch/meilisearch/pull/6451

• Replace the queueDocumentsFetch experimental feature with disableDocumentsFetchQueue converting the feature from an opt-in to an opt-out By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6456
• Bump and removes unused dependencies by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6444
• Fix Ollama embeddings changes to fix CI tests by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6450

❤️ Thanks again to @genisis0x and @antcybersec

We now support using the search personalization feature on federated search requests.

Like page/hitPerPage or limit/offset, the personalization option must be specified in the federation attribute to work properly. Otherwise, an error will be returned reminding you to move the attribute in federation.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6414

• We now better support the tokenizer-related settings
• We improved the quality of the new settings indexer to enhance the engine's performance when changing the locales, the dictionary, synonyms, stop words, separator, and non-separator tokens.
• This makes the new settings indexer feature-complete, meaning that, unless you set the MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_SETTINGS environment variable to true, all settings tasks can now be handled by the new settings indexer, bringing a better scaling behavior, much faster cancellation, and a more precise progress view.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6409

We expose more Prometheus metrics to improve observability, specifically to show more metrics on document throughput and ease debugging.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6430

• Putting attributeRank/ wordPosition before words in the rankingRules list will no longer remove hits from the response, by @pjdurden in https://github.com/meilisearch/meilisearch/pull/6437
• Meilisearch will no longer ignore the searchCutoffMs in some conditions when embedding documents, by @dureuill in https://github.com/meilisearch/meilisearch/pull/6447
• Meilisearch will no longer fail to proxy a search request with a filter containing a ' during remote federated search or useNetwork: true search requests, by @dureuill in https://github.com/meilisearch/meilisearch/pull/6445

• CI: Prevent shell injection in GitHub Actions release workflows by @curquiza in https://github.com/meilisearch/meilisearch/pull/6420

Refactor the search pipeline to mutualize the code. The new implementation will always perform a federated search under the hood, and then the output will be transformed into the expected route's output.

Noticeable changes from the user perspective:

• Some error messages have been modified
• Small breaking change: a few error codes change, such as MultiSearch<Error> <--> Search<Error>

• Make it easier and less error-prone to declare a type that is used as a body parameter on a Meilisearch endpoint, by @dureuill in https://github.com/meilisearch/meilisearch/pull/6429
• Remove the now unused vectorStoreBackend setting from the settings, by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6399
• Replace custom hf-hub git dependency by the official one by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6442
• Update Python SDK test CI by @Strift in https://github.com/meilisearch/meilisearch/pull/6439
• Refactor the code to use the MustStopProcessing type everywhere by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6423

• @pjdurden made their first contribution in https://github.com/meilisearch/meilisearch/pull/6437

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.46.1...v1.47.0

Add an experimental feature, queueDocumentsFetch, forcing the routes GET indexes/:uid/documents and POST indexes/:uid/documents/fetch to wait in the search queue if there is no available thread to process them.

This release introduces fixes for a regression in v1.45.0, where we were batching deletions by filter with other deletions or additions. It also enables the new settings indexer to support more parameters, making the engine faster to index documents when those settings are specified.

• Support exact and disable on numbers in the new settings indexer by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6398 Introduce support for exact words and disable-on-words parameters in the new settings indexer, making the engine more efficient when changing these settings.

• Support computing prefixes in the new settings indexer by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6391 Support for the prefix search settings in the new settings indexer, so that changing this parameter makes the engine more efficient.

• Better limit read bytes when creating the S3 multipart part by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6405 This fixes an issue we had with the multipart part size by ensuring we never construct a part larger than the defined multipart part size. With this fix, we always create a multipart with the provided multipart part size, except for the last part. Thanks, @vidit-virmani, for the help investigating the issue.

• Batch of documentDeletionByFilter with documentAdditionOrUpdate by @Kerollmops and @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6415 Correctly implement the support for auto-batching deletion by filter with document replacement and updates. This fixes a regression introduced in v1.45.0.

• Fix a panic with incomplete filters by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6421 Fixes an internal panic when a filter is incomplete by returning an error instead.

• Bump tar from 0.4.45 to 0.4.46 by @dependabot[bot] in https://github.com/meilisearch/meilisearch/pull/6418

• OpenAPI CI: disable rule to avoid crash (workaround) by @curquiza in https://github.com/meilisearch/meilisearch/pull/6417
• Remove milli benchmarks by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6419
• Add precision in the index swap documentation by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6408
• Fix broken links in the documentation by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6406

• Fix internal vector stores quantization config desync by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6411 We noticed issues when dumpless upgrading databases containing embedders from versions older than v1.33.1. This version fixes those databases by removing the corrupted embeddings from the impacted indexes.

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.45.1...v1.45.2

• Revert autobatch deletions by filter with additions by @kerollmops in https://github.com/meilisearch/meilisearch/pull/6412

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.45.0...v1.45.1

Meilisearch v1.45.0 mainly improves indexing performance when changing settings and also improves document fetch performance.

• Autobatch deletions by filter with additions by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6389 While Meilisearch tries to improve indexing speed when users add documents and mix those additions with deletions via a filter, it is still an anti-pattern to interleave both. It is recommended to delete documents by ID whenever possible, as the engine is optimized to merge them, thereby drastically speeding up indexation.
• Improve settings indexing performance & visibility We are introducing support for more settings in the nez settings indexer. If you find any bugs, please report them on GitHub. You can disable the new settings indexer by setting the environment variable like this: MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_SETTINGS=true or use the equivalent dedicated CLI parameter.
  • Support non-extracting parameters by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6393 Changing one of the following settings is now directly handled by the new settings indexer: displayed fields, synonyms, the primary key, authorize typos, min word len one and two typos, max values per facet, sort facet values by, pagination max total hits, search cut off, chat, and foreign keys.
  • Support the global facet search by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6390 Meilisearch can use the new settings indexer when the settings change the facet search root boolean parameter.

• Fetching documents no longer blocks the actix worker @dureuill in https://github.com/meilisearch/meilisearch/pull/6402
• Fix an internal error when changing the binary quantization by @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6396 We fixed an issue where users were changing the binary-quantized boolean in embedder configurations. The change corrupted the database, making it impossible to change the quantization in the future. Users had to create a binary-quantized embedder from scratch, or they could never change it again.

• Make timeout test less flaky on Windows by @dureuill in https://github.com/meilisearch/meilisearch/pull/6388

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.44.0...v1.45.0