Only spawn request threads if necessary

4617: Destructure `EmbedderOptions` so we don't miss some options r=dureuill a=dureuill

# Pull Request

## Related issue
#4595 was caused by the code not destructuring the embedder options.


## What does this PR do?
This PR adds the missing `url` parameter for ollama, and makes sure similar issue cannot happen in the future



Co-authored-by: Louis Dureuil <louis@meilisearch.com>

.github/ISSUE_TEMPLATE/sprint_issue.md

@@ -2,14 +2,13 @@
 name: New sprint issue
 about: ⚠️ Should only be used by the engine team ⚠️
 title: ''
-labels: ''
+labels: 'missing usage in PRD, impacts docs'
 assignees: ''
 
 ---
 
 Related product team resources: [PRD]() (_internal only_)
 Related product discussion:
-Related spec: WIP
 
 ## Motivation
 
@@ -21,11 +20,7 @@ Related spec: WIP
 
 ## TODO
 
-<!---Feel free to adapt this list with more technical/product steps-->
-
-- [ ] Release a prototype
-- [ ] If prototype validated, merge changes into `main`
-- [ ] Update the spec
+<!---If necessary, create a list with technical/product steps-->
 
 ### Reminders when modifying the Setting API
 

.github/workflows/bench-pr.yml

@@ -43,4 +43,4 @@ jobs:
 
         - name: Run benchmarks on PR ${{ github.event.issue.id }}
           run: |
-            cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" --reason "[Comment](${{ github.event.comment.url }}) on [#${{github.event.issue.id}}](${{ github.event.issue.url }})" -- ${{ steps.command.outputs.command-arguments }}
+            cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" --reason "[Comment](${{ github.event.comment.html_url }}) on [#${{ github.event.issue.number }}](${{ github.event.issue.html_url }})" -- ${{ steps.command.outputs.command-arguments }}

.github/workflows/milestone-workflow.yml

@@ -110,6 +110,44 @@ jobs:
             --milestone $MILESTONE_VERSION \
             --assignee curquiza
 
+  create-update-version-issue:
+    needs: get-release-version
+    # Create the update-version issue even if the release is a patch release
+    if: github.event.action == 'created'
+    runs-on: ubuntu-latest
+    env:
+      ISSUE_TEMPLATE: issue-template.md
+    steps:
+      - uses: actions/checkout@v3
+      - name: Download the issue template
+        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-version-issue.md > $ISSUE_TEMPLATE
+      - name: Create the issue
+        run: |
+          gh issue create \
+            --title "Update version in Cargo.toml for $MILESTONE_VERSION" \
+            --label 'maintenance' \
+            --body-file $ISSUE_TEMPLATE \
+            --milestone $MILESTONE_VERSION
+
+  create-update-openapi-issue:
+    needs: get-release-version
+    # Create the openAPI issue if the release is not only a patch release
+    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
+    runs-on: ubuntu-latest
+    env:
+      ISSUE_TEMPLATE: issue-template.md
+    steps:
+      - uses: actions/checkout@v3
+      - name: Download the issue template
+        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-openapi-issue.md > $ISSUE_TEMPLATE
+      - name: Create the issue
+        run: |
+          gh issue create \
+            --title "Update Open API file for $MILESTONE_VERSION" \
+            --label 'maintenance' \
+            --body-file $ISSUE_TEMPLATE \
+            --milestone $MILESTONE_VERSION
+
 # ----------------
 # MILESTONE CLOSED
 # ----------------

BENCHMARKS.md

@@ -0,0 +1,362 @@
+# Benchmarks
+
+Currently this repository hosts two kinds of benchmarks:
+
+1. The older "milli benchmarks", that use [criterion](https://github.com/bheisler/criterion.rs) and live in the "benchmarks" directory.
+2. The newer "bench" that are workload-based and so split between the [`workloads`](./workloads/) directory and the [`xtask::bench`](./xtask/src/bench/) module.
+
+This document describes the newer "bench" benchmarks. For more details on the "milli benchmarks", see [benchmarks/README.md](./benchmarks/README.md).
+
+## Design philosophy for the benchmarks
+
+The newer "bench" benchmarks are **integration** benchmarks, in the sense that they spawn an actual Meilisearch server and measure its performance end-to-end, including HTTP request overhead.
+
+Since this is prone to fluctuating, the benchmarks regain a bit of precision by measuring the runtime of the individual spans using the [logging machinery](./CONTRIBUTING.md#logging) of Meilisearch.
+
+A span roughly translates to a function call. The benchmark runner collects all the spans by name using the [logs route](https://github.com/orgs/meilisearch/discussions/721) and sums their runtime. The processed results are then sent to the [benchmark dashboard](https://bench.meilisearch.dev), which is in charge of storing and presenting the data.
+
+## Running the benchmarks
+
+Benchmarks can run locally or in CI.
+
+### Locally
+
+#### With a local benchmark dashboard
+
+The benchmarks dashboard lives in its [own repository](https://github.com/meilisearch/benchboard). We provide binaries for Ubuntu/Debian, but you can build from source for other platforms (MacOS should work as it was developed under that platform).
+
+Run the `benchboard` binary to create a fresh database of results. By default it will serve the results and the API to gather results on `http://localhost:9001`.
+
+From the Meilisearch repository, you can then run benchmarks with:
+
+```sh
+cargo xtask bench -- workloads/my_workload_1.json ..
+```
+
+This command will build and run Meilisearch locally on port 7700, so make sure that this port is available.
+To run benchmarks on a different commit, just use the usual git command to get back to the desired commit.
+
+#### Without a local benchmark dashboard
+
+To work with the raw results, you can also skip using a local benchmark dashboard.
+
+Run:
+
+```sh
+cargo xtask bench --no-dashboard -- workloads/my_workload_1.json workloads/my_workload_2.json ..
+```
+
+For processing the results, look at [Looking at benchmark results/Without dashboard](#without-dashboard).
+
+### In CI
+
+We have dedicated runners to run workloads on CI. Currently, there are three ways of running the CI:
+
+1. Automatically, on every push to `main`.
+2. Manually, by clicking the [`Run workflow`](https://github.com/meilisearch/meilisearch/actions/workflows/bench-manual.yml) button and specifying the target reference (tag, commit or branch) as well as one or multiple workloads to run. The workloads must exist in the Meilisearch repository (conventionally, in the [`workloads`](./workloads/) directory) on the target reference. Globbing (e.g., `workloads/*.json`) works.
+3. Manually on a PR, by posting a comment containing a `/bench` command, followed by one or multiple workloads to run. Globbing works. The workloads must exist in the Meilisearch repository in the branch of the PR.
+  ```
+  /bench workloads/movies*.json /hackernews_1M.json
+  ```
+
+## Looking at benchmark results
+
+### On the dashboard
+
+Results are available on the global dashboard used by CI at <https://bench.meilisearch.dev> or on your [local dashboard](#with-a-local-benchmark-dashboard).
+
+The dashboard homepage presents three sections:
+
+1. The latest invocations (a call to `cargo xtask bench`, either local or by CI) with their reason (generally set to some helpful link in CI) and their status.
+2. The latest workloads ran on `main`.
+3. The latest workloads ran on other references.
+
+By default, the workload shows the total runtime delta with the latest applicable commit on `main`. The latest applicable commit is the latest commit for workload invocations that do not originate on `main`, and the latest previous commit for workload invocations that originate on `main`.
+
+You can explicitly request a detailed comparison by span with the `main` branch, the branch or origin, or any previous commit, by clicking the links at the bottom of the workload invocation.
+
+In the detailed comparison view, the spans are sorted by improvements, regressions, stable (no statistically significant change) and unstable (the span runtime is comparable to its standard deviation).
+
+You can click on the name of any span to get a box plot comparing the target commit with multiple commits of the selected branch.
+
+### Without dashboard
+
+After the workloads are done running, the reports will live in the Meilisearch repository, in the `bench/reports` directory (by default).
+
+You can then convert these reports into other formats.
+
+- To [Firefox profiler](https://profiler.firefox.com) format. Run:
+  ```sh
+  cd bench/reports
+  cargo run --release --bin trace-to-firefox -- my_workload_1-0-trace.json
+  ```
+  You can then upload the resulting `firefox-my_workload_1-0-trace.json` file to the online profiler.
+
+
+## Designing benchmark workloads
+
+Benchmark workloads conventionally live in the `workloads` directory of the Meilisearch repository.
+
+They are JSON files with the following structure (comments are not actually supported, to make your own, remove them or copy some existing workload file):
+
+```jsonc
+{
+  // Name of the workload. Must be unique to the workload, as it will be used to group results on the dashboard.
+  "name": "hackernews.ndjson_1M,no-threads",
+  // Number of consecutive runs of the commands that should be performed.
+  // Each run uses a fresh instance of Meilisearch and a fresh database.
+  // Each run produces its own report file.
+  "run_count": 3,
+  // List of arguments to add to the Meilisearch command line.
+  "extra_cli_args": ["--max-indexing-threads=1"],
+  // List of named assets that can be used in the commands.
+  "assets": {
+    // name of the asset.
+    // Must be unique at the workload level.
+    // For better results, the same asset (same sha256) should have the same name accross workloads.
+    // Having multiple assets with the same name and distinct hashes is supported accross workloads,
+    // but will lead to superfluous downloads.
+    //
+    // Assets are stored in the `bench/assets/` directory by default.
+    "hackernews-100_000.ndjson": {
+      // If the assets exists in the local filesystem (Meilisearch repository or for your local workloads)
+      // Its file path can be specified here.
+      // `null` if the asset should be downloaded from a remote location.
+      "local_location": null,
+      // URL of the remote location where the asset can be downloaded.
+      // Use the `--assets-key` of the runner to pass an API key in the `Authorization: Bearer` header of the download requests.
+      // `null` if the asset should be imported from a local location.
+      // if both local and remote locations are specified, then the local one is tried first, then the remote one
+      // if the file is locally missing or its hash differs.
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-100_000.ndjson",
+      // SHA256 of the asset.
+      // Optional, the `sha256` of the asset will be displayed during a run of the workload if it is missing.
+      // If present, the hash of the asset in the `bench/assets/` directory will be compared against this hash before
+      // running the workload. If the hashes differ, the asset will be downloaded anew.
+      "sha256": "60ecd23485d560edbd90d9ca31f0e6dba1455422f2a44e402600fbb5f7f1b213",
+      // Optional, one of "Auto", "Json", "NdJson" or "Raw".
+      // If missing, assumed to be "Auto".
+      // If "Auto", the format will be determined from the extension in the asset name.
+      "format": "NdJson"
+    },
+    "hackernews-200_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-200_000.ndjson",
+      "sha256": "785b0271fdb47cba574fab617d5d332276b835c05dd86e4a95251cf7892a1685"
+    },
+    "hackernews-300_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-300_000.ndjson",
+      "sha256": "de73c7154652eddfaf69cdc3b2f824d5c452f095f40a20a1c97bb1b5c4d80ab2"
+    },
+    "hackernews-400_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-400_000.ndjson",
+      "sha256": "c1b00a24689110f366447e434c201c086d6f456d54ed1c4995894102794d8fe7"
+    },
+    "hackernews-500_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-500_000.ndjson",
+      "sha256": "ae98f9dbef8193d750e3e2dbb6a91648941a1edca5f6e82c143e7996f4840083"
+    },
+    "hackernews-600_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-600_000.ndjson",
+      "sha256": "b495fdc72c4a944801f786400f22076ab99186bee9699f67cbab2f21f5b74dbe"
+    },
+    "hackernews-700_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-700_000.ndjson",
+      "sha256": "4b2c63974f3dabaa4954e3d4598b48324d03c522321ac05b0d583f36cb78a28b"
+    },
+    "hackernews-800_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-800_000.ndjson",
+      "sha256": "cb7b6afe0e6caa1be111be256821bc63b0771b2a0e1fad95af7aaeeffd7ba546"
+    },
+    "hackernews-900_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-900_000.ndjson",
+      "sha256": "e1154ddcd398f1c867758a93db5bcb21a07b9e55530c188a2917fdef332d3ba9"
+    },
+    "hackernews-1_000_000.ndjson": {
+      "local_location": null,
+      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-1_000_000.ndjson",
+      "sha256": "27e25efd0b68b159b8b21350d9af76938710cb29ce0393fa71b41c4f3c630ffe"
+    }
+  },
+  // Core of the workload.
+  // A list of commands to run sequentially.
+  // A command is a request to the Meilisearch instance that is executed while the profiling runs.
+  "commands": [
+    {
+      // Meilisearch route to call. `http://localhost:7700/` will be prepended.
+      "route": "indexes/movies/settings",
+      // HTTP method to call.
+      "method": "PATCH",
+      // If applicable, body of the request.
+      // Optional, if missing, the body will be empty.
+      "body": {
+        // One of "empty", "inline" or "asset".
+        // If using "empty", you can skip the entire "body" key.
+        "inline": {
+          // when "inline" is used, the body is the JSON object that is the value of the `"inline"` key.
+          "displayedAttributes": [
+            "title",
+            "by",
+            "score",
+            "time"
+          ],
+          "searchableAttributes": [
+            "title"
+          ],
+          "filterableAttributes": [
+            "by"
+          ],
+          "sortableAttributes": [
+            "score",
+            "time"
+          ]
+        }
+      },
+      // Whether to wait before running the next request.
+      // One of:
+      // - DontWait: run the next command without waiting the response to this one.
+      // - WaitForResponse: run the next command as soon as the response from the server is received.
+      // - WaitForTask: run the next command once **all** the Meilisearch tasks created up to now have finished processing.
+      "synchronous": "DontWait"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        // When using "asset", use the name of an asset as value to use the content of that asset as body.
+        // the content type is derived of the format of the asset:
+        // "NdJson" => "application/x-ndjson"
+        // "Json" => "application/json"
+        // "Raw" => "application/octet-stream"
+        // See [AssetFormat::to_content_type](https://github.com/meilisearch/meilisearch/blob/7b670a4afadb132ac4a01b6403108700501a391d/xtask/src/bench/assets.rs#L30)
+        // for details and up-to-date list.
+        "asset": "hackernews-100_000.ndjson"
+      },
+      "synchronous": "WaitForTask"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-200_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-300_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-400_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-500_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-600_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-700_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-800_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-900_000.ndjson"
+      },
+      "synchronous": "WaitForResponse"
+    },
+    {
+      "route": "indexes/movies/documents",
+      "method": "POST",
+      "body": {
+        "asset": "hackernews-1_000_000.ndjson"
+      },
+      "synchronous": "WaitForTask"
+    }
+  ]
+}
+```
+
+### Adding new assets
+
+Assets reside in our DigitalOcean S3 space. Assuming you have team access to the DigitalOcean S3 space:
+
+1. go to <https://cloud.digitalocean.com/spaces/milli-benchmarks?i=d1c552&path=bench%2Fdatasets%2F>
+2. upload your dataset:
+   1. if your dataset is a single file, upload that single file using the "upload" button,
+   2. otherwise, create a folder using the "create folder" button, then inside that folder upload your individual files.
+
+## Upgrading `https://bench.meilisearch.dev`
+
+The URL of the server is in our password manager (look for "benchboard").
+
+1. Make the needed modifications on the [benchboard repository](https://github.com/meilisearch/benchboard) and merge them to main.
+2. Publish a new release to produce the Ubuntu/Debian binary.
+3. Download the binary locally, send it to the server:
+  ```
+  scp -6 ~/Downloads/benchboard root@\[<ipv6-address>\]:/bench/new-benchboard
+  ```
+  Note that the ipv6 must be between escaped square brackets for SCP.
+4. SSH to the server:
+  ```
+  ssh root@<ipv6-address>
+  ```
+  Note the ipv6 must **NOT** be between escaped square brackets for SSH 🥲
+5. On the server, set the correct permissions for the new binary:
+   ```
+   chown bench:bench /bench/new-benchboard
+   chmod 700 /bench/new-benchboard
+   ```
+6. On the server, move the new binary to the location of the running binary (if unsure, start by making a backup of the running binary):
+  ```
+  mv /bench/{new-,}benchboard
+  ```
+7. Restart the benchboard service.
+  ```
+  systemctl restart benchboard
+  ```
+8. Check that the service runs correctly.
+  ```
+  systemctl status benchboard
+  ```
+9. Check the availability of the service by going to <https://bench.meilisearch.dev> on your browser.

CONTRIBUTING.md

@@ -4,7 +4,7 @@ First, thank you for contributing to Meilisearch! The goal of this document is t
 
 Remember that there are many ways to contribute other than writing code: writing [tutorials or blog posts](https://github.com/meilisearch/awesome-meilisearch), improving [the documentation](https://github.com/meilisearch/documentation), submitting [bug reports](https://github.com/meilisearch/meilisearch/issues/new?assignees=&labels=&template=bug_report.md&title=) and [feature requests](https://github.com/meilisearch/product/discussions/categories/feedback-feature-proposal)...
 
-The code in this repository is only concerned with managing multiple indexes, handling the update store, and exposing an HTTP API. Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/milli), while tokenization is handled by [our `charabia` library](https://github.com/meilisearch/charabia/).
+Meilisearch can manage multiple indexes, handle the update store, and expose an HTTP API. Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/meilisearch/tree/main/milli), while tokenization is handled by [our `charabia` library](https://github.com/meilisearch/charabia/).
 
 If Meilisearch does not offer optimized support for your language, please consider contributing to `charabia` by following the [CONTRIBUTING.md file](https://github.com/meilisearch/charabia/blob/main/CONTRIBUTING.md) and integrating your intended normalizer/segmenter.
 
@@ -81,6 +81,30 @@ Meilisearch follows the [cargo xtask](https://github.com/matklad/cargo-xtask) wo
 
 Run `cargo xtask --help` from the root of the repository to find out what is available.
 
+### Logging
+
+Meilisearch uses [`tracing`](https://lib.rs/crates/tracing) for logging purposes. Tracing logs are structured and can be displayed as JSON to the end user, so prefer passing arguments as fields rather than interpolating them in the message.
+
+Refer to the [documentation](https://docs.rs/tracing/0.1.40/tracing/index.html#using-the-macros) for the syntax of the spans and events.
+
+Logging spans are used for 3 distinct purposes:
+
+1. Regular logging
+2. Profiling
+3. Benchmarking
+
+As a result, the spans should follow some rules:
+
+- They should not be put on functions that are called too often. That is because opening and closing a span causes some overhead. For regular logging, avoid putting spans on functions that are taking less than a few hundred nanoseconds. For profiling or benchmarking, avoid putting spans on functions that are taking less than a few microseconds.
+- For profiling and benchmarking, use the `TRACE` level.
+- For profiling and benchmarking, use the following `target` prefixes:
+  - `indexing::` for spans meant when profiling the indexing operations.
+  - `search::` for spans meant when profiling the search operations.
+
+### Benchmarking
+
+See [BENCHMARKS.md](./BENCHMARKS.md)
+
 ## Git Guidelines
 
 ### Git Branches

Cargo.toml

@@ -17,11 +17,12 @@ members = [
     "benchmarks",
     "fuzzers",
     "tracing-trace",
-    "xtask", "build-info",
+    "xtask",
+    "build-info",
 ]
 
 [workspace.package]
-version = "1.7.2"
+version = "1.8.0"
 authors = [
     "Quentin de Quelen <quentin@dequelen.me>",
     "Clément Renault <clement@meilisearch.com>",

dump/src/lib.rs

@@ -256,8 +256,8 @@ pub(crate) mod test {
 
     pub fn create_test_settings() -> Settings<Checked> {
         let settings = Settings {
-            displayed_attributes: Setting::Set(vec![S("race"), S("name")]),
-            searchable_attributes: Setting::Set(vec![S("name"), S("race")]),
+            displayed_attributes: Setting::Set(vec![S("race"), S("name")]).into(),
+            searchable_attributes: Setting::Set(vec![S("name"), S("race")]).into(),
             filterable_attributes: Setting::Set(btreeset! { S("race"), S("age") }),
             sortable_attributes: Setting::Set(btreeset! { S("age") }),
             ranking_rules: Setting::NotSet,
@@ -277,6 +277,7 @@ pub(crate) mod test {
             }),
             pagination: Setting::NotSet,
             embedders: Setting::NotSet,
+            search_cutoff_ms: Setting::NotSet,
             _kind: std::marker::PhantomData,
         };
         settings.check()

dump/src/reader/compat/v5_to_v6.rs

@@ -315,8 +315,8 @@ impl From<v5::ResponseError> for v6::ResponseError {
 impl<T> From<v5::Settings<T>> for v6::Settings<v6::Unchecked> {
     fn from(settings: v5::Settings<T>) -> Self {
         v6::Settings {
-            displayed_attributes: settings.displayed_attributes.into(),
-            searchable_attributes: settings.searchable_attributes.into(),
+            displayed_attributes: v6::Setting::from(settings.displayed_attributes).into(),
+            searchable_attributes: v6::Setting::from(settings.searchable_attributes).into(),
             filterable_attributes: settings.filterable_attributes.into(),
             sortable_attributes: settings.sortable_attributes.into(),
             ranking_rules: {
@@ -379,6 +379,7 @@ impl<T> From<v5::Settings<T>> for v6::Settings<v6::Unchecked> {
                 v5::Setting::NotSet => v6::Setting::NotSet,
             },
             embedders: v6::Setting::NotSet,
+            search_cutoff_ms: v6::Setting::NotSet,
             _kind: std::marker::PhantomData,
         }
     }

dump/src/reader/v2/updates.rs

@@ -61,7 +61,7 @@ pub enum IndexDocumentsMethod {
 #[cfg_attr(test, derive(serde::Serialize))]
 #[non_exhaustive]
 pub enum UpdateFormat {
-    /// The given update is a real **comma seperated** CSV with headers on the first line.
+    /// The given update is a real **comma separated** CSV with headers on the first line.
     Csv,
     /// The given update is a JSON array with documents inside.
     Json,

dump/src/writer.rs

@@ -219,7 +219,7 @@ pub(crate) mod test {
     fn _create_directory_hierarchy(dir: &Path, depth: usize) -> String {
         let mut ret = String::new();
 
-        // the entries are not guarenteed to be returned in the same order thus we need to sort them.
+        // the entries are not guaranteed to be returned in the same order thus we need to sort them.
         let mut entries =
             fs::read_dir(dir).unwrap().collect::<std::result::Result<Vec<_>, _>>().unwrap();
 

filter-parser/src/value.rs

@@ -42,7 +42,7 @@ fn quoted_by(quote: char, input: Span) -> IResult<Token> {
                 )));
             }
         }
-        // if it was preceeded by a `\` or if it was anything else we can continue to advance
+        // if it was preceded by a `\` or if it was anything else we can continue to advance
     }
 
     Ok((

index-scheduler/src/autobatcher.rs

@@ -870,7 +870,7 @@ mod tests {
         debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, false, None), settings(false), idx_del()]), @"Some((IndexDeletion { ids: [0, 2, 1] }, false))");
         debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments,false, None), settings(false), doc_clr(), idx_del()]), @"Some((IndexDeletion { ids: [1, 3, 0, 2] }, false))");
         debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, false, None), settings(false), doc_clr(), idx_del()]), @"Some((IndexDeletion { ids: [1, 3, 0, 2] }, false))");
-        // The third and final case is when the first task doesn't create an index but is directly followed by a task creating an index. In this case we can't batch whith what
+        // The third and final case is when the first task doesn't create an index but is directly followed by a task creating an index. In this case we can't batch whit what
         // follows because we first need to process the erronous batch.
         debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments,false, None), settings(true), idx_del()]), @"Some((DocumentOperation { method: ReplaceDocuments, allow_index_creation: false, primary_key: None, operation_ids: [0] }, false))");
         debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, false, None), settings(true), idx_del()]), @"Some((DocumentOperation { method: UpdateDocuments, allow_index_creation: false, primary_key: None, operation_ids: [0] }, false))");

index-scheduler/src/batch.rs

@@ -920,7 +920,11 @@ impl IndexScheduler {
                     }
 
                     // 3.2. Dump the settings
-                    let settings = meilisearch_types::settings::settings(index, &rtxn)?;
+                    let settings = meilisearch_types::settings::settings(
+                        index,
+                        &rtxn,
+                        meilisearch_types::settings::SecretPolicy::RevealSecrets,
+                    )?;
                     index_dumper.settings(&settings)?;
                     Ok(())
                 })?;

index-scheduler/src/lib.rs

@@ -1301,8 +1301,8 @@ impl IndexScheduler {
 
         wtxn.commit().map_err(Error::HeedTransaction)?;
 
-        // Once the tasks are commited, we should delete all the update files associated ASAP to avoid leaking files in case of a restart
-        tracing::debug!("Deleting the upadate files");
+        // Once the tasks are committed, we should delete all the update files associated ASAP to avoid leaking files in case of a restart
+        tracing::debug!("Deleting the update files");
 
         //We take one read transaction **per thread**. Then, every thread is going to pull out new IDs from the roaring bitmap with the help of an atomic shared index into the bitmap
         let idx = AtomicU32::new(0);
@@ -1332,7 +1332,7 @@ impl IndexScheduler {
         Ok(TickOutcome::TickAgain(processed_tasks))
     }
 
-    /// Once the tasks changes have been commited we must send all the tasks that were updated to our webhook if there is one.
+    /// Once the tasks changes have been committed we must send all the tasks that were updated to our webhook if there is one.
     fn notify_webhook(&self, updated: &RoaringBitmap) -> Result<()> {
         if let Some(ref url) = self.webhook_url {
             struct TaskReader<'a, 'b> {
@@ -3028,6 +3028,67 @@ mod tests {
         snapshot!(serde_json::to_string_pretty(&documents).unwrap(), name: "documents");
     }
 
+    #[test]
+    fn test_settings_update() {
+        use meilisearch_types::settings::{Settings, Unchecked};
+        use milli::update::Setting;
+
+        let (index_scheduler, mut handle) = IndexScheduler::test(true, vec![]);
+
+        let mut new_settings: Box<Settings<Unchecked>> = Box::default();
+        let mut embedders = BTreeMap::default();
+        let embedding_settings = milli::vector::settings::EmbeddingSettings {
+            source: Setting::Set(milli::vector::settings::EmbedderSource::Rest),
+            api_key: Setting::Set(S("My super secret")),
+            url: Setting::Set(S("http://localhost:7777")),
+            dimensions: Setting::Set(4),
+            ..Default::default()
+        };
+        embedders.insert(S("default"), Setting::Set(embedding_settings));
+        new_settings.embedders = Setting::Set(embedders);
+
+        index_scheduler
+            .register(
+                KindWithContent::SettingsUpdate {
+                    index_uid: S("doggos"),
+                    new_settings,
+                    is_deletion: false,
+                    allow_index_creation: true,
+                },
+                None,
+                false,
+            )
+            .unwrap();
+        index_scheduler.assert_internally_consistent();
+
+        snapshot!(snapshot_index_scheduler(&index_scheduler), name: "after_registering_settings_task");
+
+        {
+            let rtxn = index_scheduler.read_txn().unwrap();
+            let task = index_scheduler.get_task(&rtxn, 0).unwrap().unwrap();
+            let task = meilisearch_types::task_view::TaskView::from_task(&task);
+            insta::assert_json_snapshot!(task.details);
+        }
+
+        handle.advance_n_successful_batches(1);
+        snapshot!(snapshot_index_scheduler(&index_scheduler), name: "settings_update_processed");
+
+        {
+            let rtxn = index_scheduler.read_txn().unwrap();
+            let task = index_scheduler.get_task(&rtxn, 0).unwrap().unwrap();
+            let task = meilisearch_types::task_view::TaskView::from_task(&task);
+            insta::assert_json_snapshot!(task.details);
+        }
+
+        // has everything being pushed successfully in milli?
+        let index = index_scheduler.index("doggos").unwrap();
+        let rtxn = index.read_txn().unwrap();
+
+        let configs = index.embedding_configs(&rtxn).unwrap();
+        let (_, embedding_config) = configs.first().unwrap();
+        insta::assert_json_snapshot!(embedding_config.embedder_options);
+    }
+
     #[test]
     fn test_document_replace_without_autobatching() {
         let (index_scheduler, mut handle) = IndexScheduler::test(false, vec![]);

index-scheduler/src/snapshots/index_scheduler__tests__settings_update-2.snap

@@ -0,0 +1,14 @@
+---
+source: index-scheduler/src/lib.rs
+expression: task.details
+---
+{
+  "embedders": {
+    "default": {
+      "source": "rest",
+      "apiKey": "MyXXXX...",
+      "dimensions": 4,
+      "url": "http://localhost:7777"
+    }
+  }
+}

index-scheduler/src/snapshots/index_scheduler__tests__settings_update-3.snap

@@ -0,0 +1,23 @@
+---
+source: index-scheduler/src/lib.rs
+expression: embedding_config.embedder_options
+---
+{
+  "Rest": {
+    "api_key": "My super secret",
+    "distribution": null,
+    "dimensions": 4,
+    "url": "http://localhost:7777",
+    "query": null,
+    "input_field": [
+      "input"
+    ],
+    "path_to_embeddings": [
+      "data"
+    ],
+    "embedding_object": [
+      "embedding"
+    ],
+    "input_type": "text"
+  }
+}

index-scheduler/src/snapshots/index_scheduler__tests__settings_update.snap

@@ -0,0 +1,14 @@
+---
+source: index-scheduler/src/lib.rs
+expression: task.details
+---
+{
+  "embedders": {
+    "default": {
+      "source": "rest",
+      "apiKey": "MyXXXX...",
+      "dimensions": 4,
+      "url": "http://localhost:7777"
+    }
+  }
+}

index-scheduler/src/snapshots/lib.rs/test_settings_update/after_registering_settings_task.snap

@@ -0,0 +1,36 @@
+---
+source: index-scheduler/src/lib.rs
+---
+### Autobatching Enabled = true
+### Processing Tasks:
+[]
+----------------------------------------------------------------------
+### All Tasks:
+0 {uid: 0, status: enqueued, details: { settings: Settings { displayed_attributes: WildcardSetting(NotSet), searchable_attributes: WildcardSetting(NotSet), filterable_attributes: NotSet, sortable_attributes: NotSet, ranking_rules: NotSet, stop_words: NotSet, non_separator_tokens: NotSet, separator_tokens: NotSet, dictionary: NotSet, synonyms: NotSet, distinct_attribute: NotSet, proximity_precision: NotSet, typo_tolerance: NotSet, faceting: NotSet, pagination: NotSet, embedders: Set({"default": Set(EmbeddingSettings { source: Set(Rest), model: NotSet, revision: NotSet, api_key: Set("My super secret"), dimensions: Set(4), document_template: NotSet, url: Set("http://localhost:7777"), query: NotSet, input_field: NotSet, path_to_embeddings: NotSet, embedding_object: NotSet, input_type: NotSet, distribution: NotSet })}), search_cutoff_ms: NotSet, _kind: PhantomData<meilisearch_types::settings::Unchecked> } }, kind: SettingsUpdate { index_uid: "doggos", new_settings: Settings { displayed_attributes: WildcardSetting(NotSet), searchable_attributes: WildcardSetting(NotSet), filterable_attributes: NotSet, sortable_attributes: NotSet, ranking_rules: NotSet, stop_words: NotSet, non_separator_tokens: NotSet, separator_tokens: NotSet, dictionary: NotSet, synonyms: NotSet, distinct_attribute: NotSet, proximity_precision: NotSet, typo_tolerance: NotSet, faceting: NotSet, pagination: NotSet, embedders: Set({"default": Set(EmbeddingSettings { source: Set(Rest), model: NotSet, revision: NotSet, api_key: Set("My super secret"), dimensions: Set(4), document_template: NotSet, url: Set("http://localhost:7777"), query: NotSet, input_field: NotSet, path_to_embeddings: NotSet, embedding_object: NotSet, input_type: NotSet, distribution: NotSet })}), search_cutoff_ms: NotSet, _kind: PhantomData<meilisearch_types::settings::Unchecked> }, is_deletion: false, allow_index_creation: true }}
+----------------------------------------------------------------------
+### Status:
+enqueued [0,]
+----------------------------------------------------------------------
+### Kind:
+"settingsUpdate" [0,]
+----------------------------------------------------------------------
+### Index Tasks:
+doggos [0,]
+----------------------------------------------------------------------
+### Index Mapper:
+
+----------------------------------------------------------------------
+### Canceled By:
+
+----------------------------------------------------------------------
+### Enqueued At:
+[timestamp] [0,]
+----------------------------------------------------------------------
+### Started At:
+----------------------------------------------------------------------
+### Finished At:
+----------------------------------------------------------------------
+### File Store:
+
+----------------------------------------------------------------------
+

index-scheduler/src/snapshots/lib.rs/test_settings_update/settings_update_processed.snap

@@ -0,0 +1,40 @@
+---
+source: index-scheduler/src/lib.rs
+---
+### Autobatching Enabled = true
+### Processing Tasks:
+[]
+----------------------------------------------------------------------
+### All Tasks:
+0 {uid: 0, status: succeeded, details: { settings: Settings { displayed_attributes: WildcardSetting(NotSet), searchable_attributes: WildcardSetting(NotSet), filterable_attributes: NotSet, sortable_attributes: NotSet, ranking_rules: NotSet, stop_words: NotSet, non_separator_tokens: NotSet, separator_tokens: NotSet, dictionary: NotSet, synonyms: NotSet, distinct_attribute: NotSet, proximity_precision: NotSet, typo_tolerance: NotSet, faceting: NotSet, pagination: NotSet, embedders: Set({"default": Set(EmbeddingSettings { source: Set(Rest), model: NotSet, revision: NotSet, api_key: Set("My super secret"), dimensions: Set(4), document_template: NotSet, url: Set("http://localhost:7777"), query: NotSet, input_field: NotSet, path_to_embeddings: NotSet, embedding_object: NotSet, input_type: NotSet, distribution: NotSet })}), search_cutoff_ms: NotSet, _kind: PhantomData<meilisearch_types::settings::Unchecked> } }, kind: SettingsUpdate { index_uid: "doggos", new_settings: Settings { displayed_attributes: WildcardSetting(NotSet), searchable_attributes: WildcardSetting(NotSet), filterable_attributes: NotSet, sortable_attributes: NotSet, ranking_rules: NotSet, stop_words: NotSet, non_separator_tokens: NotSet, separator_tokens: NotSet, dictionary: NotSet, synonyms: NotSet, distinct_attribute: NotSet, proximity_precision: NotSet, typo_tolerance: NotSet, faceting: NotSet, pagination: NotSet, embedders: Set({"default": Set(EmbeddingSettings { source: Set(Rest), model: NotSet, revision: NotSet, api_key: Set("My super secret"), dimensions: Set(4), document_template: NotSet, url: Set("http://localhost:7777"), query: NotSet, input_field: NotSet, path_to_embeddings: NotSet, embedding_object: NotSet, input_type: NotSet, distribution: NotSet })}), search_cutoff_ms: NotSet, _kind: PhantomData<meilisearch_types::settings::Unchecked> }, is_deletion: false, allow_index_creation: true }}
+----------------------------------------------------------------------
+### Status:
+enqueued []
+succeeded [0,]
+----------------------------------------------------------------------
+### Kind:
+"settingsUpdate" [0,]
+----------------------------------------------------------------------
+### Index Tasks:
+doggos [0,]
+----------------------------------------------------------------------
+### Index Mapper:
+doggos: { number_of_documents: 0, field_distribution: {} }
+
+----------------------------------------------------------------------
+### Canceled By:
+
+----------------------------------------------------------------------
+### Enqueued At:
+[timestamp] [0,]
+----------------------------------------------------------------------
+### Started At:
+[timestamp] [0,]
+----------------------------------------------------------------------
+### Finished At:
+[timestamp] [0,]
+----------------------------------------------------------------------
+### File Store:
+
+----------------------------------------------------------------------
+

meilisearch-types/Cargo.toml

@@ -11,7 +11,7 @@ edition.workspace = true
 license.workspace = true
 
 [dependencies]
-actix-web = { version = "4.4.1", default-features = false }
+actix-web = { version = "4.5.1", default-features = false }
 anyhow = "1.0.79"
 convert_case = "0.6.0"
 csv = "1.3.0"
@@ -44,6 +44,7 @@ all-tokenizations = ["milli/all-tokenizations"]
 
 # chinese specialized tokenization
 chinese = ["milli/chinese"]
+chinese-pinyin = ["milli/chinese-pinyin"]
 # hebrew specialized tokenization
 hebrew = ["milli/hebrew"]
 # japanese specialized tokenization
@@ -56,3 +57,5 @@ greek = ["milli/greek"]
 khmer = ["milli/khmer"]
 # allow vietnamese specialized tokenization
 vietnamese = ["milli/vietnamese"]
+# force swedish character recomposition
+swedish-recomposition = ["milli/swedish-recomposition"]

meilisearch-types/src/error.rs

@@ -2,6 +2,7 @@ use std::{fmt, io};
 
 use actix_web::http::StatusCode;
 use actix_web::{self as aweb, HttpResponseBuilder};
+use aweb::http::header;
 use aweb::rt::task::JoinError;
 use convert_case::Casing;
 use milli::heed::{Error as HeedError, MdbError};
@@ -56,7 +57,14 @@ where
 impl aweb::error::ResponseError for ResponseError {
     fn error_response(&self) -> aweb::HttpResponse {
         let json = serde_json::to_vec(self).unwrap();
-        HttpResponseBuilder::new(self.status_code()).content_type("application/json").body(json)
+        let mut builder = HttpResponseBuilder::new(self.status_code());
+        builder.content_type("application/json");
+
+        if self.code == StatusCode::SERVICE_UNAVAILABLE {
+            builder.insert_header((header::RETRY_AFTER, "10"));
+        }
+
+        builder.body(json)
     }
 
     fn status_code(&self) -> StatusCode {
@@ -259,6 +267,7 @@ InvalidSettingsProximityPrecision     , InvalidRequest       , BAD_REQUEST ;
 InvalidSettingsFaceting               , InvalidRequest       , BAD_REQUEST ;
 InvalidSettingsFilterableAttributes   , InvalidRequest       , BAD_REQUEST ;
 InvalidSettingsPagination             , InvalidRequest       , BAD_REQUEST ;
+InvalidSettingsSearchCutoffMs           , InvalidRequest       , BAD_REQUEST ;
 InvalidSettingsEmbedders              , InvalidRequest       , BAD_REQUEST ;
 InvalidSettingsRankingRules           , InvalidRequest       , BAD_REQUEST ;
 InvalidSettingsSearchableAttributes   , InvalidRequest       , BAD_REQUEST ;
@@ -304,6 +313,7 @@ MissingSwapIndexes                    , InvalidRequest       , BAD_REQUEST ;
 MissingTaskFilters                    , InvalidRequest       , BAD_REQUEST ;
 NoSpaceLeftOnDevice                   , System               , UNPROCESSABLE_ENTITY;
 PayloadTooLarge                       , InvalidRequest       , PAYLOAD_TOO_LARGE ;
+TooManySearchRequests                 , System               , SERVICE_UNAVAILABLE ;
 TaskNotFound                          , InvalidRequest       , NOT_FOUND ;
 TooManyOpenFiles                      , System               , UNPROCESSABLE_ENTITY ;
 TooManyVectors                        , InvalidRequest       , BAD_REQUEST ;
@@ -352,6 +362,7 @@ impl ErrorCode for milli::Error {
                     | UserError::InvalidOpenAiModelDimensions { .. }
                     | UserError::InvalidOpenAiModelDimensionsMax { .. }
                     | UserError::InvalidSettingsDimensions { .. }
+                    | UserError::InvalidUrl { .. }
                     | UserError::InvalidPrompt(_) => Code::InvalidSettingsEmbedders,
                     UserError::TooManyEmbedders(_) => Code::InvalidSettingsEmbedders,
                     UserError::InvalidPromptForEmbeddings(..) => Code::InvalidSettingsEmbedders,

meilisearch-types/src/settings.rs

@@ -3,7 +3,7 @@ use std::convert::Infallible;
 use std::fmt;
 use std::marker::PhantomData;
 use std::num::NonZeroUsize;
-use std::ops::ControlFlow;
+use std::ops::{ControlFlow, Deref};
 use std::str::FromStr;
 
 use deserr::{DeserializeError, Deserr, ErrorKind, MergeWithError, ValuePointerRef};
@@ -143,21 +143,13 @@ impl MergeWithError<milli::CriterionError> for DeserrJsonError<InvalidSettingsRa
 )]
 #[deserr(error = DeserrJsonError, rename_all = camelCase, deny_unknown_fields)]
 pub struct Settings<T> {
-    #[serde(
-        default,
-        serialize_with = "serialize_with_wildcard",
-        skip_serializing_if = "Setting::is_not_set"
-    )]
+    #[serde(default, skip_serializing_if = "Setting::is_not_set")]
     #[deserr(default, error = DeserrJsonError<InvalidSettingsDisplayedAttributes>)]
-    pub displayed_attributes: Setting<Vec<String>>,
+    pub displayed_attributes: WildcardSetting,
 
-    #[serde(
-        default,
-        serialize_with = "serialize_with_wildcard",
-        skip_serializing_if = "Setting::is_not_set"
-    )]
+    #[serde(default, skip_serializing_if = "Setting::is_not_set")]
     #[deserr(default, error = DeserrJsonError<InvalidSettingsSearchableAttributes>)]
-    pub searchable_attributes: Setting<Vec<String>>,
+    pub searchable_attributes: WildcardSetting,
 
     #[serde(default, skip_serializing_if = "Setting::is_not_set")]
     #[deserr(default, error = DeserrJsonError<InvalidSettingsFilterableAttributes>)]
@@ -202,17 +194,57 @@ pub struct Settings<T> {
     #[serde(default, skip_serializing_if = "Setting::is_not_set")]
     #[deserr(default, error = DeserrJsonError<InvalidSettingsEmbedders>)]
     pub embedders: Setting<BTreeMap<String, Setting<milli::vector::settings::EmbeddingSettings>>>,
+    #[serde(default, skip_serializing_if = "Setting::is_not_set")]
+    #[deserr(default, error = DeserrJsonError<InvalidSettingsSearchCutoffMs>)]
+    pub search_cutoff_ms: Setting<u64>,
 
     #[serde(skip)]
     #[deserr(skip)]
     pub _kind: PhantomData<T>,
 }
 
+impl<T> Settings<T> {
+    pub fn hide_secrets(&mut self) {
+        let Setting::Set(embedders) = &mut self.embedders else {
+            return;
+        };
+
+        for mut embedder in embedders.values_mut() {
+            let Setting::Set(embedder) = &mut embedder else {
+                continue;
+            };
+
+            let Setting::Set(api_key) = &mut embedder.api_key else {
+                continue;
+            };
+
+            Self::hide_secret(api_key);
+        }
+    }
+
+    fn hide_secret(secret: &mut String) {
+        match secret.len() {
+            x if x < 10 => {
+                secret.replace_range(.., "XXX...");
+            }
+            x if x < 20 => {
+                secret.replace_range(2.., "XXXX...");
+            }
+            x if x < 30 => {
+                secret.replace_range(3.., "XXXXX...");
+            }
+            _x => {
+                secret.replace_range(5.., "XXXXXX...");
+            }
+        }
+    }
+}
+
 impl Settings<Checked> {
     pub fn cleared() -> Settings<Checked> {
         Settings {
-            displayed_attributes: Setting::Reset,
-            searchable_attributes: Setting::Reset,
+            displayed_attributes: Setting::Reset.into(),
+            searchable_attributes: Setting::Reset.into(),
             filterable_attributes: Setting::Reset,
             sortable_attributes: Setting::Reset,
             ranking_rules: Setting::Reset,
@@ -227,6 +259,7 @@ impl Settings<Checked> {
             faceting: Setting::Reset,
             pagination: Setting::Reset,
             embedders: Setting::Reset,
+            search_cutoff_ms: Setting::Reset,
             _kind: PhantomData,
         }
     }
@@ -249,6 +282,7 @@ impl Settings<Checked> {
             faceting,
             pagination,
             embedders,
+            search_cutoff_ms,
             ..
         } = self;
 
@@ -269,6 +303,7 @@ impl Settings<Checked> {
             faceting,
             pagination,
             embedders,
+            search_cutoff_ms,
             _kind: PhantomData,
         }
     }
@@ -276,7 +311,7 @@ impl Settings<Checked> {
 
 impl Settings<Unchecked> {
     pub fn check(self) -> Settings<Checked> {
-        let displayed_attributes = match self.displayed_attributes {
+        let displayed_attributes = match self.displayed_attributes.0 {
             Setting::Set(fields) => {
                 if fields.iter().any(|f| f == "*") {
                     Setting::Reset
@@ -287,7 +322,7 @@ impl Settings<Unchecked> {
             otherwise => otherwise,
         };
 
-        let searchable_attributes = match self.searchable_attributes {
+        let searchable_attributes = match self.searchable_attributes.0 {
             Setting::Set(fields) => {
                 if fields.iter().any(|f| f == "*") {
                     Setting::Reset
@@ -299,8 +334,8 @@ impl Settings<Unchecked> {
         };
 
         Settings {
-            displayed_attributes,
-            searchable_attributes,
+            displayed_attributes: displayed_attributes.into(),
+            searchable_attributes: searchable_attributes.into(),
             filterable_attributes: self.filterable_attributes,
             sortable_attributes: self.sortable_attributes,
             ranking_rules: self.ranking_rules,
@@ -315,6 +350,7 @@ impl Settings<Unchecked> {
             faceting: self.faceting,
             pagination: self.pagination,
             embedders: self.embedders,
+            search_cutoff_ms: self.search_cutoff_ms,
             _kind: PhantomData,
         }
     }
@@ -347,19 +383,40 @@ pub fn apply_settings_to_builder(
     settings: &Settings<Checked>,
     builder: &mut milli::update::Settings,
 ) {
-    match settings.searchable_attributes {
+    let Settings {
+        displayed_attributes,
+        searchable_attributes,
+        filterable_attributes,
+        sortable_attributes,
+        ranking_rules,
+        stop_words,
+        non_separator_tokens,
+        separator_tokens,
+        dictionary,
+        synonyms,
+        distinct_attribute,
+        proximity_precision,
+        typo_tolerance,
+        faceting,
+        pagination,
+        embedders,
+        search_cutoff_ms,
+        _kind,
+    } = settings;
+
+    match searchable_attributes.deref() {
         Setting::Set(ref names) => builder.set_searchable_fields(names.clone()),
         Setting::Reset => builder.reset_searchable_fields(),
         Setting::NotSet => (),
     }
 
-    match settings.displayed_attributes {
+    match displayed_attributes.deref() {
         Setting::Set(ref names) => builder.set_displayed_fields(names.clone()),
         Setting::Reset => builder.reset_displayed_fields(),
         Setting::NotSet => (),
     }
 
-    match settings.filterable_attributes {
+    match filterable_attributes {
         Setting::Set(ref facets) => {
             builder.set_filterable_fields(facets.clone().into_iter().collect())
         }
@@ -367,13 +424,13 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match settings.sortable_attributes {
+    match sortable_attributes {
         Setting::Set(ref fields) => builder.set_sortable_fields(fields.iter().cloned().collect()),
         Setting::Reset => builder.reset_sortable_fields(),
         Setting::NotSet => (),
     }
 
-    match settings.ranking_rules {
+    match ranking_rules {
         Setting::Set(ref criteria) => {
             builder.set_criteria(criteria.iter().map(|c| c.clone().into()).collect())
         }
@@ -381,13 +438,13 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match settings.stop_words {
+    match stop_words {
         Setting::Set(ref stop_words) => builder.set_stop_words(stop_words.clone()),
         Setting::Reset => builder.reset_stop_words(),
         Setting::NotSet => (),
     }
 
-    match settings.non_separator_tokens {
+    match non_separator_tokens {
         Setting::Set(ref non_separator_tokens) => {
             builder.set_non_separator_tokens(non_separator_tokens.clone())
         }
@@ -395,7 +452,7 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match settings.separator_tokens {
+    match separator_tokens {
         Setting::Set(ref separator_tokens) => {
             builder.set_separator_tokens(separator_tokens.clone())
         }
@@ -403,31 +460,31 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match settings.dictionary {
+    match dictionary {
         Setting::Set(ref dictionary) => builder.set_dictionary(dictionary.clone()),
         Setting::Reset => builder.reset_dictionary(),
         Setting::NotSet => (),
     }
 
-    match settings.synonyms {
+    match synonyms {
         Setting::Set(ref synonyms) => builder.set_synonyms(synonyms.clone().into_iter().collect()),
         Setting::Reset => builder.reset_synonyms(),
         Setting::NotSet => (),
     }
 
-    match settings.distinct_attribute {
+    match distinct_attribute {
         Setting::Set(ref attr) => builder.set_distinct_field(attr.clone()),
         Setting::Reset => builder.reset_distinct_field(),
         Setting::NotSet => (),
     }
 
-    match settings.proximity_precision {
+    match proximity_precision {
         Setting::Set(ref precision) => builder.set_proximity_precision((*precision).into()),
         Setting::Reset => builder.reset_proximity_precision(),
         Setting::NotSet => (),
     }
 
-    match settings.typo_tolerance {
+    match typo_tolerance {
         Setting::Set(ref value) => {
             match value.enabled {
                 Setting::Set(val) => builder.set_autorize_typos(val),
@@ -482,7 +539,7 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match &settings.faceting {
+    match faceting {
         Setting::Set(FacetingSettings { max_values_per_facet, sort_facet_values_by }) => {
             match max_values_per_facet {
                 Setting::Set(val) => builder.set_max_values_per_facet(*val),
@@ -504,7 +561,7 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match settings.pagination {
+    match pagination {
         Setting::Set(ref value) => match value.max_total_hits {
             Setting::Set(val) => builder.set_pagination_max_total_hits(val),
             Setting::Reset => builder.reset_pagination_max_total_hits(),
@@ -514,16 +571,28 @@ pub fn apply_settings_to_builder(
         Setting::NotSet => (),
     }
 
-    match settings.embedders.clone() {
-        Setting::Set(value) => builder.set_embedder_settings(value),
+    match embedders {
+        Setting::Set(value) => builder.set_embedder_settings(value.clone()),
         Setting::Reset => builder.reset_embedder_settings(),
         Setting::NotSet => (),
     }
+
+    match search_cutoff_ms {
+        Setting::Set(cutoff) => builder.set_search_cutoff(*cutoff),
+        Setting::Reset => builder.reset_search_cutoff(),
+        Setting::NotSet => (),
+    }
+}
+
+pub enum SecretPolicy {
+    RevealSecrets,
+    HideSecrets,
 }
 
 pub fn settings(
     index: &Index,
     rtxn: &crate::heed::RoTxn,
+    secret_policy: SecretPolicy,
 ) -> Result<Settings<Checked>, milli::Error> {
     let displayed_attributes =
         index.displayed_fields(rtxn)?.map(|fields| fields.into_iter().map(String::from).collect());
@@ -607,15 +676,19 @@ pub fn settings(
         .collect();
     let embedders = if embedders.is_empty() { Setting::NotSet } else { Setting::Set(embedders) };
 
-    Ok(Settings {
+    let search_cutoff_ms = index.search_cutoff(rtxn)?;
+
+    let mut settings = Settings {
         displayed_attributes: match displayed_attributes {
             Some(attrs) => Setting::Set(attrs),
             None => Setting::Reset,
-        },
+        }
+        .into(),
         searchable_attributes: match searchable_attributes {
             Some(attrs) => Setting::Set(attrs),
             None => Setting::Reset,
-        },
+        }
+        .into(),
         filterable_attributes: Setting::Set(filterable_attributes),
         sortable_attributes: Setting::Set(sortable_attributes),
         ranking_rules: Setting::Set(criteria.iter().map(|c| c.clone().into()).collect()),
@@ -633,8 +706,18 @@ pub fn settings(
         faceting: Setting::Set(faceting),
         pagination: Setting::Set(pagination),
         embedders,
+        search_cutoff_ms: match search_cutoff_ms {
+            Some(cutoff) => Setting::Set(cutoff),
+            None => Setting::Reset,
+        },
         _kind: PhantomData,
-    })
+    };
+
+    if let SecretPolicy::HideSecrets = secret_policy {
+        settings.hide_secrets()
+    }
+
+    Ok(settings)
 }
 
 #[derive(Debug, Clone, PartialEq, Eq, Deserr)]
@@ -759,6 +842,41 @@ impl From<ProximityPrecisionView> for ProximityPrecision {
     }
 }
 
+#[derive(Debug, Clone, Default, Deserialize, PartialEq, Eq)]
+pub struct WildcardSetting(Setting<Vec<String>>);
+
+impl From<Setting<Vec<String>>> for WildcardSetting {
+    fn from(setting: Setting<Vec<String>>) -> Self {
+        Self(setting)
+    }
+}
+
+impl Serialize for WildcardSetting {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        serialize_with_wildcard(&self.0, serializer)
+    }
+}
+
+impl<E: deserr::DeserializeError> Deserr<E> for WildcardSetting {
+    fn deserialize_from_value<V: deserr::IntoValue>(
+        value: deserr::Value<V>,
+        location: ValuePointerRef<'_>,
+    ) -> Result<Self, E> {
+        Ok(Self(Setting::deserialize_from_value(value, location)?))
+    }
+}
+
+impl std::ops::Deref for WildcardSetting {
+    type Target = Setting<Vec<String>>;
+
+    fn deref(&self) -> &Self::Target {
+        &self.0
+    }
+}
+
 #[cfg(test)]
 pub(crate) mod test {
     use super::*;
@@ -767,8 +885,8 @@ pub(crate) mod test {
     fn test_setting_check() {
         // test no changes
         let settings = Settings {
-            displayed_attributes: Setting::Set(vec![String::from("hello")]),
-            searchable_attributes: Setting::Set(vec![String::from("hello")]),
+            displayed_attributes: Setting::Set(vec![String::from("hello")]).into(),
+            searchable_attributes: Setting::Set(vec![String::from("hello")]).into(),
             filterable_attributes: Setting::NotSet,
             sortable_attributes: Setting::NotSet,
             ranking_rules: Setting::NotSet,
@@ -783,6 +901,7 @@ pub(crate) mod test {
             faceting: Setting::NotSet,
             pagination: Setting::NotSet,
             embedders: Setting::NotSet,
+            search_cutoff_ms: Setting::NotSet,
             _kind: PhantomData::<Unchecked>,
         };
 
@@ -793,8 +912,9 @@ pub(crate) mod test {
         // test wildcard
         // test no changes
         let settings = Settings {
-            displayed_attributes: Setting::Set(vec![String::from("*")]),
-            searchable_attributes: Setting::Set(vec![String::from("hello"), String::from("*")]),
+            displayed_attributes: Setting::Set(vec![String::from("*")]).into(),
+            searchable_attributes: Setting::Set(vec![String::from("hello"), String::from("*")])
+                .into(),
             filterable_attributes: Setting::NotSet,
             sortable_attributes: Setting::NotSet,
             ranking_rules: Setting::NotSet,
@@ -809,11 +929,12 @@ pub(crate) mod test {
             faceting: Setting::NotSet,
             pagination: Setting::NotSet,
             embedders: Setting::NotSet,
+            search_cutoff_ms: Setting::NotSet,
             _kind: PhantomData::<Unchecked>,
         };
 
         let checked = settings.check();
-        assert_eq!(checked.displayed_attributes, Setting::Reset);
-        assert_eq!(checked.searchable_attributes, Setting::Reset);
+        assert_eq!(checked.displayed_attributes, Setting::Reset.into());
+        assert_eq!(checked.searchable_attributes, Setting::Reset.into());
     }
 }

meilisearch-types/src/task_view.rs

@@ -86,7 +86,8 @@ impl From<Details> for DetailsView {
                     ..DetailsView::default()
                 }
             }
-            Details::SettingsUpdate { settings } => {
+            Details::SettingsUpdate { mut settings } => {
+                settings.hide_secrets();
                 DetailsView { settings: Some(settings), ..DetailsView::default() }
             }
             Details::IndexInfo { primary_key } => {

meilisearch/Cargo.toml

@@ -14,18 +14,18 @@ default-run = "meilisearch"
 
 [dependencies]
 actix-cors = "0.7.0"
-actix-http = { version = "3.5.1", default-features = false, features = [
+actix-http = { version = "3.6.0", default-features = false, features = [
     "compress-brotli",
     "compress-gzip",
-    "rustls",
+    "rustls-0_21",
 ] }
 actix-utils = "3.0.1"
-actix-web = { version = "4.4.1", default-features = false, features = [
+actix-web = { version = "4.5.1", default-features = false, features = [
     "macros",
     "compress-brotli",
     "compress-gzip",
     "cookies",
-    "rustls",
+    "rustls-0_21",
 ] }
 actix-web-static-files = { git = "https://github.com/kilork/actix-web-static-files.git", rev = "2d3b6160", optional = true }
 anyhow = { version = "1.0.79", features = ["backtrace"] }
@@ -52,7 +52,7 @@ index-scheduler = { path = "../index-scheduler" }
 indexmap = { version = "2.1.0", features = ["serde"] }
 is-terminal = "0.4.10"
 itertools = "0.11.0"
-jsonwebtoken = "8.3.0"
+jsonwebtoken = "9.2.0"
 lazy_static = "1.4.0"
 meilisearch-auth = { path = "../meilisearch-auth" }
 meilisearch-types = { path = "../meilisearch-types" }
@@ -75,7 +75,7 @@ reqwest = { version = "0.11.23", features = [
     "rustls-tls",
     "json",
 ], default-features = false }
-rustls = "0.20.8"
+rustls = "0.21.6"
 rustls-pemfile = "1.0.2"
 segment = { version = "0.2.3", optional = true }
 serde = { version = "1.0.195", features = ["derive"] }
@@ -149,12 +149,14 @@ mini-dashboard = [
     "zip",
 ]
 chinese = ["meilisearch-types/chinese"]
+chinese-pinyin = ["meilisearch-types/chinese-pinyin"]
 hebrew = ["meilisearch-types/hebrew"]
 japanese = ["meilisearch-types/japanese"]
 thai = ["meilisearch-types/thai"]
 greek = ["meilisearch-types/greek"]
 khmer = ["meilisearch-types/khmer"]
 vietnamese = ["meilisearch-types/vietnamese"]
+swedish-recomposition = ["meilisearch-types/swedish-recomposition"]
 
 [package.metadata.mini-dashboard]
 assets-url = "https://github.com/meilisearch/mini-dashboard/releases/download/v0.2.13/build.zip"

meilisearch/src/analytics/mock_analytics.rs

@@ -7,7 +7,6 @@ use serde_json::Value;
 
 use super::{find_user_id, Analytics, DocumentDeletionKind, DocumentFetchKind};
 use crate::routes::indexes::documents::UpdateDocumentsQuery;
-use crate::routes::tasks::TasksFilterQuery;
 use crate::Opt;
 
 pub struct MockAnalytics {
@@ -86,6 +85,4 @@ impl Analytics for MockAnalytics {
     }
     fn get_fetch_documents(&self, _documents_query: &DocumentFetchKind, _request: &HttpRequest) {}
     fn post_fetch_documents(&self, _documents_query: &DocumentFetchKind, _request: &HttpRequest) {}
-    fn get_tasks(&self, _query: &TasksFilterQuery, _request: &HttpRequest) {}
-    fn health_seen(&self, _request: &HttpRequest) {}
 }

meilisearch/src/analytics/mod.rs

@@ -14,7 +14,6 @@ use platform_dirs::AppDirs;
 use serde_json::Value;
 
 use crate::routes::indexes::documents::UpdateDocumentsQuery;
-use crate::routes::tasks::TasksFilterQuery;
 
 // if the analytics feature is disabled
 // the `SegmentAnalytics` point to the mock instead of the real analytics
@@ -117,10 +116,4 @@ pub trait Analytics: Sync + Send {
         index_creation: bool,
         request: &HttpRequest,
     );
-
-    // this method should be called to aggregate the get tasks requests.
-    fn get_tasks(&self, query: &TasksFilterQuery, request: &HttpRequest);
-
-    // this method should be called to aggregate a add documents request
-    fn health_seen(&self, request: &HttpRequest);
 }

meilisearch/src/analytics/segment_analytics.rs

@@ -33,7 +33,6 @@ use crate::option::{
 };
 use crate::routes::indexes::documents::UpdateDocumentsQuery;
 use crate::routes::indexes::facet_search::FacetSearchQuery;
-use crate::routes::tasks::TasksFilterQuery;
 use crate::routes::{create_all_stats, Stats};
 use crate::search::{
     FacetSearchResult, MatchingStrategy, SearchQuery, SearchQueryWithIndex, SearchResult,
@@ -81,8 +80,6 @@ pub enum AnalyticsMsg {
     AggregateUpdateDocuments(DocumentsAggregator),
     AggregateGetFetchDocuments(DocumentsFetchAggregator),
     AggregatePostFetchDocuments(DocumentsFetchAggregator),
-    AggregateTasks(TasksAggregator),
-    AggregateHealth(HealthAggregator),
 }
 
 pub struct SegmentAnalytics {
@@ -152,8 +149,6 @@ impl SegmentAnalytics {
             update_documents_aggregator: DocumentsAggregator::default(),
             get_fetch_documents_aggregator: DocumentsFetchAggregator::default(),
             post_fetch_documents_aggregator: DocumentsFetchAggregator::default(),
-            get_tasks_aggregator: TasksAggregator::default(),
-            health_aggregator: HealthAggregator::default(),
         });
         tokio::spawn(segment.run(index_scheduler.clone(), auth_controller.clone()));
 
@@ -231,16 +226,6 @@ impl super::Analytics for SegmentAnalytics {
         let aggregate = DocumentsFetchAggregator::from_query(documents_query, request);
         let _ = self.sender.try_send(AnalyticsMsg::AggregatePostFetchDocuments(aggregate));
     }
-
-    fn get_tasks(&self, query: &TasksFilterQuery, request: &HttpRequest) {
-        let aggregate = TasksAggregator::from_query(query, request);
-        let _ = self.sender.try_send(AnalyticsMsg::AggregateTasks(aggregate));
-    }
-
-    fn health_seen(&self, request: &HttpRequest) {
-        let aggregate = HealthAggregator::from_query(request);
-        let _ = self.sender.try_send(AnalyticsMsg::AggregateHealth(aggregate));
-    }
 }
 
 /// This structure represent the `infos` field we send in the analytics.
@@ -252,6 +237,7 @@ impl super::Analytics for SegmentAnalytics {
 struct Infos {
     env: String,
     experimental_enable_metrics: bool,
+    experimental_search_queue_size: usize,
     experimental_logs_mode: LogMode,
     experimental_replication_parameters: bool,
     experimental_enable_logs_route: bool,
@@ -293,6 +279,7 @@ impl From<Opt> for Infos {
         let Opt {
             db_path,
             experimental_enable_metrics,
+            experimental_search_queue_size,
             experimental_logs_mode,
             experimental_replication_parameters,
             experimental_enable_logs_route,
@@ -342,6 +329,7 @@ impl From<Opt> for Infos {
         Self {
             env,
             experimental_enable_metrics,
+            experimental_search_queue_size,
             experimental_logs_mode,
             experimental_replication_parameters,
             experimental_enable_logs_route,
@@ -391,8 +379,6 @@ pub struct Segment {
     update_documents_aggregator: DocumentsAggregator,
     get_fetch_documents_aggregator: DocumentsFetchAggregator,
     post_fetch_documents_aggregator: DocumentsFetchAggregator,
-    get_tasks_aggregator: TasksAggregator,
-    health_aggregator: HealthAggregator,
 }
 
 impl Segment {
@@ -455,8 +441,6 @@ impl Segment {
                         Some(AnalyticsMsg::AggregateUpdateDocuments(agreg)) => self.update_documents_aggregator.aggregate(agreg),
                         Some(AnalyticsMsg::AggregateGetFetchDocuments(agreg)) => self.get_fetch_documents_aggregator.aggregate(agreg),
                         Some(AnalyticsMsg::AggregatePostFetchDocuments(agreg)) => self.post_fetch_documents_aggregator.aggregate(agreg),
-                        Some(AnalyticsMsg::AggregateTasks(agreg)) => self.get_tasks_aggregator.aggregate(agreg),
-                        Some(AnalyticsMsg::AggregateHealth(agreg)) => self.health_aggregator.aggregate(agreg),
                         None => (),
                     }
                 }
@@ -510,8 +494,6 @@ impl Segment {
             update_documents_aggregator,
             get_fetch_documents_aggregator,
             post_fetch_documents_aggregator,
-            get_tasks_aggregator,
-            health_aggregator,
         } = self;
 
         if let Some(get_search) =
@@ -559,12 +541,6 @@ impl Segment {
         {
             let _ = self.batcher.push(post_fetch_documents).await;
         }
-        if let Some(get_tasks) = take(get_tasks_aggregator).into_event(user, "Tasks Seen") {
-            let _ = self.batcher.push(get_tasks).await;
-        }
-        if let Some(health) = take(health_aggregator).into_event(user, "Health Seen") {
-            let _ = self.batcher.push(health).await;
-        }
         let _ = self.batcher.flush().await;
     }
 }
@@ -579,6 +555,8 @@ pub struct SearchAggregator {
     // requests
     total_received: usize,
     total_succeeded: usize,
+    total_degraded: usize,
+    total_used_negative_operator: usize,
     time_spent: BinaryHeap<usize>,
 
     // sort
@@ -753,14 +731,22 @@ impl SearchAggregator {
         let SearchResult {
             hits: _,
             query: _,
-            vector: _,
             processing_time_ms,
             hits_info: _,
+            semantic_hit_count: _,
             facet_distribution: _,
             facet_stats: _,
+            degraded,
+            used_negative_operator,
         } = result;
 
         self.total_succeeded = self.total_succeeded.saturating_add(1);
+        if *degraded {
+            self.total_degraded = self.total_degraded.saturating_add(1);
+        }
+        if *used_negative_operator {
+            self.total_used_negative_operator = self.total_used_negative_operator.saturating_add(1);
+        }
         self.time_spent.push(*processing_time_ms as usize);
     }
 
@@ -802,6 +788,8 @@ impl SearchAggregator {
             semantic_ratio,
             embedder,
             hybrid,
+            total_degraded,
+            total_used_negative_operator,
         } = other;
 
         if self.timestamp.is_none() {
@@ -816,6 +804,9 @@ impl SearchAggregator {
         // request
         self.total_received = self.total_received.saturating_add(total_received);
         self.total_succeeded = self.total_succeeded.saturating_add(total_succeeded);
+        self.total_degraded = self.total_degraded.saturating_add(total_degraded);
+        self.total_used_negative_operator =
+            self.total_used_negative_operator.saturating_add(total_used_negative_operator);
         self.time_spent.append(time_spent);
 
         // sort
@@ -921,6 +912,8 @@ impl SearchAggregator {
             semantic_ratio,
             embedder,
             hybrid,
+            total_degraded,
+            total_used_negative_operator,
         } = self;
 
         if total_received == 0 {
@@ -940,6 +933,8 @@ impl SearchAggregator {
                     "total_succeeded": total_succeeded,
                     "total_failed": total_received.saturating_sub(total_succeeded), // just to be sure we never panics
                     "total_received": total_received,
+                    "total_degraded": total_degraded,
+                    "total_used_negative_operator": total_used_negative_operator,
                 },
                 "sort": {
                     "with_geoPoint": sort_with_geo_point,
@@ -1481,176 +1476,6 @@ impl DocumentsDeletionAggregator {
     }
 }
 
-#[derive(Default, Serialize)]
-pub struct TasksAggregator {
-    #[serde(skip)]
-    timestamp: Option<OffsetDateTime>,
-
-    // context
-    #[serde(rename = "user-agent")]
-    user_agents: HashSet<String>,
-
-    filtered_by_uid: bool,
-    filtered_by_index_uid: bool,
-    filtered_by_type: bool,
-    filtered_by_status: bool,
-    filtered_by_canceled_by: bool,
-    filtered_by_before_enqueued_at: bool,
-    filtered_by_after_enqueued_at: bool,
-    filtered_by_before_started_at: bool,
-    filtered_by_after_started_at: bool,
-    filtered_by_before_finished_at: bool,
-    filtered_by_after_finished_at: bool,
-    total_received: usize,
-}
-
-impl TasksAggregator {
-    pub fn from_query(query: &TasksFilterQuery, request: &HttpRequest) -> Self {
-        let TasksFilterQuery {
-            limit: _,
-            from: _,
-            uids,
-            index_uids,
-            types,
-            statuses,
-            canceled_by,
-            before_enqueued_at,
-            after_enqueued_at,
-            before_started_at,
-            after_started_at,
-            before_finished_at,
-            after_finished_at,
-        } = query;
-
-        Self {
-            timestamp: Some(OffsetDateTime::now_utc()),
-            user_agents: extract_user_agents(request).into_iter().collect(),
-            filtered_by_uid: uids.is_some(),
-            filtered_by_index_uid: index_uids.is_some(),
-            filtered_by_type: types.is_some(),
-            filtered_by_status: statuses.is_some(),
-            filtered_by_canceled_by: canceled_by.is_some(),
-            filtered_by_before_enqueued_at: before_enqueued_at.is_some(),
-            filtered_by_after_enqueued_at: after_enqueued_at.is_some(),
-            filtered_by_before_started_at: before_started_at.is_some(),
-            filtered_by_after_started_at: after_started_at.is_some(),
-            filtered_by_before_finished_at: before_finished_at.is_some(),
-            filtered_by_after_finished_at: after_finished_at.is_some(),
-            total_received: 1,
-        }
-    }
-
-    /// Aggregate one [TasksAggregator] into another.
-    pub fn aggregate(&mut self, other: Self) {
-        let Self {
-            timestamp,
-            user_agents,
-            total_received,
-            filtered_by_uid,
-            filtered_by_index_uid,
-            filtered_by_type,
-            filtered_by_status,
-            filtered_by_canceled_by,
-            filtered_by_before_enqueued_at,
-            filtered_by_after_enqueued_at,
-            filtered_by_before_started_at,
-            filtered_by_after_started_at,
-            filtered_by_before_finished_at,
-            filtered_by_after_finished_at,
-        } = other;
-
-        if self.timestamp.is_none() {
-            self.timestamp = timestamp;
-        }
-
-        // we can't create a union because there is no `into_union` method
-        for user_agent in user_agents {
-            self.user_agents.insert(user_agent);
-        }
-
-        self.filtered_by_uid |= filtered_by_uid;
-        self.filtered_by_index_uid |= filtered_by_index_uid;
-        self.filtered_by_type |= filtered_by_type;
-        self.filtered_by_status |= filtered_by_status;
-        self.filtered_by_canceled_by |= filtered_by_canceled_by;
-        self.filtered_by_before_enqueued_at |= filtered_by_before_enqueued_at;
-        self.filtered_by_after_enqueued_at |= filtered_by_after_enqueued_at;
-        self.filtered_by_before_started_at |= filtered_by_before_started_at;
-        self.filtered_by_after_started_at |= filtered_by_after_started_at;
-        self.filtered_by_before_finished_at |= filtered_by_before_finished_at;
-        self.filtered_by_after_finished_at |= filtered_by_after_finished_at;
-        self.filtered_by_after_finished_at |= filtered_by_after_finished_at;
-
-        self.total_received = self.total_received.saturating_add(total_received);
-    }
-
-    pub fn into_event(self, user: &User, event_name: &str) -> Option<Track> {
-        // if we had no timestamp it means we never encountered any events and
-        // thus we don't need to send this event.
-        let timestamp = self.timestamp?;
-
-        Some(Track {
-            timestamp: Some(timestamp),
-            user: user.clone(),
-            event: event_name.to_string(),
-            properties: serde_json::to_value(self).ok()?,
-            ..Default::default()
-        })
-    }
-}
-
-#[derive(Default, Serialize)]
-pub struct HealthAggregator {
-    #[serde(skip)]
-    timestamp: Option<OffsetDateTime>,
-
-    // context
-    #[serde(rename = "user-agent")]
-    user_agents: HashSet<String>,
-
-    #[serde(rename = "requests.total_received")]
-    total_received: usize,
-}
-
-impl HealthAggregator {
-    pub fn from_query(request: &HttpRequest) -> Self {
-        Self {
-            timestamp: Some(OffsetDateTime::now_utc()),
-            user_agents: extract_user_agents(request).into_iter().collect(),
-            total_received: 1,
-        }
-    }
-
-    /// Aggregate one [HealthAggregator] into another.
-    pub fn aggregate(&mut self, other: Self) {
-        let Self { timestamp, user_agents, total_received } = other;
-
-        if self.timestamp.is_none() {
-            self.timestamp = timestamp;
-        }
-
-        // we can't create a union because there is no `into_union` method
-        for user_agent in user_agents {
-            self.user_agents.insert(user_agent);
-        }
-        self.total_received = self.total_received.saturating_add(total_received);
-    }
-
-    pub fn into_event(self, user: &User, event_name: &str) -> Option<Track> {
-        // if we had no timestamp it means we never encountered any events and
-        // thus we don't need to send this event.
-        let timestamp = self.timestamp?;
-
-        Some(Track {
-            timestamp: Some(timestamp),
-            user: user.clone(),
-            event: event_name.to_string(),
-            properties: serde_json::to_value(self).ok()?,
-            ..Default::default()
-        })
-    }
-}
-
 #[derive(Default, Serialize)]
 pub struct DocumentsFetchAggregator {
     #[serde(skip)]

meilisearch/src/error.rs

@@ -29,6 +29,10 @@ pub enum MeilisearchHttpError {
     InvalidExpression(&'static [&'static str], Value),
     #[error("A {0} payload is missing.")]
     MissingPayload(PayloadType),
+    #[error("Too many search requests running at the same time: {0}. Retry after 10s.")]
+    TooManySearchRequests(usize),
+    #[error("Internal error: Search limiter is down.")]
+    SearchLimiterIsDown,
     #[error("The provided payload reached the size limit. The maximum accepted payload size is {}.",  Byte::from_bytes(*.0 as u64).get_appropriate_unit(true))]
     PayloadTooLarge(usize),
     #[error("Two indexes must be given for each swap. The list `[{}]` contains {} indexes.",
@@ -69,6 +73,8 @@ impl ErrorCode for MeilisearchHttpError {
             MeilisearchHttpError::EmptyFilter => Code::InvalidDocumentFilter,
             MeilisearchHttpError::InvalidExpression(_, _) => Code::InvalidSearchFilter,
             MeilisearchHttpError::PayloadTooLarge(_) => Code::PayloadTooLarge,
+            MeilisearchHttpError::TooManySearchRequests(_) => Code::TooManySearchRequests,
+            MeilisearchHttpError::SearchLimiterIsDown => Code::Internal,
             MeilisearchHttpError::SwapIndexPayloadWrongLength(_) => Code::InvalidSwapIndexes,
             MeilisearchHttpError::IndexUid(e) => e.error_code(),
             MeilisearchHttpError::SerdeJson(_) => Code::Internal,

meilisearch/src/lib.rs

@@ -9,12 +9,14 @@ pub mod middleware;
 pub mod option;
 pub mod routes;
 pub mod search;
+pub mod search_queue;
 
 use std::fs::File;
 use std::io::{BufReader, BufWriter};
+use std::num::NonZeroUsize;
 use std::path::Path;
 use std::sync::Arc;
-use std::thread;
+use std::thread::{self, available_parallelism};
 use std::time::Duration;
 
 use actix_cors::Cors;
@@ -38,6 +40,7 @@ use meilisearch_types::versioning::{check_version_file, create_version_file};
 use meilisearch_types::{compression, milli, VERSION_FILE_NAME};
 pub use option::Opt;
 use option::ScheduleSnapshot;
+use search_queue::SearchQueue;
 use tracing::{error, info_span};
 use tracing_subscriber::filter::Targets;
 
@@ -469,10 +472,15 @@ pub fn configure_data(
     (logs_route, logs_stderr): (LogRouteHandle, LogStderrHandle),
     analytics: Arc<dyn Analytics>,
 ) {
+    let search_queue = SearchQueue::new(
+        opt.experimental_search_queue_size,
+        available_parallelism().unwrap_or(NonZeroUsize::new(2).unwrap()),
+    );
     let http_payload_size_limit = opt.http_payload_size_limit.get_bytes() as usize;
     config
         .app_data(index_scheduler)
         .app_data(auth)
+        .app_data(web::Data::new(search_queue))
         .app_data(web::Data::from(analytics))
         .app_data(web::Data::new(logs_route))
         .app_data(web::Data::new(logs_stderr))

meilisearch/src/main.rs

@@ -151,7 +151,7 @@ async fn run_http(
     .keep_alive(KeepAlive::Os);
 
     if let Some(config) = opt_clone.get_ssl_config()? {
-        http_server.bind_rustls(opt_clone.http_addr, config)?.run().await?;
+        http_server.bind_rustls_021(opt_clone.http_addr, config)?.run().await?;
     } else {
         http_server.bind(&opt_clone.http_addr)?.run().await?;
     }

meilisearch/src/metrics.rs

@@ -4,24 +4,17 @@ use prometheus::{
     register_int_gauge_vec, HistogramVec, IntCounterVec, IntGauge, IntGaugeVec,
 };
 
-/// Create evenly distributed buckets
-fn create_buckets() -> [f64; 29] {
-    (0..10)
-        .chain((10..100).step_by(10))
-        .chain((100..=1000).step_by(100))
-        .map(|i| i as f64 / 1000.)
-        .collect::<Vec<_>>()
-        .try_into()
-        .unwrap()
-}
-
 lazy_static! {
-    pub static ref MEILISEARCH_HTTP_RESPONSE_TIME_CUSTOM_BUCKETS: [f64; 29] = create_buckets();
     pub static ref MEILISEARCH_HTTP_REQUESTS_TOTAL: IntCounterVec = register_int_counter_vec!(
         opts!("meilisearch_http_requests_total", "Meilisearch HTTP requests total"),
-        &["method", "path"]
+        &["method", "path", "status"]
     )
     .expect("Can't create a metric");
+    pub static ref MEILISEARCH_DEGRADED_SEARCH_REQUESTS: IntGauge = register_int_gauge!(opts!(
+        "meilisearch_degraded_search_requests",
+        "Meilisearch number of degraded search requests"
+    ))
+    .expect("Can't create a metric");
     pub static ref MEILISEARCH_DB_SIZE_BYTES: IntGauge =
         register_int_gauge!(opts!("meilisearch_db_size_bytes", "Meilisearch DB Size In Bytes"))
             .expect("Can't create a metric");
@@ -42,7 +35,7 @@ lazy_static! {
         "meilisearch_http_response_time_seconds",
         "Meilisearch HTTP response times",
         &["method", "path"],
-        MEILISEARCH_HTTP_RESPONSE_TIME_CUSTOM_BUCKETS.to_vec()
+        vec![0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0]
     )
     .expect("Can't create a metric");
     pub static ref MEILISEARCH_NB_TASKS: IntGaugeVec = register_int_gauge_vec!(

meilisearch/src/middleware.rs

@@ -65,9 +65,6 @@ where
                         .with_label_values(&[&request_method, request_path])
                         .start_timer(),
                 );
-                crate::metrics::MEILISEARCH_HTTP_REQUESTS_TOTAL
-                    .with_label_values(&[&request_method, request_path])
-                    .inc();
             }
         };
 
@@ -76,6 +73,14 @@ where
         Box::pin(async move {
             let res = fut.await?;
 
+            crate::metrics::MEILISEARCH_HTTP_REQUESTS_TOTAL
+                .with_label_values(&[
+                    res.request().method().as_str(),
+                    res.request().path(),
+                    res.status().as_str(),
+                ])
+                .inc();
+
             if let Some(histogram_timer) = histogram_timer {
                 histogram_timer.observe_duration();
             };

meilisearch/src/option.rs

@@ -13,6 +13,7 @@ use byte_unit::{Byte, ByteError};
 use clap::Parser;
 use meilisearch_types::features::InstanceTogglableFeatures;
 use meilisearch_types::milli::update::IndexerConfig;
+use meilisearch_types::milli::ThreadPoolNoAbortBuilder;
 use rustls::server::{
     AllowAnyAnonymousOrAuthenticatedClient, AllowAnyAuthenticatedClient, ServerSessionMemoryCache,
 };
@@ -54,6 +55,7 @@ const MEILI_EXPERIMENTAL_LOGS_MODE: &str = "MEILI_EXPERIMENTAL_LOGS_MODE";
 const MEILI_EXPERIMENTAL_REPLICATION_PARAMETERS: &str = "MEILI_EXPERIMENTAL_REPLICATION_PARAMETERS";
 const MEILI_EXPERIMENTAL_ENABLE_LOGS_ROUTE: &str = "MEILI_EXPERIMENTAL_ENABLE_LOGS_ROUTE";
 const MEILI_EXPERIMENTAL_ENABLE_METRICS: &str = "MEILI_EXPERIMENTAL_ENABLE_METRICS";
+const MEILI_EXPERIMENTAL_SEARCH_QUEUE_SIZE: &str = "MEILI_EXPERIMENTAL_SEARCH_QUEUE_SIZE";
 const MEILI_EXPERIMENTAL_REDUCE_INDEXING_MEMORY_USAGE: &str =
     "MEILI_EXPERIMENTAL_REDUCE_INDEXING_MEMORY_USAGE";
 const MEILI_EXPERIMENTAL_MAX_NUMBER_OF_BATCHED_TASKS: &str =
@@ -344,6 +346,15 @@ pub struct Opt {
     #[serde(default)]
     pub experimental_enable_metrics: bool,
 
+    /// Experimental search queue size. For more information, see: <https://github.com/orgs/meilisearch/discussions/729>
+    ///
+    /// Lets you customize the size of the search queue. Meilisearch processes your search requests as fast as possible but once the
+    /// queue is full it starts returning HTTP 503, Service Unavailable.
+    /// The default value is 1000.
+    #[clap(long, env = MEILI_EXPERIMENTAL_SEARCH_QUEUE_SIZE, default_value_t = 1000)]
+    #[serde(default)]
+    pub experimental_search_queue_size: usize,
+
     /// Experimental logs mode feature. For more information, see: <https://github.com/orgs/meilisearch/discussions/723>
     ///
     /// Change the mode of the logs on the console.
@@ -473,6 +484,7 @@ impl Opt {
             #[cfg(feature = "analytics")]
             no_analytics,
             experimental_enable_metrics,
+            experimental_search_queue_size,
             experimental_logs_mode,
             experimental_enable_logs_route,
             experimental_replication_parameters,
@@ -532,6 +544,10 @@ impl Opt {
             MEILI_EXPERIMENTAL_ENABLE_METRICS,
             experimental_enable_metrics.to_string(),
         );
+        export_to_env_if_not_present(
+            MEILI_EXPERIMENTAL_SEARCH_QUEUE_SIZE,
+            experimental_search_queue_size.to_string(),
+        );
         export_to_env_if_not_present(
             MEILI_EXPERIMENTAL_LOGS_MODE,
             experimental_logs_mode.to_string(),
@@ -564,11 +580,11 @@ impl Opt {
                     }
                     if self.ssl_require_auth {
                         let verifier = AllowAnyAuthenticatedClient::new(client_auth_roots);
-                        config.with_client_cert_verifier(verifier)
+                        config.with_client_cert_verifier(Arc::from(verifier))
                     } else {
                         let verifier =
                             AllowAnyAnonymousOrAuthenticatedClient::new(client_auth_roots);
-                        config.with_client_cert_verifier(verifier)
+                        config.with_client_cert_verifier(Arc::from(verifier))
                     }
                 }
                 None => config.with_no_client_auth(),
@@ -651,7 +667,7 @@ impl TryFrom<&IndexerOpts> for IndexerConfig {
     type Error = anyhow::Error;
 
     fn try_from(other: &IndexerOpts) -> Result<Self, Self::Error> {
-        let thread_pool = rayon::ThreadPoolBuilder::new()
+        let thread_pool = ThreadPoolNoAbortBuilder::new()
             .thread_name(|index| format!("indexing-thread:{index}"))
             .num_threads(*other.max_indexing_threads)
             .build()?;

meilisearch/src/routes/indexes/facet_search.rs

@@ -12,11 +12,13 @@ use tracing::debug;
 use crate::analytics::{Analytics, FacetSearchAggregator};
 use crate::extractors::authentication::policies::*;
 use crate::extractors::authentication::GuardedData;
+use crate::routes::indexes::search::search_kind;
 use crate::search::{
     add_search_rules, perform_facet_search, HybridQuery, MatchingStrategy, SearchQuery,
     DEFAULT_CROP_LENGTH, DEFAULT_CROP_MARKER, DEFAULT_HIGHLIGHT_POST_TAG,
     DEFAULT_HIGHLIGHT_PRE_TAG, DEFAULT_SEARCH_LIMIT, DEFAULT_SEARCH_OFFSET,
 };
+use crate::search_queue::SearchQueue;
 
 pub fn configure(cfg: &mut web::ServiceConfig) {
     cfg.service(web::resource("").route(web::post().to(search)));
@@ -48,6 +50,7 @@ pub struct FacetSearchQuery {
 
 pub async fn search(
     index_scheduler: GuardedData<ActionPolicy<{ actions::SEARCH }>, Data<IndexScheduler>>,
+    search_queue: Data<SearchQueue>,
     index_uid: web::Path<String>,
     params: AwebJson<FacetSearchQuery, DeserrJsonError>,
     req: HttpRequest,
@@ -71,8 +74,10 @@ pub async fn search(
 
     let index = index_scheduler.index(&index_uid)?;
     let features = index_scheduler.features();
+    let search_kind = search_kind(&search_query, &index_scheduler, &index, features)?;
+    let _permit = search_queue.try_get_search_permit().await?;
     let search_result = tokio::task::spawn_blocking(move || {
-        perform_facet_search(&index, search_query, facet_query, facet_name, features)
+        perform_facet_search(&index, search_query, facet_query, facet_name, search_kind)
     })
     .await?;
 

meilisearch/src/routes/indexes/mod.rs

@@ -269,12 +269,8 @@ impl From<index_scheduler::IndexStats> for IndexStats {
 pub async fn get_index_stats(
     index_scheduler: GuardedData<ActionPolicy<{ actions::STATS_GET }>, Data<IndexScheduler>>,
     index_uid: web::Path<String>,
-    req: HttpRequest,
-    analytics: web::Data<dyn Analytics>,
 ) -> Result<HttpResponse, ResponseError> {
     let index_uid = IndexUid::try_from(index_uid.into_inner())?;
-    analytics.publish("Stats Seen".to_string(), json!({ "per_index_uid": true }), Some(&req));
-
     let stats = IndexStats::from(index_scheduler.index_stats(&index_uid)?);
 
     debug!(returns = ?stats, "Get index stats");

meilisearch/src/routes/indexes/search.rs

@@ -1,27 +1,29 @@
 use actix_web::web::Data;
 use actix_web::{web, HttpRequest, HttpResponse};
 use deserr::actix_web::{AwebJson, AwebQueryParameter};
-use index_scheduler::IndexScheduler;
+use index_scheduler::{IndexScheduler, RoFeatures};
 use meilisearch_types::deserr::query_params::Param;
 use meilisearch_types::deserr::{DeserrJsonError, DeserrQueryParamError};
 use meilisearch_types::error::deserr_codes::*;
 use meilisearch_types::error::ResponseError;
 use meilisearch_types::index_uid::IndexUid;
 use meilisearch_types::milli;
-use meilisearch_types::milli::vector::DistributionShift;
 use meilisearch_types::serde_cs::vec::CS;
 use serde_json::Value;
-use tracing::{debug, warn};
+use tracing::debug;
 
 use crate::analytics::{Analytics, SearchAggregator};
+use crate::error::MeilisearchHttpError;
 use crate::extractors::authentication::policies::*;
 use crate::extractors::authentication::GuardedData;
 use crate::extractors::sequential_extractor::SeqHandler;
+use crate::metrics::MEILISEARCH_DEGRADED_SEARCH_REQUESTS;
 use crate::search::{
-    add_search_rules, perform_search, HybridQuery, MatchingStrategy, SearchQuery, SemanticRatio,
-    DEFAULT_CROP_LENGTH, DEFAULT_CROP_MARKER, DEFAULT_HIGHLIGHT_POST_TAG,
+    add_search_rules, perform_search, HybridQuery, MatchingStrategy, SearchKind, SearchQuery,
+    SemanticRatio, DEFAULT_CROP_LENGTH, DEFAULT_CROP_MARKER, DEFAULT_HIGHLIGHT_POST_TAG,
     DEFAULT_HIGHLIGHT_PRE_TAG, DEFAULT_SEARCH_LIMIT, DEFAULT_SEARCH_OFFSET, DEFAULT_SEMANTIC_RATIO,
 };
+use crate::search_queue::SearchQueue;
 
 pub fn configure(cfg: &mut web::ServiceConfig) {
     cfg.service(
@@ -181,6 +183,7 @@ fn fix_sort_query_parameters(sort_query: &str) -> Vec<String> {
 
 pub async fn search_with_url_query(
     index_scheduler: GuardedData<ActionPolicy<{ actions::SEARCH }>, Data<IndexScheduler>>,
+    search_queue: web::Data<SearchQueue>,
     index_uid: web::Path<String>,
     params: AwebQueryParameter<SearchQueryGet, DeserrQueryParamError>,
     req: HttpRequest,
@@ -201,11 +204,11 @@ pub async fn search_with_url_query(
     let index = index_scheduler.index(&index_uid)?;
     let features = index_scheduler.features();
 
-    let distribution = embed(&mut query, index_scheduler.get_ref(), &index).await?;
+    let search_kind = search_kind(&query, index_scheduler.get_ref(), &index, features)?;
 
+    let _permit = search_queue.try_get_search_permit().await?;
     let search_result =
-        tokio::task::spawn_blocking(move || perform_search(&index, query, features, distribution))
-            .await?;
+        tokio::task::spawn_blocking(move || perform_search(&index, query, search_kind)).await?;
     if let Ok(ref search_result) = search_result {
         aggregate.succeed(search_result);
     }
@@ -219,6 +222,7 @@ pub async fn search_with_url_query(
 
 pub async fn search_with_post(
     index_scheduler: GuardedData<ActionPolicy<{ actions::SEARCH }>, Data<IndexScheduler>>,
+    search_queue: web::Data<SearchQueue>,
     index_uid: web::Path<String>,
     params: AwebJson<SearchQuery, DeserrJsonError>,
     req: HttpRequest,
@@ -240,13 +244,16 @@ pub async fn search_with_post(
 
     let features = index_scheduler.features();
 
-    let distribution = embed(&mut query, index_scheduler.get_ref(), &index).await?;
+    let search_kind = search_kind(&query, index_scheduler.get_ref(), &index, features)?;
 
+    let _permit = search_queue.try_get_search_permit().await?;
     let search_result =
-        tokio::task::spawn_blocking(move || perform_search(&index, query, features, distribution))
-            .await?;
+        tokio::task::spawn_blocking(move || perform_search(&index, query, search_kind)).await?;
     if let Ok(ref search_result) = search_result {
         aggregate.succeed(search_result);
+        if search_result.degraded {
+            MEILISEARCH_DEGRADED_SEARCH_REQUESTS.inc();
+        }
     }
     analytics.post_search(aggregate);
 
@@ -256,77 +263,58 @@ pub async fn search_with_post(
     Ok(HttpResponse::Ok().json(search_result))
 }
 
-pub async fn embed(
-    query: &mut SearchQuery,
+pub fn search_kind(
+    query: &SearchQuery,
     index_scheduler: &IndexScheduler,
     index: &milli::Index,
-) -> Result<Option<DistributionShift>, ResponseError> {
-    match (&query.hybrid, &query.vector, &query.q) {
-        (Some(HybridQuery { semantic_ratio: _, embedder }), None, Some(q))
-            if !q.trim().is_empty() =>
-        {
-            let embedder_configs = index.embedding_configs(&index.read_txn()?)?;
-            let embedders = index_scheduler.embedders(embedder_configs)?;
+    features: RoFeatures,
+) -> Result<SearchKind, ResponseError> {
+    if query.vector.is_some() {
+        features.check_vector("Passing `vector` as a query parameter")?;
+    }
 
-            let embedder = if let Some(embedder_name) = embedder {
-                embedders.get(embedder_name)
-            } else {
-                embedders.get_default()
-            };
+    if query.hybrid.is_some() {
+        features.check_vector("Passing `hybrid` as a query parameter")?;
+    }
 
-            let embedder = embedder
-                .ok_or(milli::UserError::InvalidEmbedder("default".to_owned()))
-                .map_err(milli::Error::from)?
-                .0;
-
-            let distribution = embedder.distribution();
-
-            let embeddings = embedder
-                .embed(vec![q.to_owned()])
-                .await
-                .map_err(milli::vector::Error::from)
-                .map_err(milli::Error::from)?
-                .pop()
-                .expect("No vector returned from embedding");
-
-            if embeddings.iter().nth(1).is_some() {
-                warn!("Ignoring embeddings past the first one in long search query");
-                query.vector = Some(embeddings.iter().next().unwrap().to_vec());
-            } else {
-                query.vector = Some(embeddings.into_inner());
-            }
-            Ok(distribution)
+    // regardless of anything, always do a keyword search when we don't have a vector and the query is whitespace or missing
+    if query.vector.is_none() {
+        match &query.q {
+            Some(q) if q.trim().is_empty() => return Ok(SearchKind::KeywordOnly),
+            None => return Ok(SearchKind::KeywordOnly),
+            _ => {}
         }
-        (Some(hybrid), vector, _) => {
-            let embedder_configs = index.embedding_configs(&index.read_txn()?)?;
-            let embedders = index_scheduler.embedders(embedder_configs)?;
+    }
 
-            let embedder = if let Some(embedder_name) = &hybrid.embedder {
-                embedders.get(embedder_name)
-            } else {
-                embedders.get_default()
-            };
-
-            let embedder = embedder
-                .ok_or(milli::UserError::InvalidEmbedder("default".to_owned()))
-                .map_err(milli::Error::from)?
-                .0;
-
-            if let Some(vector) = vector {
-                if vector.len() != embedder.dimensions() {
-                    return Err(meilisearch_types::milli::Error::UserError(
-                        meilisearch_types::milli::UserError::InvalidVectorDimensions {
-                            expected: embedder.dimensions(),
-                            found: vector.len(),
-                        },
-                    )
-                    .into());
-                }
-            }
-
-            Ok(embedder.distribution())
+    match &query.hybrid {
+        Some(HybridQuery { semantic_ratio, embedder }) if **semantic_ratio == 1.0 => {
+            Ok(SearchKind::semantic(
+                index_scheduler,
+                index,
+                embedder.as_deref(),
+                query.vector.as_ref().map(Vec::len),
+            )?)
         }
-        _ => Ok(None),
+        Some(HybridQuery { semantic_ratio, embedder: _ }) if **semantic_ratio == 0.0 => {
+            Ok(SearchKind::KeywordOnly)
+        }
+        Some(HybridQuery { semantic_ratio, embedder }) => Ok(SearchKind::hybrid(
+            index_scheduler,
+            index,
+            embedder.as_deref(),
+            **semantic_ratio,
+            query.vector.as_ref().map(Vec::len),
+        )?),
+        None => match (query.q.as_deref(), query.vector.as_deref()) {
+            (_query, None) => Ok(SearchKind::KeywordOnly),
+            (None, Some(_vector)) => Ok(SearchKind::semantic(
+                index_scheduler,
+                index,
+                None,
+                query.vector.as_ref().map(Vec::len),
+            )?),
+            (Some(_), Some(_)) => Err(MeilisearchHttpError::MissingSearchHybrid.into()),
+        },
     }
 }
 

meilisearch/src/routes/indexes/settings.rs

@@ -7,7 +7,7 @@ use meilisearch_types::error::ResponseError;
 use meilisearch_types::facet_values_sort::FacetValuesSort;
 use meilisearch_types::index_uid::IndexUid;
 use meilisearch_types::milli::update::Setting;
-use meilisearch_types::settings::{settings, RankingRuleView, Settings, Unchecked};
+use meilisearch_types::settings::{settings, RankingRuleView, SecretPolicy, Settings, Unchecked};
 use meilisearch_types::tasks::KindWithContent;
 use serde_json::json;
 use tracing::debug;
@@ -134,13 +134,11 @@ macro_rules! make_setting_route {
 
                 let index = index_scheduler.index(&index_uid)?;
                 let rtxn = index.read_txn()?;
-                let settings = settings(&index, &rtxn)?;
+                let settings = settings(&index, &rtxn, meilisearch_types::settings::SecretPolicy::HideSecrets)?;
 
                 debug!(returns = ?settings, "Update settings");
-                let mut json = serde_json::json!(&settings);
-                let val = json[$camelcase_attr].take();
 
-                Ok(HttpResponse::Ok().json(val))
+                Ok(HttpResponse::Ok().json(settings.$attr))
             }
 
             pub fn resources() -> Resource {
@@ -604,6 +602,8 @@ fn embedder_analytics(
                 EmbedderSource::OpenAi => sources.insert("openAi"),
                 EmbedderSource::HuggingFace => sources.insert("huggingFace"),
                 EmbedderSource::UserProvided => sources.insert("userProvided"),
+                EmbedderSource::Ollama => sources.insert("ollama"),
+                EmbedderSource::Rest => sources.insert("rest"),
             };
         }
     };
@@ -623,6 +623,25 @@ fn embedder_analytics(
     )
 }
 
+make_setting_route!(
+    "/search-cutoff-ms",
+    put,
+    u64,
+    meilisearch_types::deserr::DeserrJsonError<
+        meilisearch_types::error::deserr_codes::InvalidSettingsSearchCutoffMs,
+    >,
+    search_cutoff_ms,
+    "searchCutoffMs",
+    analytics,
+    |setting: &Option<u64>, req: &HttpRequest| {
+        analytics.publish(
+            "Search Cutoff Updated".to_string(),
+            serde_json::json!({"search_cutoff_ms": setting }),
+            Some(req),
+        );
+    }
+);
+
 macro_rules! generate_configure {
     ($($mod:ident),*) => {
         pub fn configure(cfg: &mut web::ServiceConfig) {
@@ -653,7 +672,8 @@ generate_configure!(
     typo_tolerance,
     pagination,
     faceting,
-    embedders
+    embedders,
+    search_cutoff_ms
 );
 
 pub async fn update_all(
@@ -764,7 +784,8 @@ pub async fn update_all(
             "synonyms": {
                 "total": new_settings.synonyms.as_ref().set().map(|synonyms| synonyms.len()),
             },
-            "embedders": crate::routes::indexes::settings::embedder_analytics(new_settings.embedders.as_ref().set())
+            "embedders": crate::routes::indexes::settings::embedder_analytics(new_settings.embedders.as_ref().set()),
+            "search_cutoff_ms": new_settings.search_cutoff_ms.as_ref().set(),
         }),
         Some(&req),
     );
@@ -796,7 +817,7 @@ pub async fn get_all(
 
     let index = index_scheduler.index(&index_uid)?;
     let rtxn = index.read_txn()?;
-    let new_settings = settings(&index, &rtxn)?;
+    let new_settings = settings(&index, &rtxn, SecretPolicy::HideSecrets)?;
     debug!(returns = ?new_settings, "Get all settings");
     Ok(HttpResponse::Ok().json(new_settings))
 }

meilisearch/src/routes/mod.rs

@@ -8,13 +8,12 @@ use meilisearch_types::error::{Code, ResponseError};
 use meilisearch_types::settings::{Settings, Unchecked};
 use meilisearch_types::tasks::{Kind, Status, Task, TaskId};
 use serde::{Deserialize, Serialize};
-use serde_json::json;
 use time::OffsetDateTime;
 use tracing::debug;
 
-use crate::analytics::Analytics;
 use crate::extractors::authentication::policies::*;
 use crate::extractors::authentication::GuardedData;
+use crate::search_queue::SearchQueue;
 use crate::Opt;
 
 const PAGINATION_DEFAULT_LIMIT: usize = 20;
@@ -295,10 +294,7 @@ pub struct Stats {
 async fn get_stats(
     index_scheduler: GuardedData<ActionPolicy<{ actions::STATS_GET }>, Data<IndexScheduler>>,
     auth_controller: GuardedData<ActionPolicy<{ actions::STATS_GET }>, Data<AuthController>>,
-    req: HttpRequest,
-    analytics: web::Data<dyn Analytics>,
 ) -> Result<HttpResponse, ResponseError> {
-    analytics.publish("Stats Seen".to_string(), json!({ "per_index_uid": false }), Some(&req));
     let filters = index_scheduler.filters();
 
     let stats = create_all_stats((*index_scheduler).clone(), (*auth_controller).clone(), filters)?;
@@ -354,11 +350,7 @@ struct VersionResponse {
 
 async fn get_version(
     _index_scheduler: GuardedData<ActionPolicy<{ actions::VERSION }>, Data<IndexScheduler>>,
-    req: HttpRequest,
-    analytics: web::Data<dyn Analytics>,
 ) -> HttpResponse {
-    analytics.publish("Version Seen".to_string(), json!(null), Some(&req));
-
     let build_info = build_info::BuildInfo::from_build();
 
     HttpResponse::Ok().json(VersionResponse {
@@ -375,20 +367,12 @@ async fn get_version(
     })
 }
 
-#[derive(Serialize)]
-struct KeysResponse {
-    private: Option<String>,
-    public: Option<String>,
-}
-
 pub async fn get_health(
-    req: HttpRequest,
     index_scheduler: Data<IndexScheduler>,
     auth_controller: Data<AuthController>,
-    analytics: web::Data<dyn Analytics>,
+    search_queue: Data<SearchQueue>,
 ) -> Result<HttpResponse, ResponseError> {
-    analytics.health_seen(&req);
-
+    search_queue.health().unwrap();
     index_scheduler.health().unwrap();
     auth_controller.health().unwrap();
 

meilisearch/src/routes/multi_search.rs

@@ -13,10 +13,11 @@ use crate::analytics::{Analytics, MultiSearchAggregator};
 use crate::extractors::authentication::policies::ActionPolicy;
 use crate::extractors::authentication::{AuthenticationError, GuardedData};
 use crate::extractors::sequential_extractor::SeqHandler;
-use crate::routes::indexes::search::embed;
+use crate::routes::indexes::search::search_kind;
 use crate::search::{
     add_search_rules, perform_search, SearchQueryWithIndex, SearchResultWithIndex,
 };
+use crate::search_queue::SearchQueue;
 
 pub fn configure(cfg: &mut web::ServiceConfig) {
     cfg.service(web::resource("").route(web::post().to(SeqHandler(multi_search_with_post))));
@@ -35,6 +36,7 @@ pub struct SearchQueries {
 
 pub async fn multi_search_with_post(
     index_scheduler: GuardedData<ActionPolicy<{ actions::SEARCH }>, Data<IndexScheduler>>,
+    search_queue: Data<SearchQueue>,
     params: AwebJson<SearchQueries, DeserrJsonError>,
     req: HttpRequest,
     analytics: web::Data<dyn Analytics>,
@@ -44,6 +46,10 @@ pub async fn multi_search_with_post(
     let mut multi_aggregate = MultiSearchAggregator::from_queries(&queries, &req);
     let features = index_scheduler.features();
 
+    // Since we don't want to process half of the search requests and then get a permit refused
+    // we're going to get one permit for the whole duration of the multi-search request.
+    let _permit = search_queue.try_get_search_permit().await?;
+
     // Explicitly expect a `(ResponseError, usize)` for the error type rather than `ResponseError` only,
     // so that `?` doesn't work if it doesn't use `with_index`, ensuring that it is not forgotten in case of code
     // changes.
@@ -75,15 +81,13 @@ pub async fn multi_search_with_post(
                 })
                 .with_index(query_index)?;
 
-            let distribution = embed(&mut query, index_scheduler.get_ref(), &index)
-                .await
+            let search_kind = search_kind(&query, index_scheduler.get_ref(), &index, features)
                 .with_index(query_index)?;
 
-            let search_result = tokio::task::spawn_blocking(move || {
-                perform_search(&index, query, features, distribution)
-            })
-            .await
-            .with_index(query_index)?;
+            let search_result =
+                tokio::task::spawn_blocking(move || perform_search(&index, query, search_kind))
+                    .await
+                    .with_index(query_index)?;
 
             search_results.push(SearchResultWithIndex {
                 index_uid: index_uid.into_inner(),

meilisearch/src/routes/tasks.rs

@@ -270,12 +270,8 @@ pub struct AllTasks {
 async fn get_tasks(
     index_scheduler: GuardedData<ActionPolicy<{ actions::TASKS_GET }>, Data<IndexScheduler>>,
     params: AwebQueryParameter<TasksFilterQuery, DeserrQueryParamError>,
-    req: HttpRequest,
-    analytics: web::Data<dyn Analytics>,
 ) -> Result<HttpResponse, ResponseError> {
     let mut params = params.into_inner();
-    analytics.get_tasks(&params, &req);
-
     // We +1 just to know if there is more after this "page" or not.
     params.limit.0 = params.limit.0.saturating_add(1);
     let limit = params.limit.0;
@@ -298,8 +294,6 @@ async fn get_tasks(
 async fn get_task(
     index_scheduler: GuardedData<ActionPolicy<{ actions::TASKS_GET }>, Data<IndexScheduler>>,
     task_uid: web::Path<String>,
-    req: HttpRequest,
-    analytics: web::Data<dyn Analytics>,
 ) -> Result<HttpResponse, ResponseError> {
     let task_uid_string = task_uid.into_inner();
 
@@ -310,8 +304,6 @@ async fn get_task(
         }
     };
 
-    analytics.publish("Tasks Seen".to_string(), json!({ "per_task_uid": true }), Some(&req));
-
     let query = index_scheduler::Query { uids: Some(vec![task_uid]), ..Query::default() };
     let filters = index_scheduler.filters();
     let (tasks, _) = index_scheduler.get_tasks_from_authorized_indexes(query, filters)?;

meilisearch/src/search.rs

@@ -1,20 +1,22 @@
+use core::fmt;
 use std::cmp::min;
 use std::collections::{BTreeMap, BTreeSet, HashSet};
 use std::str::FromStr;
-use std::time::Instant;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
 
 use deserr::Deserr;
 use either::Either;
-use index_scheduler::RoFeatures;
 use indexmap::IndexMap;
 use meilisearch_auth::IndexSearchRules;
 use meilisearch_types::deserr::DeserrJsonError;
 use meilisearch_types::error::deserr_codes::*;
+use meilisearch_types::error::ResponseError;
 use meilisearch_types::heed::RoTxn;
 use meilisearch_types::index_uid::IndexUid;
-use meilisearch_types::milli::score_details::{self, ScoreDetails, ScoringStrategy};
-use meilisearch_types::milli::vector::DistributionShift;
-use meilisearch_types::milli::{FacetValueHit, OrderBy, SearchForFacetValues};
+use meilisearch_types::milli::score_details::{ScoreDetails, ScoringStrategy};
+use meilisearch_types::milli::vector::Embedder;
+use meilisearch_types::milli::{FacetValueHit, OrderBy, SearchForFacetValues, TimeBudget};
 use meilisearch_types::settings::DEFAULT_PAGINATION_MAX_TOTAL_HITS;
 use meilisearch_types::{milli, Document};
 use milli::tokenizer::TokenizerBuilder;
@@ -38,7 +40,7 @@ pub const DEFAULT_HIGHLIGHT_PRE_TAG: fn() -> String = || "<em>".to_string();
 pub const DEFAULT_HIGHLIGHT_POST_TAG: fn() -> String = || "</em>".to_string();
 pub const DEFAULT_SEMANTIC_RATIO: fn() -> SemanticRatio = || SemanticRatio(0.5);
 
-#[derive(Debug, Clone, Default, PartialEq, Deserr)]
+#[derive(Clone, Default, PartialEq, Deserr)]
 #[deserr(error = DeserrJsonError, rename_all = camelCase, deny_unknown_fields)]
 pub struct SearchQuery {
     #[deserr(default, error = DeserrJsonError<InvalidSearchQ>)]
@@ -87,16 +89,182 @@ pub struct SearchQuery {
     pub attributes_to_search_on: Option<Vec<String>>,
 }
 
+// Since this structure is logged A LOT we're going to reduce the number of things it logs to the bare minimum.
+// - Only what IS used, we know everything else is set to None so there is no need to print it
+// - Re-order the most important field to debug first
+impl fmt::Debug for SearchQuery {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        let Self {
+            q,
+            vector,
+            hybrid,
+            offset,
+            limit,
+            page,
+            hits_per_page,
+            attributes_to_retrieve,
+            attributes_to_crop,
+            crop_length,
+            attributes_to_highlight,
+            show_matches_position,
+            show_ranking_score,
+            show_ranking_score_details,
+            filter,
+            sort,
+            facets,
+            highlight_pre_tag,
+            highlight_post_tag,
+            crop_marker,
+            matching_strategy,
+            attributes_to_search_on,
+        } = self;
+
+        let mut debug = f.debug_struct("SearchQuery");
+
+        // First, everything related to the number of documents to retrieve
+        debug.field("limit", &limit).field("offset", &offset);
+        if let Some(page) = page {
+            debug.field("page", &page);
+        }
+        if let Some(hits_per_page) = hits_per_page {
+            debug.field("hits_per_page", &hits_per_page);
+        }
+
+        // Then, everything related to the queries
+        if let Some(q) = q {
+            debug.field("q", &q);
+        }
+        if let Some(v) = vector {
+            if v.len() < 10 {
+                debug.field("vector", &v);
+            } else {
+                debug.field(
+                    "vector",
+                    &format!("[{}, {}, {}, ... {} dimensions]", v[0], v[1], v[2], v.len()),
+                );
+            }
+        }
+        if let Some(hybrid) = hybrid {
+            debug.field("hybrid", &hybrid);
+        }
+        if let Some(attributes_to_search_on) = attributes_to_search_on {
+            debug.field("attributes_to_search_on", &attributes_to_search_on);
+        }
+        if let Some(filter) = filter {
+            debug.field("filter", &filter);
+        }
+        if let Some(sort) = sort {
+            debug.field("sort", &sort);
+        }
+        if let Some(facets) = facets {
+            debug.field("facets", &facets);
+        }
+        debug.field("matching_strategy", &matching_strategy);
+
+        // Then everything related to the formatting
+        debug.field("crop_length", &crop_length);
+        if *show_matches_position {
+            debug.field("show_matches_position", show_matches_position);
+        }
+        if *show_ranking_score {
+            debug.field("show_ranking_score", show_ranking_score);
+        }
+        if *show_ranking_score_details {
+            debug.field("self.show_ranking_score_details", show_ranking_score_details);
+        }
+        debug.field("crop_length", &crop_length);
+        if let Some(facets) = facets {
+            debug.field("facets", &facets);
+        }
+        if let Some(attributes_to_retrieve) = attributes_to_retrieve {
+            debug.field("attributes_to_retrieve", &attributes_to_retrieve);
+        }
+        if let Some(attributes_to_crop) = attributes_to_crop {
+            debug.field("attributes_to_crop", &attributes_to_crop);
+        }
+        if let Some(attributes_to_highlight) = attributes_to_highlight {
+            debug.field("attributes_to_highlight", &attributes_to_highlight);
+        }
+        debug.field("highlight_pre_tag", &highlight_pre_tag);
+        debug.field("highlight_post_tag", &highlight_post_tag);
+        debug.field("crop_marker", &crop_marker);
+
+        debug.finish()
+    }
+}
+
 #[derive(Debug, Clone, Default, PartialEq, Deserr)]
 #[deserr(error = DeserrJsonError<InvalidHybridQuery>, rename_all = camelCase, deny_unknown_fields)]
 pub struct HybridQuery {
-    /// TODO validate that sementic ratio is between 0.0 and 1,0
     #[deserr(default, error = DeserrJsonError<InvalidSearchSemanticRatio>, default)]
     pub semantic_ratio: SemanticRatio,
     #[deserr(default, error = DeserrJsonError<InvalidEmbedder>, default)]
     pub embedder: Option<String>,
 }
 
+pub enum SearchKind {
+    KeywordOnly,
+    SemanticOnly { embedder_name: String, embedder: Arc<Embedder> },
+    Hybrid { embedder_name: String, embedder: Arc<Embedder>, semantic_ratio: f32 },
+}
+impl SearchKind {
+    pub(crate) fn semantic(
+        index_scheduler: &index_scheduler::IndexScheduler,
+        index: &Index,
+        embedder_name: Option<&str>,
+        vector_len: Option<usize>,
+    ) -> Result<Self, ResponseError> {
+        let (embedder_name, embedder) =
+            Self::embedder(index_scheduler, index, embedder_name, vector_len)?;
+        Ok(Self::SemanticOnly { embedder_name, embedder })
+    }
+
+    pub(crate) fn hybrid(
+        index_scheduler: &index_scheduler::IndexScheduler,
+        index: &Index,
+        embedder_name: Option<&str>,
+        semantic_ratio: f32,
+        vector_len: Option<usize>,
+    ) -> Result<Self, ResponseError> {
+        let (embedder_name, embedder) =
+            Self::embedder(index_scheduler, index, embedder_name, vector_len)?;
+        Ok(Self::Hybrid { embedder_name, embedder, semantic_ratio })
+    }
+
+    fn embedder(
+        index_scheduler: &index_scheduler::IndexScheduler,
+        index: &Index,
+        embedder_name: Option<&str>,
+        vector_len: Option<usize>,
+    ) -> Result<(String, Arc<Embedder>), ResponseError> {
+        let embedder_configs = index.embedding_configs(&index.read_txn()?)?;
+        let embedders = index_scheduler.embedders(embedder_configs)?;
+
+        let embedder_name = embedder_name.unwrap_or_else(|| embedders.get_default_embedder_name());
+
+        let embedder = embedders.get(embedder_name);
+
+        let embedder = embedder
+            .ok_or(milli::UserError::InvalidEmbedder(embedder_name.to_owned()))
+            .map_err(milli::Error::from)?
+            .0;
+
+        if let Some(vector_len) = vector_len {
+            if vector_len != embedder.dimensions() {
+                return Err(meilisearch_types::milli::Error::UserError(
+                    meilisearch_types::milli::UserError::InvalidVectorDimensions {
+                        expected: embedder.dimensions(),
+                        found: vector_len,
+                    },
+                )
+                .into());
+            }
+        }
+
+        Ok((embedder_name.to_owned(), embedder))
+    }
+}
+
 #[derive(Debug, Clone, Copy, PartialEq, Deserr)]
 #[deserr(try_from(f32) = TryFrom::try_from -> InvalidSearchSemanticRatio)]
 pub struct SemanticRatio(f32);
@@ -305,17 +473,13 @@ pub struct SearchHit {
     pub ranking_score: Option<f64>,
     #[serde(rename = "_rankingScoreDetails", skip_serializing_if = "Option::is_none")]
     pub ranking_score_details: Option<serde_json::Map<String, serde_json::Value>>,
-    #[serde(rename = "_semanticScore", skip_serializing_if = "Option::is_none")]
-    pub semantic_score: Option<f32>,
 }
 
-#[derive(Serialize, Debug, Clone, PartialEq)]
+#[derive(Serialize, Clone, PartialEq)]
 #[serde(rename_all = "camelCase")]
 pub struct SearchResult {
     pub hits: Vec<SearchHit>,
     pub query: String,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub vector: Option<Vec<f32>>,
     pub processing_time_ms: u128,
     #[serde(flatten)]
     pub hits_info: HitsInfo,
@@ -323,6 +487,55 @@ pub struct SearchResult {
     pub facet_distribution: Option<BTreeMap<String, IndexMap<String, u64>>>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub facet_stats: Option<BTreeMap<String, FacetStats>>,
+
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub semantic_hit_count: Option<u32>,
+
+    // These fields are only used for analytics purposes
+    #[serde(skip)]
+    pub degraded: bool,
+    #[serde(skip)]
+    pub used_negative_operator: bool,
+}
+
+impl fmt::Debug for SearchResult {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        let SearchResult {
+            hits,
+            query,
+            processing_time_ms,
+            hits_info,
+            facet_distribution,
+            facet_stats,
+            semantic_hit_count,
+            degraded,
+            used_negative_operator,
+        } = self;
+
+        let mut debug = f.debug_struct("SearchResult");
+        // The most important thing when looking at a search result is the time it took to process
+        debug.field("processing_time_ms", &processing_time_ms);
+        debug.field("hits", &format!("[{} hits returned]", hits.len()));
+        debug.field("query", &query);
+        debug.field("hits_info", &hits_info);
+        if *used_negative_operator {
+            debug.field("used_negative_operator", used_negative_operator);
+        }
+        if *degraded {
+            debug.field("degraded", degraded);
+        }
+        if let Some(facet_distribution) = facet_distribution {
+            debug.field("facet_distribution", &facet_distribution);
+        }
+        if let Some(facet_stats) = facet_stats {
+            debug.field("facet_stats", &facet_stats);
+        }
+        if let Some(semantic_hit_count) = semantic_hit_count {
+            debug.field("semantic_hit_count", &semantic_hit_count);
+        }
+
+        debug.finish()
+    }
 }
 
 #[derive(Serialize, Debug, Clone, PartialEq)]
@@ -380,45 +593,36 @@ fn prepare_search<'t>(
     index: &'t Index,
     rtxn: &'t RoTxn,
     query: &'t SearchQuery,
-    features: RoFeatures,
-    distribution: Option<DistributionShift>,
+    search_kind: &SearchKind,
+    time_budget: TimeBudget,
 ) -> Result<(milli::Search<'t>, bool, usize, usize), MeilisearchHttpError> {
     let mut search = index.search(rtxn);
+    search.time_budget(time_budget);
 
-    if query.vector.is_some() {
-        features.check_vector("Passing `vector` as a query parameter")?;
-    }
-
-    if query.hybrid.is_some() {
-        features.check_vector("Passing `hybrid` as a query parameter")?;
-    }
-
-    if query.hybrid.is_none() && query.q.is_some() && query.vector.is_some() {
-        return Err(MeilisearchHttpError::MissingSearchHybrid);
-    }
-
-    search.distribution_shift(distribution);
-
-    if let Some(ref vector) = query.vector {
-        match &query.hybrid {
-            // If semantic ratio is 0.0, only the query search will impact the search results,
-            // skip the vector
-            Some(hybrid) if *hybrid.semantic_ratio == 0.0 => (),
-            _otherwise => {
-                search.vector(vector.clone());
-            }
-        }
-    }
-
-    if let Some(ref q) = query.q {
-        match &query.hybrid {
-            // If semantic ratio is 1.0, only the vector search will impact the search results,
-            // skip the query
-            Some(hybrid) if *hybrid.semantic_ratio == 1.0 => (),
-            _otherwise => {
+    match search_kind {
+        SearchKind::KeywordOnly => {
+            if let Some(q) = &query.q {
                 search.query(q);
             }
         }
+        SearchKind::SemanticOnly { embedder_name, embedder } => {
+            let vector = match query.vector.clone() {
+                Some(vector) => vector,
+                None => embedder
+                    .embed_one(query.q.clone().unwrap())
+                    .map_err(milli::vector::Error::from)
+                    .map_err(milli::Error::from)?,
+            };
+
+            search.semantic(embedder_name.clone(), embedder.clone(), Some(vector));
+        }
+        SearchKind::Hybrid { embedder_name, embedder, semantic_ratio: _ } => {
+            if let Some(q) = &query.q {
+                search.query(q);
+            }
+            // will be embedded in hybrid search if necessary
+            search.semantic(embedder_name.clone(), embedder.clone(), query.vector.clone());
+        }
     }
 
     if let Some(ref searchable) = query.attributes_to_search_on {
@@ -441,10 +645,6 @@ fn prepare_search<'t>(
         ScoringStrategy::Skip
     });
 
-    if let Some(HybridQuery { embedder: Some(embedder), .. }) = &query.hybrid {
-        search.embedder_name(embedder);
-    }
-
     // compute the offset on the limit depending on the pagination mode.
     let (offset, limit) = if is_finite_pagination {
         let limit = query.hits_per_page.unwrap_or_else(DEFAULT_SEARCH_LIMIT);
@@ -487,23 +687,37 @@ fn prepare_search<'t>(
 pub fn perform_search(
     index: &Index,
     query: SearchQuery,
-    features: RoFeatures,
-    distribution: Option<DistributionShift>,
+    search_kind: SearchKind,
 ) -> Result<SearchResult, MeilisearchHttpError> {
     let before_search = Instant::now();
     let rtxn = index.read_txn()?;
+    let time_budget = match index.search_cutoff(&rtxn)? {
+        Some(cutoff) => TimeBudget::new(Duration::from_millis(cutoff)),
+        None => TimeBudget::default(),
+    };
 
     let (search, is_finite_pagination, max_total_hits, offset) =
-        prepare_search(index, &rtxn, &query, features, distribution)?;
+        prepare_search(index, &rtxn, &query, &search_kind, time_budget)?;
 
-    let milli::SearchResult { documents_ids, matching_words, candidates, document_scores, .. } =
-        match &query.hybrid {
-            Some(hybrid) => match *hybrid.semantic_ratio {
-                ratio if ratio == 0.0 || ratio == 1.0 => search.execute()?,
-                ratio => search.execute_hybrid(ratio)?,
-            },
-            None => search.execute()?,
-        };
+    let (
+        milli::SearchResult {
+            documents_ids,
+            matching_words,
+            candidates,
+            document_scores,
+            degraded,
+            used_negative_operator,
+        },
+        semantic_hit_count,
+    ) = match &search_kind {
+        SearchKind::KeywordOnly => (search.execute()?, None),
+        SearchKind::SemanticOnly { .. } => {
+            let results = search.execute()?;
+            let semantic_hit_count = results.document_scores.len() as u32;
+            (results, Some(semantic_hit_count))
+        }
+        SearchKind::Hybrid { semantic_ratio, .. } => search.execute_hybrid(*semantic_ratio)?,
+    };
 
     let fields_ids_map = index.fields_ids_map(&rtxn).unwrap();
 
@@ -530,7 +744,7 @@ pub fn perform_search(
     // The attributes to retrieve are the ones explicitly marked as to retrieve (all by default),
     // but these attributes must be also be present
     // - in the fields_ids_map
-    // - in the the displayed attributes
+    // - in the displayed attributes
     let to_retrieve_ids: BTreeSet<_> = query
         .attributes_to_retrieve
         .as_ref()
@@ -612,18 +826,6 @@ pub fn perform_search(
             insert_geo_distance(sort, &mut document);
         }
 
-        let mut semantic_score = None;
-        for details in &score {
-            if let ScoreDetails::Vector(score_details::Vector {
-                target_vector: _,
-                value_similarity: Some((_matching_vector, similarity)),
-            }) = details
-            {
-                semantic_score = Some(*similarity);
-                break;
-            }
-        }
-
         let ranking_score =
             query.show_ranking_score.then(|| ScoreDetails::global_score(score.iter()));
         let ranking_score_details =
@@ -635,7 +837,6 @@ pub fn perform_search(
             matches_position,
             ranking_score_details,
             ranking_score,
-            semantic_score,
         };
         documents.push(hit);
     }
@@ -671,27 +872,16 @@ pub fn perform_search(
 
             let sort_facet_values_by =
                 index.sort_facet_values_by(&rtxn).map_err(milli::Error::from)?;
-            let default_sort_facet_values_by =
-                sort_facet_values_by.get("*").copied().unwrap_or_default();
 
             if fields.iter().all(|f| f != "*") {
-                let fields: Vec<_> = fields
-                    .iter()
-                    .map(|n| {
-                        (
-                            n,
-                            sort_facet_values_by
-                                .get(n)
-                                .copied()
-                                .unwrap_or(default_sort_facet_values_by),
-                        )
-                    })
-                    .collect();
+                let fields: Vec<_> =
+                    fields.iter().map(|n| (n, sort_facet_values_by.get(n))).collect();
                 facet_distribution.facets(fields);
             }
+
             let distribution = facet_distribution
                 .candidates(candidates)
-                .default_order_by(default_sort_facet_values_by)
+                .default_order_by(sort_facet_values_by.get("*"))
                 .execute()?;
             let stats = facet_distribution.compute_stats()?;
             (Some(distribution), Some(stats))
@@ -707,10 +897,12 @@ pub fn perform_search(
         hits: documents,
         hits_info,
         query: query.q.unwrap_or_default(),
-        vector: query.vector,
         processing_time_ms: before_search.elapsed().as_millis(),
         facet_distribution,
         facet_stats,
+        degraded,
+        used_negative_operator,
+        semantic_hit_count,
     };
     Ok(result)
 }
@@ -720,14 +912,21 @@ pub fn perform_facet_search(
     search_query: SearchQuery,
     facet_query: Option<String>,
     facet_name: String,
-    features: RoFeatures,
+    search_kind: SearchKind,
 ) -> Result<FacetSearchResult, MeilisearchHttpError> {
     let before_search = Instant::now();
     let rtxn = index.read_txn()?;
+    let time_budget = match index.search_cutoff(&rtxn)? {
+        Some(cutoff) => TimeBudget::new(Duration::from_millis(cutoff)),
+        None => TimeBudget::default(),
+    };
 
-    let (search, _, _, _) = prepare_search(index, &rtxn, &search_query, features, None)?;
-    let mut facet_search =
-        SearchForFacetValues::new(facet_name, search, search_query.hybrid.is_some());
+    let (search, _, _, _) = prepare_search(index, &rtxn, &search_query, &search_kind, time_budget)?;
+    let mut facet_search = SearchForFacetValues::new(
+        facet_name,
+        search,
+        matches!(search_kind, SearchKind::Hybrid { .. }),
+    );
     if let Some(facet_query) = &facet_query {
         facet_search.query(facet_query);
     }

meilisearch/src/search_queue.rs

@@ -0,0 +1,130 @@
+//! This file implements a queue of searches to process and the ability to control how many searches can be run in parallel.
+//! We need this because we don't want to process more search requests than we have cores.
+//! That slows down everything and consumes RAM for no reason.
+//! The steps to do a search are to get the `SearchQueue` data structure and try to get a search permit.
+//! This can fail if the queue is full, and we need to drop your search request to register a new one.
+//!
+//! ### How to do a search request
+//!
+//! In order to do a search request you should try to get a search permit.
+//! Retrieve the `SearchQueue` structure from actix-web (`search_queue: Data<SearchQueue>`)
+//! and right before processing the search, calls the `SearchQueue::try_get_search_permit` method: `search_queue.try_get_search_permit().await?;`
+//!
+//! What is going to happen at this point is that you're going to send a oneshot::Sender over an async mpsc channel.
+//! Then, the queue/scheduler is going to either:
+//! - Drop your oneshot channel => that means there are too many searches going on, and yours won't be executed.
+//!                                You should exit and free all the RAM you use ASAP.
+//! - Sends you a Permit => that will unlock the method, and you will be able to process your search.
+//!                         And should drop the Permit only once you have freed all the RAM consumed by the method.
+
+use std::num::NonZeroUsize;
+
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::sync::{mpsc, oneshot};
+
+use crate::error::MeilisearchHttpError;
+
+#[derive(Debug)]
+pub struct SearchQueue {
+    sender: mpsc::Sender<oneshot::Sender<Permit>>,
+    capacity: usize,
+}
+
+/// You should only run search requests while holding this permit.
+/// Once it's dropped, a new search request will be able to process.
+#[derive(Debug)]
+pub struct Permit {
+    sender: mpsc::Sender<()>,
+}
+
+impl Drop for Permit {
+    fn drop(&mut self) {
+        // if the channel is closed then the whole instance is down
+        let _ = futures::executor::block_on(self.sender.send(()));
+    }
+}
+
+impl SearchQueue {
+    pub fn new(capacity: usize, paralellism: NonZeroUsize) -> Self {
+        // Search requests are going to wait until we're available anyway,
+        // so let's not allocate any RAM and keep a capacity of 1.
+        let (sender, receiver) = mpsc::channel(1);
+
+        tokio::task::spawn(Self::run(capacity, paralellism, receiver));
+        Self { sender, capacity }
+    }
+
+    /// This function is the main loop, it's in charge on scheduling which search request should execute first and
+    /// how many should executes at the same time.
+    ///
+    /// It **must never** panic or exit.
+    async fn run(
+        capacity: usize,
+        parallelism: NonZeroUsize,
+        mut receive_new_searches: mpsc::Receiver<oneshot::Sender<Permit>>,
+    ) {
+        let mut queue: Vec<oneshot::Sender<Permit>> = Default::default();
+        let mut rng: StdRng = StdRng::from_entropy();
+        let mut searches_running: usize = 0;
+        // By having a capacity of parallelism we ensures that every time a search finish it can release its RAM asap
+        let (sender, mut search_finished) = mpsc::channel(parallelism.into());
+
+        loop {
+            tokio::select! {
+                // biased select because we wants to free up space before trying to register new tasks
+                biased;
+                _ = search_finished.recv() => {
+                    searches_running = searches_running.saturating_sub(1);
+                    if !queue.is_empty() {
+                        // Can't panic: the queue wasn't empty thus the range isn't empty.
+                        let remove = rng.gen_range(0..queue.len());
+                        let channel = queue.swap_remove(remove);
+                        let _ = channel.send(Permit { sender: sender.clone() });
+                    }
+                },
+
+                search_request = receive_new_searches.recv() => {
+                    // this unwrap is safe because we're sure the `SearchQueue` still lives somewhere in actix-web
+                    let search_request = search_request.unwrap();
+                    if searches_running < usize::from(parallelism) && queue.is_empty() {
+                        searches_running += 1;
+                        // if the search requests die it's not a hard error on our side
+                        let _ = search_request.send(Permit { sender: sender.clone() });
+                        continue;
+                    } else if capacity == 0 {
+                        // in the very specific case where we have a capacity of zero
+                        // we must refuse the request straight away without going through
+                        // the queue stuff.
+                        drop(search_request);
+                        continue;
+
+                    } else if queue.len() >= capacity {
+                        let remove = rng.gen_range(0..queue.len());
+                        let thing = queue.swap_remove(remove); // this will drop the channel and notify the search that it won't be processed
+                        drop(thing);
+                    }
+                    queue.push(search_request);
+                },
+            }
+        }
+    }
+
+    /// Returns a search `Permit`.
+    /// It should be dropped as soon as you've freed all the RAM associated with the search request being processed.
+    pub async fn try_get_search_permit(&self) -> Result<Permit, MeilisearchHttpError> {
+        let (sender, receiver) = oneshot::channel();
+        self.sender.send(sender).await.map_err(|_| MeilisearchHttpError::SearchLimiterIsDown)?;
+        receiver.await.map_err(|_| MeilisearchHttpError::TooManySearchRequests(self.capacity))
+    }
+
+    /// Returns `Ok(())` if everything seems normal.
+    /// Returns `Err(MeilisearchHttpError::SearchLimiterIsDown)` if the search limiter seems down.
+    pub fn health(&self) -> Result<(), MeilisearchHttpError> {
+        if self.sender.is_closed() {
+            Err(MeilisearchHttpError::SearchLimiterIsDown)
+        } else {
+            Ok(())
+        }
+    }
+}

meilisearch/tests/common/index.rs

@@ -328,6 +328,11 @@ impl Index<'_> {
         self.service.patch_encoded(url, settings, self.encoder).await
     }
 
+    pub async fn update_settings_search_cutoff_ms(&self, settings: Value) -> (Value, StatusCode) {
+        let url = format!("/indexes/{}/settings/search-cutoff-ms", urlencode(self.uid.as_ref()));
+        self.service.put_encoded(url, settings, self.encoder).await
+    }
+
     pub async fn delete_settings(&self) -> (Value, StatusCode) {
         let url = format!("/indexes/{}/settings", urlencode(self.uid.as_ref()));
         self.service.delete(url).await

meilisearch/tests/common/mod.rs

@@ -16,6 +16,7 @@ pub use server::{default_settings, Server};
 pub struct Value(pub serde_json::Value);
 
 impl Value {
+    #[track_caller]
     pub fn uid(&self) -> u64 {
         if let Some(uid) = self["uid"].as_u64() {
             uid

meilisearch/tests/documents/add_documents.rs

@@ -1237,8 +1237,8 @@ async fn error_add_documents_missing_document_id() {
 }
 
 #[actix_rt::test]
-#[ignore] // // TODO: Fix in an other PR: this does not provoke any error.
-async fn error_document_field_limit_reached() {
+#[should_panic]
+async fn error_document_field_limit_reached_in_one_document() {
     let server = Server::new().await;
     let index = server.index("test");
 
@@ -1246,22 +1246,241 @@ async fn error_document_field_limit_reached() {
 
     let mut big_object = std::collections::HashMap::new();
     big_object.insert("id".to_owned(), "wow");
-    for i in 0..65535 {
+    for i in 0..(u16::MAX as usize + 1) {
         let key = i.to_string();
         big_object.insert(key, "I am a text!");
     }
 
     let documents = json!([big_object]);
 
-    let (_response, code) = index.update_documents(documents, Some("id")).await;
-    snapshot!(code, @"202");
+    let (response, code) = index.update_documents(documents, Some("id")).await;
+    snapshot!(code, @"500 Internal Server Error");
 
-    index.wait_task(0).await;
-    let (response, code) = index.get_task(0).await;
-    snapshot!(code, @"200");
+    let response = index.wait_task(response.uid()).await;
+    snapshot!(code, @"202 Accepted");
     // Documents without a primary key are not accepted.
-    snapshot!(json_string!(response, { ".duration" => "[duration]", ".enqueuedAt" => "[date]", ".startedAt" => "[date]", ".finishedAt" => "[date]" }),
-        @"");
+    snapshot!(response,
+        @r###"
+    {
+      "uid": 1,
+      "indexUid": "test",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "[duration]",
+      "enqueuedAt": "[date]",
+      "startedAt": "[date]",
+      "finishedAt": "[date]"
+    }
+    "###);
+}
+
+#[actix_rt::test]
+async fn error_document_field_limit_reached_over_multiple_documents() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    index.create(Some("id")).await;
+
+    let mut big_object = std::collections::HashMap::new();
+    big_object.insert("id".to_owned(), "wow");
+    for i in 0..(u16::MAX / 2) {
+        let key = i.to_string();
+        big_object.insert(key, "I am a text!");
+    }
+
+    let documents = json!([big_object]);
+
+    let (response, code) = index.update_documents(documents, Some("id")).await;
+    snapshot!(code, @"202 Accepted");
+
+    let response = index.wait_task(response.uid()).await;
+    snapshot!(code, @"202 Accepted");
+    snapshot!(response,
+        @r###"
+    {
+      "uid": 1,
+      "indexUid": "test",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "[duration]",
+      "enqueuedAt": "[date]",
+      "startedAt": "[date]",
+      "finishedAt": "[date]"
+    }
+    "###);
+
+    let mut big_object = std::collections::HashMap::new();
+    big_object.insert("id".to_owned(), "waw");
+    for i in (u16::MAX as usize / 2)..(u16::MAX as usize + 1) {
+        let key = i.to_string();
+        big_object.insert(key, "I am a text!");
+    }
+
+    let documents = json!([big_object]);
+
+    let (response, code) = index.update_documents(documents, Some("id")).await;
+    snapshot!(code, @"202 Accepted");
+
+    let response = index.wait_task(response.uid()).await;
+    snapshot!(code, @"202 Accepted");
+    snapshot!(response,
+        @r###"
+    {
+      "uid": 2,
+      "indexUid": "test",
+      "status": "failed",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "error": {
+        "message": "A document cannot contain more than 65,535 fields.",
+        "code": "max_fields_limit_exceeded",
+        "type": "invalid_request",
+        "link": "https://docs.meilisearch.com/errors#max_fields_limit_exceeded"
+      },
+      "duration": "[duration]",
+      "enqueuedAt": "[date]",
+      "startedAt": "[date]",
+      "finishedAt": "[date]"
+    }
+    "###);
+}
+
+#[actix_rt::test]
+async fn error_document_field_limit_reached_in_one_nested_document() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    index.create(Some("id")).await;
+
+    let mut nested = std::collections::HashMap::new();
+    for i in 0..(u16::MAX as usize + 1) {
+        let key = i.to_string();
+        nested.insert(key, "I am a text!");
+    }
+    let mut big_object = std::collections::HashMap::new();
+    big_object.insert("id".to_owned(), "wow");
+
+    let documents = json!([big_object]);
+
+    let (response, code) = index.update_documents(documents, Some("id")).await;
+    snapshot!(code, @"202 Accepted");
+
+    let response = index.wait_task(response.uid()).await;
+    snapshot!(code, @"202 Accepted");
+    // Documents without a primary key are not accepted.
+    snapshot!(response,
+        @r###"
+    {
+      "uid": 1,
+      "indexUid": "test",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "[duration]",
+      "enqueuedAt": "[date]",
+      "startedAt": "[date]",
+      "finishedAt": "[date]"
+    }
+    "###);
+}
+
+#[actix_rt::test]
+async fn error_document_field_limit_reached_over_multiple_documents_with_nested_fields() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    index.create(Some("id")).await;
+
+    let mut nested = std::collections::HashMap::new();
+    for i in 0..(u16::MAX / 2) {
+        let key = i.to_string();
+        nested.insert(key, "I am a text!");
+    }
+    let mut big_object = std::collections::HashMap::new();
+    big_object.insert("id".to_owned(), "wow");
+
+    let documents = json!([big_object]);
+
+    let (response, code) = index.update_documents(documents, Some("id")).await;
+    snapshot!(code, @"202 Accepted");
+
+    let response = index.wait_task(response.uid()).await;
+    snapshot!(code, @"202 Accepted");
+    snapshot!(response,
+        @r###"
+    {
+      "uid": 1,
+      "indexUid": "test",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "[duration]",
+      "enqueuedAt": "[date]",
+      "startedAt": "[date]",
+      "finishedAt": "[date]"
+    }
+    "###);
+
+    let mut nested = std::collections::HashMap::new();
+    for i in 0..(u16::MAX / 2) {
+        let key = i.to_string();
+        nested.insert(key, "I am a text!");
+    }
+    let mut big_object = std::collections::HashMap::new();
+    big_object.insert("id".to_owned(), "wow");
+
+    let documents = json!([big_object]);
+
+    let (response, code) = index.update_documents(documents, Some("id")).await;
+    snapshot!(code, @"202 Accepted");
+
+    let response = index.wait_task(response.uid()).await;
+    snapshot!(code, @"202 Accepted");
+    snapshot!(response,
+        @r###"
+    {
+      "uid": 2,
+      "indexUid": "test",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "[duration]",
+      "enqueuedAt": "[date]",
+      "startedAt": "[date]",
+      "finishedAt": "[date]"
+    }
+    "###);
 }
 
 #[actix_rt::test]

meilisearch/tests/dumps/mod.rs

@@ -77,7 +77,8 @@ async fn import_dump_v1_movie_raw() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -238,7 +239,8 @@ async fn import_dump_v1_movie_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -385,7 +387,8 @@ async fn import_dump_v1_rubygems_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -518,7 +521,8 @@ async fn import_dump_v2_movie_raw() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -663,7 +667,8 @@ async fn import_dump_v2_movie_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -807,7 +812,8 @@ async fn import_dump_v2_rubygems_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -940,7 +946,8 @@ async fn import_dump_v3_movie_raw() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -1085,7 +1092,8 @@ async fn import_dump_v3_movie_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -1229,7 +1237,8 @@ async fn import_dump_v3_rubygems_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -1362,7 +1371,8 @@ async fn import_dump_v4_movie_raw() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -1507,7 +1517,8 @@ async fn import_dump_v4_movie_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -1651,7 +1662,8 @@ async fn import_dump_v4_rubygems_with_settings() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###
     );
@@ -1895,7 +1907,8 @@ async fn import_dump_v6_containing_experimental_features() {
       },
       "pagination": {
         "maxTotalHits": 1000
-      }
+      },
+      "searchCutoffMs": null
     }
     "###);
 

meilisearch/tests/search/facet_search.rs

@@ -123,6 +123,28 @@ async fn simple_facet_search_with_max_values() {
     assert_eq!(dbg!(response)["facetHits"].as_array().unwrap().len(), 1);
 }
 
+#[actix_rt::test]
+async fn simple_facet_search_by_count_with_max_values() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = DOCUMENTS.clone();
+    index
+        .update_settings_faceting(
+            json!({ "maxValuesPerFacet": 1, "sortFacetValuesBy": { "*": "count" } }),
+        )
+        .await;
+    index.update_settings_filterable_attributes(json!(["genres"])).await;
+    index.add_documents(documents, None).await;
+    index.wait_task(2).await;
+
+    let (response, code) =
+        index.facet_search(json!({"facetName": "genres", "facetQuery": "a"})).await;
+
+    assert_eq!(code, 200, "{}", response);
+    assert_eq!(dbg!(response)["facetHits"].as_array().unwrap().len(), 1);
+}
+
 #[actix_rt::test]
 async fn non_filterable_facet_search_error() {
     let server = Server::new().await;
@@ -157,3 +179,24 @@ async fn facet_search_dont_support_words() {
     assert_eq!(code, 200, "{}", response);
     assert_eq!(response["facetHits"].as_array().unwrap().len(), 0);
 }
+
+#[actix_rt::test]
+async fn simple_facet_search_with_sort_by_count() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = DOCUMENTS.clone();
+    index.update_settings_faceting(json!({ "sortFacetValuesBy": { "*": "count" } })).await;
+    index.update_settings_filterable_attributes(json!(["genres"])).await;
+    index.add_documents(documents, None).await;
+    index.wait_task(2).await;
+
+    let (response, code) =
+        index.facet_search(json!({"facetName": "genres", "facetQuery": "a"})).await;
+
+    assert_eq!(code, 200, "{}", response);
+    let hits = response["facetHits"].as_array().unwrap();
+    assert_eq!(hits.len(), 2);
+    assert_eq!(hits[0], json!({ "value": "Action", "count": 3 }));
+    assert_eq!(hits[1], json!({ "value": "Adventure", "count": 2 }));
+}

meilisearch/tests/search/hybrid.rs

@@ -77,14 +77,57 @@ async fn simple_search() {
         .await;
     snapshot!(code, @"200 OK");
     snapshot!(response["hits"], @r###"[{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]}},{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]}},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]}}]"###);
+    snapshot!(response["semanticHitCount"], @"0");
 
     let (response, code) = index
         .search_post(
-            json!({"q": "Captain", "vector": [1.0, 1.0], "hybrid": {"semanticRatio": 0.8}}),
+            json!({"q": "Captain", "vector": [1.0, 1.0], "hybrid": {"semanticRatio": 0.5}, "showRankingScore": true}),
         )
         .await;
     snapshot!(code, @"200 OK");
-    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_semanticScore":0.99029034},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_semanticScore":0.97434163},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_semanticScore":0.9472136}]"###);
+    snapshot!(response["hits"], @r###"[{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":0.996969696969697},{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":0.996969696969697},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":0.9472135901451112}]"###);
+    snapshot!(response["semanticHitCount"], @"1");
+
+    let (response, code) = index
+        .search_post(
+            json!({"q": "Captain", "vector": [1.0, 1.0], "hybrid": {"semanticRatio": 0.8}, "showRankingScore": true}),
+        )
+        .await;
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":0.990290343761444},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":0.974341630935669},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":0.9472135901451112}]"###);
+    snapshot!(response["semanticHitCount"], @"3");
+}
+
+#[actix_rt::test]
+async fn distribution_shift() {
+    let server = Server::new().await;
+    let index = index_with_documents(&server, &SIMPLE_SEARCH_DOCUMENTS).await;
+
+    let search = json!({"q": "Captain", "vector": [1.0, 1.0], "showRankingScore": true, "hybrid": {"semanticRatio": 1.0}});
+    let (response, code) = index.search_post(search.clone()).await;
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":0.990290343761444},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":0.974341630935669},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":0.9472135901451112}]"###);
+
+    let (response, code) = index
+        .update_settings(json!({
+            "embedders": {
+                "default": {
+                    "distribution": {
+                        "mean": 0.998,
+                        "sigma": 0.01
+                    }
+                }
+            }
+        }))
+        .await;
+
+    snapshot!(code, @"202 Accepted");
+    let response = server.wait_task(response.uid()).await;
+    snapshot!(response["details"], @r###"{"embedders":{"default":{"distribution":{"mean":0.998,"sigma":0.01}}}}"###);
+
+    let (response, code) = index.search_post(search).await;
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":0.19161224365234375},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":1.1920928955078125e-7},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":1.1920928955078125e-7}]"###);
 }
 
 #[actix_rt::test]
@@ -104,10 +147,12 @@ async fn highlighter() {
         .await;
     snapshot!(code, @"200 OK");
     snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_formatted":{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":["2.0","3.0"]}}},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_formatted":{"title":"Shazam!","desc":"a **BEGIN**Captain**END** **BEGIN**Marvel**END** ersatz","id":"1","_vectors":{"default":["1.0","3.0"]}}},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_formatted":{"title":"Captain Planet","desc":"He's not part of the **BEGIN**Marvel**END** Cinematic Universe","id":"2","_vectors":{"default":["1.0","2.0"]}}}]"###);
+    snapshot!(response["semanticHitCount"], @"0");
 
     let (response, code) = index
         .search_post(json!({"q": "Captain Marvel", "vector": [1.0, 1.0],
             "hybrid": {"semanticRatio": 0.8},
+            "showRankingScore": true,
             "attributesToHighlight": [
                      "desc"
                    ],
@@ -116,12 +161,14 @@ async fn highlighter() {
         }))
         .await;
     snapshot!(code, @"200 OK");
-    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_formatted":{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":["2.0","3.0"]}},"_semanticScore":0.99029034},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_formatted":{"title":"Captain Planet","desc":"He's not part of the **BEGIN**Marvel**END** Cinematic Universe","id":"2","_vectors":{"default":["1.0","2.0"]}},"_semanticScore":0.97434163},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_formatted":{"title":"Shazam!","desc":"a **BEGIN**Captain**END** **BEGIN**Marvel**END** ersatz","id":"1","_vectors":{"default":["1.0","3.0"]}},"_semanticScore":0.9472136}]"###);
+    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_formatted":{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":["2.0","3.0"]}},"_rankingScore":0.990290343761444},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_formatted":{"title":"Captain Planet","desc":"He's not part of the **BEGIN**Marvel**END** Cinematic Universe","id":"2","_vectors":{"default":["1.0","2.0"]}},"_rankingScore":0.974341630935669},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_formatted":{"title":"Shazam!","desc":"a **BEGIN**Captain**END** **BEGIN**Marvel**END** ersatz","id":"1","_vectors":{"default":["1.0","3.0"]}},"_rankingScore":0.9472135901451112}]"###);
+    snapshot!(response["semanticHitCount"], @"3");
 
     // no highlighting on full semantic
     let (response, code) = index
         .search_post(json!({"q": "Captain Marvel", "vector": [1.0, 1.0],
             "hybrid": {"semanticRatio": 1.0},
+            "showRankingScore": true,
             "attributesToHighlight": [
                      "desc"
                    ],
@@ -130,7 +177,8 @@ async fn highlighter() {
         }))
         .await;
     snapshot!(code, @"200 OK");
-    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_formatted":{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":["2.0","3.0"]}},"_semanticScore":0.99029034},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_formatted":{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":["1.0","2.0"]}},"_semanticScore":0.97434163},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_formatted":{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":["1.0","3.0"]}}}]"###);
+    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_formatted":{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":["2.0","3.0"]}},"_rankingScore":0.990290343761444},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_formatted":{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":["1.0","2.0"]}},"_rankingScore":0.974341630935669},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_formatted":{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":["1.0","3.0"]}},"_rankingScore":0.9472135901451112}]"###);
+    snapshot!(response["semanticHitCount"], @"3");
 }
 
 #[actix_rt::test]
@@ -217,5 +265,115 @@ async fn single_document() {
     .await;
 
     snapshot!(code, @"200 OK");
-    snapshot!(response["hits"][0], @r###"{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":1.0,"_semanticScore":1.0}"###);
+    snapshot!(response["hits"][0], @r###"{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":1.0}"###);
+    snapshot!(response["semanticHitCount"], @"1");
+}
+
+#[actix_rt::test]
+async fn query_combination() {
+    let server = Server::new().await;
+    let index = index_with_documents(&server, &SIMPLE_SEARCH_DOCUMENTS).await;
+
+    // search without query and vector, but with hybrid => still placeholder
+    let (response, code) = index
+        .search_post(json!({"hybrid": {"semanticRatio": 1.0}, "showRankingScore": true}))
+        .await;
+
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":1.0},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":1.0},{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":1.0}]"###);
+    snapshot!(response["semanticHitCount"], @"null");
+
+    // same with a different semantic ratio
+    let (response, code) = index
+        .search_post(json!({"hybrid": {"semanticRatio": 0.76}, "showRankingScore": true}))
+        .await;
+
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":1.0},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":1.0},{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":1.0}]"###);
+    snapshot!(response["semanticHitCount"], @"null");
+
+    // wrong vector dimensions
+    let (response, code) = index
+    .search_post(json!({"vector": [1.0, 0.0, 1.0], "hybrid": {"semanticRatio": 1.0}, "showRankingScore": true}))
+    .await;
+
+    snapshot!(code, @"400 Bad Request");
+    snapshot!(response, @r###"
+    {
+      "message": "Invalid vector dimensions: expected: `2`, found: `3`.",
+      "code": "invalid_vector_dimensions",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#invalid_vector_dimensions"
+    }
+    "###);
+
+    // full vector
+    let (response, code) = index
+    .search_post(json!({"vector": [1.0, 0.0], "hybrid": {"semanticRatio": 1.0}, "showRankingScore": true}))
+    .await;
+
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":0.7773500680923462},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":0.7236068248748779},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":0.6581138968467712}]"###);
+    snapshot!(response["semanticHitCount"], @"3");
+
+    // full keyword, without a query
+    let (response, code) = index
+    .search_post(json!({"vector": [1.0, 0.0], "hybrid": {"semanticRatio": 0.0}, "showRankingScore": true}))
+    .await;
+
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":1.0},{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":1.0},{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":1.0}]"###);
+    snapshot!(response["semanticHitCount"], @"null");
+
+    // query + vector, full keyword => keyword
+    let (response, code) = index
+    .search_post(json!({"q": "Captain", "vector": [1.0, 0.0], "hybrid": {"semanticRatio": 0.0}, "showRankingScore": true}))
+    .await;
+
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":0.996969696969697},{"title":"Captain Marvel","desc":"a Shazam ersatz","id":"3","_vectors":{"default":[2.0,3.0]},"_rankingScore":0.996969696969697},{"title":"Shazam!","desc":"a Captain Marvel ersatz","id":"1","_vectors":{"default":[1.0,3.0]},"_rankingScore":0.8848484848484849}]"###);
+    snapshot!(response["semanticHitCount"], @"null");
+
+    // query + vector, no hybrid keyword =>
+    let (response, code) = index
+        .search_post(json!({"q": "Captain", "vector": [1.0, 0.0], "showRankingScore": true}))
+        .await;
+
+    snapshot!(code, @"400 Bad Request");
+    snapshot!(response, @r###"
+    {
+      "message": "Invalid request: missing `hybrid` parameter when both `q` and `vector` are present.",
+      "code": "missing_search_hybrid",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#missing_search_hybrid"
+    }
+    "###);
+
+    // full vector, without a vector => error
+    let (response, code) = index
+        .search_post(
+            json!({"q": "Captain", "hybrid": {"semanticRatio": 1.0}, "showRankingScore": true}),
+        )
+        .await;
+
+    snapshot!(code, @"400 Bad Request");
+    snapshot!(response, @r###"
+    {
+      "message": "Error while generating embeddings: user error: attempt to embed the following text in a configuration where embeddings must be user provided: \"Captain\"",
+      "code": "vector_embedding_error",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#vector_embedding_error"
+    }
+    "###);
+
+    // hybrid without a vector => full keyword
+    let (response, code) = index
+        .search_post(
+            json!({"q": "Planet", "hybrid": {"semanticRatio": 0.99}, "showRankingScore": true}),
+        )
+        .await;
+
+    snapshot!(code, @"200 OK");
+    snapshot!(response["hits"], @r###"[{"title":"Captain Planet","desc":"He's not part of the Marvel Cinematic Universe","id":"2","_vectors":{"default":[1.0,2.0]},"_rankingScore":0.9848484848484848}]"###);
+    snapshot!(response["semanticHitCount"], @"0");
 }

meilisearch/tests/search/mod.rs

@@ -10,6 +10,7 @@ mod hybrid;
 mod multi;
 mod pagination;
 mod restrict_searchable;
+mod search_queue;
 
 use once_cell::sync::Lazy;
 
@@ -184,6 +185,110 @@ async fn phrase_search_with_stop_word() {
         .await;
 }
 
+#[actix_rt::test]
+async fn negative_phrase_search() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = DOCUMENTS.clone();
+    index.add_documents(documents, None).await;
+    index.wait_task(0).await;
+
+    index
+        .search(json!({"q": "-\"train your dragon\"" }), |response, code| {
+            assert_eq!(code, 200, "{}", response);
+            let hits = response["hits"].as_array().unwrap();
+            assert_eq!(hits.len(), 4);
+            assert_eq!(hits[0]["id"], "287947");
+            assert_eq!(hits[1]["id"], "299537");
+            assert_eq!(hits[2]["id"], "522681");
+            assert_eq!(hits[3]["id"], "450465");
+        })
+        .await;
+}
+
+#[actix_rt::test]
+async fn negative_word_search() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = DOCUMENTS.clone();
+    index.add_documents(documents, None).await;
+    index.wait_task(0).await;
+
+    index
+        .search(json!({"q": "-escape" }), |response, code| {
+            assert_eq!(code, 200, "{}", response);
+            let hits = response["hits"].as_array().unwrap();
+            assert_eq!(hits.len(), 4);
+            assert_eq!(hits[0]["id"], "287947");
+            assert_eq!(hits[1]["id"], "299537");
+            assert_eq!(hits[2]["id"], "166428");
+            assert_eq!(hits[3]["id"], "450465");
+        })
+        .await;
+
+    // Everything that contains derivates of escape but not escape: nothing
+    index
+        .search(json!({"q": "-escape escape" }), |response, code| {
+            assert_eq!(code, 200, "{}", response);
+            let hits = response["hits"].as_array().unwrap();
+            assert_eq!(hits.len(), 0);
+        })
+        .await;
+}
+
+#[actix_rt::test]
+async fn non_negative_search() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = DOCUMENTS.clone();
+    index.add_documents(documents, None).await;
+    index.wait_task(0).await;
+
+    index
+        .search(json!({"q": "- escape" }), |response, code| {
+            assert_eq!(code, 200, "{}", response);
+            let hits = response["hits"].as_array().unwrap();
+            assert_eq!(hits.len(), 1);
+            assert_eq!(hits[0]["id"], "522681");
+        })
+        .await;
+
+    index
+        .search(json!({"q": "- \"train your dragon\"" }), |response, code| {
+            assert_eq!(code, 200, "{}", response);
+            let hits = response["hits"].as_array().unwrap();
+            assert_eq!(hits.len(), 1);
+            assert_eq!(hits[0]["id"], "166428");
+        })
+        .await;
+}
+
+#[actix_rt::test]
+async fn negative_special_cases_search() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = DOCUMENTS.clone();
+    index.add_documents(documents, None).await;
+    index.wait_task(0).await;
+
+    index.update_settings(json!({"synonyms": { "escape": ["glass"] }})).await;
+    index.wait_task(1).await;
+
+    // There is a synonym for escape -> glass but we don't want "escape", only the derivates: glass
+    index
+        .search(json!({"q": "-escape escape" }), |response, code| {
+            assert_eq!(code, 200, "{}", response);
+            let hits = response["hits"].as_array().unwrap();
+            assert_eq!(hits.len(), 1);
+            assert_eq!(hits[0]["id"], "450465");
+        })
+        .await;
+}
+
 #[cfg(feature = "default")]
 #[actix_rt::test]
 async fn test_kanji_language_detection() {
@@ -834,6 +939,94 @@ async fn test_score_details() {
         .await;
 }
 
+#[actix_rt::test]
+async fn test_degraded_score_details() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let documents = NESTED_DOCUMENTS.clone();
+
+    index.add_documents(json!(documents), None).await;
+    // We can't really use anything else than 0ms here; otherwise, the test will get flaky.
+    let (res, _code) = index.update_settings(json!({ "searchCutoffMs": 0 })).await;
+    index.wait_task(res.uid()).await;
+
+    index
+        .search(
+            json!({
+                "q": "b",
+                "attributesToRetrieve": ["doggos.name", "cattos"],
+                "showRankingScoreDetails": true,
+            }),
+            |response, code| {
+                meili_snap::snapshot!(code, @"200 OK");
+                meili_snap::snapshot!(meili_snap::json_string!(response, { ".processingTimeMs" => "[duration]" }), @r###"
+                {
+                  "hits": [
+                    {
+                      "doggos": [
+                        {
+                          "name": "bobby"
+                        },
+                        {
+                          "name": "buddy"
+                        }
+                      ],
+                      "cattos": "pésti",
+                      "_rankingScoreDetails": {
+                        "skipped": {
+                          "order": 0
+                        }
+                      }
+                    },
+                    {
+                      "doggos": [
+                        {
+                          "name": "gros bill"
+                        }
+                      ],
+                      "cattos": [
+                        "simba",
+                        "pestiféré"
+                      ],
+                      "_rankingScoreDetails": {
+                        "skipped": {
+                          "order": 0
+                        }
+                      }
+                    },
+                    {
+                      "doggos": [
+                        {
+                          "name": "turbo"
+                        },
+                        {
+                          "name": "fast"
+                        }
+                      ],
+                      "cattos": [
+                        "moumoute",
+                        "gomez"
+                      ],
+                      "_rankingScoreDetails": {
+                        "skipped": {
+                          "order": 0
+                        }
+                      }
+                    }
+                  ],
+                  "query": "b",
+                  "processingTimeMs": "[duration]",
+                  "limit": 20,
+                  "offset": 0,
+                  "estimatedTotalHits": 3
+                }
+                "###);
+            },
+        )
+        .await;
+}
+
 #[actix_rt::test]
 async fn experimental_feature_vector_store() {
     let server = Server::new().await;
@@ -847,6 +1040,7 @@ async fn experimental_feature_vector_store() {
     let (response, code) = index
         .search_post(json!({
             "vector": [1.0, 2.0, 3.0],
+            "showRankingScore": true
         }))
         .await;
     meili_snap::snapshot!(code, @"400 Bad Request");
@@ -889,6 +1083,7 @@ async fn experimental_feature_vector_store() {
     let (response, code) = index
         .search_post(json!({
             "vector": [1.0, 2.0, 3.0],
+            "showRankingScore": true,
         }))
         .await;
 
@@ -906,7 +1101,7 @@ async fn experimental_feature_vector_store() {
             3
           ]
         },
-        "_semanticScore": 1.0
+        "_rankingScore": 1.0
       },
       {
         "title": "Captain Marvel",
@@ -918,7 +1113,7 @@ async fn experimental_feature_vector_store() {
             54
           ]
         },
-        "_semanticScore": 0.9129112
+        "_rankingScore": 0.9129111766815186
       },
       {
         "title": "Gläss",
@@ -930,7 +1125,7 @@ async fn experimental_feature_vector_store() {
             90
           ]
         },
-        "_semanticScore": 0.8106413
+        "_rankingScore": 0.8106412887573242
       },
       {
         "title": "How to Train Your Dragon: The Hidden World",
@@ -942,7 +1137,7 @@ async fn experimental_feature_vector_store() {
             32
           ]
         },
-        "_semanticScore": 0.74120104
+        "_rankingScore": 0.7412010431289673
       },
       {
         "title": "Escape Room",
@@ -953,7 +1148,8 @@ async fn experimental_feature_vector_store() {
             -23,
             32
           ]
-        }
+        },
+        "_rankingScore": 0.6972063183784485
       }
     ]
     "###);

meilisearch/tests/search/search_queue.rs

@@ -0,0 +1,184 @@
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+use std::time::Duration;
+
+use actix_web::ResponseError;
+use meili_snap::snapshot;
+use meilisearch::search_queue::SearchQueue;
+
+#[actix_rt::test]
+async fn search_queue_register() {
+    let queue = SearchQueue::new(4, NonZeroUsize::new(2).unwrap());
+
+    // First, use all the cores
+    let permit1 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+    let _permit2 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+
+    // If we free one spot we should be able to register one new search
+    drop(permit1);
+
+    let permit3 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+
+    // And again
+    drop(permit3);
+
+    let _permit4 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+}
+
+#[actix_rt::test]
+async fn wait_till_cores_are_available() {
+    let queue = Arc::new(SearchQueue::new(4, NonZeroUsize::new(1).unwrap()));
+
+    // First, use all the cores
+    let permit1 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+
+    let ret = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit()).await;
+    assert!(ret.is_err(), "The capacity is full, we should not get a permit");
+
+    let q = queue.clone();
+    let task = tokio::task::spawn(async move { q.try_get_search_permit().await });
+
+    // after dropping a permit the previous task should be able to finish
+    drop(permit1);
+    let _permit2 = tokio::time::timeout(Duration::from_secs(1), task)
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+}
+
+#[actix_rt::test]
+async fn refuse_search_requests_when_queue_is_full() {
+    let queue = Arc::new(SearchQueue::new(1, NonZeroUsize::new(1).unwrap()));
+
+    // First, use the whole capacity of the
+    let _permit1 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+
+    let q = queue.clone();
+    let permit2 = tokio::task::spawn(async move { q.try_get_search_permit().await });
+
+    // Here the queue is full. By registering two new search requests the permit 2 and 3 should be thrown out
+    let q = queue.clone();
+    let _permit3 = tokio::task::spawn(async move { q.try_get_search_permit().await });
+
+    let permit2 = tokio::time::timeout(Duration::from_secs(1), permit2)
+        .await
+        .expect("I should get a result straight away")
+        .unwrap(); // task should end successfully
+
+    let err = meilisearch_types::error::ResponseError::from(permit2.unwrap_err());
+    let http_response = err.error_response();
+    let mut headers: Vec<_> = http_response
+        .headers()
+        .iter()
+        .map(|(name, value)| (name.to_string(), value.to_str().unwrap().to_string()))
+        .collect();
+    headers.sort();
+    snapshot!(format!("{headers:?}"), @r###"[("content-type", "application/json"), ("retry-after", "10")]"###);
+
+    let err = serde_json::to_string_pretty(&err).unwrap();
+    snapshot!(err, @r###"
+    {
+      "message": "Too many search requests running at the same time: 1. Retry after 10s.",
+      "code": "too_many_search_requests",
+      "type": "system",
+      "link": "https://docs.meilisearch.com/errors#too_many_search_requests"
+    }
+    "###);
+}
+
+#[actix_rt::test]
+async fn search_request_crashes_while_holding_permits() {
+    let queue = Arc::new(SearchQueue::new(1, NonZeroUsize::new(1).unwrap()));
+
+    let (send, recv) = tokio::sync::oneshot::channel();
+
+    // This first request take a cpu
+    let q = queue.clone();
+    tokio::task::spawn(async move {
+        let _permit = q.try_get_search_permit().await.unwrap();
+        recv.await.unwrap();
+        panic!("oops an unexpected crash happened")
+    });
+
+    // This second request waits in the queue till the first request finishes
+    let q = queue.clone();
+    let task = tokio::task::spawn(async move {
+        let _permit = q.try_get_search_permit().await.unwrap();
+    });
+
+    // By sending something in the channel the request holding a CPU will panic and should lose its permit
+    send.send(()).unwrap();
+
+    // Then the second request should be able to process and finishes correctly without panic
+    tokio::time::timeout(Duration::from_secs(1), task)
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+
+    // I should even be able to take second permit here
+    let _permit1 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+}
+
+#[actix_rt::test]
+async fn works_with_capacity_of_zero() {
+    let queue = Arc::new(SearchQueue::new(0, NonZeroUsize::new(1).unwrap()));
+
+    // First, use the whole capacity of the
+    let permit1 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+
+    // then we should get an error if we try to register a second search request.
+    let permit2 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a result straight away");
+
+    let err = meilisearch_types::error::ResponseError::from(permit2.unwrap_err());
+    let http_response = err.error_response();
+    let mut headers: Vec<_> = http_response
+        .headers()
+        .iter()
+        .map(|(name, value)| (name.to_string(), value.to_str().unwrap().to_string()))
+        .collect();
+    headers.sort();
+    snapshot!(format!("{headers:?}"), @r###"[("content-type", "application/json"), ("retry-after", "10")]"###);
+
+    let err = serde_json::to_string_pretty(&err).unwrap();
+    snapshot!(err, @r###"
+    {
+      "message": "Too many search requests running at the same time: 0. Retry after 10s.",
+      "code": "too_many_search_requests",
+      "type": "system",
+      "link": "https://docs.meilisearch.com/errors#too_many_search_requests"
+    }
+    "###);
+
+    drop(permit1);
+    // After dropping the first permit we should be able to get a new permit
+    let _permit3 = tokio::time::timeout(Duration::from_secs(1), queue.try_get_search_permit())
+        .await
+        .expect("I should get a permit straight away")
+        .unwrap();
+}

meilisearch/tests/settings/errors.rs

@@ -337,3 +337,31 @@ async fn settings_bad_pagination() {
     }
     "###);
 }
+
+#[actix_rt::test]
+async fn settings_bad_search_cutoff_ms() {
+    let server = Server::new().await;
+    let index = server.index("test");
+
+    let (response, code) = index.update_settings(json!({ "searchCutoffMs": "doggo" })).await;
+    snapshot!(code, @"400 Bad Request");
+    snapshot!(json_string!(response), @r###"
+    {
+      "message": "Invalid value type at `.searchCutoffMs`: expected a positive integer, but found a string: `\"doggo\"`",
+      "code": "invalid_settings_search_cutoff_ms",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#invalid_settings_search_cutoff_ms"
+    }
+    "###);
+
+    let (response, code) = index.update_settings_search_cutoff_ms(json!("doggo")).await;
+    snapshot!(code, @"400 Bad Request");
+    snapshot!(json_string!(response), @r###"
+    {
+      "message": "Invalid value type: expected a positive integer, but found a string: `\"doggo\"`",
+      "code": "invalid_settings_search_cutoff_ms",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#invalid_settings_search_cutoff_ms"
+    }
+    "###);
+}

meilisearch/tests/settings/get_settings.rs

@@ -35,6 +35,7 @@ static DEFAULT_SETTINGS_VALUES: Lazy<HashMap<&'static str, Value>> = Lazy::new(|
             "maxTotalHits": json!(1000),
         }),
     );
+    map.insert("search_cutoff_ms", json!(null));
     map
 });
 
@@ -49,12 +50,12 @@ async fn get_settings_unexisting_index() {
 async fn get_settings() {
     let server = Server::new().await;
     let index = server.index("test");
-    index.create(None).await;
-    index.wait_task(0).await;
+    let (response, _code) = index.create(None).await;
+    index.wait_task(response.uid()).await;
     let (response, code) = index.settings().await;
     assert_eq!(code, 200);
     let settings = response.as_object().unwrap();
-    assert_eq!(settings.keys().len(), 15);
+    assert_eq!(settings.keys().len(), 16);
     assert_eq!(settings["displayedAttributes"], json!(["*"]));
     assert_eq!(settings["searchableAttributes"], json!(["*"]));
     assert_eq!(settings["filterableAttributes"], json!([]));
@@ -84,6 +85,140 @@ async fn get_settings() {
         })
     );
     assert_eq!(settings["proximityPrecision"], json!("byWord"));
+    assert_eq!(settings["searchCutoffMs"], json!(null));
+}
+
+#[actix_rt::test]
+async fn secrets_are_hidden_in_settings() {
+    let server = Server::new().await;
+    let (response, code) = server.set_features(json!({"vectorStore": true})).await;
+
+    meili_snap::snapshot!(code, @"200 OK");
+    meili_snap::snapshot!(meili_snap::json_string!(response), @r###"
+    {
+      "vectorStore": true,
+      "metrics": false,
+      "logsRoute": false,
+      "exportPuffinReports": false
+    }
+    "###);
+
+    let index = server.index("test");
+    let (response, _code) = index.create(None).await;
+    index.wait_task(response.uid()).await;
+
+    let (response, code) = index
+        .update_settings(json!({
+            "embedders": {
+                "default": {
+                    "source": "rest",
+                    "url": "https://localhost:7777",
+                    "apiKey": "My super secret value you will never guess",
+                    "dimensions": 4,
+                }
+            }
+        }))
+        .await;
+    meili_snap::snapshot!(code, @"202 Accepted");
+
+    meili_snap::snapshot!(meili_snap::json_string!(response, { ".duration" => "[duration]", ".enqueuedAt" => "[date]", ".startedAt" => "[date]", ".finishedAt" => "[date]" }),
+    @r###"
+    {
+      "taskUid": 1,
+      "indexUid": "test",
+      "status": "enqueued",
+      "type": "settingsUpdate",
+      "enqueuedAt": "[date]"
+    }
+    "###);
+
+    let settings_update_uid = response.uid();
+
+    index.wait_task(settings_update_uid).await;
+
+    let (response, code) = index.settings().await;
+    meili_snap::snapshot!(code, @"200 OK");
+    meili_snap::snapshot!(meili_snap::json_string!(response), @r###"
+    {
+      "displayedAttributes": [
+        "*"
+      ],
+      "searchableAttributes": [
+        "*"
+      ],
+      "filterableAttributes": [],
+      "sortableAttributes": [],
+      "rankingRules": [
+        "words",
+        "typo",
+        "proximity",
+        "attribute",
+        "sort",
+        "exactness"
+      ],
+      "stopWords": [],
+      "nonSeparatorTokens": [],
+      "separatorTokens": [],
+      "dictionary": [],
+      "synonyms": {},
+      "distinctAttribute": null,
+      "proximityPrecision": "byWord",
+      "typoTolerance": {
+        "enabled": true,
+        "minWordSizeForTypos": {
+          "oneTypo": 5,
+          "twoTypos": 9
+        },
+        "disableOnWords": [],
+        "disableOnAttributes": []
+      },
+      "faceting": {
+        "maxValuesPerFacet": 100,
+        "sortFacetValuesBy": {
+          "*": "alpha"
+        }
+      },
+      "pagination": {
+        "maxTotalHits": 1000
+      },
+      "embedders": {
+        "default": {
+          "source": "rest",
+          "apiKey": "My suXXXXXX...",
+          "dimensions": 4,
+          "documentTemplate": "{% for field in fields %} {{ field.name }}: {{ field.value }}\n{% endfor %}",
+          "url": "https://localhost:7777",
+          "query": null,
+          "inputField": [
+            "input"
+          ],
+          "pathToEmbeddings": [
+            "data"
+          ],
+          "embeddingObject": [
+            "embedding"
+          ],
+          "inputType": "text"
+        }
+      },
+      "searchCutoffMs": null
+    }
+    "###);
+
+    let (response, code) = server.get_task(settings_update_uid).await;
+    meili_snap::snapshot!(code, @"200 OK");
+    meili_snap::snapshot!(meili_snap::json_string!(response["details"]), @r###"
+    {
+      "embedders": {
+        "default": {
+          "source": "rest",
+          "apiKey": "My suXXXXXX...",
+          "dimensions": 4,
+          "url": "https://localhost:7777"
+        }
+      }
+    }
+    "###);
 }
 
 #[actix_rt::test]
@@ -285,7 +420,8 @@ test_setting_routes!(
     ranking_rules put,
     synonyms put,
     pagination patch,
-    faceting patch
+    faceting patch,
+    search_cutoff_ms put
 );
 
 #[actix_rt::test]

meilitool/src/main.rs

@@ -291,7 +291,11 @@ fn export_a_dump(
         }
 
         // 4.2. Dump the settings
-        let settings = meilisearch_types::settings::settings(&index, &rtxn)?;
+        let settings = meilisearch_types::settings::settings(
+            &index,
+            &rtxn,
+            meilisearch_types::settings::SecretPolicy::RevealSecrets,
+        )?;
         index_dumper.settings(&settings)?;
         count += 1;
     }

milli/Cargo.toml

@@ -17,7 +17,7 @@ bincode = "1.3.3"
 bstr = "1.9.0"
 bytemuck = { version = "1.14.0", features = ["extern_crate_alloc"] }
 byteorder = "1.5.0"
-charabia = { version = "0.8.7", default-features = false }
+charabia = { version = "0.8.10", default-features = false }
 concat-arrays = "0.1.2"
 crossbeam-channel = "0.5.11"
 deserr = "0.6.1"
@@ -26,7 +26,7 @@ flatten-serde-json = { path = "../flatten-serde-json" }
 fst = "0.4.7"
 fxhash = "0.2.1"
 geoutils = "0.5.1"
-grenad = { version = "0.4.5", default-features = false, features = [
+grenad = { version = "0.4.6", default-features = false, features = [
     "rayon",
     "tempfile",
 ] }
@@ -71,26 +71,22 @@ itertools = "0.11.0"
 puffin = "0.16.0"
 
 csv = "1.3.0"
-candle-core = { git = "https://github.com/huggingface/candle.git", version = "0.3.1" }
-candle-transformers = { git = "https://github.com/huggingface/candle.git", version = "0.3.1" }
-candle-nn = { git = "https://github.com/huggingface/candle.git", version = "0.3.1" }
-tokenizers = { git = "https://github.com/huggingface/tokenizers.git", tag = "v0.14.1", version = "0.14.1", default_features = false, features = [
+candle-core = { version = "0.4.1" }
+candle-transformers = { version = "0.4.1" }
+candle-nn = { version = "0.4.1" }
+tokenizers = { git = "https://github.com/huggingface/tokenizers.git", tag = "v0.15.2", version = "0.15.2", default_features = false, features = [
     "onig",
 ] }
 hf-hub = { git = "https://github.com/dureuill/hf-hub.git", branch = "rust_tls", default_features = false, features = [
     "online",
 ] }
-tokio = { version = "1.35.1", features = ["rt"] }
-futures = "0.3.30"
-reqwest = { version = "0.11.23", features = [
-    "rustls-tls",
-    "json",
-], default-features = false }
 tiktoken-rs = "0.5.8"
 liquid = "0.26.4"
 arroy = "0.2.0"
 rand = "0.8.5"
 tracing = "0.1.40"
+ureq = { version = "2.9.6", features = ["json"] }
+url = "2.5.0"
 
 [dev-dependencies]
 mimalloc = { version = "0.1.39", default-features = false }
@@ -119,6 +115,7 @@ lmdb-posix-sem = ["heed/posix-sem"]
 
 # allow chinese specialized tokenization
 chinese = ["charabia/chinese"]
+chinese-pinyin = ["chinese", "charabia/chinese-normalization-pinyin"]
 
 # allow hebrew specialized tokenization
 hebrew = ["charabia/hebrew"]
@@ -139,7 +136,11 @@ greek = ["charabia/greek"]
 # allow khmer specialized tokenization
 khmer = ["charabia/khmer"]
 
+# allow vietnamese specialized tokenization
 vietnamese = ["charabia/vietnamese"]
 
+# force swedish character recomposition
+swedish-recomposition = ["charabia/swedish-recomposition"]
+
 # allow CUDA support, see <https://github.com/meilisearch/meilisearch/issues/4306>
 cuda = ["candle-core/cuda"]

milli/examples/search.rs

@@ -6,7 +6,7 @@ use std::time::Instant;
 use heed::EnvOpenOptions;
 use milli::{
     execute_search, filtered_universe, DefaultSearchLogger, GeoSortStrategy, Index, SearchContext,
-    SearchLogger, TermsMatchingStrategy,
+    SearchLogger, TermsMatchingStrategy, TimeBudget,
 };
 
 #[global_allocator]
@@ -65,6 +65,7 @@ fn main() -> Result<(), Box<dyn Error>> {
                 None,
                 &mut DefaultSearchLogger,
                 logger,
+                TimeBudget::max(),
             )?;
             if let Some((logger, dir)) = detailed_logger {
                 logger.finish(&mut ctx, Path::new(dir))?;

milli/src/error.rs

@@ -9,6 +9,7 @@ use serde_json::Value;
 use thiserror::Error;
 
 use crate::documents::{self, DocumentsBatchCursorError};
+use crate::thread_pool_no_abort::PanicCatched;
 use crate::{CriterionError, DocumentId, FieldId, Object, SortError};
 
 pub fn is_reserved_keyword(keyword: &str) -> bool {
@@ -39,17 +40,19 @@ pub enum InternalError {
     Fst(#[from] fst::Error),
     #[error(transparent)]
     DocumentsError(#[from] documents::Error),
-    #[error("Invalid compression type have been specified to grenad.")]
+    #[error("Invalid compression type have been specified to grenad")]
     GrenadInvalidCompressionType,
-    #[error("Invalid grenad file with an invalid version format.")]
+    #[error("Invalid grenad file with an invalid version format")]
     GrenadInvalidFormatVersion,
-    #[error("Invalid merge while processing {process}.")]
+    #[error("Invalid merge while processing {process}")]
     IndexingMergingKeys { process: &'static str },
     #[error("{}", HeedError::InvalidDatabaseTyping)]
     InvalidDatabaseTyping,
     #[error(transparent)]
     RayonThreadPool(#[from] ThreadPoolBuildError),
     #[error(transparent)]
+    PanicInThreadPool(#[from] PanicCatched),
+    #[error(transparent)]
     SerdeJson(#[from] serde_json::Error),
     #[error(transparent)]
     Serialization(#[from] SerializationError),
@@ -57,9 +60,9 @@ pub enum InternalError {
     Store(#[from] MdbError),
     #[error(transparent)]
     Utf8(#[from] str::Utf8Error),
-    #[error("An indexation process was explicitly aborted.")]
+    #[error("An indexation process was explicitly aborted")]
     AbortedIndexation,
-    #[error("The matching words list contains at least one invalid member.")]
+    #[error("The matching words list contains at least one invalid member")]
     InvalidMatchingWords,
     #[error(transparent)]
     ArroyError(#[from] arroy::Error),
@@ -196,7 +199,7 @@ only composed of alphanumeric characters (a-z A-Z 0-9), hyphens (-) and undersco
     InvalidPromptForEmbeddings(String, crate::prompt::error::NewPromptError),
     #[error("Too many embedders in the configuration. Found {0}, but limited to 256.")]
     TooManyEmbedders(usize),
-    #[error("Cannot find embedder with name {0}.")]
+    #[error("Cannot find embedder with name `{0}`.")]
     InvalidEmbedder(String),
     #[error("Too many vectors for document with id {0}: found {1}, but limited to 256.")]
     TooManyVectors(String, usize),
@@ -243,6 +246,8 @@ only composed of alphanumeric characters (a-z A-Z 0-9), hyphens (-) and undersco
     },
     #[error("`.embedders.{embedder_name}.dimensions`: `dimensions` cannot be zero")]
     InvalidSettingsDimensions { embedder_name: String },
+    #[error("`.embedders.{embedder_name}.url`: could not parse `{url}`: {inner_error}")]
+    InvalidUrl { embedder_name: String, inner_error: url::ParseError, url: String },
 }
 
 impl From<crate::vector::Error> for Error {

milli/src/index.rs

@@ -20,13 +20,13 @@ use crate::heed_codec::facet::{
 use crate::heed_codec::{
     BEU16StrCodec, FstSetCodec, ScriptLanguageCodec, StrBEU16Codec, StrRefCodec,
 };
+use crate::order_by_map::OrderByMap;
 use crate::proximity::ProximityPrecision;
 use crate::vector::EmbeddingConfig;
 use crate::{
     default_criteria, CboRoaringBitmapCodec, Criterion, DocumentId, ExternalDocumentsIds,
     FacetDistribution, FieldDistribution, FieldId, FieldIdWordCountCodec, GeoPoint, ObkvCodec,
-    OrderBy, Result, RoaringBitmapCodec, RoaringBitmapLenCodec, Search, U8StrStrCodec, BEU16,
-    BEU32, BEU64,
+    Result, RoaringBitmapCodec, RoaringBitmapLenCodec, Search, U8StrStrCodec, BEU16, BEU32, BEU64,
 };
 
 pub const DEFAULT_MIN_WORD_LEN_ONE_TYPO: u8 = 5;
@@ -67,6 +67,7 @@ pub mod main_key {
     pub const PAGINATION_MAX_TOTAL_HITS: &str = "pagination-max-total-hits";
     pub const PROXIMITY_PRECISION: &str = "proximity-precision";
     pub const EMBEDDING_CONFIGS: &str = "embedding_configs";
+    pub const SEARCH_CUTOFF: &str = "search_cutoff";
 }
 
 pub mod db_name {
@@ -677,6 +678,23 @@ impl Index {
             .get(rtxn, main_key::USER_DEFINED_SEARCHABLE_FIELDS_KEY)
     }
 
+    /// Identical to `user_defined_searchable_fields`, but returns ids instead.
+    pub fn user_defined_searchable_fields_ids(&self, rtxn: &RoTxn) -> Result<Option<Vec<FieldId>>> {
+        match self.user_defined_searchable_fields(rtxn)? {
+            Some(fields) => {
+                let fields_ids_map = self.fields_ids_map(rtxn)?;
+                let mut fields_ids = Vec::new();
+                for name in fields {
+                    if let Some(field_id) = fields_ids_map.id(name) {
+                        fields_ids.push(field_id);
+                    }
+                }
+                Ok(Some(fields_ids))
+            }
+            None => Ok(None),
+        }
+    }
+
     /* filterable fields */
 
     /// Writes the filterable fields names in the database.
@@ -823,11 +841,11 @@ impl Index {
 
     /// Identical to `user_defined_faceted_fields`, but returns ids instead.
     pub fn user_defined_faceted_fields_ids(&self, rtxn: &RoTxn) -> Result<HashSet<FieldId>> {
-        let fields = self.faceted_fields(rtxn)?;
+        let fields = self.user_defined_faceted_fields(rtxn)?;
         let fields_ids_map = self.fields_ids_map(rtxn)?;
 
         let mut fields_ids = HashSet::new();
-        for name in fields.into_iter() {
+        for name in fields {
             if let Some(field_id) = fields_ids_map.id(&name) {
                 fields_ids.insert(field_id);
             }
@@ -1115,7 +1133,7 @@ impl Index {
 
     /* words prefixes fst */
 
-    /// Writes the FST which is the words prefixes dictionnary of the engine.
+    /// Writes the FST which is the words prefixes dictionary of the engine.
     pub(crate) fn put_words_prefixes_fst<A: AsRef<[u8]>>(
         &self,
         wtxn: &mut RwTxn,
@@ -1128,7 +1146,7 @@ impl Index {
         )
     }
 
-    /// Returns the FST which is the words prefixes dictionnary of the engine.
+    /// Returns the FST which is the words prefixes dictionary of the engine.
     pub fn words_prefixes_fst<'t>(&self, rtxn: &'t RoTxn) -> Result<fst::Set<Cow<'t, [u8]>>> {
         match self.main.remap_types::<Str, Bytes>().get(rtxn, main_key::WORDS_PREFIXES_FST_KEY)? {
             Some(bytes) => Ok(fst::Set::new(bytes)?.map_data(Cow::Borrowed)?),
@@ -1373,21 +1391,19 @@ impl Index {
         self.main.remap_key_type::<Str>().delete(txn, main_key::MAX_VALUES_PER_FACET)
     }
 
-    pub fn sort_facet_values_by(&self, txn: &RoTxn) -> heed::Result<HashMap<String, OrderBy>> {
-        let mut orders = self
+    pub fn sort_facet_values_by(&self, txn: &RoTxn) -> heed::Result<OrderByMap> {
+        let orders = self
             .main
-            .remap_types::<Str, SerdeJson<HashMap<String, OrderBy>>>()
+            .remap_types::<Str, SerdeJson<OrderByMap>>()
             .get(txn, main_key::SORT_FACET_VALUES_BY)?
             .unwrap_or_default();
-        // Insert the default ordering if it is not already overwritten by the user.
-        orders.entry("*".to_string()).or_insert(OrderBy::Lexicographic);
         Ok(orders)
     }
 
     pub(crate) fn put_sort_facet_values_by(
         &self,
         txn: &mut RwTxn,
-        val: &HashMap<String, OrderBy>,
+        val: &OrderByMap,
     ) -> heed::Result<()> {
         self.main.remap_types::<Str, SerdeJson<_>>().put(txn, main_key::SORT_FACET_VALUES_BY, &val)
     }
@@ -1500,12 +1516,16 @@ impl Index {
             .unwrap_or_default())
     }
 
-    pub fn default_embedding_name(&self, rtxn: &RoTxn<'_>) -> Result<String> {
-        let configs = self.embedding_configs(rtxn)?;
-        Ok(match configs.as_slice() {
-            [(ref first_name, _)] => first_name.clone(),
-            _ => "default".to_owned(),
-        })
+    pub(crate) fn put_search_cutoff(&self, wtxn: &mut RwTxn<'_>, cutoff: u64) -> heed::Result<()> {
+        self.main.remap_types::<Str, BEU64>().put(wtxn, main_key::SEARCH_CUTOFF, &cutoff)
+    }
+
+    pub fn search_cutoff(&self, rtxn: &RoTxn<'_>) -> Result<Option<u64>> {
+        Ok(self.main.remap_types::<Str, BEU64>().get(rtxn, main_key::SEARCH_CUTOFF)?)
+    }
+
+    pub(crate) fn delete_search_cutoff(&self, wtxn: &mut RwTxn<'_>) -> heed::Result<bool> {
+        self.main.remap_key_type::<Str>().delete(wtxn, main_key::SEARCH_CUTOFF)
     }
 }
 
@@ -2423,6 +2443,8 @@ pub(crate) mod tests {
             candidates: _,
             document_scores: _,
             mut documents_ids,
+            degraded: _,
+            used_negative_operator: _,
         } = search.execute().unwrap();
         let primary_key_id = index.fields_ids_map(&rtxn).unwrap().id("primary_key").unwrap();
         documents_ids.sort_unstable();

milli/src/lib.rs

@@ -16,10 +16,12 @@ pub mod facet;
 mod fields_ids_map;
 pub mod heed_codec;
 pub mod index;
+pub mod order_by_map;
 pub mod prompt;
 pub mod proximity;
 pub mod score_details;
 mod search;
+mod thread_pool_no_abort;
 pub mod update;
 pub mod vector;
 
@@ -29,6 +31,7 @@ pub mod snapshot_tests;
 
 use std::collections::{BTreeMap, HashMap};
 use std::convert::{TryFrom, TryInto};
+use std::fmt;
 use std::hash::BuildHasherDefault;
 
 use charabia::normalizer::{CharNormalizer, CompatibilityDecompositionNormalizer};
@@ -40,6 +43,7 @@ pub use search::new::{
     SearchLogger, VisualSearchLogger,
 };
 use serde_json::Value;
+pub use thread_pool_no_abort::{PanicCatched, ThreadPoolNoAbort, ThreadPoolNoAbortBuilder};
 pub use {charabia as tokenizer, heed};
 
 pub use self::asc_desc::{AscDesc, AscDescError, Member, SortError};
@@ -56,10 +60,10 @@ pub use self::heed_codec::{
     UncheckedU8StrStrCodec,
 };
 pub use self::index::Index;
+pub use self::search::facet::{FacetValueHit, SearchForFacetValues};
 pub use self::search::{
-    FacetDistribution, FacetValueHit, Filter, FormatOptions, MatchBounds, MatcherBuilder,
-    MatchingWords, OrderBy, Search, SearchForFacetValues, SearchResult, TermsMatchingStrategy,
-    DEFAULT_VALUES_PER_FACET,
+    FacetDistribution, Filter, FormatOptions, MatchBounds, MatcherBuilder, MatchingWords, OrderBy,
+    Search, SearchResult, SemanticSearch, TermsMatchingStrategy, DEFAULT_VALUES_PER_FACET,
 };
 
 pub type Result<T> = std::result::Result<T, error::Error>;
@@ -103,6 +107,73 @@ pub const MAX_WORD_LENGTH: usize = MAX_LMDB_KEY_LENGTH / 2;
 
 pub const MAX_POSITION_PER_ATTRIBUTE: u32 = u16::MAX as u32 + 1;
 
+#[derive(Clone)]
+pub struct TimeBudget {
+    started_at: std::time::Instant,
+    budget: std::time::Duration,
+
+    /// When testing the time budget, ensuring we did more than iteration of the bucket sort can be useful.
+    /// But to avoid being flaky, the only option is to add the ability to stop after a specific number of calls instead of a `Duration`.
+    #[cfg(test)]
+    stop_after: Option<(std::sync::Arc<std::sync::atomic::AtomicUsize>, usize)>,
+}
+
+impl fmt::Debug for TimeBudget {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("TimeBudget")
+            .field("started_at", &self.started_at)
+            .field("budget", &self.budget)
+            .field("left", &(self.budget - self.started_at.elapsed()))
+            .finish()
+    }
+}
+
+impl Default for TimeBudget {
+    fn default() -> Self {
+        Self::new(std::time::Duration::from_millis(1500))
+    }
+}
+
+impl TimeBudget {
+    pub fn new(budget: std::time::Duration) -> Self {
+        Self {
+            started_at: std::time::Instant::now(),
+            budget,
+
+            #[cfg(test)]
+            stop_after: None,
+        }
+    }
+
+    pub fn max() -> Self {
+        Self::new(std::time::Duration::from_secs(u64::MAX))
+    }
+
+    #[cfg(test)]
+    pub fn with_stop_after(mut self, stop_after: usize) -> Self {
+        use std::sync::atomic::AtomicUsize;
+        use std::sync::Arc;
+
+        self.stop_after = Some((Arc::new(AtomicUsize::new(0)), stop_after));
+        self
+    }
+
+    pub fn exceeded(&self) -> bool {
+        #[cfg(test)]
+        if let Some((current, stop_after)) = &self.stop_after {
+            let current = current.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+            if current >= *stop_after {
+                return true;
+            } else {
+                // if a number has been specified then we ignore entirely the time budget
+                return false;
+            }
+        }
+
+        self.started_at.elapsed() > self.budget
+    }
+}
+
 // Convert an absolute word position into a relative position.
 // Return the field id of the attribute related to the absolute position
 // and the relative position in the attribute.

milli/src/order_by_map.rs

@@ -0,0 +1,57 @@
+use std::collections::{hash_map, HashMap};
+use std::iter::FromIterator;
+
+use serde::{Deserialize, Deserializer, Serialize};
+
+use crate::OrderBy;
+
+#[derive(Serialize)]
+pub struct OrderByMap(HashMap<String, OrderBy>);
+
+impl OrderByMap {
+    pub fn get(&self, key: impl AsRef<str>) -> OrderBy {
+        self.0
+            .get(key.as_ref())
+            .copied()
+            .unwrap_or_else(|| self.0.get("*").copied().unwrap_or_default())
+    }
+
+    pub fn insert(&mut self, key: String, value: OrderBy) -> Option<OrderBy> {
+        self.0.insert(key, value)
+    }
+}
+
+impl Default for OrderByMap {
+    fn default() -> Self {
+        let mut map = HashMap::new();
+        map.insert("*".to_string(), OrderBy::Lexicographic);
+        OrderByMap(map)
+    }
+}
+
+impl FromIterator<(String, OrderBy)> for OrderByMap {
+    fn from_iter<T: IntoIterator<Item = (String, OrderBy)>>(iter: T) -> Self {
+        OrderByMap(iter.into_iter().collect())
+    }
+}
+
+impl IntoIterator for OrderByMap {
+    type Item = (String, OrderBy);
+    type IntoIter = hash_map::IntoIter<String, OrderBy>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.0.into_iter()
+    }
+}
+
+impl<'de> Deserialize<'de> for OrderByMap {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let mut map = Deserialize::deserialize(deserializer).map(OrderByMap)?;
+        // Insert the default ordering if it is not already overwritten by the user.
+        map.0.entry("*".to_string()).or_insert(OrderBy::default());
+        Ok(map)
+    }
+}

milli/src/score_details.rs

@@ -17,6 +17,9 @@ pub enum ScoreDetails {
     Sort(Sort),
     Vector(Vector),
     GeoSort(GeoSort),
+
+    /// Returned when we don't have the time to finish applying all the subsequent ranking-rules
+    Skipped,
 }
 
 #[derive(Clone, Copy)]
@@ -50,6 +53,7 @@ impl ScoreDetails {
             ScoreDetails::Sort(_) => None,
             ScoreDetails::GeoSort(_) => None,
             ScoreDetails::Vector(_) => None,
+            ScoreDetails::Skipped => Some(Rank { rank: 0, max_rank: 1 }),
         }
     }
 
@@ -94,9 +98,10 @@ impl ScoreDetails {
             ScoreDetails::ExactWords(e) => RankOrValue::Rank(e.rank()),
             ScoreDetails::Sort(sort) => RankOrValue::Sort(sort),
             ScoreDetails::GeoSort(geosort) => RankOrValue::GeoSort(geosort),
-            ScoreDetails::Vector(vector) => RankOrValue::Score(
-                vector.value_similarity.as_ref().map(|(_, s)| *s as f64).unwrap_or(0.0f64),
-            ),
+            ScoreDetails::Vector(vector) => {
+                RankOrValue::Score(vector.similarity.as_ref().map(|s| *s as f64).unwrap_or(0.0f64))
+            }
+            ScoreDetails::Skipped => RankOrValue::Rank(Rank { rank: 0, max_rank: 1 }),
         }
     }
 
@@ -244,16 +249,18 @@ impl ScoreDetails {
                     order += 1;
                 }
                 ScoreDetails::Vector(s) => {
-                    let vector = format!("vectorSort({:?})", s.target_vector);
-                    let value = s.value_similarity.as_ref().map(|(v, _)| v);
-                    let similarity = s.value_similarity.as_ref().map(|(_, s)| s);
+                    let similarity = s.similarity.as_ref();
 
                     let details = serde_json::json!({
                         "order": order,
-                        "value": value,
                         "similarity": similarity,
                     });
-                    details_map.insert(vector, details);
+                    details_map.insert("vectorSort".into(), details);
+                    order += 1;
+                }
+                ScoreDetails::Skipped => {
+                    details_map
+                        .insert("skipped".to_string(), serde_json::json!({ "order": order }));
                     order += 1;
                 }
             }
@@ -484,8 +491,7 @@ impl PartialOrd for GeoSort {
 
 #[derive(Debug, Clone, PartialEq, PartialOrd)]
 pub struct Vector {
-    pub target_vector: Vec<f32>,
-    pub value_similarity: Option<(Vec<f32>, f32)>,
+    pub similarity: Option<f32>,
 }
 
 impl GeoSort {

milli/src/search/facet/facet_distribution.rs

@@ -97,6 +97,7 @@ impl<'a> FacetDistribution<'a> {
     ) -> heed::Result<()> {
         match facet_type {
             FacetType::Number => {
+                let mut lexicographic_distribution = BTreeMap::new();
                 let mut key_buffer: Vec<_> = field_id.to_be_bytes().to_vec();
 
                 let distribution_prelength = distribution.len();
@@ -111,14 +112,17 @@ impl<'a> FacetDistribution<'a> {
 
                     for result in iter {
                         let ((_, _, value), ()) = result?;
-                        *distribution.entry(value.to_string()).or_insert(0) += 1;
+                        *lexicographic_distribution.entry(value.to_string()).or_insert(0) += 1;
 
-                        if distribution.len() - distribution_prelength == self.max_values_per_facet
+                        if lexicographic_distribution.len() - distribution_prelength
+                            == self.max_values_per_facet
                         {
                             break;
                         }
                     }
                 }
+
+                distribution.extend(lexicographic_distribution);
             }
             FacetType::String => {
                 let mut normalized_distribution = BTreeMap::new();

milli/src/search/facet/facet_range_search.rs

@@ -168,7 +168,7 @@ impl<'t, 'b, 'bitmap> FacetRangeSearch<'t, 'b, 'bitmap> {
             }
 
             // should we stop?
-            // We should if the the search range doesn't include any
+            // We should if the search range doesn't include any
             // element from the previous key or its successors
             let should_stop = {
                 match self.right {
@@ -232,7 +232,7 @@ impl<'t, 'b, 'bitmap> FacetRangeSearch<'t, 'b, 'bitmap> {
         }
 
         // should we stop?
-        // We should if the the search range doesn't include any
+        // We should if the search range doesn't include any
         // element from the previous key or its successors
         let should_stop = {
             match self.right {

milli/src/search/facet/mod.rs

@@ -6,15 +6,18 @@ use roaring::RoaringBitmap;
 
 pub use self::facet_distribution::{FacetDistribution, OrderBy, DEFAULT_VALUES_PER_FACET};
 pub use self::filter::{BadGeoError, Filter};
+pub use self::search::{FacetValueHit, SearchForFacetValues};
 use crate::heed_codec::facet::{FacetGroupKeyCodec, FacetGroupValueCodec, OrderedF64Codec};
 use crate::heed_codec::BytesRefCodec;
 use crate::{Index, Result};
+
 mod facet_distribution;
 mod facet_distribution_iter;
 mod facet_range_search;
 mod facet_sort_ascending;
 mod facet_sort_descending;
 mod filter;
+mod search;
 
 fn facet_extreme_value<'t>(
     mut extreme_it: impl Iterator<Item = heed::Result<(RoaringBitmap, &'t [u8])>> + 't,

milli/src/search/facet/search.rs

@@ -0,0 +1,332 @@
+use std::cmp::{Ordering, Reverse};
+use std::collections::BinaryHeap;
+use std::ops::ControlFlow;
+
+use charabia::normalizer::NormalizerOption;
+use charabia::Normalize;
+use fst::automaton::{Automaton, Str};
+use fst::{IntoStreamer, Streamer};
+use roaring::RoaringBitmap;
+use tracing::error;
+
+use crate::error::UserError;
+use crate::heed_codec::facet::{FacetGroupKey, FacetGroupValue};
+use crate::search::build_dfa;
+use crate::{DocumentId, FieldId, OrderBy, Result, Search};
+
+/// The maximum number of values per facet returned by the facet search route.
+const DEFAULT_MAX_NUMBER_OF_VALUES_PER_FACET: usize = 100;
+
+pub struct SearchForFacetValues<'a> {
+    query: Option<String>,
+    facet: String,
+    search_query: Search<'a>,
+    max_values: usize,
+    is_hybrid: bool,
+}
+
+impl<'a> SearchForFacetValues<'a> {
+    pub fn new(
+        facet: String,
+        search_query: Search<'a>,
+        is_hybrid: bool,
+    ) -> SearchForFacetValues<'a> {
+        SearchForFacetValues {
+            query: None,
+            facet,
+            search_query,
+            max_values: DEFAULT_MAX_NUMBER_OF_VALUES_PER_FACET,
+            is_hybrid,
+        }
+    }
+
+    pub fn query(&mut self, query: impl Into<String>) -> &mut Self {
+        self.query = Some(query.into());
+        self
+    }
+
+    pub fn max_values(&mut self, max: usize) -> &mut Self {
+        self.max_values = max;
+        self
+    }
+
+    fn one_original_value_of(
+        &self,
+        field_id: FieldId,
+        facet_str: &str,
+        any_docid: DocumentId,
+    ) -> Result<Option<String>> {
+        let index = self.search_query.index;
+        let rtxn = self.search_query.rtxn;
+        let key: (FieldId, _, &str) = (field_id, any_docid, facet_str);
+        Ok(index.field_id_docid_facet_strings.get(rtxn, &key)?.map(|v| v.to_owned()))
+    }
+
+    pub fn execute(&self) -> Result<Vec<FacetValueHit>> {
+        let index = self.search_query.index;
+        let rtxn = self.search_query.rtxn;
+
+        let filterable_fields = index.filterable_fields(rtxn)?;
+        if !filterable_fields.contains(&self.facet) {
+            let (valid_fields, hidden_fields) =
+                index.remove_hidden_fields(rtxn, filterable_fields)?;
+
+            return Err(UserError::InvalidFacetSearchFacetName {
+                field: self.facet.clone(),
+                valid_fields,
+                hidden_fields,
+            }
+            .into());
+        }
+
+        let fields_ids_map = index.fields_ids_map(rtxn)?;
+        let fid = match fields_ids_map.id(&self.facet) {
+            Some(fid) => fid,
+            // we return an empty list of results when the attribute has been
+            // set as filterable but no document contains this field (yet).
+            None => return Ok(Vec::new()),
+        };
+
+        let fst = match self.search_query.index.facet_id_string_fst.get(rtxn, &fid)? {
+            Some(fst) => fst,
+            None => return Ok(Vec::new()),
+        };
+
+        let search_candidates = self.search_query.execute_for_candidates(
+            self.is_hybrid
+                || self
+                    .search_query
+                    .semantic
+                    .as_ref()
+                    .and_then(|semantic| semantic.vector.as_ref())
+                    .is_some(),
+        )?;
+
+        let mut results = match index.sort_facet_values_by(rtxn)?.get(&self.facet) {
+            OrderBy::Lexicographic => ValuesCollection::by_lexicographic(self.max_values),
+            OrderBy::Count => ValuesCollection::by_count(self.max_values),
+        };
+
+        match self.query.as_ref() {
+            Some(query) => {
+                let options = NormalizerOption { lossy: true, ..Default::default() };
+                let query = query.normalize(&options);
+                let query = query.as_ref();
+
+                let authorize_typos = self.search_query.index.authorize_typos(rtxn)?;
+                let field_authorizes_typos =
+                    !self.search_query.index.exact_attributes_ids(rtxn)?.contains(&fid);
+
+                if authorize_typos && field_authorizes_typos {
+                    let exact_words_fst = self.search_query.index.exact_words(rtxn)?;
+                    if exact_words_fst.map_or(false, |fst| fst.contains(query)) {
+                        if fst.contains(query) {
+                            self.fetch_original_facets_using_normalized(
+                                fid,
+                                query,
+                                query,
+                                &search_candidates,
+                                &mut results,
+                            )?;
+                        }
+                    } else {
+                        let one_typo = self.search_query.index.min_word_len_one_typo(rtxn)?;
+                        let two_typos = self.search_query.index.min_word_len_two_typos(rtxn)?;
+
+                        let is_prefix = true;
+                        let automaton = if query.len() < one_typo as usize {
+                            build_dfa(query, 0, is_prefix)
+                        } else if query.len() < two_typos as usize {
+                            build_dfa(query, 1, is_prefix)
+                        } else {
+                            build_dfa(query, 2, is_prefix)
+                        };
+
+                        let mut stream = fst.search(automaton).into_stream();
+                        while let Some(facet_value) = stream.next() {
+                            let value = std::str::from_utf8(facet_value)?;
+                            if self
+                                .fetch_original_facets_using_normalized(
+                                    fid,
+                                    value,
+                                    query,
+                                    &search_candidates,
+                                    &mut results,
+                                )?
+                                .is_break()
+                            {
+                                break;
+                            }
+                        }
+                    }
+                } else {
+                    let automaton = Str::new(query).starts_with();
+                    let mut stream = fst.search(automaton).into_stream();
+                    while let Some(facet_value) = stream.next() {
+                        let value = std::str::from_utf8(facet_value)?;
+                        if self
+                            .fetch_original_facets_using_normalized(
+                                fid,
+                                value,
+                                query,
+                                &search_candidates,
+                                &mut results,
+                            )?
+                            .is_break()
+                        {
+                            break;
+                        }
+                    }
+                }
+            }
+            None => {
+                let prefix = FacetGroupKey { field_id: fid, level: 0, left_bound: "" };
+                for result in index.facet_id_string_docids.prefix_iter(rtxn, &prefix)? {
+                    let (FacetGroupKey { left_bound, .. }, FacetGroupValue { bitmap, .. }) =
+                        result?;
+                    let count = search_candidates.intersection_len(&bitmap);
+                    if count != 0 {
+                        let value = self
+                            .one_original_value_of(fid, left_bound, bitmap.min().unwrap())?
+                            .unwrap_or_else(|| left_bound.to_string());
+                        if results.insert(FacetValueHit { value, count }).is_break() {
+                            break;
+                        }
+                    }
+                }
+            }
+        }
+
+        Ok(results.into_sorted_vec())
+    }
+
+    fn fetch_original_facets_using_normalized(
+        &self,
+        fid: FieldId,
+        value: &str,
+        query: &str,
+        search_candidates: &RoaringBitmap,
+        results: &mut ValuesCollection,
+    ) -> Result<ControlFlow<()>> {
+        let index = self.search_query.index;
+        let rtxn = self.search_query.rtxn;
+
+        let database = index.facet_id_normalized_string_strings;
+        let key = (fid, value);
+        let original_strings = match database.get(rtxn, &key)? {
+            Some(original_strings) => original_strings,
+            None => {
+                error!("the facet value is missing from the facet database: {key:?}");
+                return Ok(ControlFlow::Continue(()));
+            }
+        };
+        for original in original_strings {
+            let key = FacetGroupKey { field_id: fid, level: 0, left_bound: original.as_str() };
+            let docids = match index.facet_id_string_docids.get(rtxn, &key)? {
+                Some(FacetGroupValue { bitmap, .. }) => bitmap,
+                None => {
+                    error!("the facet value is missing from the facet database: {key:?}");
+                    return Ok(ControlFlow::Continue(()));
+                }
+            };
+            let count = search_candidates.intersection_len(&docids);
+            if count != 0 {
+                let value = self
+                    .one_original_value_of(fid, &original, docids.min().unwrap())?
+                    .unwrap_or_else(|| query.to_string());
+                if results.insert(FacetValueHit { value, count }).is_break() {
+                    break;
+                }
+            }
+        }
+
+        Ok(ControlFlow::Continue(()))
+    }
+}
+
+#[derive(Debug, Clone, serde::Serialize, PartialEq)]
+pub struct FacetValueHit {
+    /// The original facet value
+    pub value: String,
+    /// The number of documents associated to this facet
+    pub count: u64,
+}
+
+impl PartialOrd for FacetValueHit {
+    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+        Some(self.cmp(other))
+    }
+}
+
+impl Ord for FacetValueHit {
+    fn cmp(&self, other: &Self) -> Ordering {
+        self.count.cmp(&other.count).then_with(|| self.value.cmp(&other.value))
+    }
+}
+
+impl Eq for FacetValueHit {}
+
+/// A wrapper type that collects the best facet values by
+/// lexicographic or number of associated values.
+enum ValuesCollection {
+    /// Keeps the top values according to the lexicographic order.
+    Lexicographic { max: usize, content: Vec<FacetValueHit> },
+    /// Keeps the top values according to the number of values associated to them.
+    ///
+    /// Note that it is a max heap and we need to move the smallest counts
+    /// at the top to be able to pop them when we reach the max_values limit.
+    Count { max: usize, content: BinaryHeap<Reverse<FacetValueHit>> },
+}
+
+impl ValuesCollection {
+    pub fn by_lexicographic(max: usize) -> Self {
+        ValuesCollection::Lexicographic { max, content: Vec::new() }
+    }
+
+    pub fn by_count(max: usize) -> Self {
+        ValuesCollection::Count { max, content: BinaryHeap::new() }
+    }
+
+    pub fn insert(&mut self, value: FacetValueHit) -> ControlFlow<()> {
+        match self {
+            ValuesCollection::Lexicographic { max, content } => {
+                if content.len() < *max {
+                    content.push(value);
+                    if content.len() < *max {
+                        return ControlFlow::Continue(());
+                    }
+                }
+                ControlFlow::Break(())
+            }
+            ValuesCollection::Count { max, content } => {
+                if content.len() == *max {
+                    // Peeking gives us the worst value in the list as
+                    // this is a max-heap and we reversed it.
+                    let Some(mut peek) = content.peek_mut() else { return ControlFlow::Break(()) };
+                    if peek.0.count <= value.count {
+                        // Replace the current worst value in the heap
+                        // with the new one we received that is better.
+                        *peek = Reverse(value);
+                    }
+                } else {
+                    content.push(Reverse(value));
+                }
+                ControlFlow::Continue(())
+            }
+        }
+    }
+
+    /// Returns the list of facet values in descending order of, either,
+    /// count or lexicographic order of the value depending on the type.
+    pub fn into_sorted_vec(self) -> Vec<FacetValueHit> {
+        match self {
+            ValuesCollection::Lexicographic { content, .. } => content.into_iter().collect(),
+            ValuesCollection::Count { content, .. } => {
+                // Convert the heap into a vec of hits by removing the Reverse wrapper.
+                // Hits are already in the right order as they were reversed and there
+                // are output in ascending order.
+                content.into_sorted_vec().into_iter().map(|Reverse(hit)| hit).collect()
+            }
+        }
+    }
+}

milli/src/search/hybrid.rs

@@ -4,12 +4,15 @@ use itertools::Itertools;
 use roaring::RoaringBitmap;
 
 use crate::score_details::{ScoreDetails, ScoreValue, ScoringStrategy};
+use crate::search::SemanticSearch;
 use crate::{MatchingWords, Result, Search, SearchResult};
 
 struct ScoreWithRatioResult {
     matching_words: MatchingWords,
     candidates: RoaringBitmap,
     document_scores: Vec<(u32, ScoreWithRatio)>,
+    degraded: bool,
+    used_negative_operator: bool,
 }
 
 type ScoreWithRatio = (Vec<ScoreDetails>, f32);
@@ -49,8 +52,12 @@ fn compare_scores(
                     order => return order,
                 }
             }
-            (Some(ScoreValue::Score(_)), Some(_)) => return Ordering::Greater,
-            (Some(_), Some(ScoreValue::Score(_))) => return Ordering::Less,
+            (Some(ScoreValue::Score(x)), Some(_)) => {
+                return if x == 0. { Ordering::Less } else { Ordering::Greater }
+            }
+            (Some(_), Some(ScoreValue::Score(x))) => {
+                return if x == 0. { Ordering::Greater } else { Ordering::Less }
+            }
             // if we have this, we're bad
             (Some(ScoreValue::GeoSort(_)), Some(ScoreValue::Sort(_)))
             | (Some(ScoreValue::Sort(_)), Some(ScoreValue::GeoSort(_))) => {
@@ -72,51 +79,82 @@ impl ScoreWithRatioResult {
             matching_words: results.matching_words,
             candidates: results.candidates,
             document_scores,
+            degraded: results.degraded,
+            used_negative_operator: results.used_negative_operator,
         }
     }
 
-    fn merge(left: Self, right: Self, from: usize, length: usize) -> SearchResult {
-        let mut documents_ids =
-            Vec::with_capacity(left.document_scores.len() + right.document_scores.len());
-        let mut document_scores =
-            Vec::with_capacity(left.document_scores.len() + right.document_scores.len());
+    fn merge(
+        vector_results: Self,
+        keyword_results: Self,
+        from: usize,
+        length: usize,
+    ) -> (SearchResult, u32) {
+        #[derive(Clone, Copy)]
+        enum ResultSource {
+            Semantic,
+            Keyword,
+        }
+        let mut semantic_hit_count = 0;
+
+        let mut documents_ids = Vec::with_capacity(
+            vector_results.document_scores.len() + keyword_results.document_scores.len(),
+        );
+        let mut document_scores = Vec::with_capacity(
+            vector_results.document_scores.len() + keyword_results.document_scores.len(),
+        );
 
         let mut documents_seen = RoaringBitmap::new();
-        for (docid, (main_score, _sub_score)) in left
+        for ((docid, (main_score, _sub_score)), source) in vector_results
             .document_scores
             .into_iter()
-            .merge_by(right.document_scores.into_iter(), |(_, left), (_, right)| {
-                // the first value is the one with the greatest score
-                compare_scores(left, right).is_ge()
-            })
+            .zip(std::iter::repeat(ResultSource::Semantic))
+            .merge_by(
+                keyword_results
+                    .document_scores
+                    .into_iter()
+                    .zip(std::iter::repeat(ResultSource::Keyword)),
+                |((_, left), _), ((_, right), _)| {
+                    // the first value is the one with the greatest score
+                    compare_scores(left, right).is_ge()
+                },
+            )
             // remove documents we already saw
-            .filter(|(docid, _)| documents_seen.insert(*docid))
+            .filter(|((docid, _), _)| documents_seen.insert(*docid))
             // start skipping **after** the filter
             .skip(from)
             // take **after** skipping
             .take(length)
         {
+            if let ResultSource::Semantic = source {
+                semantic_hit_count += 1;
+            }
             documents_ids.push(docid);
             // TODO: pass both scores to documents_score in some way?
             document_scores.push(main_score);
         }
 
-        SearchResult {
-            matching_words: right.matching_words,
-            candidates: left.candidates | right.candidates,
-            documents_ids,
-            document_scores,
-        }
+        (
+            SearchResult {
+                matching_words: keyword_results.matching_words,
+                candidates: vector_results.candidates | keyword_results.candidates,
+                documents_ids,
+                document_scores,
+                degraded: vector_results.degraded | keyword_results.degraded,
+                used_negative_operator: vector_results.used_negative_operator
+                    | keyword_results.used_negative_operator,
+            },
+            semantic_hit_count,
+        )
     }
 }
 
 impl<'a> Search<'a> {
-    pub fn execute_hybrid(&self, semantic_ratio: f32) -> Result<SearchResult> {
+    pub fn execute_hybrid(&self, semantic_ratio: f32) -> Result<(SearchResult, Option<u32>)> {
         // TODO: find classier way to achieve that than to reset vector and query params
         // create separate keyword and semantic searches
         let mut search = Search {
             query: self.query.clone(),
-            vector: self.vector.clone(),
             filter: self.filter.clone(),
             offset: 0,
             limit: self.limit + self.offset,
@@ -129,25 +167,43 @@ impl<'a> Search<'a> {
             exhaustive_number_hits: self.exhaustive_number_hits,
             rtxn: self.rtxn,
             index: self.index,
-            distribution_shift: self.distribution_shift,
-            embedder_name: self.embedder_name.clone(),
+            semantic: self.semantic.clone(),
+            time_budget: self.time_budget.clone(),
         };
 
-        let vector_query = search.vector.take();
+        let semantic = search.semantic.take();
         let keyword_results = search.execute()?;
 
-        // skip semantic search if we don't have a vector query (placeholder search)
-        let Some(vector_query) = vector_query else {
-            return Ok(keyword_results);
-        };
-
         // completely skip semantic search if the results of the keyword search are good enough
         if self.results_good_enough(&keyword_results, semantic_ratio) {
-            return Ok(keyword_results);
+            return Ok((keyword_results, Some(0)));
         }
 
-        search.vector = Some(vector_query);
-        search.query = None;
+        // no vector search against placeholder search
+        let Some(query) = search.query.take() else {
+            return Ok((keyword_results, Some(0)));
+        };
+        // no embedder, no semantic search
+        let Some(SemanticSearch { vector, embedder_name, embedder }) = semantic else {
+            return Ok((keyword_results, Some(0)));
+        };
+
+        let vector_query = match vector {
+            Some(vector_query) => vector_query,
+            None => {
+                // attempt to embed the vector
+                match embedder.embed_one(query) {
+                    Ok(embedding) => embedding,
+                    Err(error) => {
+                        tracing::error!(error=%error, "Embedding failed");
+                        return Ok((keyword_results, Some(0)));
+                    }
+                }
+            }
+        };
+
+        search.semantic =
+            Some(SemanticSearch { vector: Some(vector_query), embedder_name, embedder });
 
         // TODO: would be better to have two distinct functions at this point
         let vector_results = search.execute()?;
@@ -155,10 +211,10 @@ impl<'a> Search<'a> {
         let keyword_results = ScoreWithRatioResult::new(keyword_results, 1.0 - semantic_ratio);
         let vector_results = ScoreWithRatioResult::new(vector_results, semantic_ratio);
 
-        let merge_results =
+        let (merge_results, semantic_hit_count) =
             ScoreWithRatioResult::merge(vector_results, keyword_results, self.offset, self.limit);
         assert!(merge_results.documents_ids.len() <= self.limit);
-        Ok(merge_results)
+        Ok((merge_results, Some(semantic_hit_count)))
     }
 
     fn results_good_enough(&self, keyword_results: &SearchResult, semantic_ratio: f32) -> bool {

milli/src/search/mod.rs

@@ -1,25 +1,18 @@
 use std::fmt;
-use std::ops::ControlFlow;
+use std::sync::Arc;
 
-use charabia::normalizer::NormalizerOption;
-use charabia::Normalize;
-use fst::automaton::{Automaton, Str};
-use fst::{IntoStreamer, Streamer};
 use levenshtein_automata::{LevenshteinAutomatonBuilder as LevBuilder, DFA};
 use once_cell::sync::Lazy;
 use roaring::bitmap::RoaringBitmap;
-use tracing::error;
 
 pub use self::facet::{FacetDistribution, Filter, OrderBy, DEFAULT_VALUES_PER_FACET};
 pub use self::new::matches::{FormatOptions, MatchBounds, MatcherBuilder, MatchingWords};
 use self::new::{execute_vector_search, PartialSearchResult};
-use crate::error::UserError;
-use crate::heed_codec::facet::{FacetGroupKey, FacetGroupValue};
 use crate::score_details::{ScoreDetails, ScoringStrategy};
-use crate::vector::DistributionShift;
+use crate::vector::Embedder;
 use crate::{
-    execute_search, filtered_universe, AscDesc, DefaultSearchLogger, DocumentId, FieldId, Index,
-    Result, SearchContext,
+    execute_search, filtered_universe, AscDesc, DefaultSearchLogger, DocumentId, Index, Result,
+    SearchContext, TimeBudget,
 };
 
 // Building these factories is not free.
@@ -27,17 +20,20 @@ static LEVDIST0: Lazy<LevBuilder> = Lazy::new(|| LevBuilder::new(0, true));
 static LEVDIST1: Lazy<LevBuilder> = Lazy::new(|| LevBuilder::new(1, true));
 static LEVDIST2: Lazy<LevBuilder> = Lazy::new(|| LevBuilder::new(2, true));
 
-/// The maximum number of values per facet returned by the facet search route.
-const DEFAULT_MAX_NUMBER_OF_VALUES_PER_FACET: usize = 100;
-
 pub mod facet;
 mod fst_utils;
 pub mod hybrid;
 pub mod new;
 
+#[derive(Debug, Clone)]
+pub struct SemanticSearch {
+    vector: Option<Vec<f32>>,
+    embedder_name: String,
+    embedder: Arc<Embedder>,
+}
+
 pub struct Search<'a> {
     query: Option<String>,
-    vector: Option<Vec<f32>>,
     // this should be linked to the String in the query
     filter: Option<Filter<'a>>,
     offset: usize,
@@ -49,18 +45,16 @@ pub struct Search<'a> {
     scoring_strategy: ScoringStrategy,
     words_limit: usize,
     exhaustive_number_hits: bool,
-    /// TODO: Add semantic ratio or pass it directly to execute_hybrid()
     rtxn: &'a heed::RoTxn<'a>,
     index: &'a Index,
-    distribution_shift: Option<DistributionShift>,
-    embedder_name: Option<String>,
+    semantic: Option<SemanticSearch>,
+    time_budget: TimeBudget,
 }
 
 impl<'a> Search<'a> {
     pub fn new(rtxn: &'a heed::RoTxn, index: &'a Index) -> Search<'a> {
         Search {
             query: None,
-            vector: None,
             filter: None,
             offset: 0,
             limit: 20,
@@ -73,8 +67,8 @@ impl<'a> Search<'a> {
             words_limit: 10,
             rtxn,
             index,
-            distribution_shift: None,
-            embedder_name: None,
+            semantic: None,
+            time_budget: TimeBudget::max(),
         }
     }
 
@@ -83,8 +77,13 @@ impl<'a> Search<'a> {
         self
     }
 
-    pub fn vector(&mut self, vector: Vec<f32>) -> &mut Search<'a> {
-        self.vector = Some(vector);
+    pub fn semantic(
+        &mut self,
+        embedder_name: String,
+        embedder: Arc<Embedder>,
+        vector: Option<Vec<f32>>,
+    ) -> &mut Search<'a> {
+        self.semantic = Some(SemanticSearch { embedder_name, embedder, vector });
         self
     }
 
@@ -141,16 +140,8 @@ impl<'a> Search<'a> {
         self
     }
 
-    pub fn distribution_shift(
-        &mut self,
-        distribution_shift: Option<DistributionShift>,
-    ) -> &mut Search<'a> {
-        self.distribution_shift = distribution_shift;
-        self
-    }
-
-    pub fn embedder_name(&mut self, embedder_name: impl Into<String>) -> &mut Search<'a> {
-        self.embedder_name = Some(embedder_name.into());
+    pub fn time_budget(&mut self, time_budget: TimeBudget) -> &mut Search<'a> {
+        self.time_budget = time_budget;
         self
     }
 
@@ -164,15 +155,6 @@ impl<'a> Search<'a> {
     }
 
     pub fn execute(&self) -> Result<SearchResult> {
-        let embedder_name;
-        let embedder_name = match &self.embedder_name {
-            Some(embedder_name) => embedder_name,
-            None => {
-                embedder_name = self.index.default_embedding_name(self.rtxn)?;
-                &embedder_name
-            }
-        };
-
         let mut ctx = SearchContext::new(self.index, self.rtxn);
 
         if let Some(searchable_attributes) = self.searchable_attributes {
@@ -180,9 +162,16 @@ impl<'a> Search<'a> {
         }
 
         let universe = filtered_universe(&ctx, &self.filter)?;
-        let PartialSearchResult { located_query_terms, candidates, documents_ids, document_scores } =
-            match self.vector.as_ref() {
-                Some(vector) => execute_vector_search(
+        let PartialSearchResult {
+            located_query_terms,
+            candidates,
+            documents_ids,
+            document_scores,
+            degraded,
+            used_negative_operator,
+        } = match self.semantic.as_ref() {
+            Some(SemanticSearch { vector: Some(vector), embedder_name, embedder }) => {
+                execute_vector_search(
                     &mut ctx,
                     vector,
                     self.scoring_strategy,
@@ -191,25 +180,28 @@ impl<'a> Search<'a> {
                     self.geo_strategy,
                     self.offset,
                     self.limit,
-                    self.distribution_shift,
                     embedder_name,
-                )?,
-                None => execute_search(
-                    &mut ctx,
-                    self.query.as_deref(),
-                    self.terms_matching_strategy,
-                    self.scoring_strategy,
-                    self.exhaustive_number_hits,
-                    universe,
-                    &self.sort_criteria,
-                    self.geo_strategy,
-                    self.offset,
-                    self.limit,
-                    Some(self.words_limit),
-                    &mut DefaultSearchLogger,
-                    &mut DefaultSearchLogger,
-                )?,
-            };
+                    embedder,
+                    self.time_budget.clone(),
+                )?
+            }
+            _ => execute_search(
+                &mut ctx,
+                self.query.as_deref(),
+                self.terms_matching_strategy,
+                self.scoring_strategy,
+                self.exhaustive_number_hits,
+                universe,
+                &self.sort_criteria,
+                self.geo_strategy,
+                self.offset,
+                self.limit,
+                Some(self.words_limit),
+                &mut DefaultSearchLogger,
+                &mut DefaultSearchLogger,
+                self.time_budget.clone(),
+            )?,
+        };
 
         // consume context and located_query_terms to build MatchingWords.
         let matching_words = match located_query_terms {
@@ -217,7 +209,14 @@ impl<'a> Search<'a> {
             None => MatchingWords::default(),
         };
 
-        Ok(SearchResult { matching_words, candidates, document_scores, documents_ids })
+        Ok(SearchResult {
+            matching_words,
+            candidates,
+            document_scores,
+            documents_ids,
+            degraded,
+            used_negative_operator,
+        })
     }
 }
 
@@ -225,7 +224,6 @@ impl fmt::Debug for Search<'_> {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         let Search {
             query,
-            vector: _,
             filter,
             offset,
             limit,
@@ -238,8 +236,8 @@ impl fmt::Debug for Search<'_> {
             exhaustive_number_hits,
             rtxn: _,
             index: _,
-            distribution_shift,
-            embedder_name,
+            semantic,
+            time_budget,
         } = self;
         f.debug_struct("Search")
             .field("query", query)
@@ -253,8 +251,11 @@ impl fmt::Debug for Search<'_> {
             .field("scoring_strategy", scoring_strategy)
             .field("exhaustive_number_hits", exhaustive_number_hits)
             .field("words_limit", words_limit)
-            .field("distribution_shift", distribution_shift)
-            .field("embedder_name", embedder_name)
+            .field(
+                "semantic.embedder_name",
+                &semantic.as_ref().map(|semantic| &semantic.embedder_name),
+            )
+            .field("time_budget", time_budget)
             .finish()
     }
 }
@@ -265,6 +266,8 @@ pub struct SearchResult {
     pub candidates: RoaringBitmap,
     pub documents_ids: Vec<DocumentId>,
     pub document_scores: Vec<Vec<ScoreDetails>>,
+    pub degraded: bool,
+    pub used_negative_operator: bool,
 }
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -302,240 +305,6 @@ pub fn build_dfa(word: &str, typos: u8, is_prefix: bool) -> DFA {
     }
 }
 
-pub struct SearchForFacetValues<'a> {
-    query: Option<String>,
-    facet: String,
-    search_query: Search<'a>,
-    max_values: usize,
-    is_hybrid: bool,
-}
-
-impl<'a> SearchForFacetValues<'a> {
-    pub fn new(
-        facet: String,
-        search_query: Search<'a>,
-        is_hybrid: bool,
-    ) -> SearchForFacetValues<'a> {
-        SearchForFacetValues {
-            query: None,
-            facet,
-            search_query,
-            max_values: DEFAULT_MAX_NUMBER_OF_VALUES_PER_FACET,
-            is_hybrid,
-        }
-    }
-
-    pub fn query(&mut self, query: impl Into<String>) -> &mut Self {
-        self.query = Some(query.into());
-        self
-    }
-
-    pub fn max_values(&mut self, max: usize) -> &mut Self {
-        self.max_values = max;
-        self
-    }
-
-    fn one_original_value_of(
-        &self,
-        field_id: FieldId,
-        facet_str: &str,
-        any_docid: DocumentId,
-    ) -> Result<Option<String>> {
-        let index = self.search_query.index;
-        let rtxn = self.search_query.rtxn;
-        let key: (FieldId, _, &str) = (field_id, any_docid, facet_str);
-        Ok(index.field_id_docid_facet_strings.get(rtxn, &key)?.map(|v| v.to_owned()))
-    }
-
-    pub fn execute(&self) -> Result<Vec<FacetValueHit>> {
-        let index = self.search_query.index;
-        let rtxn = self.search_query.rtxn;
-
-        let filterable_fields = index.filterable_fields(rtxn)?;
-        if !filterable_fields.contains(&self.facet) {
-            let (valid_fields, hidden_fields) =
-                index.remove_hidden_fields(rtxn, filterable_fields)?;
-
-            return Err(UserError::InvalidFacetSearchFacetName {
-                field: self.facet.clone(),
-                valid_fields,
-                hidden_fields,
-            }
-            .into());
-        }
-
-        let fields_ids_map = index.fields_ids_map(rtxn)?;
-        let fid = match fields_ids_map.id(&self.facet) {
-            Some(fid) => fid,
-            // we return an empty list of results when the attribute has been
-            // set as filterable but no document contains this field (yet).
-            None => return Ok(Vec::new()),
-        };
-
-        let fst = match self.search_query.index.facet_id_string_fst.get(rtxn, &fid)? {
-            Some(fst) => fst,
-            None => return Ok(vec![]),
-        };
-
-        let search_candidates = self
-            .search_query
-            .execute_for_candidates(self.is_hybrid || self.search_query.vector.is_some())?;
-
-        match self.query.as_ref() {
-            Some(query) => {
-                let options = NormalizerOption { lossy: true, ..Default::default() };
-                let query = query.normalize(&options);
-                let query = query.as_ref();
-
-                let authorize_typos = self.search_query.index.authorize_typos(rtxn)?;
-                let field_authorizes_typos =
-                    !self.search_query.index.exact_attributes_ids(rtxn)?.contains(&fid);
-
-                if authorize_typos && field_authorizes_typos {
-                    let exact_words_fst = self.search_query.index.exact_words(rtxn)?;
-                    if exact_words_fst.map_or(false, |fst| fst.contains(query)) {
-                        let mut results = vec![];
-                        if fst.contains(query) {
-                            self.fetch_original_facets_using_normalized(
-                                fid,
-                                query,
-                                query,
-                                &search_candidates,
-                                &mut results,
-                            )?;
-                        }
-                        Ok(results)
-                    } else {
-                        let one_typo = self.search_query.index.min_word_len_one_typo(rtxn)?;
-                        let two_typos = self.search_query.index.min_word_len_two_typos(rtxn)?;
-
-                        let is_prefix = true;
-                        let automaton = if query.len() < one_typo as usize {
-                            build_dfa(query, 0, is_prefix)
-                        } else if query.len() < two_typos as usize {
-                            build_dfa(query, 1, is_prefix)
-                        } else {
-                            build_dfa(query, 2, is_prefix)
-                        };
-
-                        let mut stream = fst.search(automaton).into_stream();
-                        let mut results = vec![];
-                        while let Some(facet_value) = stream.next() {
-                            let value = std::str::from_utf8(facet_value)?;
-                            if self
-                                .fetch_original_facets_using_normalized(
-                                    fid,
-                                    value,
-                                    query,
-                                    &search_candidates,
-                                    &mut results,
-                                )?
-                                .is_break()
-                            {
-                                break;
-                            }
-                        }
-
-                        Ok(results)
-                    }
-                } else {
-                    let automaton = Str::new(query).starts_with();
-                    let mut stream = fst.search(automaton).into_stream();
-                    let mut results = vec![];
-                    while let Some(facet_value) = stream.next() {
-                        let value = std::str::from_utf8(facet_value)?;
-                        if self
-                            .fetch_original_facets_using_normalized(
-                                fid,
-                                value,
-                                query,
-                                &search_candidates,
-                                &mut results,
-                            )?
-                            .is_break()
-                        {
-                            break;
-                        }
-                    }
-
-                    Ok(results)
-                }
-            }
-            None => {
-                let mut results = vec![];
-                let prefix = FacetGroupKey { field_id: fid, level: 0, left_bound: "" };
-                for result in index.facet_id_string_docids.prefix_iter(rtxn, &prefix)? {
-                    let (FacetGroupKey { left_bound, .. }, FacetGroupValue { bitmap, .. }) =
-                        result?;
-                    let count = search_candidates.intersection_len(&bitmap);
-                    if count != 0 {
-                        let value = self
-                            .one_original_value_of(fid, left_bound, bitmap.min().unwrap())?
-                            .unwrap_or_else(|| left_bound.to_string());
-                        results.push(FacetValueHit { value, count });
-                    }
-                    if results.len() >= self.max_values {
-                        break;
-                    }
-                }
-                Ok(results)
-            }
-        }
-    }
-
-    fn fetch_original_facets_using_normalized(
-        &self,
-        fid: FieldId,
-        value: &str,
-        query: &str,
-        search_candidates: &RoaringBitmap,
-        results: &mut Vec<FacetValueHit>,
-    ) -> Result<ControlFlow<()>> {
-        let index = self.search_query.index;
-        let rtxn = self.search_query.rtxn;
-
-        let database = index.facet_id_normalized_string_strings;
-        let key = (fid, value);
-        let original_strings = match database.get(rtxn, &key)? {
-            Some(original_strings) => original_strings,
-            None => {
-                error!("the facet value is missing from the facet database: {key:?}");
-                return Ok(ControlFlow::Continue(()));
-            }
-        };
-        for original in original_strings {
-            let key = FacetGroupKey { field_id: fid, level: 0, left_bound: original.as_str() };
-            let docids = match index.facet_id_string_docids.get(rtxn, &key)? {
-                Some(FacetGroupValue { bitmap, .. }) => bitmap,
-                None => {
-                    error!("the facet value is missing from the facet database: {key:?}");
-                    return Ok(ControlFlow::Continue(()));
-                }
-            };
-            let count = search_candidates.intersection_len(&docids);
-            if count != 0 {
-                let value = self
-                    .one_original_value_of(fid, &original, docids.min().unwrap())?
-                    .unwrap_or_else(|| query.to_string());
-                results.push(FacetValueHit { value, count });
-            }
-            if results.len() >= self.max_values {
-                return Ok(ControlFlow::Break(()));
-            }
-        }
-
-        Ok(ControlFlow::Continue(()))
-    }
-}
-
-#[derive(Debug, Clone, serde::Serialize, PartialEq)]
-pub struct FacetValueHit {
-    /// The original facet value
-    pub value: String,
-    /// The number of documents associated to this facet
-    pub count: u64,
-}
-
 #[cfg(test)]
 mod test {
     #[allow(unused_imports)]

milli/src/search/new/bucket_sort.rs

@@ -5,12 +5,14 @@ use super::ranking_rules::{BoxRankingRule, RankingRuleQueryTrait};
 use super::SearchContext;
 use crate::score_details::{ScoreDetails, ScoringStrategy};
 use crate::search::new::distinct::{apply_distinct_rule, distinct_single_docid, DistinctOutput};
-use crate::Result;
+use crate::{Result, TimeBudget};
 
 pub struct BucketSortOutput {
     pub docids: Vec<u32>,
     pub scores: Vec<Vec<ScoreDetails>>,
     pub all_candidates: RoaringBitmap,
+
+    pub degraded: bool,
 }
 
 // TODO: would probably be good to regroup some of these inside of a struct?
@@ -25,6 +27,7 @@ pub fn bucket_sort<'ctx, Q: RankingRuleQueryTrait>(
     length: usize,
     scoring_strategy: ScoringStrategy,
     logger: &mut dyn SearchLogger<Q>,
+    time_budget: TimeBudget,
 ) -> Result<BucketSortOutput> {
     logger.initial_query(query);
     logger.ranking_rules(&ranking_rules);
@@ -41,6 +44,7 @@ pub fn bucket_sort<'ctx, Q: RankingRuleQueryTrait>(
             docids: vec![],
             scores: vec![],
             all_candidates: universe.clone(),
+            degraded: false,
         });
     }
     if ranking_rules.is_empty() {
@@ -74,6 +78,7 @@ pub fn bucket_sort<'ctx, Q: RankingRuleQueryTrait>(
                 scores: vec![Default::default(); results.len()],
                 docids: results,
                 all_candidates,
+                degraded: false,
             });
         } else {
             let docids: Vec<u32> = universe.iter().skip(from).take(length).collect();
@@ -81,6 +86,7 @@ pub fn bucket_sort<'ctx, Q: RankingRuleQueryTrait>(
                 scores: vec![Default::default(); docids.len()],
                 docids,
                 all_candidates: universe.clone(),
+                degraded: false,
             });
         };
     }
@@ -154,6 +160,28 @@ pub fn bucket_sort<'ctx, Q: RankingRuleQueryTrait>(
     }
 
     while valid_docids.len() < length {
+        if time_budget.exceeded() {
+            loop {
+                let bucket = std::mem::take(&mut ranking_rule_universes[cur_ranking_rule_index]);
+                ranking_rule_scores.push(ScoreDetails::Skipped);
+                maybe_add_to_results!(bucket);
+                ranking_rule_scores.pop();
+
+                if cur_ranking_rule_index == 0 {
+                    break;
+                }
+
+                back!();
+            }
+
+            return Ok(BucketSortOutput {
+                scores: valid_scores,
+                docids: valid_docids,
+                all_candidates,
+                degraded: true,
+            });
+        }
+
         // The universe for this bucket is zero, so we don't need to sort
         // anything, just go back to the parent ranking rule.
         if ranking_rule_universes[cur_ranking_rule_index].is_empty()
@@ -219,7 +247,12 @@ pub fn bucket_sort<'ctx, Q: RankingRuleQueryTrait>(
         )?;
     }
 
-    Ok(BucketSortOutput { docids: valid_docids, scores: valid_scores, all_candidates })
+    Ok(BucketSortOutput {
+        docids: valid_docids,
+        scores: valid_scores,
+        all_candidates,
+        degraded: false,
+    })
 }
 
 /// Add the candidates to the results. Take `distinct`, `from`, `length`, and `cur_offset`

milli/src/search/new/matches/matching_words.rs

@@ -240,6 +240,7 @@ pub(crate) mod tests {
     use super::super::super::located_query_terms_from_tokens;
     use super::*;
     use crate::index::tests::TempIndex;
+    use crate::search::new::query_term::ExtractedTokens;
 
     pub(crate) fn temp_index_with_documents() -> TempIndex {
         let temp_index = TempIndex::new();
@@ -261,7 +262,8 @@ pub(crate) mod tests {
         let mut builder = TokenizerBuilder::default();
         let tokenizer = builder.build();
         let tokens = tokenizer.tokenize("split this world");
-        let query_terms = located_query_terms_from_tokens(&mut ctx, tokens, None).unwrap();
+        let ExtractedTokens { query_terms, .. } =
+            located_query_terms_from_tokens(&mut ctx, tokens, None).unwrap();
         let matching_words = MatchingWords::new(ctx, query_terms);
 
         assert_eq!(

milli/src/search/new/matches/mod.rs

@@ -502,7 +502,7 @@ mod tests {
 
     use super::*;
     use crate::index::tests::TempIndex;
-    use crate::{execute_search, filtered_universe, SearchContext};
+    use crate::{execute_search, filtered_universe, SearchContext, TimeBudget};
 
     impl<'a> MatcherBuilder<'a> {
         fn new_test(rtxn: &'a heed::RoTxn, index: &'a TempIndex, query: &str) -> Self {
@@ -522,6 +522,7 @@ mod tests {
                 Some(10),
                 &mut crate::DefaultSearchLogger,
                 &mut crate::DefaultSearchLogger,
+                TimeBudget::max(),
             )
             .unwrap();
 

milli/src/search/new/mod.rs

@@ -33,7 +33,9 @@ use interner::{DedupInterner, Interner};
 pub use logger::visual::VisualSearchLogger;
 pub use logger::{DefaultSearchLogger, SearchLogger};
 use query_graph::{QueryGraph, QueryNode};
-use query_term::{located_query_terms_from_tokens, LocatedQueryTerm, Phrase, QueryTerm};
+use query_term::{
+    located_query_terms_from_tokens, ExtractedTokens, LocatedQueryTerm, Phrase, QueryTerm,
+};
 use ranking_rules::{
     BoxRankingRule, PlaceholderQuery, RankingRule, RankingRuleOutput, RankingRuleQueryTrait,
 };
@@ -50,9 +52,10 @@ use self::vector_sort::VectorSort;
 use crate::error::FieldIdMapMissingEntry;
 use crate::score_details::{ScoreDetails, ScoringStrategy};
 use crate::search::new::distinct::apply_distinct_rule;
-use crate::vector::DistributionShift;
+use crate::vector::Embedder;
 use crate::{
-    AscDesc, DocumentId, FieldId, Filter, Index, Member, Result, TermsMatchingStrategy, UserError,
+    AscDesc, DocumentId, FieldId, Filter, Index, Member, Result, TermsMatchingStrategy, TimeBudget,
+    UserError,
 };
 
 /// A structure used throughout the execution of a search query.
@@ -208,6 +211,35 @@ fn resolve_universe(
     )
 }
 
+#[tracing::instrument(level = "trace", skip_all, target = "search")]
+fn resolve_negative_words(
+    ctx: &mut SearchContext,
+    negative_words: &[Word],
+) -> Result<RoaringBitmap> {
+    let mut negative_bitmap = RoaringBitmap::new();
+    for &word in negative_words {
+        if let Some(bitmap) = ctx.word_docids(word)? {
+            negative_bitmap |= bitmap;
+        }
+    }
+    Ok(negative_bitmap)
+}
+
+#[tracing::instrument(level = "trace", skip_all, target = "search")]
+fn resolve_negative_phrases(
+    ctx: &mut SearchContext,
+    negative_phrases: &[LocatedQueryTerm],
+) -> Result<RoaringBitmap> {
+    let mut negative_bitmap = RoaringBitmap::new();
+    for term in negative_phrases {
+        let query_term = ctx.term_interner.get(term.value);
+        if let Some(phrase) = query_term.original_phrase() {
+            negative_bitmap |= ctx.get_phrase_docids(phrase)?;
+        }
+    }
+    Ok(negative_bitmap)
+}
+
 /// Return the list of initialised ranking rules to be used for a placeholder search.
 fn get_ranking_rules_for_placeholder_search<'ctx>(
     ctx: &SearchContext<'ctx>,
@@ -266,8 +298,8 @@ fn get_ranking_rules_for_vector<'ctx>(
     geo_strategy: geo_sort::Strategy,
     limit_plus_offset: usize,
     target: &[f32],
-    distribution_shift: Option<DistributionShift>,
     embedder_name: &str,
+    embedder: &Embedder,
 ) -> Result<Vec<BoxRankingRule<'ctx, PlaceholderQuery>>> {
     // query graph search
 
@@ -293,8 +325,8 @@ fn get_ranking_rules_for_vector<'ctx>(
                         target.to_vec(),
                         vector_candidates,
                         limit_plus_offset,
-                        distribution_shift,
                         embedder_name,
+                        embedder,
                     )?;
                     ranking_rules.push(Box::new(vector_sort));
                     vector = true;
@@ -516,8 +548,9 @@ pub fn execute_vector_search(
     geo_strategy: geo_sort::Strategy,
     from: usize,
     length: usize,
-    distribution_shift: Option<DistributionShift>,
     embedder_name: &str,
+    embedder: &Embedder,
+    time_budget: TimeBudget,
 ) -> Result<PartialSearchResult> {
     check_sort_criteria(ctx, sort_criteria.as_ref())?;
 
@@ -529,15 +562,15 @@ pub fn execute_vector_search(
         geo_strategy,
         from + length,
         vector,
-        distribution_shift,
         embedder_name,
+        embedder,
     )?;
 
     let mut placeholder_search_logger = logger::DefaultSearchLogger;
     let placeholder_search_logger: &mut dyn SearchLogger<PlaceholderQuery> =
         &mut placeholder_search_logger;
 
-    let BucketSortOutput { docids, scores, all_candidates } = bucket_sort(
+    let BucketSortOutput { docids, scores, all_candidates, degraded } = bucket_sort(
         ctx,
         ranking_rules,
         &PlaceholderQuery,
@@ -546,6 +579,7 @@ pub fn execute_vector_search(
         length,
         scoring_strategy,
         placeholder_search_logger,
+        time_budget,
     )?;
 
     Ok(PartialSearchResult {
@@ -553,6 +587,8 @@ pub fn execute_vector_search(
         document_scores: scores,
         documents_ids: docids,
         located_query_terms: None,
+        degraded,
+        used_negative_operator: false,
     })
 }
 
@@ -572,9 +608,11 @@ pub fn execute_search(
     words_limit: Option<usize>,
     placeholder_search_logger: &mut dyn SearchLogger<PlaceholderQuery>,
     query_graph_logger: &mut dyn SearchLogger<QueryGraph>,
+    time_budget: TimeBudget,
 ) -> Result<PartialSearchResult> {
     check_sort_criteria(ctx, sort_criteria.as_ref())?;
 
+    let mut used_negative_operator = false;
     let mut located_query_terms = None;
     let query_terms = if let Some(query) = query {
         let span = tracing::trace_span!(target: "search::tokens", "tokenizer_builder");
@@ -615,7 +653,16 @@ pub fn execute_search(
         let tokens = tokenizer.tokenize(query);
         drop(entered);
 
-        let query_terms = located_query_terms_from_tokens(ctx, tokens, words_limit)?;
+        let ExtractedTokens { query_terms, negative_words, negative_phrases } =
+            located_query_terms_from_tokens(ctx, tokens, words_limit)?;
+        used_negative_operator = !negative_words.is_empty() || !negative_phrases.is_empty();
+
+        let ignored_documents = resolve_negative_words(ctx, &negative_words)?;
+        let ignored_phrases = resolve_negative_phrases(ctx, &negative_phrases)?;
+
+        universe -= ignored_documents;
+        universe -= ignored_phrases;
+
         if query_terms.is_empty() {
             // Do a placeholder search instead
             None
@@ -625,6 +672,7 @@ pub fn execute_search(
     } else {
         None
     };
+
     let bucket_sort_output = if let Some(query_terms) = query_terms {
         let (graph, new_located_query_terms) = QueryGraph::from_query(ctx, &query_terms)?;
         located_query_terms = Some(new_located_query_terms);
@@ -648,6 +696,7 @@ pub fn execute_search(
             length,
             scoring_strategy,
             query_graph_logger,
+            time_budget,
         )?
     } else {
         let ranking_rules =
@@ -661,10 +710,11 @@ pub fn execute_search(
             length,
             scoring_strategy,
             placeholder_search_logger,
+            time_budget,
         )?
     };
 
-    let BucketSortOutput { docids, scores, mut all_candidates } = bucket_sort_output;
+    let BucketSortOutput { docids, scores, mut all_candidates, degraded } = bucket_sort_output;
     let fields_ids_map = ctx.index.fields_ids_map(ctx.txn)?;
 
     // The candidates is the universe unless the exhaustive number of hits
@@ -682,6 +732,8 @@ pub fn execute_search(
         document_scores: scores,
         documents_ids: docids,
         located_query_terms,
+        degraded,
+        used_negative_operator,
     })
 }
 
@@ -742,4 +794,7 @@ pub struct PartialSearchResult {
     pub candidates: RoaringBitmap,
     pub documents_ids: Vec<DocumentId>,
     pub document_scores: Vec<Vec<ScoreDetails>>,
+
+    pub degraded: bool,
+    pub used_negative_operator: bool,
 }

milli/src/search/new/query_term/mod.rs

@@ -9,7 +9,9 @@ use std::ops::RangeInclusive;
 
 use either::Either;
 pub use ntypo_subset::NTypoTermSubset;
-pub use parse_query::{located_query_terms_from_tokens, make_ngram, number_of_typos_allowed};
+pub use parse_query::{
+    located_query_terms_from_tokens, make_ngram, number_of_typos_allowed, ExtractedTokens,
+};
 pub use phrase::Phrase;
 
 use super::interner::{DedupInterner, Interned};
@@ -478,6 +480,11 @@ impl QueryTerm {
     pub fn original_word(&self, ctx: &SearchContext) -> String {
         ctx.word_interner.get(self.original).clone()
     }
+
+    pub fn original_phrase(&self) -> Option<Interned<Phrase>> {
+        self.zero_typo.phrase
+    }
+
     pub fn all_computed_derivations(&self) -> (Vec<Interned<String>>, Vec<Interned<Phrase>>) {
         let mut words = BTreeSet::new();
         let mut phrases = BTreeSet::new();

milli/src/search/new/query_term/parse_query.rs

@@ -6,20 +6,37 @@ use charabia::{SeparatorKind, TokenKind};
 use super::compute_derivations::partially_initialized_term_from_word;
 use super::{LocatedQueryTerm, ZeroTypoTerm};
 use crate::search::new::query_term::{Lazy, Phrase, QueryTerm};
+use crate::search::new::Word;
 use crate::{Result, SearchContext, MAX_WORD_LENGTH};
 
+#[derive(Clone)]
+/// Extraction of the content of a query.
+pub struct ExtractedTokens {
+    /// The terms to search for in the database.
+    pub query_terms: Vec<LocatedQueryTerm>,
+    /// The words that must not appear in the results.
+    pub negative_words: Vec<Word>,
+    /// The phrases that must not appear in the results.
+    pub negative_phrases: Vec<LocatedQueryTerm>,
+}
+
 /// Convert the tokenised search query into a list of located query terms.
 #[tracing::instrument(level = "trace", skip_all, target = "search::query")]
 pub fn located_query_terms_from_tokens(
     ctx: &mut SearchContext,
     query: NormalizedTokenIter,
     words_limit: Option<usize>,
-) -> Result<Vec<LocatedQueryTerm>> {
+) -> Result<ExtractedTokens> {
     let nbr_typos = number_of_typos_allowed(ctx)?;
 
-    let mut located_terms = Vec::new();
+    let mut query_terms = Vec::new();
 
+    let mut negative_phrase = false;
     let mut phrase: Option<PhraseBuilder> = None;
+    let mut encountered_whitespace = true;
+    let mut negative_next_token = false;
+    let mut negative_words = Vec::new();
+    let mut negative_phrases = Vec::new();
 
     let parts_limit = words_limit.unwrap_or(usize::MAX);
 
@@ -31,9 +48,10 @@ pub fn located_query_terms_from_tokens(
         if token.lemma().is_empty() {
             continue;
         }
+
         // early return if word limit is exceeded
-        if located_terms.len() >= parts_limit {
-            return Ok(located_terms);
+        if query_terms.len() >= parts_limit {
+            return Ok(ExtractedTokens { query_terms, negative_words, negative_phrases });
         }
 
         match token.kind {
@@ -46,6 +64,11 @@ pub fn located_query_terms_from_tokens(
                 // 3. if the word is the last token of the query we push it as a prefix word.
                 if let Some(phrase) = &mut phrase {
                     phrase.push_word(ctx, &token, position)
+                } else if negative_next_token {
+                    let word = token.lemma().to_string();
+                    let word = Word::Original(ctx.word_interner.insert(word));
+                    negative_words.push(word);
+                    negative_next_token = false;
                 } else if peekable.peek().is_some() {
                     match token.kind {
                         TokenKind::Word => {
@@ -61,9 +84,9 @@ pub fn located_query_terms_from_tokens(
                                 value: ctx.term_interner.push(term),
                                 positions: position..=position,
                             };
-                            located_terms.push(located_term);
+                            query_terms.push(located_term);
                         }
-                        TokenKind::StopWord | TokenKind::Separator(_) | TokenKind::Unknown => {}
+                        TokenKind::StopWord | TokenKind::Separator(_) | TokenKind::Unknown => (),
                     }
                 } else {
                     let word = token.lemma();
@@ -78,7 +101,7 @@ pub fn located_query_terms_from_tokens(
                         value: ctx.term_interner.push(term),
                         positions: position..=position,
                     };
-                    located_terms.push(located_term);
+                    query_terms.push(located_term);
                 }
             }
             TokenKind::Separator(separator_kind) => {
@@ -94,7 +117,14 @@ pub fn located_query_terms_from_tokens(
                     let phrase = if separator_kind == SeparatorKind::Hard {
                         if let Some(phrase) = phrase {
                             if let Some(located_query_term) = phrase.build(ctx) {
-                                located_terms.push(located_query_term)
+                                // as we are evaluating a negative operator we put the phrase
+                                // in the negative one *but* we don't reset the negative operator
+                                // as we are immediatly starting a new negative phrase.
+                                if negative_phrase {
+                                    negative_phrases.push(located_query_term);
+                                } else {
+                                    query_terms.push(located_query_term);
+                                }
                             }
                             Some(PhraseBuilder::empty())
                         } else {
@@ -115,26 +145,49 @@ pub fn located_query_terms_from_tokens(
                         // Per the check above, quote_count > 0
                         quote_count -= 1;
                         if let Some(located_query_term) = phrase.build(ctx) {
-                            located_terms.push(located_query_term)
+                            // we were evaluating a negative operator so we
+                            // put the phrase in the negative phrases
+                            if negative_phrase {
+                                negative_phrases.push(located_query_term);
+                                negative_phrase = false;
+                            } else {
+                                query_terms.push(located_query_term);
+                            }
                         }
                     }
 
                     // Start new phrase if the token ends with an opening quote
-                    (quote_count % 2 == 1).then_some(PhraseBuilder::empty())
+                    if quote_count % 2 == 1 {
+                        negative_phrase = negative_next_token;
+                        Some(PhraseBuilder::empty())
+                    } else {
+                        None
+                    }
                 };
+
+                negative_next_token =
+                    phrase.is_none() && token.lemma() == "-" && encountered_whitespace;
             }
             _ => (),
         }
+
+        encountered_whitespace =
+            token.lemma().chars().last().filter(|c| c.is_whitespace()).is_some();
     }
 
     // If a quote is never closed, we consider all of the end of the query as a phrase.
     if let Some(phrase) = phrase.take() {
         if let Some(located_query_term) = phrase.build(ctx) {
-            located_terms.push(located_query_term);
+            // put the phrase in the negative set if we are evaluating a negative operator.
+            if negative_phrase {
+                negative_phrases.push(located_query_term);
+            } else {
+                query_terms.push(located_query_term);
+            }
         }
     }
 
-    Ok(located_terms)
+    Ok(ExtractedTokens { query_terms, negative_words, negative_phrases })
 }
 
 pub fn number_of_typos_allowed<'ctx>(
@@ -315,8 +368,10 @@ mod tests {
         let rtxn = index.read_txn()?;
         let mut ctx = SearchContext::new(&index, &rtxn);
         // panics with `attempt to add with overflow` before <https://github.com/meilisearch/meilisearch/issues/3785>
-        let located_query_terms = located_query_terms_from_tokens(&mut ctx, tokens, None)?;
-        assert!(located_query_terms.is_empty());
+        let ExtractedTokens { query_terms, .. } =
+            located_query_terms_from_tokens(&mut ctx, tokens, None)?;
+        assert!(query_terms.is_empty());
+
         Ok(())
     }
 }

milli/src/search/new/tests/cutoff.rs

@@ -0,0 +1,429 @@
+//! This module test the search cutoff and ensure a few things:
+//! 1. A basic test works and mark the search as degraded
+//! 2. A test that ensure the filters are affectively applied even with a cutoff of 0
+//! 3. A test that ensure the cutoff works well with the ranking scores
+
+use std::time::Duration;
+
+use big_s::S;
+use maplit::hashset;
+use meili_snap::snapshot;
+
+use crate::index::tests::TempIndex;
+use crate::score_details::{ScoreDetails, ScoringStrategy};
+use crate::{Criterion, Filter, Search, TimeBudget};
+
+fn create_index() -> TempIndex {
+    let index = TempIndex::new();
+
+    index
+        .update_settings(|s| {
+            s.set_primary_key("id".to_owned());
+            s.set_searchable_fields(vec!["text".to_owned()]);
+            s.set_filterable_fields(hashset! { S("id") });
+            s.set_criteria(vec![Criterion::Words, Criterion::Typo]);
+        })
+        .unwrap();
+
+    // reverse the ID / insertion order so we see better what was sorted from what got the insertion order ordering
+    index
+        .add_documents(documents!([
+            {
+                "id": 4,
+                "text": "hella puppo kefir",
+            },
+            {
+                "id": 3,
+                "text": "hella puppy kefir",
+            },
+            {
+                "id": 2,
+                "text": "hello",
+            },
+            {
+                "id": 1,
+                "text": "hello puppy",
+            },
+            {
+                "id": 0,
+                "text": "hello puppy kefir",
+            },
+        ]))
+        .unwrap();
+    index
+}
+
+#[test]
+fn basic_degraded_search() {
+    let index = create_index();
+    let rtxn = index.read_txn().unwrap();
+
+    let mut search = Search::new(&rtxn, &index);
+    search.query("hello puppy kefir");
+    search.limit(3);
+    search.time_budget(TimeBudget::new(Duration::from_millis(0)));
+
+    let result = search.execute().unwrap();
+    assert!(result.degraded);
+}
+
+#[test]
+fn degraded_search_cannot_skip_filter() {
+    let index = create_index();
+    let rtxn = index.read_txn().unwrap();
+
+    let mut search = Search::new(&rtxn, &index);
+    search.query("hello puppy kefir");
+    search.limit(100);
+    search.time_budget(TimeBudget::new(Duration::from_millis(0)));
+    let filter_condition = Filter::from_str("id > 2").unwrap().unwrap();
+    search.filter(filter_condition);
+
+    let result = search.execute().unwrap();
+    assert!(result.degraded);
+    snapshot!(format!("{:?}\n{:?}", result.candidates, result.documents_ids), @r###"
+    RoaringBitmap<[0, 1]>
+    [0, 1]
+    "###);
+}
+
+#[test]
+#[allow(clippy::format_collect)] // the test is already quite big
+fn degraded_search_and_score_details() {
+    let index = create_index();
+    let rtxn = index.read_txn().unwrap();
+
+    let mut search = Search::new(&rtxn, &index);
+    search.query("hello puppy kefir");
+    search.limit(4);
+    search.scoring_strategy(ScoringStrategy::Detailed);
+    search.time_budget(TimeBudget::max());
+
+    let result = search.execute().unwrap();
+    snapshot!(format!("IDs: {:?}\nScores: {}\nScore Details:\n{:#?}", result.documents_ids, result.document_scores.iter().map(|scores| format!("{:.4} ", ScoreDetails::global_score(scores.iter()))).collect::<String>(), result.document_scores), @r###"
+    IDs: [4, 1, 0, 3]
+    Scores: 1.0000 0.9167 0.8333 0.6667 
+    Score Details:
+    [
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 0,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 1,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 2,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 2,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 0,
+                    max_typo_count: 2,
+                },
+            ),
+        ],
+    ]
+    "###);
+
+    // Do ONE loop iteration. Not much can be deduced, almost everyone matched the words first bucket.
+    search.time_budget(TimeBudget::max().with_stop_after(1));
+
+    let result = search.execute().unwrap();
+    snapshot!(format!("IDs: {:?}\nScores: {}\nScore Details:\n{:#?}", result.documents_ids, result.document_scores.iter().map(|scores| format!("{:.4} ", ScoreDetails::global_score(scores.iter()))).collect::<String>(), result.document_scores), @r###"
+    IDs: [0, 1, 4, 2]
+    Scores: 0.6667 0.6667 0.6667 0.0000 
+    Score Details:
+    [
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+        [
+            Skipped,
+        ],
+    ]
+    "###);
+
+    // Do TWO loop iterations. The first document should be entirely sorted
+    search.time_budget(TimeBudget::max().with_stop_after(2));
+
+    let result = search.execute().unwrap();
+    snapshot!(format!("IDs: {:?}\nScores: {}\nScore Details:\n{:#?}", result.documents_ids, result.document_scores.iter().map(|scores| format!("{:.4} ", ScoreDetails::global_score(scores.iter()))).collect::<String>(), result.document_scores), @r###"
+    IDs: [4, 0, 1, 2]
+    Scores: 1.0000 0.6667 0.6667 0.0000 
+    Score Details:
+    [
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 0,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+        [
+            Skipped,
+        ],
+    ]
+    "###);
+
+    // Do THREE loop iterations. The second document should be entirely sorted as well
+    search.time_budget(TimeBudget::max().with_stop_after(3));
+
+    let result = search.execute().unwrap();
+    snapshot!(format!("IDs: {:?}\nScores: {}\nScore Details:\n{:#?}", result.documents_ids, result.document_scores.iter().map(|scores| format!("{:.4} ", ScoreDetails::global_score(scores.iter()))).collect::<String>(), result.document_scores), @r###"
+    IDs: [4, 1, 0, 2]
+    Scores: 1.0000 0.9167 0.6667 0.0000 
+    Score Details:
+    [
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 0,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 1,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+        [
+            Skipped,
+        ],
+    ]
+    "###);
+
+    // Do FOUR loop iterations. The third document should be entirely sorted as well
+    // The words bucket have still not progressed thus the last document doesn't have any info yet.
+    search.time_budget(TimeBudget::max().with_stop_after(4));
+
+    let result = search.execute().unwrap();
+    snapshot!(format!("IDs: {:?}\nScores: {}\nScore Details:\n{:#?}", result.documents_ids, result.document_scores.iter().map(|scores| format!("{:.4} ", ScoreDetails::global_score(scores.iter()))).collect::<String>(), result.document_scores), @r###"
+    IDs: [4, 1, 0, 2]
+    Scores: 1.0000 0.9167 0.8333 0.0000 
+    Score Details:
+    [
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 0,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 1,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 2,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Skipped,
+        ],
+    ]
+    "###);
+
+    // After SIX loop iteration. The words ranking rule gave us a new bucket.
+    // Since we reached the limit we were able to early exit without checking the typo ranking rule.
+    search.time_budget(TimeBudget::max().with_stop_after(6));
+
+    let result = search.execute().unwrap();
+    snapshot!(format!("IDs: {:?}\nScores: {}\nScore Details:\n{:#?}", result.documents_ids, result.document_scores.iter().map(|scores| format!("{:.4} ", ScoreDetails::global_score(scores.iter()))).collect::<String>(), result.document_scores), @r###"
+    IDs: [4, 1, 0, 3]
+    Scores: 1.0000 0.9167 0.8333 0.3333 
+    Score Details:
+    [
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 0,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 1,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 3,
+                    max_matching_words: 3,
+                },
+            ),
+            Typo(
+                Typo {
+                    typo_count: 2,
+                    max_typo_count: 3,
+                },
+            ),
+        ],
+        [
+            Words(
+                Words {
+                    matching_words: 2,
+                    max_matching_words: 3,
+                },
+            ),
+            Skipped,
+        ],
+    ]
+    "###);
+}

milli/src/search/new/tests/mod.rs

@@ -1,5 +1,6 @@
 pub mod attribute_fid;
 pub mod attribute_position;
+pub mod cutoff;
 pub mod distinct;
 pub mod exactness;
 pub mod geo_sort;

milli/src/search/new/tests/typo_proximity.rs

@@ -5,7 +5,7 @@ The typo ranking rule should transform the query graph such that it only contain
 the combinations of word derivations that it used to compute its bucket.
 
 The proximity ranking rule should then look for proximities only between those specific derivations.
-For example, given the the search query `beautiful summer` and the dataset:
+For example, given the search query `beautiful summer` and the dataset:
 ```text
 { "id": 0, "text": "beautigul summer...... beautiful day in the summer" }
 { "id": 1, "text": "beautiful summer" }

milli/src/search/new/vector_sort.rs

@@ -5,14 +5,14 @@ use roaring::RoaringBitmap;
 
 use super::ranking_rules::{RankingRule, RankingRuleOutput, RankingRuleQueryTrait};
 use crate::score_details::{self, ScoreDetails};
-use crate::vector::DistributionShift;
+use crate::vector::{DistributionShift, Embedder};
 use crate::{DocumentId, Result, SearchContext, SearchLogger};
 
 pub struct VectorSort<Q: RankingRuleQueryTrait> {
     query: Option<Q>,
     target: Vec<f32>,
     vector_candidates: RoaringBitmap,
-    cached_sorted_docids: std::vec::IntoIter<(DocumentId, f32, Vec<f32>)>,
+    cached_sorted_docids: std::vec::IntoIter<(DocumentId, f32)>,
     limit: usize,
     distribution_shift: Option<DistributionShift>,
     embedder_index: u8,
@@ -24,8 +24,8 @@ impl<Q: RankingRuleQueryTrait> VectorSort<Q> {
         target: Vec<f32>,
         vector_candidates: RoaringBitmap,
         limit: usize,
-        distribution_shift: Option<DistributionShift>,
         embedder_name: &str,
+        embedder: &Embedder,
     ) -> Result<Self> {
         let embedder_index = ctx
             .index
@@ -39,7 +39,7 @@ impl<Q: RankingRuleQueryTrait> VectorSort<Q> {
             vector_candidates,
             cached_sorted_docids: Default::default(),
             limit,
-            distribution_shift,
+            distribution_shift: embedder.distribution(),
             embedder_index,
         })
     }
@@ -70,14 +70,9 @@ impl<Q: RankingRuleQueryTrait> VectorSort<Q> {
         for reader in readers.iter() {
             let nns_by_vector =
                 reader.nns_by_vector(ctx.txn, target, self.limit, None, Some(vector_candidates))?;
-            let vectors: std::result::Result<Vec<_>, _> = nns_by_vector
-                .iter()
-                .map(|(docid, _)| reader.item_vector(ctx.txn, *docid).transpose().unwrap())
-                .collect();
-            let vectors = vectors?;
-            results.extend(nns_by_vector.into_iter().zip(vectors).map(|((x, y), z)| (x, y, z)));
+            results.extend(nns_by_vector.into_iter());
         }
-        results.sort_unstable_by_key(|(_, distance, _)| OrderedFloat(*distance));
+        results.sort_unstable_by_key(|(_, distance)| OrderedFloat(*distance));
         self.cached_sorted_docids = results.into_iter();
 
         Ok(())
@@ -118,14 +113,11 @@ impl<'ctx, Q: RankingRuleQueryTrait> RankingRule<'ctx, Q> for VectorSort<Q> {
             return Ok(Some(RankingRuleOutput {
                 query,
                 candidates: universe.clone(),
-                score: ScoreDetails::Vector(score_details::Vector {
-                    target_vector: self.target.clone(),
-                    value_similarity: None,
-                }),
+                score: ScoreDetails::Vector(score_details::Vector { similarity: None }),
             }));
         }
 
-        for (docid, distance, vector) in self.cached_sorted_docids.by_ref() {
+        for (docid, distance) in self.cached_sorted_docids.by_ref() {
             if vector_candidates.contains(docid) {
                 let score = 1.0 - distance;
                 let score = self
@@ -135,10 +127,7 @@ impl<'ctx, Q: RankingRuleQueryTrait> RankingRule<'ctx, Q> for VectorSort<Q> {
                 return Ok(Some(RankingRuleOutput {
                     query,
                     candidates: RoaringBitmap::from_iter([docid]),
-                    score: ScoreDetails::Vector(score_details::Vector {
-                        target_vector: self.target.clone(),
-                        value_similarity: Some((vector, score)),
-                    }),
+                    score: ScoreDetails::Vector(score_details::Vector { similarity: Some(score) }),
                 }));
             }
         }
@@ -154,10 +143,7 @@ impl<'ctx, Q: RankingRuleQueryTrait> RankingRule<'ctx, Q> for VectorSort<Q> {
             return Ok(Some(RankingRuleOutput {
                 query,
                 candidates: universe.clone(),
-                score: ScoreDetails::Vector(score_details::Vector {
-                    target_vector: self.target.clone(),
-                    value_similarity: None,
-                }),
+                score: ScoreDetails::Vector(score_details::Vector { similarity: None }),
             }));
         }
 

milli/src/thread_pool_no_abort.rs

@@ -0,0 +1,69 @@
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::sync::Arc;
+
+use rayon::{ThreadPool, ThreadPoolBuilder};
+use thiserror::Error;
+
+/// A rayon ThreadPool wrapper that can catch panics in the pool
+/// and modifies the install function accordingly.
+#[derive(Debug)]
+pub struct ThreadPoolNoAbort {
+    thread_pool: ThreadPool,
+    /// Set to true if the thread pool catched a panic.
+    pool_catched_panic: Arc<AtomicBool>,
+}
+
+impl ThreadPoolNoAbort {
+    pub fn install<OP, R>(&self, op: OP) -> Result<R, PanicCatched>
+    where
+        OP: FnOnce() -> R + Send,
+        R: Send,
+    {
+        let output = self.thread_pool.install(op);
+        // While reseting the pool panic catcher we return an error if we catched one.
+        if self.pool_catched_panic.swap(false, Ordering::SeqCst) {
+            Err(PanicCatched)
+        } else {
+            Ok(output)
+        }
+    }
+
+    pub fn current_num_threads(&self) -> usize {
+        self.thread_pool.current_num_threads()
+    }
+}
+
+#[derive(Error, Debug)]
+#[error("A panic occured. Read the logs to find more information about it")]
+pub struct PanicCatched;
+
+#[derive(Default)]
+pub struct ThreadPoolNoAbortBuilder(ThreadPoolBuilder);
+
+impl ThreadPoolNoAbortBuilder {
+    pub fn new() -> ThreadPoolNoAbortBuilder {
+        ThreadPoolNoAbortBuilder::default()
+    }
+
+    pub fn thread_name<F>(mut self, closure: F) -> Self
+    where
+        F: FnMut(usize) -> String + 'static,
+    {
+        self.0 = self.0.thread_name(closure);
+        self
+    }
+
+    pub fn num_threads(mut self, num_threads: usize) -> ThreadPoolNoAbortBuilder {
+        self.0 = self.0.num_threads(num_threads);
+        self
+    }
+
+    pub fn build(mut self) -> Result<ThreadPoolNoAbort, rayon::ThreadPoolBuildError> {
+        let pool_catched_panic = Arc::new(AtomicBool::new(false));
+        self.0 = self.0.panic_handler({
+            let catched_panic = pool_catched_panic.clone();
+            move |_result| catched_panic.store(true, Ordering::SeqCst)
+        });
+        Ok(ThreadPoolNoAbort { thread_pool: self.0.build()?, pool_catched_panic })
+    }
+}

milli/src/update/del_add.rs

@@ -71,8 +71,8 @@ pub enum DelAddOperation {
 /// putting each deletion obkv's keys under an DelAdd::Deletion
 /// and putting each addition obkv's keys under an DelAdd::Addition
 pub fn del_add_from_two_obkvs<K: obkv::Key + PartialOrd + Ord>(
-    deletion: obkv::KvReader<K>,
-    addition: obkv::KvReader<K>,
+    deletion: &obkv::KvReader<K>,
+    addition: &obkv::KvReader<K>,
     buffer: &mut Vec<u8>,
 ) -> Result<(), std::io::Error> {
     use itertools::merge_join_by;

milli/src/update/index_documents/extract/extract_docid_word_positions.rs

@@ -1,4 +1,4 @@
-use std::collections::{HashMap, HashSet};
+use std::collections::HashMap;
 use std::convert::TryInto;
 use std::fs::File;
 use std::io::BufReader;
@@ -12,6 +12,7 @@ use serde_json::Value;
 use super::helpers::{create_sorter, keep_latest_obkv, sorter_into_reader, GrenadParameters};
 use crate::error::{InternalError, SerializationError};
 use crate::update::del_add::{del_add_from_two_obkvs, DelAdd, KvReaderDelAdd};
+use crate::update::settings::{InnerIndexSettings, InnerIndexSettingsDiff};
 use crate::{FieldId, Result, MAX_POSITION_PER_ATTRIBUTE, MAX_WORD_LENGTH};
 
 pub type ScriptLanguageDocidsMap = HashMap<(Script, Language), (RoaringBitmap, RoaringBitmap)>;
@@ -25,10 +26,7 @@ pub type ScriptLanguageDocidsMap = HashMap<(Script, Language), (RoaringBitmap, R
 pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
     obkv_documents: grenad::Reader<R>,
     indexer: GrenadParameters,
-    searchable_fields: &Option<HashSet<FieldId>>,
-    stop_words: Option<&fst::Set<Vec<u8>>>,
-    allowed_separators: Option<&[&str]>,
-    dictionary: Option<&[&str]>,
+    settings_diff: &InnerIndexSettingsDiff,
     max_positions_per_attributes: Option<u32>,
 ) -> Result<(grenad::Reader<BufReader<File>>, ScriptLanguageDocidsMap)> {
     puffin::profile_function!();
@@ -36,6 +34,7 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
     let max_positions_per_attributes = max_positions_per_attributes
         .map_or(MAX_POSITION_PER_ATTRIBUTE, |max| max.min(MAX_POSITION_PER_ATTRIBUTE));
     let max_memory = indexer.max_memory_by_thread();
+    let force_reindexing = settings_diff.reindex_searchable();
 
     // initialize destination values.
     let mut documents_ids = RoaringBitmap::new();
@@ -56,8 +55,37 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
     let mut value_buffer = Vec::new();
 
     // initialize tokenizer.
-    let mut builder = tokenizer_builder(stop_words, allowed_separators, dictionary, None);
-    let tokenizer = builder.build();
+    let old_stop_words = settings_diff.old.stop_words.as_ref();
+    let old_separators: Option<Vec<_>> = settings_diff
+        .old
+        .allowed_separators
+        .as_ref()
+        .map(|s| s.iter().map(String::as_str).collect());
+    let old_dictionary: Option<Vec<_>> =
+        settings_diff.old.dictionary.as_ref().map(|s| s.iter().map(String::as_str).collect());
+    let mut del_builder = tokenizer_builder(
+        old_stop_words,
+        old_separators.as_deref(),
+        old_dictionary.as_deref(),
+        None,
+    );
+    let del_tokenizer = del_builder.build();
+
+    let new_stop_words = settings_diff.new.stop_words.as_ref();
+    let new_separators: Option<Vec<_>> = settings_diff
+        .new
+        .allowed_separators
+        .as_ref()
+        .map(|s| s.iter().map(String::as_str).collect());
+    let new_dictionary: Option<Vec<_>> =
+        settings_diff.new.dictionary.as_ref().map(|s| s.iter().map(String::as_str).collect());
+    let mut add_builder = tokenizer_builder(
+        new_stop_words,
+        new_separators.as_deref(),
+        new_dictionary.as_deref(),
+        None,
+    );
+    let add_tokenizer = add_builder.build();
 
     // iterate over documents.
     let mut cursor = obkv_documents.into_cursor()?;
@@ -69,7 +97,7 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
         let obkv = KvReader::<FieldId>::new(value);
 
         // if the searchable fields didn't change, skip the searchable indexing for this document.
-        if !searchable_fields_changed(&KvReader::<FieldId>::new(value), searchable_fields) {
+        if !force_reindexing && !searchable_fields_changed(&obkv, settings_diff) {
             continue;
         }
 
@@ -85,11 +113,8 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
                 // deletions
                 lang_safe_tokens_from_document(
                     &obkv,
-                    searchable_fields,
-                    &tokenizer,
-                    stop_words,
-                    allowed_separators,
-                    dictionary,
+                    &settings_diff.old,
+                    &del_tokenizer,
                     max_positions_per_attributes,
                     DelAdd::Deletion,
                     &mut del_buffers,
@@ -99,11 +124,8 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
                 // additions
                 lang_safe_tokens_from_document(
                     &obkv,
-                    searchable_fields,
-                    &tokenizer,
-                    stop_words,
-                    allowed_separators,
-                    dictionary,
+                    &settings_diff.new,
+                    &add_tokenizer,
                     max_positions_per_attributes,
                     DelAdd::Addition,
                     &mut add_buffers,
@@ -118,8 +140,8 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
         // transforming two KV<FieldId, KV<u16, String>> into one KV<FieldId, KV<DelAdd, KV<u16, String>>>
         value_buffer.clear();
         del_add_from_two_obkvs(
-            KvReader::<FieldId>::new(del_obkv),
-            KvReader::<FieldId>::new(add_obkv),
+            &KvReader::<FieldId>::new(del_obkv),
+            &KvReader::<FieldId>::new(add_obkv),
             &mut value_buffer,
         )?;
 
@@ -160,8 +182,9 @@ pub fn extract_docid_word_positions<R: io::Read + io::Seek>(
 /// Check if any searchable fields of a document changed.
 fn searchable_fields_changed(
     obkv: &KvReader<FieldId>,
-    searchable_fields: &Option<HashSet<FieldId>>,
+    settings_diff: &InnerIndexSettingsDiff,
 ) -> bool {
+    let searchable_fields = &settings_diff.new.searchable_fields_ids;
     for (field_id, field_bytes) in obkv.iter() {
         if searchable_fields.as_ref().map_or(true, |sf| sf.contains(&field_id)) {
             let del_add = KvReaderDelAdd::new(field_bytes);
@@ -206,14 +229,10 @@ fn tokenizer_builder<'a>(
 
 /// Extract words mapped with their positions of a document,
 /// ensuring no Language detection mistakes was made.
-#[allow(clippy::too_many_arguments)] // FIXME: consider grouping arguments in a struct
 fn lang_safe_tokens_from_document<'a>(
     obkv: &KvReader<FieldId>,
-    searchable_fields: &Option<HashSet<FieldId>>,
+    settings: &InnerIndexSettings,
     tokenizer: &Tokenizer,
-    stop_words: Option<&fst::Set<Vec<u8>>>,
-    allowed_separators: Option<&[&str]>,
-    dictionary: Option<&[&str]>,
     max_positions_per_attributes: u32,
     del_add: DelAdd,
     buffers: &'a mut Buffers,
@@ -222,7 +241,7 @@ fn lang_safe_tokens_from_document<'a>(
 
     tokens_from_document(
         obkv,
-        searchable_fields,
+        &settings.searchable_fields_ids,
         tokenizer,
         max_positions_per_attributes,
         del_add,
@@ -246,12 +265,15 @@ fn lang_safe_tokens_from_document<'a>(
         // then we don't rerun the extraction.
         if !script_language.is_empty() {
             // build a new temporary tokenizer including the allow list.
-            let mut builder = tokenizer_builder(
-                stop_words,
-                allowed_separators,
-                dictionary,
-                Some(&script_language),
-            );
+            let stop_words = settings.stop_words.as_ref();
+            let separators: Option<Vec<_>> = settings
+                .allowed_separators
+                .as_ref()
+                .map(|s| s.iter().map(String::as_str).collect());
+            let dictionary: Option<Vec<_>> =
+                settings.dictionary.as_ref().map(|s| s.iter().map(String::as_str).collect());
+            let mut builder =
+                tokenizer_builder(stop_words, separators.as_deref(), dictionary.as_deref(), None);
             let tokenizer = builder.build();
 
             script_language_word_count.clear();
@@ -259,7 +281,7 @@ fn lang_safe_tokens_from_document<'a>(
             // rerun the extraction.
             tokens_from_document(
                 obkv,
-                searchable_fields,
+                &settings.searchable_fields_ids,
                 &tokenizer,
                 max_positions_per_attributes,
                 del_add,
@@ -276,7 +298,7 @@ fn lang_safe_tokens_from_document<'a>(
 /// Extract words mapped with their positions of a document.
 fn tokens_from_document<'a>(
     obkv: &KvReader<FieldId>,
-    searchable_fields: &Option<HashSet<FieldId>>,
+    searchable_fields: &Option<Vec<FieldId>>,
     tokenizer: &Tokenizer,
     max_positions_per_attributes: u32,
     del_add: DelAdd,

milli/src/update/index_documents/extract/extract_facet_number_docids.rs

@@ -10,6 +10,7 @@ use crate::heed_codec::facet::{
     FacetGroupKey, FacetGroupKeyCodec, FieldDocIdFacetF64Codec, OrderedF64Codec,
 };
 use crate::update::del_add::{KvReaderDelAdd, KvWriterDelAdd};
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::Result;
 
 /// Extracts the facet number and the documents ids where this facet number appear.
@@ -20,6 +21,7 @@ use crate::Result;
 pub fn extract_facet_number_docids<R: io::Read + io::Seek>(
     fid_docid_facet_number: grenad::Reader<R>,
     indexer: GrenadParameters,
+    _settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<grenad::Reader<BufReader<File>>> {
     puffin::profile_function!();
 

milli/src/update/index_documents/extract/extract_facet_string_docids.rs

@@ -15,6 +15,7 @@ use crate::update::del_add::{DelAdd, KvReaderDelAdd, KvWriterDelAdd};
 use crate::update::index_documents::helpers::{
     merge_deladd_btreeset_string, merge_deladd_cbo_roaring_bitmaps,
 };
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::{FieldId, Result, MAX_FACET_VALUE_LENGTH};
 
 /// Extracts the facet string and the documents ids where this facet string appear.
@@ -25,6 +26,7 @@ use crate::{FieldId, Result, MAX_FACET_VALUE_LENGTH};
 pub fn extract_facet_string_docids<R: io::Read + io::Seek>(
     docid_fid_facet_string: grenad::Reader<R>,
     indexer: GrenadParameters,
+    _settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<(grenad::Reader<BufReader<File>>, grenad::Reader<BufReader<File>>)> {
     puffin::profile_function!();
 

milli/src/update/index_documents/extract/extract_fid_docid_facet_values.rs

@@ -1,5 +1,5 @@
 use std::borrow::Cow;
-use std::collections::{BTreeMap, HashSet};
+use std::collections::BTreeMap;
 use std::convert::TryInto;
 use std::fs::File;
 use std::io::{self, BufReader};
@@ -20,6 +20,7 @@ use crate::error::InternalError;
 use crate::facet::value_encoding::f64_into_bytes;
 use crate::update::del_add::{DelAdd, KvWriterDelAdd};
 use crate::update::index_documents::{create_writer, writer_into_reader};
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::{CboRoaringBitmapCodec, DocumentId, Error, FieldId, Result, MAX_FACET_VALUE_LENGTH};
 
 /// The length of the elements that are always in the buffer when inserting new values.
@@ -43,7 +44,7 @@ pub struct ExtractedFacetValues {
 pub fn extract_fid_docid_facet_values<R: io::Read + io::Seek>(
     obkv_documents: grenad::Reader<R>,
     indexer: GrenadParameters,
-    faceted_fields: &HashSet<FieldId>,
+    settings_diff: &InnerIndexSettingsDiff,
     geo_fields_ids: Option<(FieldId, FieldId)>,
 ) -> Result<ExtractedFacetValues> {
     puffin::profile_function!();
@@ -82,7 +83,9 @@ pub fn extract_fid_docid_facet_values<R: io::Read + io::Seek>(
         let obkv = obkv::KvReader::new(value);
 
         for (field_id, field_bytes) in obkv.iter() {
-            if faceted_fields.contains(&field_id) {
+            let delete_faceted = settings_diff.old.faceted_fields_ids.contains(&field_id);
+            let add_faceted = settings_diff.new.faceted_fields_ids.contains(&field_id);
+            if delete_faceted || add_faceted {
                 numbers_key_buffer.clear();
                 strings_key_buffer.clear();
 
@@ -99,11 +102,12 @@ pub fn extract_fid_docid_facet_values<R: io::Read + io::Seek>(
                 strings_key_buffer.extend_from_slice(docid_bytes);
 
                 let del_add_obkv = obkv::KvReader::new(field_bytes);
-                let del_value = match del_add_obkv.get(DelAdd::Deletion) {
+                let del_value = match del_add_obkv.get(DelAdd::Deletion).filter(|_| delete_faceted)
+                {
                     Some(bytes) => Some(from_slice(bytes).map_err(InternalError::SerdeJson)?),
                     None => None,
                 };
-                let add_value = match del_add_obkv.get(DelAdd::Addition) {
+                let add_value = match del_add_obkv.get(DelAdd::Addition).filter(|_| add_faceted) {
                     Some(bytes) => Some(from_slice(bytes).map_err(InternalError::SerdeJson)?),
                     None => None,
                 };

milli/src/update/index_documents/extract/extract_fid_word_count_docids.rs

@@ -10,6 +10,7 @@ use super::helpers::{
 use crate::error::SerializationError;
 use crate::index::db_name::DOCID_WORD_POSITIONS;
 use crate::update::del_add::{DelAdd, KvReaderDelAdd, KvWriterDelAdd};
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::Result;
 
 const MAX_COUNTED_WORDS: usize = 30;
@@ -23,6 +24,7 @@ const MAX_COUNTED_WORDS: usize = 30;
 pub fn extract_fid_word_count_docids<R: io::Read + io::Seek>(
     docid_word_positions: grenad::Reader<R>,
     indexer: GrenadParameters,
+    _settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<grenad::Reader<BufReader<File>>> {
     puffin::profile_function!();
 

milli/src/update/index_documents/extract/extract_vector_points.rs

@@ -17,8 +17,9 @@ use crate::error::UserError;
 use crate::prompt::Prompt;
 use crate::update::del_add::{DelAdd, KvReaderDelAdd, KvWriterDelAdd};
 use crate::update::index_documents::helpers::try_split_at;
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::vector::Embedder;
-use crate::{DocumentId, FieldsIdsMap, InternalError, Result, VectorOrArrayOfVectors};
+use crate::{DocumentId, InternalError, Result, ThreadPoolNoAbort, VectorOrArrayOfVectors};
 
 /// The length of the elements that are always in the buffer when inserting new values.
 const TRUNCATE_SIZE: usize = size_of::<DocumentId>();
@@ -71,12 +72,15 @@ impl VectorStateDelta {
 pub fn extract_vector_points<R: io::Read + io::Seek>(
     obkv_documents: grenad::Reader<R>,
     indexer: GrenadParameters,
-    field_id_map: &FieldsIdsMap,
+    settings_diff: &InnerIndexSettingsDiff,
     prompt: &Prompt,
     embedder_name: &str,
 ) -> Result<ExtractedVectorPoints> {
     puffin::profile_function!();
 
+    let old_fields_ids_map = &settings_diff.old.fields_ids_map;
+    let new_fields_ids_map = &settings_diff.new.fields_ids_map;
+
     // (docid, _index) -> KvWriterDelAdd -> Vector
     let mut manual_vectors_writer = create_writer(
         indexer.chunk_compression_type,
@@ -98,8 +102,6 @@ pub fn extract_vector_points<R: io::Read + io::Seek>(
         tempfile::tempfile()?,
     );
 
-    let vectors_fid = field_id_map.id("_vectors");
-
     let mut key_buffer = Vec::new();
     let mut cursor = obkv_documents.into_cursor()?;
     while let Some((key, value)) = cursor.move_on_next()? {
@@ -116,15 +118,29 @@ pub fn extract_vector_points<R: io::Read + io::Seek>(
         // lazily get it when needed
         let document_id = || -> Value { from_utf8(external_id_bytes).unwrap().into() };
 
-        let vectors_field = vectors_fid
-            .and_then(|vectors_fid| obkv.get(vectors_fid))
-            .map(KvReaderDelAdd::new)
-            .map(|obkv| to_vector_maps(obkv, document_id))
-            .transpose()?;
+        // the vector field id may have changed
+        let old_vectors_fid = old_fields_ids_map.id("_vectors");
+        // filter the old vector fid if the settings has been changed forcing reindexing.
+        let old_vectors_fid = old_vectors_fid.filter(|_| !settings_diff.reindex_vectors());
 
-        let (del_map, add_map) = vectors_field.unzip();
-        let del_map = del_map.flatten();
-        let add_map = add_map.flatten();
+        let new_vectors_fid = new_fields_ids_map.id("_vectors");
+        let vectors_field = {
+            let del = old_vectors_fid
+                .and_then(|vectors_fid| obkv.get(vectors_fid))
+                .map(KvReaderDelAdd::new)
+                .map(|obkv| to_vector_map(obkv, DelAdd::Deletion, &document_id))
+                .transpose()?
+                .flatten();
+            let add = new_vectors_fid
+                .and_then(|vectors_fid| obkv.get(vectors_fid))
+                .map(KvReaderDelAdd::new)
+                .map(|obkv| to_vector_map(obkv, DelAdd::Addition, &document_id))
+                .transpose()?
+                .flatten();
+            (del, add)
+        };
+
+        let (del_map, add_map) = vectors_field;
 
         let del_value = del_map.and_then(|mut map| map.remove(embedder_name));
         let add_value = add_map.and_then(|mut map| map.remove(embedder_name));
@@ -155,7 +171,7 @@ pub fn extract_vector_points<R: io::Read + io::Seek>(
                     VectorStateDelta::NowGenerated(prompt.render(
                         obkv,
                         DelAdd::Addition,
-                        field_id_map,
+                        new_fields_ids_map,
                     )?)
                 } else {
                     VectorStateDelta::NowRemoved
@@ -182,10 +198,16 @@ pub fn extract_vector_points<R: io::Read + io::Seek>(
 
                 if document_is_kept {
                     // Don't give up if the old prompt was failing
-                    let old_prompt =
-                        prompt.render(obkv, DelAdd::Deletion, field_id_map).unwrap_or_default();
-                    let new_prompt = prompt.render(obkv, DelAdd::Addition, field_id_map)?;
-                    if old_prompt != new_prompt {
+                    let old_prompt = Some(prompt)
+                        // TODO: this filter works because we erase the vec database when a embedding setting changes.
+                        // When vector pipeline will be optimized, this should be removed.
+                        .filter(|_| !settings_diff.reindex_vectors())
+                        .map(|p| {
+                            p.render(obkv, DelAdd::Deletion, old_fields_ids_map).unwrap_or_default()
+                        });
+                    let new_prompt = prompt.render(obkv, DelAdd::Addition, new_fields_ids_map)?;
+                    if old_prompt.as_ref() != Some(&new_prompt) {
+                        let old_prompt = old_prompt.unwrap_or_default();
                         tracing::trace!(
                             "🚀 Changing prompt from\n{old_prompt}\n===to===\n{new_prompt}"
                         );
@@ -207,6 +229,7 @@ pub fn extract_vector_points<R: io::Read + io::Seek>(
             &mut manual_vectors_writer,
             &mut key_buffer,
             delta,
+            settings_diff,
         )?;
     }
 
@@ -220,15 +243,6 @@ pub fn extract_vector_points<R: io::Read + io::Seek>(
     })
 }
 
-fn to_vector_maps(
-    obkv: KvReaderDelAdd,
-    document_id: impl Fn() -> Value,
-) -> Result<(Option<serde_json::Map<String, Value>>, Option<serde_json::Map<String, Value>>)> {
-    let del = to_vector_map(obkv, DelAdd::Deletion, &document_id)?;
-    let add = to_vector_map(obkv, DelAdd::Addition, &document_id)?;
-    Ok((del, add))
-}
-
 fn to_vector_map(
     obkv: KvReaderDelAdd,
     side: DelAdd,
@@ -256,10 +270,15 @@ fn push_vectors_diff(
     manual_vectors_writer: &mut Writer<BufWriter<File>>,
     key_buffer: &mut Vec<u8>,
     delta: VectorStateDelta,
+    settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<()> {
     puffin::profile_function!();
     let (must_remove, prompt, (mut del_vectors, mut add_vectors)) = delta.into_values();
-    if must_remove {
+    if must_remove
+    // TODO: the below condition works because we erase the vec database when a embedding setting changes.
+    // When vector pipeline will be optimized, this should be removed.
+    && !settings_diff.reindex_vectors()
+    {
         key_buffer.truncate(TRUNCATE_SIZE);
         remove_vectors_writer.insert(&key_buffer, [])?;
     }
@@ -287,12 +306,16 @@ fn push_vectors_diff(
         match eob {
             EitherOrBoth::Both(_, _) => (), // no need to touch anything
             EitherOrBoth::Left(vector) => {
-                // We insert only the Del part of the Obkv to inform
-                // that we only want to remove all those vectors.
-                let mut obkv = KvWriterDelAdd::memory();
-                obkv.insert(DelAdd::Deletion, cast_slice(&vector))?;
-                let bytes = obkv.into_inner()?;
-                manual_vectors_writer.insert(&key_buffer, bytes)?;
+                // TODO: the below condition works because we erase the vec database when a embedding setting changes.
+                // When vector pipeline will be optimized, this should be removed.
+                if !settings_diff.reindex_vectors() {
+                    // We insert only the Del part of the Obkv to inform
+                    // that we only want to remove all those vectors.
+                    let mut obkv = KvWriterDelAdd::memory();
+                    obkv.insert(DelAdd::Deletion, cast_slice(&vector))?;
+                    let bytes = obkv.into_inner()?;
+                    manual_vectors_writer.insert(&key_buffer, bytes)?;
+                }
             }
             EitherOrBoth::Right(vector) => {
                 // We insert only the Add part of the Obkv to inform
@@ -339,6 +362,7 @@ pub fn extract_embeddings<R: io::Read + io::Seek>(
     prompt_reader: grenad::Reader<R>,
     indexer: GrenadParameters,
     embedder: Arc<Embedder>,
+    request_threads: &ThreadPoolNoAbort,
 ) -> Result<grenad::Reader<BufReader<File>>> {
     puffin::profile_function!();
     let n_chunks = embedder.chunk_count_hint(); // chunk level parallelism
@@ -376,7 +400,10 @@ pub fn extract_embeddings<R: io::Read + io::Seek>(
 
         if chunks.len() == chunks.capacity() {
             let chunked_embeds = embedder
-                .embed_chunks(std::mem::replace(&mut chunks, Vec::with_capacity(n_chunks)))
+                .embed_chunks(
+                    std::mem::replace(&mut chunks, Vec::with_capacity(n_chunks)),
+                    request_threads,
+                )
                 .map_err(crate::vector::Error::from)
                 .map_err(crate::Error::from)?;
 
@@ -394,7 +421,7 @@ pub fn extract_embeddings<R: io::Read + io::Seek>(
     // send last chunk
     if !chunks.is_empty() {
         let chunked_embeds = embedder
-            .embed_chunks(std::mem::take(&mut chunks))
+            .embed_chunks(std::mem::take(&mut chunks), request_threads)
             .map_err(crate::vector::Error::from)
             .map_err(crate::Error::from)?;
         for (docid, embeddings) in chunks_ids
@@ -408,7 +435,7 @@ pub fn extract_embeddings<R: io::Read + io::Seek>(
 
     if !current_chunk.is_empty() {
         let embeds = embedder
-            .embed_chunks(vec![std::mem::take(&mut current_chunk)])
+            .embed_chunks(vec![std::mem::take(&mut current_chunk)], request_threads)
             .map_err(crate::vector::Error::from)
             .map_err(crate::Error::from)?;
 

milli/src/update/index_documents/extract/extract_word_docids.rs

@@ -1,20 +1,23 @@
-use std::collections::{BTreeSet, HashSet};
+use std::collections::BTreeSet;
 use std::fs::File;
 use std::io::{self, BufReader};
 
-use heed::BytesDecode;
+use heed::{BytesDecode, BytesEncode};
 use obkv::KvReaderU16;
+use roaring::RoaringBitmap;
 
 use super::helpers::{
-    create_sorter, create_writer, merge_deladd_cbo_roaring_bitmaps, sorter_into_reader,
-    try_split_array_at, writer_into_reader, GrenadParameters,
+    create_sorter, create_writer, merge_deladd_cbo_roaring_bitmaps, try_split_array_at,
+    writer_into_reader, GrenadParameters,
 };
 use crate::error::SerializationError;
 use crate::heed_codec::StrBEU16Codec;
 use crate::index::db_name::DOCID_WORD_POSITIONS;
 use crate::update::del_add::{is_noop_del_add_obkv, DelAdd, KvReaderDelAdd, KvWriterDelAdd};
+use crate::update::index_documents::helpers::sorter_into_reader;
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::update::MergeFn;
-use crate::{DocumentId, FieldId, Result};
+use crate::{CboRoaringBitmapCodec, DocumentId, FieldId, Result};
 
 /// Extracts the word and the documents ids where this word appear.
 ///
@@ -27,7 +30,7 @@ use crate::{DocumentId, FieldId, Result};
 pub fn extract_word_docids<R: io::Read + io::Seek>(
     docid_word_positions: grenad::Reader<R>,
     indexer: GrenadParameters,
-    exact_attributes: &HashSet<FieldId>,
+    settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<(
     grenad::Reader<BufReader<File>>,
     grenad::Reader<BufReader<File>>,
@@ -43,7 +46,7 @@ pub fn extract_word_docids<R: io::Read + io::Seek>(
         indexer.chunk_compression_type,
         indexer.chunk_compression_level,
         indexer.max_nb_chunks,
-        max_memory.map(|x| x / 3),
+        max_memory.map(|m| m / 3),
     );
     let mut key_buffer = Vec::new();
     let mut del_words = BTreeSet::new();
@@ -85,13 +88,19 @@ pub fn extract_word_docids<R: io::Read + io::Seek>(
         add_words.clear();
     }
 
+    let mut word_fid_docids_writer = create_writer(
+        indexer.chunk_compression_type,
+        indexer.chunk_compression_level,
+        tempfile::tempfile()?,
+    );
+
     let mut word_docids_sorter = create_sorter(
         grenad::SortAlgorithm::Unstable,
         merge_deladd_cbo_roaring_bitmaps,
         indexer.chunk_compression_type,
         indexer.chunk_compression_level,
         indexer.max_nb_chunks,
-        max_memory.map(|x| x / 3),
+        max_memory.map(|m| m / 3),
     );
 
     let mut exact_word_docids_sorter = create_sorter(
@@ -100,31 +109,45 @@ pub fn extract_word_docids<R: io::Read + io::Seek>(
         indexer.chunk_compression_type,
         indexer.chunk_compression_level,
         indexer.max_nb_chunks,
-        max_memory.map(|x| x / 3),
-    );
-
-    let mut word_fid_docids_writer = create_writer(
-        indexer.chunk_compression_type,
-        indexer.chunk_compression_level,
-        tempfile::tempfile()?,
+        max_memory.map(|m| m / 3),
     );
 
     let mut iter = word_fid_docids_sorter.into_stream_merger_iter()?;
-    // TODO: replace sorters by writers by accumulating values into a buffer before inserting them.
+    let mut buffer = Vec::new();
+    // NOTE: replacing sorters by bitmap merging is less efficient, so, use sorters.
     while let Some((key, value)) = iter.next()? {
         // only keep the value if their is a change to apply in the DB.
         if !is_noop_del_add_obkv(KvReaderDelAdd::new(value)) {
             word_fid_docids_writer.insert(key, value)?;
         }
 
-        let (word, fid) = StrBEU16Codec::bytes_decode(key)
+        let (w, fid) = StrBEU16Codec::bytes_decode(key)
             .map_err(|_| SerializationError::Decoding { db_name: Some(DOCID_WORD_POSITIONS) })?;
 
-        // every words contained in an attribute set to exact must be pushed in the exact_words list.
-        if exact_attributes.contains(&fid) {
-            exact_word_docids_sorter.insert(word.as_bytes(), value)?;
-        } else {
-            word_docids_sorter.insert(word.as_bytes(), value)?;
+        // merge all deletions
+        let obkv = KvReaderDelAdd::new(value);
+        if let Some(value) = obkv.get(DelAdd::Deletion) {
+            let delete_from_exact = settings_diff.old.exact_attributes.contains(&fid);
+            buffer.clear();
+            let mut obkv = KvWriterDelAdd::new(&mut buffer);
+            obkv.insert(DelAdd::Deletion, value)?;
+            if delete_from_exact {
+                exact_word_docids_sorter.insert(w, obkv.into_inner().unwrap())?;
+            } else {
+                word_docids_sorter.insert(w, obkv.into_inner().unwrap())?;
+            }
+        }
+        // merge all additions
+        if let Some(value) = obkv.get(DelAdd::Addition) {
+            let add_in_exact = settings_diff.new.exact_attributes.contains(&fid);
+            buffer.clear();
+            let mut obkv = KvWriterDelAdd::new(&mut buffer);
+            obkv.insert(DelAdd::Addition, value)?;
+            if add_in_exact {
+                exact_word_docids_sorter.insert(w, obkv.into_inner().unwrap())?;
+            } else {
+                word_docids_sorter.insert(w, obkv.into_inner().unwrap())?;
+            }
         }
     }
 
@@ -178,3 +201,45 @@ fn words_into_sorter(
 
     Ok(())
 }
+
+#[tracing::instrument(level = "trace", skip_all, target = "indexing::extract")]
+fn docids_into_writers<W>(
+    word: &str,
+    deletions: &RoaringBitmap,
+    additions: &RoaringBitmap,
+    writer: &mut grenad::Writer<W>,
+) -> Result<()>
+where
+    W: std::io::Write,
+{
+    if deletions == additions {
+        // if the same value is deleted and added, do nothing.
+        return Ok(());
+    }
+
+    // Write each value in the same KvDelAdd before inserting it in the final writer.
+    let mut obkv = KvWriterDelAdd::memory();
+    // deletions:
+    if !deletions.is_empty() && !deletions.is_subset(additions) {
+        obkv.insert(
+            DelAdd::Deletion,
+            CboRoaringBitmapCodec::bytes_encode(deletions).map_err(|_| {
+                SerializationError::Encoding { db_name: Some(DOCID_WORD_POSITIONS) }
+            })?,
+        )?;
+    }
+    // additions:
+    if !additions.is_empty() {
+        obkv.insert(
+            DelAdd::Addition,
+            CboRoaringBitmapCodec::bytes_encode(additions).map_err(|_| {
+                SerializationError::Encoding { db_name: Some(DOCID_WORD_POSITIONS) }
+            })?,
+        )?;
+    }
+
+    // insert everything in the same writer.
+    writer.insert(word.as_bytes(), obkv.into_inner().unwrap())?;
+
+    Ok(())
+}

milli/src/update/index_documents/extract/extract_word_pair_proximity_docids.rs

@@ -11,8 +11,9 @@ use super::helpers::{
 };
 use crate::error::SerializationError;
 use crate::index::db_name::DOCID_WORD_POSITIONS;
-use crate::proximity::{index_proximity, MAX_DISTANCE};
+use crate::proximity::{index_proximity, ProximityPrecision, MAX_DISTANCE};
 use crate::update::del_add::{DelAdd, KvReaderDelAdd, KvWriterDelAdd};
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::{DocumentId, Result};
 
 /// Extracts the best proximity between pairs of words and the documents ids where this pair appear.
@@ -23,8 +24,21 @@ use crate::{DocumentId, Result};
 pub fn extract_word_pair_proximity_docids<R: io::Read + io::Seek>(
     docid_word_positions: grenad::Reader<R>,
     indexer: GrenadParameters,
+    settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<grenad::Reader<BufReader<File>>> {
     puffin::profile_function!();
+    let any_deletion = settings_diff.old.proximity_precision == ProximityPrecision::ByWord;
+    let any_addition = settings_diff.new.proximity_precision == ProximityPrecision::ByWord;
+
+    // early return if the data shouldn't be deleted nor created.
+    if !any_deletion && !any_addition {
+        let writer = create_writer(
+            indexer.chunk_compression_type,
+            indexer.chunk_compression_level,
+            tempfile::tempfile()?,
+        );
+        return writer_into_reader(writer);
+    }
 
     let max_memory = indexer.max_memory_by_thread();
 
@@ -77,6 +91,10 @@ pub fn extract_word_pair_proximity_docids<R: io::Read + io::Seek>(
 
         let (del, add): (Result<_>, Result<_>) = rayon::join(
             || {
+                if !any_deletion {
+                    return Ok(());
+                }
+
                 // deletions
                 if let Some(deletion) = KvReaderDelAdd::new(value).get(DelAdd::Deletion) {
                     for (position, word) in KvReaderU16::new(deletion).iter() {
@@ -106,6 +124,10 @@ pub fn extract_word_pair_proximity_docids<R: io::Read + io::Seek>(
                 Ok(())
             },
             || {
+                if !any_addition {
+                    return Ok(());
+                }
+
                 // additions
                 if let Some(addition) = KvReaderDelAdd::new(value).get(DelAdd::Addition) {
                     for (position, word) in KvReaderU16::new(addition).iter() {

milli/src/update/index_documents/extract/extract_word_position_docids.rs

@@ -11,6 +11,7 @@ use super::helpers::{
 use crate::error::SerializationError;
 use crate::index::db_name::DOCID_WORD_POSITIONS;
 use crate::update::del_add::{DelAdd, KvReaderDelAdd, KvWriterDelAdd};
+use crate::update::settings::InnerIndexSettingsDiff;
 use crate::update::MergeFn;
 use crate::{bucketed_position, DocumentId, Result};
 
@@ -22,6 +23,7 @@ use crate::{bucketed_position, DocumentId, Result};
 pub fn extract_word_position_docids<R: io::Read + io::Seek>(
     docid_word_positions: grenad::Reader<R>,
     indexer: GrenadParameters,
+    _settings_diff: &InnerIndexSettingsDiff,
 ) -> Result<grenad::Reader<BufReader<File>>> {
     puffin::profile_function!();
 

milli/src/update/index_documents/extract/mod.rs

@@ -9,9 +9,9 @@ mod extract_word_docids;
 mod extract_word_pair_proximity_docids;
 mod extract_word_position_docids;
 
-use std::collections::HashSet;
 use std::fs::File;
 use std::io::BufReader;
+use std::sync::Arc;
 
 use crossbeam_channel::Sender;
 use rayon::prelude::*;
@@ -30,9 +30,8 @@ use self::extract_word_pair_proximity_docids::extract_word_pair_proximity_docids
 use self::extract_word_position_docids::extract_word_position_docids;
 use super::helpers::{as_cloneable_grenad, CursorClonableMmap, GrenadParameters};
 use super::{helpers, TypedChunk};
-use crate::proximity::ProximityPrecision;
-use crate::vector::EmbeddingConfigs;
-use crate::{FieldId, FieldsIdsMap, Result};
+use crate::update::settings::InnerIndexSettingsDiff;
+use crate::{FieldId, Result, ThreadPoolNoAbortBuilder};
 
 /// Extract data for each databases from obkv documents in parallel.
 /// Send data in grenad file over provided Sender.
@@ -43,18 +42,10 @@ pub(crate) fn data_from_obkv_documents(
     flattened_obkv_chunks: impl Iterator<Item = Result<grenad::Reader<BufReader<File>>>> + Send,
     indexer: GrenadParameters,
     lmdb_writer_sx: Sender<Result<TypedChunk>>,
-    searchable_fields: Option<HashSet<FieldId>>,
-    faceted_fields: HashSet<FieldId>,
     primary_key_id: FieldId,
     geo_fields_ids: Option<(FieldId, FieldId)>,
-    field_id_map: FieldsIdsMap,
-    stop_words: Option<fst::Set<Vec<u8>>>,
-    allowed_separators: Option<&[&str]>,
-    dictionary: Option<&[&str]>,
+    settings_diff: Arc<InnerIndexSettingsDiff>,
     max_positions_per_attributes: Option<u32>,
-    exact_attributes: HashSet<FieldId>,
-    proximity_precision: ProximityPrecision,
-    embedders: EmbeddingConfigs,
 ) -> Result<()> {
     puffin::profile_function!();
 
@@ -67,8 +58,7 @@ pub(crate) fn data_from_obkv_documents(
                         original_documents_chunk,
                         indexer,
                         lmdb_writer_sx.clone(),
-                        field_id_map.clone(),
-                        embedders.clone(),
+                        settings_diff.clone(),
                     )
                 })
                 .collect::<Result<()>>()
@@ -81,13 +71,9 @@ pub(crate) fn data_from_obkv_documents(
                         flattened_obkv_chunks,
                         indexer,
                         lmdb_writer_sx.clone(),
-                        &searchable_fields,
-                        &faceted_fields,
                         primary_key_id,
                         geo_fields_ids,
-                        &stop_words,
-                        &allowed_separators,
-                        &dictionary,
+                        settings_diff.clone(),
                         max_positions_per_attributes,
                     )
                 })
@@ -100,13 +86,12 @@ pub(crate) fn data_from_obkv_documents(
                         run_extraction_task::<_, _, grenad::Reader<BufReader<File>>>(
                             docid_word_positions_chunk.clone(),
                             indexer,
+                            settings_diff.clone(),
                             lmdb_writer_sx.clone(),
                             extract_fid_word_count_docids,
                             TypedChunk::FieldIdWordCountDocids,
                             "field-id-wordcount-docids",
                         );
-
-                        let exact_attributes = exact_attributes.clone();
                         run_extraction_task::<
                             _,
                             _,
@@ -118,10 +103,9 @@ pub(crate) fn data_from_obkv_documents(
                         >(
                             docid_word_positions_chunk.clone(),
                             indexer,
+                            settings_diff.clone(),
                             lmdb_writer_sx.clone(),
-                            move |doc_word_pos, indexer| {
-                                extract_word_docids(doc_word_pos, indexer, &exact_attributes)
-                            },
+                            extract_word_docids,
                             |(
                                 word_docids_reader,
                                 exact_word_docids_reader,
@@ -139,6 +123,7 @@ pub(crate) fn data_from_obkv_documents(
                         run_extraction_task::<_, _, grenad::Reader<BufReader<File>>>(
                             docid_word_positions_chunk.clone(),
                             indexer,
+                            settings_diff.clone(),
                             lmdb_writer_sx.clone(),
                             extract_word_position_docids,
                             TypedChunk::WordPositionDocids,
@@ -152,6 +137,7 @@ pub(crate) fn data_from_obkv_documents(
                         >(
                             fid_docid_facet_strings_chunk.clone(),
                             indexer,
+                            settings_diff.clone(),
                             lmdb_writer_sx.clone(),
                             extract_facet_string_docids,
                             TypedChunk::FieldIdFacetStringDocids,
@@ -161,22 +147,22 @@ pub(crate) fn data_from_obkv_documents(
                         run_extraction_task::<_, _, grenad::Reader<BufReader<File>>>(
                             fid_docid_facet_numbers_chunk.clone(),
                             indexer,
+                            settings_diff.clone(),
                             lmdb_writer_sx.clone(),
                             extract_facet_number_docids,
                             TypedChunk::FieldIdFacetNumberDocids,
                             "field-id-facet-number-docids",
                         );
 
-                        if proximity_precision == ProximityPrecision::ByWord {
-                            run_extraction_task::<_, _, grenad::Reader<BufReader<File>>>(
-                                docid_word_positions_chunk.clone(),
-                                indexer,
-                                lmdb_writer_sx.clone(),
-                                extract_word_pair_proximity_docids,
-                                TypedChunk::WordPairProximityDocids,
-                                "word-pair-proximity-docids",
-                            );
-                        }
+                        run_extraction_task::<_, _, grenad::Reader<BufReader<File>>>(
+                            docid_word_positions_chunk.clone(),
+                            indexer,
+                            settings_diff.clone(),
+                            lmdb_writer_sx.clone(),
+                            extract_word_pair_proximity_docids,
+                            TypedChunk::WordPairProximityDocids,
+                            "word-pair-proximity-docids",
+                        );
                     }
 
                     Ok(())
@@ -195,12 +181,17 @@ pub(crate) fn data_from_obkv_documents(
 fn run_extraction_task<FE, FS, M>(
     chunk: grenad::Reader<CursorClonableMmap>,
     indexer: GrenadParameters,
+    settings_diff: Arc<InnerIndexSettingsDiff>,
     lmdb_writer_sx: Sender<Result<TypedChunk>>,
     extract_fn: FE,
     serialize_fn: FS,
     name: &'static str,
 ) where
-    FE: Fn(grenad::Reader<CursorClonableMmap>, GrenadParameters) -> Result<M>
+    FE: Fn(
+            grenad::Reader<CursorClonableMmap>,
+            GrenadParameters,
+            &InnerIndexSettingsDiff,
+        ) -> Result<M>
         + Sync
         + Send
         + 'static,
@@ -213,7 +204,7 @@ fn run_extraction_task<FE, FS, M>(
         let child_span = tracing::trace_span!(target: "indexing::extract::details", parent: &current_span, "extract_multiple_chunks");
         let _entered = child_span.enter();
         puffin::profile_scope!("extract_multiple_chunks", name);
-        match extract_fn(chunk, indexer) {
+        match extract_fn(chunk, indexer, &settings_diff) {
             Ok(chunk) => {
                 let _ = lmdb_writer_sx.send(Ok(serialize_fn(chunk)));
             }
@@ -230,53 +221,69 @@ fn send_original_documents_data(
     original_documents_chunk: Result<grenad::Reader<BufReader<File>>>,
     indexer: GrenadParameters,
     lmdb_writer_sx: Sender<Result<TypedChunk>>,
-    field_id_map: FieldsIdsMap,
-    embedders: EmbeddingConfigs,
+    settings_diff: Arc<InnerIndexSettingsDiff>,
 ) -> Result<()> {
     let original_documents_chunk =
         original_documents_chunk.and_then(|c| unsafe { as_cloneable_grenad(&c) })?;
 
     let documents_chunk_cloned = original_documents_chunk.clone();
     let lmdb_writer_sx_cloned = lmdb_writer_sx.clone();
-    rayon::spawn(move || {
-        for (name, (embedder, prompt)) in embedders {
-            let result = extract_vector_points(
-                documents_chunk_cloned.clone(),
-                indexer,
-                &field_id_map,
-                &prompt,
-                &name,
-            );
-            match result {
-                Ok(ExtractedVectorPoints { manual_vectors, remove_vectors, prompts }) => {
-                    let embeddings = match extract_embeddings(prompts, indexer, embedder.clone()) {
-                        Ok(results) => Some(results),
-                        Err(error) => {
-                            let _ = lmdb_writer_sx_cloned.send(Err(error));
-                            None
-                        }
-                    };
 
-                    if !(remove_vectors.is_empty()
-                        && manual_vectors.is_empty()
-                        && embeddings.as_ref().map_or(true, |e| e.is_empty()))
-                    {
-                        let _ = lmdb_writer_sx_cloned.send(Ok(TypedChunk::VectorPoints {
-                            remove_vectors,
-                            embeddings,
-                            expected_dimension: embedder.dimensions(),
-                            manual_vectors,
-                            embedder_name: name,
-                        }));
+    let new_embedding_configs = settings_diff.new.embedding_configs.clone();
+
+    if (settings_diff.reindex_vectors() || !settings_diff.settings_update_only())
+        && new_embedding_configs.get_default().is_some()
+    {
+        let request_threads = ThreadPoolNoAbortBuilder::new()
+            .num_threads(crate::vector::REQUEST_PARALLELISM)
+            .thread_name(|index| format!("embedding-request-{index}"))
+            .build()?;
+        let settings_diff = settings_diff.clone();
+        rayon::spawn(move || {
+            for (name, (embedder, prompt)) in settings_diff.new.embedding_configs.clone() {
+                let result = extract_vector_points(
+                    documents_chunk_cloned.clone(),
+                    indexer,
+                    &settings_diff,
+                    &prompt,
+                    &name,
+                );
+                match result {
+                    Ok(ExtractedVectorPoints { manual_vectors, remove_vectors, prompts }) => {
+                        let embeddings = match extract_embeddings(
+                            prompts,
+                            indexer,
+                            embedder.clone(),
+                            &request_threads,
+                        ) {
+                            Ok(results) => Some(results),
+                            Err(error) => {
+                                let _ = lmdb_writer_sx_cloned.send(Err(error));
+                                None
+                            }
+                        };
+
+                        if !(remove_vectors.is_empty()
+                            && manual_vectors.is_empty()
+                            && embeddings.as_ref().map_or(true, |e| e.is_empty()))
+                        {
+                            let _ = lmdb_writer_sx_cloned.send(Ok(TypedChunk::VectorPoints {
+                                remove_vectors,
+                                embeddings,
+                                expected_dimension: embedder.dimensions(),
+                                manual_vectors,
+                                embedder_name: name,
+                            }));
+                        }
+                    }
+
+                    Err(error) => {
+                        let _ = lmdb_writer_sx_cloned.send(Err(error));
                     }
                 }
-
-                Err(error) => {
-                    let _ = lmdb_writer_sx_cloned.send(Err(error));
-                }
             }
-        }
-    });
+        });
+    }
 
     // TODO: create a custom internal error
     let _ = lmdb_writer_sx.send(Ok(TypedChunk::Documents(original_documents_chunk)));
@@ -295,13 +302,9 @@ fn send_and_extract_flattened_documents_data(
     flattened_documents_chunk: Result<grenad::Reader<BufReader<File>>>,
     indexer: GrenadParameters,
     lmdb_writer_sx: Sender<Result<TypedChunk>>,
-    searchable_fields: &Option<HashSet<FieldId>>,
-    faceted_fields: &HashSet<FieldId>,
     primary_key_id: FieldId,
     geo_fields_ids: Option<(FieldId, FieldId)>,
-    stop_words: &Option<fst::Set<Vec<u8>>>,
-    allowed_separators: &Option<&[&str]>,
-    dictionary: &Option<&[&str]>,
+    settings_diff: Arc<InnerIndexSettingsDiff>,
     max_positions_per_attributes: Option<u32>,
 ) -> Result<(
     grenad::Reader<CursorClonableMmap>,
@@ -330,10 +333,7 @@ fn send_and_extract_flattened_documents_data(
                     extract_docid_word_positions(
                         flattened_documents_chunk.clone(),
                         indexer,
-                        searchable_fields,
-                        stop_words.as_ref(),
-                        *allowed_separators,
-                        *dictionary,
+                        &settings_diff,
                         max_positions_per_attributes,
                     )?;
 
@@ -356,7 +356,7 @@ fn send_and_extract_flattened_documents_data(
                 } = extract_fid_docid_facet_values(
                     flattened_documents_chunk.clone(),
                     indexer,
-                    faceted_fields,
+                    &settings_diff,
                     geo_fields_ids,
                 )?;
 

milli/src/update/index_documents/mod.rs

@@ -6,9 +6,9 @@ mod typed_chunk;
 
 use std::collections::{HashMap, HashSet};
 use std::io::{Read, Seek};
-use std::iter::FromIterator;
 use std::num::NonZeroU32;
 use std::result::Result as StdResult;
+use std::sync::Arc;
 
 use crossbeam_channel::{Receiver, Sender};
 use grenad::{Merger, MergerBuilder};
@@ -33,6 +33,7 @@ use self::helpers::{grenad_obkv_into_chunks, GrenadParameters};
 pub use self::transform::{Transform, TransformOutput};
 use crate::documents::{obkv_to_object, DocumentsBatchReader};
 use crate::error::{Error, InternalError, UserError};
+use crate::thread_pool_no_abort::ThreadPoolNoAbortBuilder;
 pub use crate::update::index_documents::helpers::CursorClonableMmap;
 use crate::update::{
     IndexerConfig, UpdateIndexingStep, WordPrefixDocids, WordPrefixIntegerDocids, WordsPrefixesFst,
@@ -259,21 +260,6 @@ where
             .expect("Invalid document addition state")
             .output_from_sorter(self.wtxn, &self.progress)?;
 
-        let new_facets = output.compute_real_facets(self.wtxn, self.index)?;
-        self.index.put_faceted_fields(self.wtxn, &new_facets)?;
-
-        // in case new fields were introduced we're going to recreate the searchable fields.
-        if let Some(faceted_fields) = self.index.user_defined_searchable_fields(self.wtxn)? {
-            // we can't keep references on the faceted fields while we update the index thus we need to own it.
-            let faceted_fields: Vec<String> =
-                faceted_fields.into_iter().map(str::to_string).collect();
-            self.index.put_all_searchable_fields_from_fields_ids_map(
-                self.wtxn,
-                &faceted_fields.iter().map(String::as_ref).collect::<Vec<_>>(),
-                &output.fields_ids_map,
-            )?;
-        }
-
         let indexed_documents = output.documents_count as u64;
         let number_of_documents = self.execute_raw(output)?;
 
@@ -296,32 +282,35 @@ where
 
         let TransformOutput {
             primary_key,
-            fields_ids_map,
+            mut settings_diff,
             field_distribution,
             documents_count,
             original_documents,
             flattened_documents,
         } = output;
 
-        // The fields_ids_map is put back to the store now so the rest of the transaction sees an
-        // up to date field map.
-        self.index.put_fields_ids_map(self.wtxn, &fields_ids_map)?;
+        // update the internal facet and searchable list,
+        // because they might have changed due to the nested documents flattening.
+        settings_diff.new.recompute_facets(self.wtxn, self.index)?;
+        settings_diff.new.recompute_searchables(self.wtxn, self.index)?;
+
+        let settings_diff = Arc::new(settings_diff);
 
         let backup_pool;
         let pool = match self.indexer_config.thread_pool {
             Some(ref pool) => pool,
-            #[cfg(not(test))]
             None => {
-                // We initialize a bakcup pool with the default
+                // We initialize a backup pool with the default
                 // settings if none have already been set.
-                backup_pool = rayon::ThreadPoolBuilder::new().build()?;
-                &backup_pool
-            }
-            #[cfg(test)]
-            None => {
-                // We initialize a bakcup pool with the default
-                // settings if none have already been set.
-                backup_pool = rayon::ThreadPoolBuilder::new().num_threads(1).build()?;
+                #[allow(unused_mut)]
+                let mut pool_builder = ThreadPoolNoAbortBuilder::new();
+
+                #[cfg(test)]
+                {
+                    pool_builder = pool_builder.num_threads(1);
+                }
+
+                backup_pool = pool_builder.build()?;
                 &backup_pool
             }
         };
@@ -333,13 +322,8 @@ where
         ) = crossbeam_channel::unbounded();
 
         // get the primary key field id
-        let primary_key_id = fields_ids_map.id(&primary_key).unwrap();
+        let primary_key_id = settings_diff.new.fields_ids_map.id(&primary_key).unwrap();
 
-        // get searchable fields for word databases
-        let searchable_fields =
-            self.index.searchable_fields_ids(self.wtxn)?.map(HashSet::from_iter);
-        // get filterable fields for facet databases
-        let faceted_fields = self.index.faceted_fields_ids(self.wtxn)?;
         // get the fid of the `_geo.lat` and `_geo.lng` fields.
         let mut field_id_map = self.index.fields_ids_map(self.wtxn)?;
 
@@ -362,12 +346,6 @@ where
             None => None,
         };
 
-        let stop_words = self.index.stop_words(self.wtxn)?;
-        let separators = self.index.allowed_separators(self.wtxn)?;
-        let dictionary = self.index.dictionary(self.wtxn)?;
-        let exact_attributes = self.index.exact_attributes_ids(self.wtxn)?;
-        let proximity_precision = self.index.proximity_precision(self.wtxn)?.unwrap_or_default();
-
         let pool_params = GrenadParameters {
             chunk_compression_type: self.indexer_config.chunk_compression_type,
             chunk_compression_level: self.indexer_config.chunk_compression_level,
@@ -400,8 +378,6 @@ where
 
         let max_positions_per_attributes = self.indexer_config.max_positions_per_attributes;
 
-        let cloned_embedder = self.embedders.clone();
-
         let mut final_documents_ids = RoaringBitmap::new();
         let mut databases_seen = 0;
         let mut word_position_docids = None;
@@ -410,7 +386,6 @@ where
         let mut exact_word_docids = None;
         let mut chunk_accumulator = ChunkAccumulator::default();
         let mut dimension = HashMap::new();
-        let stop_words = stop_words.map(|sw| sw.map_data(Vec::from).unwrap());
 
         let current_span = tracing::Span::current();
 
@@ -428,10 +403,6 @@ where
                 let flattened_chunk_iter =
                     grenad_obkv_into_chunks(flattened_documents, pool_params, documents_chunk_size);
 
-                let separators: Option<Vec<_>> =
-                    separators.as_ref().map(|x| x.iter().map(String::as_str).collect());
-                let dictionary: Option<Vec<_>> =
-                    dictionary.as_ref().map(|x| x.iter().map(String::as_str).collect());
                 let result = original_chunk_iter.and_then(|original_chunk| {
                     let flattened_chunk = flattened_chunk_iter?;
                     // extract all databases from the chunked obkv douments
@@ -440,18 +411,10 @@ where
                         flattened_chunk,
                         pool_params,
                         lmdb_writer_sx.clone(),
-                        searchable_fields,
-                        faceted_fields,
                         primary_key_id,
                         geo_fields_ids,
-                        field_id_map,
-                        stop_words,
-                        separators.as_deref(),
-                        dictionary.as_deref(),
+                        settings_diff.clone(),
                         max_positions_per_attributes,
-                        exact_attributes,
-                        proximity_precision,
-                        cloned_embedder,
                     )
                 });
 
@@ -571,7 +534,7 @@ where
             }
 
             Ok(())
-        })?;
+        }).map_err(InternalError::from)??;
 
         // We write the field distribution into the main database
         self.index.put_field_distribution(self.wtxn, &field_distribution)?;
@@ -600,7 +563,8 @@ where
                     writer.build(wtxn, &mut rng, None)?;
                 }
                 Result::Ok(())
-            })?;
+            })
+            .map_err(InternalError::from)??;
         }
 
         self.execute_prefix_databases(
@@ -2646,6 +2610,13 @@ mod tests {
                         api_key: Setting::NotSet,
                         dimensions: Setting::Set(3),
                         document_template: Setting::NotSet,
+                        url: Setting::NotSet,
+                        query: Setting::NotSet,
+                        input_field: Setting::NotSet,
+                        path_to_embeddings: Setting::NotSet,
+                        embedding_object: Setting::NotSet,
+                        input_type: Setting::NotSet,
+                        distribution: Setting::NotSet,
                     }),
                 );
                 settings.set_embedder_settings(embedders);
@@ -2665,7 +2636,16 @@ mod tests {
                .unwrap();
 
         let rtxn = index.read_txn().unwrap();
-        let res = index.search(&rtxn).vector([0.0, 1.0, 2.0].to_vec()).execute().unwrap();
+        let mut embedding_configs = index.embedding_configs(&rtxn).unwrap();
+        let (embedder_name, embedder) = embedding_configs.pop().unwrap();
+        let embedder =
+            std::sync::Arc::new(crate::vector::Embedder::new(embedder.embedder_options).unwrap());
+        assert_eq!("manual", embedder_name);
+        let res = index
+            .search(&rtxn)
+            .semantic(embedder_name, embedder, Some([0.0, 1.0, 2.0].to_vec()))
+            .execute()
+            .unwrap();
         assert_eq!(res.documents_ids.len(), 3);
     }
 

milli/src/update/index_documents/transform.rs

@@ -1,12 +1,11 @@
 use std::borrow::Cow;
 use std::collections::btree_map::Entry as BEntry;
 use std::collections::hash_map::Entry as HEntry;
-use std::collections::{HashMap, HashSet};
+use std::collections::HashMap;
 use std::fs::File;
 use std::io::{Read, Seek};
 
 use fxhash::FxHashMap;
-use heed::RoTxn;
 use itertools::Itertools;
 use obkv::{KvReader, KvReaderU16, KvWriter};
 use roaring::RoaringBitmap;
@@ -21,14 +20,17 @@ use super::{IndexDocumentsMethod, IndexerConfig};
 use crate::documents::{DocumentsBatchIndex, EnrichedDocument, EnrichedDocumentsBatchReader};
 use crate::error::{Error, InternalError, UserError};
 use crate::index::{db_name, main_key};
-use crate::update::del_add::{into_del_add_obkv, DelAdd, DelAddOperation, KvReaderDelAdd};
+use crate::update::del_add::{
+    del_add_from_two_obkvs, into_del_add_obkv, DelAdd, DelAddOperation, KvReaderDelAdd,
+};
 use crate::update::index_documents::GrenadParameters;
-use crate::update::{AvailableDocumentsIds, ClearDocuments, UpdateIndexingStep};
+use crate::update::settings::{InnerIndexSettings, InnerIndexSettingsDiff};
+use crate::update::{AvailableDocumentsIds, UpdateIndexingStep};
 use crate::{FieldDistribution, FieldId, FieldIdMapMissingEntry, FieldsIdsMap, Index, Result};
 
 pub struct TransformOutput {
     pub primary_key: String,
-    pub fields_ids_map: FieldsIdsMap,
+    pub settings_diff: InnerIndexSettingsDiff,
     pub field_distribution: FieldDistribution,
     pub documents_count: usize,
     pub original_documents: File,
@@ -282,7 +284,9 @@ impl<'a, 'i> Transform<'a, 'i> {
                     self.original_sorter
                         .insert(&document_sorter_key_buffer, &document_sorter_value_buffer)?;
                     let base_obkv = KvReader::new(base_obkv);
-                    if let Some(flattened_obkv) = self.flatten_from_fields_ids_map(base_obkv)? {
+                    if let Some(flattened_obkv) =
+                        Self::flatten_from_fields_ids_map(&base_obkv, &mut self.fields_ids_map)?
+                    {
                         // we recreate our buffer with the flattened documents
                         document_sorter_value_buffer.clear();
                         document_sorter_value_buffer.push(Operation::Addition as u8);
@@ -315,7 +319,9 @@ impl<'a, 'i> Transform<'a, 'i> {
                     .insert(&document_sorter_key_buffer, &document_sorter_value_buffer)?;
 
                 let flattened_obkv = KvReader::new(&obkv_buffer);
-                if let Some(obkv) = self.flatten_from_fields_ids_map(flattened_obkv)? {
+                if let Some(obkv) =
+                    Self::flatten_from_fields_ids_map(&flattened_obkv, &mut self.fields_ids_map)?
+                {
                     document_sorter_value_buffer.clear();
                     document_sorter_value_buffer.push(Operation::Addition as u8);
                     into_del_add_obkv(
@@ -524,7 +530,9 @@ impl<'a, 'i> Transform<'a, 'i> {
 
         // flatten it and push it as to delete in the flattened_sorter
         let flattened_obkv = KvReader::new(base_obkv);
-        if let Some(obkv) = self.flatten_from_fields_ids_map(flattened_obkv)? {
+        if let Some(obkv) =
+            Self::flatten_from_fields_ids_map(&flattened_obkv, &mut self.fields_ids_map)?
+        {
             // we recreate our buffer with the flattened documents
             document_sorter_value_buffer.clear();
             document_sorter_value_buffer.push(Operation::Deletion as u8);
@@ -541,8 +549,15 @@ impl<'a, 'i> Transform<'a, 'i> {
 
     // Flatten a document from the fields ids map contained in self and insert the new
     // created fields. Returns `None` if the document doesn't need to be flattened.
-    #[tracing::instrument(level = "trace", skip(self, obkv), target = "indexing::transform")]
-    fn flatten_from_fields_ids_map(&mut self, obkv: KvReader<FieldId>) -> Result<Option<Vec<u8>>> {
+    #[tracing::instrument(
+        level = "trace",
+        skip(obkv, fields_ids_map),
+        target = "indexing::transform"
+    )]
+    fn flatten_from_fields_ids_map(
+        obkv: &KvReader<FieldId>,
+        fields_ids_map: &mut FieldsIdsMap,
+    ) -> Result<Option<Vec<u8>>> {
         if obkv
             .iter()
             .all(|(_, value)| !json_depth_checker::should_flatten_from_unchecked_slice(value))
@@ -563,7 +578,7 @@ impl<'a, 'i> Transform<'a, 'i> {
         // all the raw values get inserted directly in the `key_value` vec.
         for (key, value) in obkv.iter() {
             if json_depth_checker::should_flatten_from_unchecked_slice(value) {
-                let key = self.fields_ids_map.name(key).ok_or(FieldIdMapMissingEntry::FieldId {
+                let key = fields_ids_map.name(key).ok_or(FieldIdMapMissingEntry::FieldId {
                     field_id: key,
                     process: "Flatten from fields ids map.",
                 })?;
@@ -581,7 +596,7 @@ impl<'a, 'i> Transform<'a, 'i> {
         // Once we have the flattened version we insert all the new generated fields_ids
         // (if any) in the fields ids map and serialize the value.
         for (key, value) in flattened.into_iter() {
-            let fid = self.fields_ids_map.insert(&key).ok_or(UserError::AttributeLimitReached)?;
+            let fid = fields_ids_map.insert(&key).ok_or(UserError::AttributeLimitReached)?;
             let value = serde_json::to_vec(&value).map_err(InternalError::SerdeJson)?;
             key_value.push((fid, value.into()));
         }
@@ -792,9 +807,19 @@ impl<'a, 'i> Transform<'a, 'i> {
             fst_new_external_documents_ids_builder.insert(key, value)
         })?;
 
+        let old_inner_settings = InnerIndexSettings::from_index(self.index, wtxn)?;
+        let mut new_inner_settings = old_inner_settings.clone();
+        new_inner_settings.fields_ids_map = self.fields_ids_map;
+        let settings_diff = InnerIndexSettingsDiff {
+            old: old_inner_settings,
+            new: new_inner_settings,
+            embedding_configs_updated: false,
+            settings_update_only: false,
+        };
+
         Ok(TransformOutput {
             primary_key,
-            fields_ids_map: self.fields_ids_map,
+            settings_diff,
             field_distribution,
             documents_count: self.documents_count,
             original_documents: original_documents.into_inner().map_err(|err| err.into_error())?,
@@ -804,6 +829,44 @@ impl<'a, 'i> Transform<'a, 'i> {
         })
     }
 
+    /// Rebind the field_ids of the provided document to their values
+    /// based on the field_ids_maps difference between the old and the new settings,
+    /// then fill the provided buffers with delta documents using KvWritterDelAdd.
+    fn rebind_existing_document(
+        old_obkv: KvReader<FieldId>,
+        settings_diff: &InnerIndexSettingsDiff,
+        original_obkv_buffer: &mut Vec<u8>,
+        flattened_obkv_buffer: &mut Vec<u8>,
+    ) -> Result<()> {
+        let mut old_fields_ids_map = settings_diff.old.fields_ids_map.clone();
+        let mut new_fields_ids_map = settings_diff.new.fields_ids_map.clone();
+        let mut obkv_writer = KvWriter::<_, FieldId>::memory();
+        // We iterate over the new `FieldsIdsMap` ids in order and construct the new obkv.
+        for (id, name) in new_fields_ids_map.iter() {
+            if let Some(val) = old_fields_ids_map.id(name).and_then(|id| old_obkv.get(id)) {
+                obkv_writer.insert(id, val)?;
+            }
+        }
+        let data = obkv_writer.into_inner()?;
+        let new_obkv = KvReader::<FieldId>::new(&data);
+
+        // take the non-flattened version if flatten_from_fields_ids_map returns None.
+        let old_flattened = Self::flatten_from_fields_ids_map(&old_obkv, &mut old_fields_ids_map)?;
+        let old_flattened =
+            old_flattened.as_deref().map_or_else(|| old_obkv, KvReader::<FieldId>::new);
+        let new_flattened = Self::flatten_from_fields_ids_map(&new_obkv, &mut new_fields_ids_map)?;
+        let new_flattened =
+            new_flattened.as_deref().map_or_else(|| new_obkv, KvReader::<FieldId>::new);
+
+        original_obkv_buffer.clear();
+        flattened_obkv_buffer.clear();
+
+        del_add_from_two_obkvs(&old_obkv, &new_obkv, original_obkv_buffer)?;
+        del_add_from_two_obkvs(&old_flattened, &new_flattened, flattened_obkv_buffer)?;
+
+        Ok(())
+    }
+
     /// Clear all databases. Returns a `TransformOutput` with a file that contains the documents
     /// of the index with the attributes reordered accordingly to the `FieldsIdsMap` given as argument.
     ///
@@ -811,8 +874,7 @@ impl<'a, 'i> Transform<'a, 'i> {
     pub fn prepare_for_documents_reindexing(
         self,
         wtxn: &mut heed::RwTxn<'i>,
-        old_fields_ids_map: FieldsIdsMap,
-        mut new_fields_ids_map: FieldsIdsMap,
+        settings_diff: InnerIndexSettingsDiff,
     ) -> Result<TransformOutput> {
         // There already has been a document addition, the primary key should be set by now.
         let primary_key = self
@@ -848,78 +910,27 @@ impl<'a, 'i> Transform<'a, 'i> {
             self.indexer_settings.max_memory.map(|mem| mem / 2),
         );
 
-        let mut obkv_buffer = Vec::new();
+        let mut original_obkv_buffer = Vec::new();
+        let mut flattened_obkv_buffer = Vec::new();
         let mut document_sorter_key_buffer = Vec::new();
-        let mut document_sorter_value_buffer = Vec::new();
         for result in self.index.external_documents_ids().iter(wtxn)? {
             let (external_id, docid) = result?;
-            let obkv = self.index.documents.get(wtxn, &docid)?.ok_or(
+            let old_obkv = self.index.documents.get(wtxn, &docid)?.ok_or(
                 InternalError::DatabaseMissingEntry { db_name: db_name::DOCUMENTS, key: None },
             )?;
 
-            obkv_buffer.clear();
-            let mut obkv_writer = KvWriter::<_, FieldId>::new(&mut obkv_buffer);
-
-            // We iterate over the new `FieldsIdsMap` ids in order and construct the new obkv.
-            for (id, name) in new_fields_ids_map.iter() {
-                if let Some(val) = old_fields_ids_map.id(name).and_then(|id| obkv.get(id)) {
-                    obkv_writer.insert(id, val)?;
-                }
-            }
-
-            let buffer = obkv_writer.into_inner()?;
+            Self::rebind_existing_document(
+                old_obkv,
+                &settings_diff,
+                &mut original_obkv_buffer,
+                &mut flattened_obkv_buffer,
+            )?;
 
             document_sorter_key_buffer.clear();
             document_sorter_key_buffer.extend_from_slice(&docid.to_be_bytes());
             document_sorter_key_buffer.extend_from_slice(external_id.as_bytes());
-            document_sorter_value_buffer.clear();
-            into_del_add_obkv(
-                KvReaderU16::new(buffer),
-                DelAddOperation::Addition,
-                &mut document_sorter_value_buffer,
-            )?;
-            original_sorter.insert(&document_sorter_key_buffer, &document_sorter_value_buffer)?;
-
-            // Once we have the document. We're going to flatten it
-            // and insert it in the flattened sorter.
-            let mut doc = serde_json::Map::new();
-
-            let reader = obkv::KvReader::new(buffer);
-            for (k, v) in reader.iter() {
-                let key = new_fields_ids_map.name(k).ok_or(FieldIdMapMissingEntry::FieldId {
-                    field_id: k,
-                    process: "Accessing field distribution in transform.",
-                })?;
-                let value = serde_json::from_slice::<serde_json::Value>(v)
-                    .map_err(InternalError::SerdeJson)?;
-                doc.insert(key.to_string(), value);
-            }
-
-            let flattened = flatten_serde_json::flatten(&doc);
-
-            // Once we have the flattened version we can convert it back to obkv and
-            // insert all the new generated fields_ids (if any) in the fields ids map.
-            let mut buffer: Vec<u8> = Vec::new();
-            let mut writer = KvWriter::new(&mut buffer);
-            let mut flattened: Vec<_> = flattened.into_iter().collect();
-            // we reorder the field to get all the known field first
-            flattened.sort_unstable_by_key(|(key, _)| {
-                new_fields_ids_map.id(key).unwrap_or(FieldId::MAX)
-            });
-
-            for (key, value) in flattened {
-                let fid =
-                    new_fields_ids_map.insert(&key).ok_or(UserError::AttributeLimitReached)?;
-                let value = serde_json::to_vec(&value).map_err(InternalError::SerdeJson)?;
-                writer.insert(fid, &value)?;
-            }
-            document_sorter_value_buffer.clear();
-            into_del_add_obkv(
-                KvReaderU16::new(&buffer),
-                DelAddOperation::Addition,
-                &mut document_sorter_value_buffer,
-            )?;
-            flattened_sorter.insert(docid.to_be_bytes(), &document_sorter_value_buffer)?;
+            original_sorter.insert(&document_sorter_key_buffer, &original_obkv_buffer)?;
+            flattened_sorter.insert(docid.to_be_bytes(), &flattened_obkv_buffer)?;
         }
 
         let grenad_params = GrenadParameters {
@@ -934,22 +945,14 @@ impl<'a, 'i> Transform<'a, 'i> {
 
         let flattened_documents = sorter_into_reader(flattened_sorter, grenad_params)?;
 
-        let output = TransformOutput {
+        Ok(TransformOutput {
             primary_key,
-            fields_ids_map: new_fields_ids_map,
             field_distribution,
+            settings_diff,
             documents_count,
             original_documents: original_documents.into_inner().into_inner(),
             flattened_documents: flattened_documents.into_inner().into_inner(),
-        };
-
-        let new_facets = output.compute_real_facets(wtxn, self.index)?;
-        self.index.put_faceted_fields(wtxn, &new_facets)?;
-
-        // We clear the full database (words-fst, documents ids and documents content).
-        ClearDocuments::new(wtxn, self.index).execute()?;
-
-        Ok(output)
+        })
     }
 }
 
@@ -964,20 +967,6 @@ fn drop_and_reuse<U, T>(mut vec: Vec<U>) -> Vec<T> {
     vec.into_iter().map(|_| unreachable!()).collect()
 }
 
-impl TransformOutput {
-    // find and insert the new field ids
-    pub fn compute_real_facets(&self, rtxn: &RoTxn, index: &Index) -> Result<HashSet<String>> {
-        let user_defined_facets = index.user_defined_faceted_fields(rtxn)?;
-
-        Ok(self
-            .fields_ids_map
-            .names()
-            .filter(|&field| crate::is_faceted(field, &user_defined_facets))
-            .map(|field| field.to_string())
-            .collect())
-    }
-}
-
 #[cfg(test)]
 mod test {
     use super::*;

milli/src/update/indexer_config.rs

@@ -1,5 +1,6 @@
 use grenad::CompressionType;
-use rayon::ThreadPool;
+
+use crate::thread_pool_no_abort::ThreadPoolNoAbort;
 
 #[derive(Debug)]
 pub struct IndexerConfig {
@@ -9,7 +10,7 @@ pub struct IndexerConfig {
     pub max_memory: Option<usize>,
     pub chunk_compression_type: CompressionType,
     pub chunk_compression_level: Option<u32>,
-    pub thread_pool: Option<ThreadPool>,
+    pub thread_pool: Option<ThreadPoolNoAbort>,
     pub max_positions_per_attributes: Option<u32>,
     pub skip_index_budget: bool,
 }

milli/src/update/settings.rs

@@ -14,12 +14,13 @@ use super::IndexerConfig;
 use crate::criterion::Criterion;
 use crate::error::UserError;
 use crate::index::{DEFAULT_MIN_WORD_LEN_ONE_TYPO, DEFAULT_MIN_WORD_LEN_TWO_TYPOS};
+use crate::order_by_map::OrderByMap;
 use crate::proximity::ProximityPrecision;
 use crate::update::index_documents::IndexDocumentsMethod;
 use crate::update::{IndexDocuments, UpdateIndexingStep};
 use crate::vector::settings::{check_set, check_unset, EmbedderSource, EmbeddingSettings};
 use crate::vector::{Embedder, EmbeddingConfig, EmbeddingConfigs};
-use crate::{FieldsIdsMap, Index, OrderBy, Result};
+use crate::{FieldId, FieldsIdsMap, Index, Result};
 
 #[derive(Debug, Clone, PartialEq, Eq, Copy)]
 pub enum Setting<T> {
@@ -145,10 +146,11 @@ pub struct Settings<'a, 't, 'i> {
     /// Attributes on which typo tolerance is disabled.
     exact_attributes: Setting<HashSet<String>>,
     max_values_per_facet: Setting<usize>,
-    sort_facet_values_by: Setting<HashMap<String, OrderBy>>,
+    sort_facet_values_by: Setting<OrderByMap>,
     pagination_max_total_hits: Setting<usize>,
     proximity_precision: Setting<ProximityPrecision>,
     embedder_settings: Setting<BTreeMap<String, Setting<EmbeddingSettings>>>,
+    search_cutoff: Setting<u64>,
 }
 
 impl<'a, 't, 'i> Settings<'a, 't, 'i> {
@@ -182,6 +184,7 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
             pagination_max_total_hits: Setting::NotSet,
             proximity_precision: Setting::NotSet,
             embedder_settings: Setting::NotSet,
+            search_cutoff: Setting::NotSet,
             indexer_config,
         }
     }
@@ -340,7 +343,7 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
         self.max_values_per_facet = Setting::Reset;
     }
 
-    pub fn set_sort_facet_values_by(&mut self, value: HashMap<String, OrderBy>) {
+    pub fn set_sort_facet_values_by(&mut self, value: OrderByMap) {
         self.sort_facet_values_by = Setting::Set(value);
     }
 
@@ -372,16 +375,24 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
         self.embedder_settings = Setting::Reset;
     }
 
+    pub fn set_search_cutoff(&mut self, value: u64) {
+        self.search_cutoff = Setting::Set(value);
+    }
+
+    pub fn reset_search_cutoff(&mut self) {
+        self.search_cutoff = Setting::Reset;
+    }
+
     #[tracing::instrument(
         level = "trace"
-        skip(self, progress_callback, should_abort, old_fields_ids_map),
+        skip(self, progress_callback, should_abort, settings_diff),
         target = "indexing::documents"
     )]
     fn reindex<FP, FA>(
         &mut self,
         progress_callback: &FP,
         should_abort: &FA,
-        old_fields_ids_map: FieldsIdsMap,
+        settings_diff: InnerIndexSettingsDiff,
     ) -> Result<()>
     where
         FP: Fn(UpdateIndexingStep) + Sync,
@@ -389,7 +400,6 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
     {
         puffin::profile_function!();
 
-        let fields_ids_map = self.index.fields_ids_map(self.wtxn)?;
         // if the settings are set before any document update, we don't need to do anything, and
         // will set the primary key during the first document addition.
         if self.index.number_of_documents(self.wtxn)? == 0 {
@@ -405,14 +415,7 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
         )?;
 
         // We clear the databases and remap the documents fields based on the new `FieldsIdsMap`.
-        let output = transform.prepare_for_documents_reindexing(
-            self.wtxn,
-            old_fields_ids_map,
-            fields_ids_map,
-        )?;
-
-        let embedder_configs = self.index.embedding_configs(self.wtxn)?;
-        let embedders = self.embedders(embedder_configs)?;
+        let output = transform.prepare_for_documents_reindexing(self.wtxn, settings_diff)?;
 
         // We index the generated `TransformOutput` which must contain
         // all the documents with fields in the newly defined searchable order.
@@ -425,32 +428,11 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
             &should_abort,
         )?;
 
-        let indexing_builder = indexing_builder.with_embedders(embedders);
         indexing_builder.execute_raw(output)?;
 
         Ok(())
     }
 
-    fn embedders(
-        &self,
-        embedding_configs: Vec<(String, EmbeddingConfig)>,
-    ) -> Result<EmbeddingConfigs> {
-        let res: Result<_> = embedding_configs
-            .into_iter()
-            .map(|(name, EmbeddingConfig { embedder_options, prompt })| {
-                let prompt = Arc::new(prompt.try_into().map_err(crate::Error::from)?);
-
-                let embedder = Arc::new(
-                    Embedder::new(embedder_options.clone())
-                        .map_err(crate::vector::Error::from)
-                        .map_err(crate::Error::from)?,
-                );
-                Ok((name, (embedder, prompt)))
-            })
-            .collect();
-        res.map(EmbeddingConfigs::new)
-    }
-
     fn update_displayed(&mut self) -> Result<bool> {
         match self.displayed_fields {
             Setting::Set(ref fields) => {
@@ -965,7 +947,12 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
                     match joined {
                         // updated config
                         EitherOrBoth::Both((name, mut old), (_, new)) => {
-                            changed |= old.apply(new);
+                            changed |= EmbeddingSettings::apply_and_need_reindex(&mut old, new);
+                            if changed {
+                                tracing::debug!(embedder = name, "need reindex");
+                            } else {
+                                tracing::debug!(embedder = name, "skip reindex");
+                            }
                             let new = validate_embedding_settings(old, &name)?;
                             new_configs.insert(name, new);
                         }
@@ -1022,9 +1009,34 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
             }
             Setting::NotSet => false,
         };
+
+        // if any changes force a reindexing
+        // clear the vector database.
+        if update {
+            self.index.vector_arroy.clear(self.wtxn)?;
+        }
+
         Ok(update)
     }
 
+    fn update_search_cutoff(&mut self) -> Result<bool> {
+        let changed = match self.search_cutoff {
+            Setting::Set(new) => {
+                let old = self.index.search_cutoff(self.wtxn)?;
+                if old == Some(new) {
+                    false
+                } else {
+                    self.index.put_search_cutoff(self.wtxn, new)?;
+                    true
+                }
+            }
+            Setting::Reset => self.index.delete_search_cutoff(self.wtxn)?,
+            Setting::NotSet => false,
+        };
+
+        Ok(changed)
+    }
+
     pub fn execute<FP, FA>(mut self, progress_callback: FP, should_abort: FA) -> Result<()>
     where
         FP: Fn(UpdateIndexingStep) + Sync,
@@ -1032,19 +1044,10 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
     {
         self.index.set_updated_at(self.wtxn, &OffsetDateTime::now_utc())?;
 
-        let existing_fields: HashSet<_> = self
-            .index
-            .field_distribution(self.wtxn)?
-            .into_iter()
-            .filter_map(|(field, count)| (count != 0).then_some(field))
-            .collect();
-
-        let old_faceted_fields = self.index.user_defined_faceted_fields(self.wtxn)?;
-        let old_fields_ids_map = self.index.fields_ids_map(self.wtxn)?;
+        let old_inner_settings = InnerIndexSettings::from_index(self.index, self.wtxn)?;
 
+        // never trigger re-indexing
         self.update_displayed()?;
-        self.update_filterable()?;
-        self.update_sortable()?;
         self.update_distinct_field()?;
         self.update_criteria()?;
         self.update_primary_key()?;
@@ -1054,22 +1057,19 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
         self.update_max_values_per_facet()?;
         self.update_sort_facet_values_by()?;
         self.update_pagination_max_total_hits()?;
+        self.update_search_cutoff()?;
 
-        // If there is new faceted fields we indicate that we must reindex as we must
-        // index new fields as facets. It means that the distinct attribute,
-        // an Asc/Desc criterion or a filtered attribute as be added or removed.
-        let new_faceted_fields = self.index.user_defined_faceted_fields(self.wtxn)?;
-        let faceted_updated =
-            (&existing_fields - &old_faceted_fields) != (&existing_fields - &new_faceted_fields);
-
-        let stop_words_updated = self.update_stop_words()?;
-        let non_separator_tokens_updated = self.update_non_separator_tokens()?;
-        let separator_tokens_updated = self.update_separator_tokens()?;
-        let dictionary_updated = self.update_dictionary()?;
-        let synonyms_updated = self.update_synonyms()?;
-        let searchable_updated = self.update_searchable()?;
-        let exact_attributes_updated = self.update_exact_attributes()?;
-        let proximity_precision = self.update_proximity_precision()?;
+        // could trigger re-indexing
+        self.update_filterable()?;
+        self.update_sortable()?;
+        self.update_stop_words()?;
+        self.update_non_separator_tokens()?;
+        self.update_separator_tokens()?;
+        self.update_dictionary()?;
+        self.update_synonyms()?;
+        self.update_searchable()?;
+        self.update_exact_attributes()?;
+        self.update_proximity_precision()?;
         // TODO: very rough approximation of the needs for reindexing where any change will result in
         // a full reindexing.
         // What can be done instead:
@@ -1078,24 +1078,195 @@ impl<'a, 't, 'i> Settings<'a, 't, 'i> {
         // 3. Keep the old vectors but reattempt indexing on a prompt change: only actually changed prompt will need embedding + storage
         let embedding_configs_updated = self.update_embedding_configs()?;
 
-        if stop_words_updated
-            || non_separator_tokens_updated
-            || separator_tokens_updated
-            || dictionary_updated
-            || faceted_updated
-            || synonyms_updated
-            || searchable_updated
-            || exact_attributes_updated
-            || proximity_precision
-            || embedding_configs_updated
-        {
-            self.reindex(&progress_callback, &should_abort, old_fields_ids_map)?;
+        let new_inner_settings = InnerIndexSettings::from_index(self.index, self.wtxn)?;
+        let inner_settings_diff = InnerIndexSettingsDiff {
+            old: old_inner_settings,
+            new: new_inner_settings,
+            embedding_configs_updated,
+            settings_update_only: true,
+        };
+
+        if inner_settings_diff.any_reindexing_needed() {
+            self.reindex(&progress_callback, &should_abort, inner_settings_diff)?;
         }
 
         Ok(())
     }
 }
 
+pub struct InnerIndexSettingsDiff {
+    pub(crate) old: InnerIndexSettings,
+    pub(crate) new: InnerIndexSettings,
+
+    // TODO: compare directly the embedders.
+    pub(crate) embedding_configs_updated: bool,
+
+    pub(crate) settings_update_only: bool,
+}
+
+impl InnerIndexSettingsDiff {
+    pub fn any_reindexing_needed(&self) -> bool {
+        self.reindex_searchable() || self.reindex_facets() || self.reindex_vectors()
+    }
+
+    pub fn reindex_searchable(&self) -> bool {
+        self.old
+            .fields_ids_map
+            .iter()
+            .zip(self.new.fields_ids_map.iter())
+            .any(|(old, new)| old != new)
+            || self.old.stop_words.as_ref().map(|set| set.as_fst().as_bytes())
+                != self.new.stop_words.as_ref().map(|set| set.as_fst().as_bytes())
+            || self.old.allowed_separators != self.new.allowed_separators
+            || self.old.dictionary != self.new.dictionary
+            || self.old.user_defined_searchable_fields != self.new.user_defined_searchable_fields
+            || self.old.exact_attributes != self.new.exact_attributes
+            || self.old.proximity_precision != self.new.proximity_precision
+    }
+
+    pub fn reindex_facets(&self) -> bool {
+        let existing_fields = &self.new.existing_fields;
+        if existing_fields.iter().any(|field| field.contains('.')) {
+            return true;
+        }
+
+        let old_faceted_fields = &self.old.user_defined_faceted_fields;
+        if old_faceted_fields.iter().any(|field| field.contains('.')) {
+            return true;
+        }
+
+        // If there is new faceted fields we indicate that we must reindex as we must
+        // index new fields as facets. It means that the distinct attribute,
+        // an Asc/Desc criterion or a filtered attribute as be added or removed.
+        let new_faceted_fields = &self.new.user_defined_faceted_fields;
+        if new_faceted_fields.iter().any(|field| field.contains('.')) {
+            return true;
+        }
+
+        let faceted_updated =
+            (existing_fields - old_faceted_fields) != (existing_fields - new_faceted_fields);
+
+        self.old
+            .fields_ids_map
+            .iter()
+            .zip(self.new.fields_ids_map.iter())
+            .any(|(old, new)| old != new)
+            || faceted_updated
+    }
+
+    pub fn reindex_vectors(&self) -> bool {
+        self.embedding_configs_updated
+    }
+
+    pub fn settings_update_only(&self) -> bool {
+        self.settings_update_only
+    }
+}
+
+#[derive(Clone)]
+pub(crate) struct InnerIndexSettings {
+    pub stop_words: Option<fst::Set<Vec<u8>>>,
+    pub allowed_separators: Option<BTreeSet<String>>,
+    pub dictionary: Option<BTreeSet<String>>,
+    pub fields_ids_map: FieldsIdsMap,
+    pub user_defined_faceted_fields: HashSet<String>,
+    pub user_defined_searchable_fields: Option<Vec<String>>,
+    pub faceted_fields_ids: HashSet<FieldId>,
+    pub searchable_fields_ids: Option<Vec<FieldId>>,
+    pub exact_attributes: HashSet<FieldId>,
+    pub proximity_precision: ProximityPrecision,
+    pub embedding_configs: EmbeddingConfigs,
+    pub existing_fields: HashSet<String>,
+}
+
+impl InnerIndexSettings {
+    pub fn from_index(index: &Index, rtxn: &heed::RoTxn) -> Result<Self> {
+        let stop_words = index.stop_words(rtxn)?;
+        let stop_words = stop_words.map(|sw| sw.map_data(Vec::from).unwrap());
+        let allowed_separators = index.allowed_separators(rtxn)?;
+        let dictionary = index.dictionary(rtxn)?;
+        let fields_ids_map = index.fields_ids_map(rtxn)?;
+        let user_defined_searchable_fields = index.user_defined_searchable_fields(rtxn)?;
+        let user_defined_searchable_fields =
+            user_defined_searchable_fields.map(|sf| sf.into_iter().map(String::from).collect());
+        let user_defined_faceted_fields = index.user_defined_faceted_fields(rtxn)?;
+        let searchable_fields_ids = index.searchable_fields_ids(rtxn)?;
+        let faceted_fields_ids = index.faceted_fields_ids(rtxn)?;
+        let exact_attributes = index.exact_attributes_ids(rtxn)?;
+        let proximity_precision = index.proximity_precision(rtxn)?.unwrap_or_default();
+        let embedding_configs = embedders(index.embedding_configs(rtxn)?)?;
+        let existing_fields: HashSet<_> = index
+            .field_distribution(rtxn)?
+            .into_iter()
+            .filter_map(|(field, count)| (count != 0).then_some(field))
+            .collect();
+
+        Ok(Self {
+            stop_words,
+            allowed_separators,
+            dictionary,
+            fields_ids_map,
+            user_defined_faceted_fields,
+            user_defined_searchable_fields,
+            faceted_fields_ids,
+            searchable_fields_ids,
+            exact_attributes,
+            proximity_precision,
+            embedding_configs,
+            existing_fields,
+        })
+    }
+
+    // find and insert the new field ids
+    pub fn recompute_facets(&mut self, wtxn: &mut heed::RwTxn, index: &Index) -> Result<()> {
+        let new_facets = self
+            .fields_ids_map
+            .names()
+            .filter(|&field| crate::is_faceted(field, &self.user_defined_faceted_fields))
+            .map(|field| field.to_string())
+            .collect();
+        index.put_faceted_fields(wtxn, &new_facets)?;
+
+        self.faceted_fields_ids = index.faceted_fields_ids(wtxn)?;
+        Ok(())
+    }
+
+    // find and insert the new field ids
+    pub fn recompute_searchables(&mut self, wtxn: &mut heed::RwTxn, index: &Index) -> Result<()> {
+        // in case new fields were introduced we're going to recreate the searchable fields.
+        if let Some(searchable_fields) = self.user_defined_searchable_fields.as_ref() {
+            let searchable_fields =
+                searchable_fields.iter().map(String::as_ref).collect::<Vec<_>>();
+            index.put_all_searchable_fields_from_fields_ids_map(
+                wtxn,
+                &searchable_fields,
+                &self.fields_ids_map,
+            )?;
+            let searchable_fields_ids = index.searchable_fields_ids(wtxn)?;
+            self.searchable_fields_ids = searchable_fields_ids;
+        }
+
+        Ok(())
+    }
+}
+
+fn embedders(embedding_configs: Vec<(String, EmbeddingConfig)>) -> Result<EmbeddingConfigs> {
+    let res: Result<_> = embedding_configs
+        .into_iter()
+        .map(|(name, EmbeddingConfig { embedder_options, prompt })| {
+            let prompt = Arc::new(prompt.try_into().map_err(crate::Error::from)?);
+
+            let embedder = Arc::new(
+                Embedder::new(embedder_options.clone())
+                    .map_err(crate::vector::Error::from)
+                    .map_err(crate::Error::from)?,
+            );
+            Ok((name, (embedder, prompt)))
+        })
+        .collect();
+    res.map(EmbeddingConfigs::new)
+}
+
 fn validate_prompt(
     name: &str,
     new: Setting<EmbeddingSettings>,
@@ -1108,6 +1279,13 @@ fn validate_prompt(
             api_key,
             dimensions,
             document_template: Setting::Set(template),
+            url,
+            query,
+            input_field,
+            path_to_embeddings,
+            embedding_object,
+            input_type,
+            distribution,
         }) => {
             // validate
             let template = crate::prompt::Prompt::new(template)
@@ -1121,6 +1299,13 @@ fn validate_prompt(
                 api_key,
                 dimensions,
                 document_template: Setting::Set(template),
+                url,
+                query,
+                input_field,
+                path_to_embeddings,
+                embedding_object,
+                input_type,
+                distribution,
             }))
         }
         new => Ok(new),
@@ -1133,8 +1318,21 @@ pub fn validate_embedding_settings(
 ) -> Result<Setting<EmbeddingSettings>> {
     let settings = validate_prompt(name, settings)?;
     let Setting::Set(settings) = settings else { return Ok(settings) };
-    let EmbeddingSettings { source, model, revision, api_key, dimensions, document_template } =
-        settings;
+    let EmbeddingSettings {
+        source,
+        model,
+        revision,
+        api_key,
+        dimensions,
+        document_template,
+        url,
+        query,
+        input_field,
+        path_to_embeddings,
+        embedding_object,
+        input_type,
+        distribution,
+    } = settings;
 
     if let Some(0) = dimensions.set() {
         return Err(crate::error::UserError::InvalidSettingsDimensions {
@@ -1143,6 +1341,14 @@ pub fn validate_embedding_settings(
         .into());
     }
 
+    if let Some(url) = url.as_ref().set() {
+        url::Url::parse(url).map_err(|error| crate::error::UserError::InvalidUrl {
+            embedder_name: name.to_owned(),
+            inner_error: error,
+            url: url.to_owned(),
+        })?;
+    }
+
     let Some(inferred_source) = source.set() else {
         return Ok(Setting::Set(EmbeddingSettings {
             source,
@@ -1151,11 +1357,36 @@ pub fn validate_embedding_settings(
             api_key,
             dimensions,
             document_template,
+            url,
+            query,
+            input_field,
+            path_to_embeddings,
+            embedding_object,
+            input_type,
+            distribution,
         }));
     };
     match inferred_source {
         EmbedderSource::OpenAi => {
-            check_unset(&revision, "revision", inferred_source, name)?;
+            check_unset(&revision, EmbeddingSettings::REVISION, inferred_source, name)?;
+
+            check_unset(&url, EmbeddingSettings::URL, inferred_source, name)?;
+            check_unset(&query, EmbeddingSettings::QUERY, inferred_source, name)?;
+            check_unset(&input_field, EmbeddingSettings::INPUT_FIELD, inferred_source, name)?;
+            check_unset(
+                &path_to_embeddings,
+                EmbeddingSettings::PATH_TO_EMBEDDINGS,
+                inferred_source,
+                name,
+            )?;
+            check_unset(
+                &embedding_object,
+                EmbeddingSettings::EMBEDDING_OBJECT,
+                inferred_source,
+                name,
+            )?;
+            check_unset(&input_type, EmbeddingSettings::INPUT_TYPE, inferred_source, name)?;
+
             if let Setting::Set(model) = &model {
                 let model = crate::vector::openai::EmbeddingModel::from_name(model.as_str())
                     .ok_or(crate::error::UserError::InvalidOpenAiModel {
@@ -1186,16 +1417,82 @@ pub fn validate_embedding_settings(
                 }
             }
         }
+        EmbedderSource::Ollama => {
+            // Dimensions get inferred, only model name is required
+            check_unset(&dimensions, EmbeddingSettings::DIMENSIONS, inferred_source, name)?;
+            check_set(&model, EmbeddingSettings::MODEL, inferred_source, name)?;
+            check_unset(&revision, EmbeddingSettings::REVISION, inferred_source, name)?;
+
+            check_unset(&query, EmbeddingSettings::QUERY, inferred_source, name)?;
+            check_unset(&input_field, EmbeddingSettings::INPUT_FIELD, inferred_source, name)?;
+            check_unset(
+                &path_to_embeddings,
+                EmbeddingSettings::PATH_TO_EMBEDDINGS,
+                inferred_source,
+                name,
+            )?;
+            check_unset(
+                &embedding_object,
+                EmbeddingSettings::EMBEDDING_OBJECT,
+                inferred_source,
+                name,
+            )?;
+            check_unset(&input_type, EmbeddingSettings::INPUT_TYPE, inferred_source, name)?;
+        }
         EmbedderSource::HuggingFace => {
-            check_unset(&api_key, "apiKey", inferred_source, name)?;
-            check_unset(&dimensions, "dimensions", inferred_source, name)?;
+            check_unset(&api_key, EmbeddingSettings::API_KEY, inferred_source, name)?;
+            check_unset(&dimensions, EmbeddingSettings::DIMENSIONS, inferred_source, name)?;
+
+            check_unset(&url, EmbeddingSettings::URL, inferred_source, name)?;
+            check_unset(&query, EmbeddingSettings::QUERY, inferred_source, name)?;
+            check_unset(&input_field, EmbeddingSettings::INPUT_FIELD, inferred_source, name)?;
+            check_unset(
+                &path_to_embeddings,
+                EmbeddingSettings::PATH_TO_EMBEDDINGS,
+                inferred_source,
+                name,
+            )?;
+            check_unset(
+                &embedding_object,
+                EmbeddingSettings::EMBEDDING_OBJECT,
+                inferred_source,
+                name,
+            )?;
+            check_unset(&input_type, EmbeddingSettings::INPUT_TYPE, inferred_source, name)?;
         }
         EmbedderSource::UserProvided => {
-            check_unset(&model, "model", inferred_source, name)?;
-            check_unset(&revision, "revision", inferred_source, name)?;
-            check_unset(&api_key, "apiKey", inferred_source, name)?;
-            check_unset(&document_template, "documentTemplate", inferred_source, name)?;
-            check_set(&dimensions, "dimensions", inferred_source, name)?;
+            check_unset(&model, EmbeddingSettings::MODEL, inferred_source, name)?;
+            check_unset(&revision, EmbeddingSettings::REVISION, inferred_source, name)?;
+            check_unset(&api_key, EmbeddingSettings::API_KEY, inferred_source, name)?;
+            check_unset(
+                &document_template,
+                EmbeddingSettings::DOCUMENT_TEMPLATE,
+                inferred_source,
+                name,
+            )?;
+            check_set(&dimensions, EmbeddingSettings::DIMENSIONS, inferred_source, name)?;
+
+            check_unset(&url, EmbeddingSettings::URL, inferred_source, name)?;
+            check_unset(&query, EmbeddingSettings::QUERY, inferred_source, name)?;
+            check_unset(&input_field, EmbeddingSettings::INPUT_FIELD, inferred_source, name)?;
+            check_unset(
+                &path_to_embeddings,
+                EmbeddingSettings::PATH_TO_EMBEDDINGS,
+                inferred_source,
+                name,
+            )?;
+            check_unset(
+                &embedding_object,
+                EmbeddingSettings::EMBEDDING_OBJECT,
+                inferred_source,
+                name,
+            )?;
+            check_unset(&input_type, EmbeddingSettings::INPUT_TYPE, inferred_source, name)?;
+        }
+        EmbedderSource::Rest => {
+            check_unset(&model, EmbeddingSettings::MODEL, inferred_source, name)?;
+            check_unset(&revision, EmbeddingSettings::REVISION, inferred_source, name)?;
+            check_set(&url, EmbeddingSettings::URL, inferred_source, name)?;
         }
     }
     Ok(Setting::Set(EmbeddingSettings {
@@ -1205,6 +1502,13 @@ pub fn validate_embedding_settings(
         api_key,
         dimensions,
         document_template,
+        url,
+        query,
+        input_field,
+        path_to_embeddings,
+        embedding_object,
+        input_type,
+        distribution,
     }))
 }
 
@@ -1450,6 +1754,70 @@ mod tests {
             .unwrap()
             .count();
         assert_eq!(count, 4);
+
+        // Set the filterable fields to be the age and the name.
+        index
+            .update_settings(|settings| {
+                settings.set_filterable_fields(hashset! { S("age"),  S("name") });
+            })
+            .unwrap();
+
+        // Check that the displayed fields are correctly set.
+        let rtxn = index.read_txn().unwrap();
+        let fields_ids = index.filterable_fields(&rtxn).unwrap();
+        assert_eq!(fields_ids, hashset! { S("age"),  S("name") });
+
+        let rtxn = index.read_txn().unwrap();
+        // Only count the field_id 0 and level 0 facet values.
+        let count = index
+            .facet_id_f64_docids
+            .remap_key_type::<Bytes>()
+            .prefix_iter(&rtxn, &[0, 1, 0])
+            .unwrap()
+            .count();
+        assert_eq!(count, 4);
+
+        let rtxn = index.read_txn().unwrap();
+        // Only count the field_id 0 and level 0 facet values.
+        let count = index
+            .facet_id_string_docids
+            .remap_key_type::<Bytes>()
+            .prefix_iter(&rtxn, &[0, 0])
+            .unwrap()
+            .count();
+        assert_eq!(count, 5);
+
+        // Remove the age from the filterable fields.
+        index
+            .update_settings(|settings| {
+                settings.set_filterable_fields(hashset! { S("name") });
+            })
+            .unwrap();
+
+        // Check that the displayed fields are correctly set.
+        let rtxn = index.read_txn().unwrap();
+        let fields_ids = index.filterable_fields(&rtxn).unwrap();
+        assert_eq!(fields_ids, hashset! { S("name") });
+
+        let rtxn = index.read_txn().unwrap();
+        // Only count the field_id 0 and level 0 facet values.
+        let count = index
+            .facet_id_f64_docids
+            .remap_key_type::<Bytes>()
+            .prefix_iter(&rtxn, &[0, 1, 0])
+            .unwrap()
+            .count();
+        assert_eq!(count, 0);
+
+        let rtxn = index.read_txn().unwrap();
+        // Only count the field_id 0 and level 0 facet values.
+        let count = index
+            .facet_id_string_docids
+            .remap_key_type::<Bytes>()
+            .prefix_iter(&rtxn, &[0, 0])
+            .unwrap()
+            .count();
+        assert_eq!(count, 5);
     }
 
     #[test]
@@ -2027,6 +2395,7 @@ mod tests {
                     pagination_max_total_hits,
                     proximity_precision,
                     embedder_settings,
+                    search_cutoff,
                 } = settings;
                 assert!(matches!(searchable_fields, Setting::NotSet));
                 assert!(matches!(displayed_fields, Setting::NotSet));
@@ -2050,6 +2419,7 @@ mod tests {
                 assert!(matches!(pagination_max_total_hits, Setting::NotSet));
                 assert!(matches!(proximity_precision, Setting::NotSet));
                 assert!(matches!(embedder_settings, Setting::NotSet));
+                assert!(matches!(search_cutoff, Setting::NotSet));
             })
             .unwrap();
     }

milli/src/update/words_prefixes_fst.rs

@@ -20,7 +20,7 @@ impl<'t, 'i> WordsPrefixesFst<'t, 'i> {
 
     /// Set the number of words required to make a prefix be part of the words prefixes
     /// database. If a word prefix is supposed to match more than this number of words in the
-    /// dictionnary, therefore this prefix is added to the words prefixes datastructures.
+    /// dictionary, therefore this prefix is added to the words prefixes datastructures.
     ///
     /// Default value is 100. This value must be higher than 50 and will be clamped
     /// to this bound otherwise.

milli/src/vector/error.rs

@@ -3,7 +3,7 @@ use std::path::PathBuf;
 use hf_hub::api::sync::ApiError;
 
 use crate::error::FaultSource;
-use crate::vector::openai::OpenAiError;
+use crate::PanicCatched;
 
 #[derive(Debug, thiserror::Error)]
 #[error("Error while generating embeddings: {inner}")]
@@ -51,26 +51,38 @@ pub enum EmbedErrorKind {
     TensorValue(candle_core::Error),
     #[error("could not run model: {0}")]
     ModelForward(candle_core::Error),
-    #[error("could not reach OpenAI: {0}")]
-    OpenAiNetwork(reqwest::Error),
-    #[error("unexpected response from OpenAI: {0}")]
-    OpenAiUnexpected(reqwest::Error),
-    #[error("could not authenticate against OpenAI: {0}")]
-    OpenAiAuth(OpenAiError),
-    #[error("sent too many requests to OpenAI: {0}")]
-    OpenAiTooManyRequests(OpenAiError),
-    #[error("received internal error from OpenAI: {0:?}")]
-    OpenAiInternalServerError(Option<OpenAiError>),
-    #[error("sent too many tokens in a request to OpenAI: {0}")]
-    OpenAiTooManyTokens(OpenAiError),
-    #[error("received unhandled HTTP status code {0} from OpenAI")]
-    OpenAiUnhandledStatusCode(u16),
     #[error("attempt to embed the following text in a configuration where embeddings must be user provided: {0:?}")]
     ManualEmbed(String),
-    #[error("could not initialize asynchronous runtime: {0}")]
-    OpenAiRuntimeInit(std::io::Error),
-    #[error("initializing web client for sending embedding requests failed: {0}")]
-    InitWebClient(reqwest::Error),
+    #[error("model not found. Meilisearch will not automatically download models from the Ollama library, please pull the model manually: {0:?}")]
+    OllamaModelNotFoundError(Option<String>),
+    #[error("error deserialization the response body as JSON: {0}")]
+    RestResponseDeserialization(std::io::Error),
+    #[error("component `{0}` not found in path `{1}` in response: `{2}`")]
+    RestResponseMissingEmbeddings(String, String, String),
+    #[error("unexpected format of the embedding response: {0}")]
+    RestResponseFormat(serde_json::Error),
+    #[error("expected a response containing {0} embeddings, got only {1}")]
+    RestResponseEmbeddingCount(usize, usize),
+    #[error("could not authenticate against embedding server: {0:?}")]
+    RestUnauthorized(Option<String>),
+    #[error("sent too many requests to embedding server: {0:?}")]
+    RestTooManyRequests(Option<String>),
+    #[error("sent a bad request to embedding server: {0:?}")]
+    RestBadRequest(Option<String>),
+    #[error("received internal error from embedding server: {0:?}")]
+    RestInternalServerError(u16, Option<String>),
+    #[error("received HTTP {0} from embedding server: {0:?}")]
+    RestOtherStatusCode(u16, Option<String>),
+    #[error("could not reach embedding server: {0}")]
+    RestNetwork(ureq::Transport),
+    #[error("was expected '{}' to be an object in query '{0}'", .1.join("."))]
+    RestNotAnObject(serde_json::Value, Vec<String>),
+    #[error("while embedding tokenized, was expecting embeddings of dimension `{0}`, got embeddings of dimensions `{1}`")]
+    OpenAiUnexpectedDimension(usize, usize),
+    #[error("no embedding was produced")]
+    MissingEmbedding,
+    #[error(transparent)]
+    PanicInThreadPool(#[from] PanicCatched),
 }
 
 impl EmbedError {
@@ -90,44 +102,101 @@ impl EmbedError {
         Self { kind: EmbedErrorKind::ModelForward(inner), fault: FaultSource::Runtime }
     }
 
-    pub fn openai_network(inner: reqwest::Error) -> Self {
-        Self { kind: EmbedErrorKind::OpenAiNetwork(inner), fault: FaultSource::Runtime }
-    }
-
-    pub fn openai_unexpected(inner: reqwest::Error) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiUnexpected(inner), fault: FaultSource::Bug }
-    }
-
-    pub(crate) fn openai_auth_error(inner: OpenAiError) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiAuth(inner), fault: FaultSource::User }
-    }
-
-    pub(crate) fn openai_too_many_requests(inner: OpenAiError) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiTooManyRequests(inner), fault: FaultSource::Runtime }
-    }
-
-    pub(crate) fn openai_internal_server_error(inner: Option<OpenAiError>) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiInternalServerError(inner), fault: FaultSource::Runtime }
-    }
-
-    pub(crate) fn openai_too_many_tokens(inner: OpenAiError) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiTooManyTokens(inner), fault: FaultSource::Bug }
-    }
-
-    pub(crate) fn openai_unhandled_status_code(code: u16) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiUnhandledStatusCode(code), fault: FaultSource::Bug }
-    }
-
     pub(crate) fn embed_on_manual_embedder(texts: String) -> EmbedError {
         Self { kind: EmbedErrorKind::ManualEmbed(texts), fault: FaultSource::User }
     }
 
-    pub(crate) fn openai_runtime_init(inner: std::io::Error) -> EmbedError {
-        Self { kind: EmbedErrorKind::OpenAiRuntimeInit(inner), fault: FaultSource::Runtime }
+    pub(crate) fn ollama_model_not_found(inner: Option<String>) -> EmbedError {
+        Self { kind: EmbedErrorKind::OllamaModelNotFoundError(inner), fault: FaultSource::User }
     }
 
-    pub fn openai_initialize_web_client(inner: reqwest::Error) -> Self {
-        Self { kind: EmbedErrorKind::InitWebClient(inner), fault: FaultSource::Runtime }
+    pub(crate) fn rest_response_deserialization(error: std::io::Error) -> EmbedError {
+        Self {
+            kind: EmbedErrorKind::RestResponseDeserialization(error),
+            fault: FaultSource::Runtime,
+        }
+    }
+
+    pub(crate) fn rest_response_missing_embeddings<S: AsRef<str>>(
+        response: serde_json::Value,
+        component: &str,
+        response_field: &[S],
+    ) -> EmbedError {
+        let response_field: Vec<&str> = response_field.iter().map(AsRef::as_ref).collect();
+        let response_field = response_field.join(".");
+
+        Self {
+            kind: EmbedErrorKind::RestResponseMissingEmbeddings(
+                component.to_owned(),
+                response_field,
+                serde_json::to_string_pretty(&response).unwrap_or_default(),
+            ),
+            fault: FaultSource::Undecided,
+        }
+    }
+
+    pub(crate) fn rest_response_format(error: serde_json::Error) -> EmbedError {
+        Self { kind: EmbedErrorKind::RestResponseFormat(error), fault: FaultSource::Undecided }
+    }
+
+    pub(crate) fn rest_response_embedding_count(expected: usize, got: usize) -> EmbedError {
+        Self {
+            kind: EmbedErrorKind::RestResponseEmbeddingCount(expected, got),
+            fault: FaultSource::Runtime,
+        }
+    }
+
+    pub(crate) fn rest_unauthorized(error_response: Option<String>) -> EmbedError {
+        Self { kind: EmbedErrorKind::RestUnauthorized(error_response), fault: FaultSource::User }
+    }
+
+    pub(crate) fn rest_too_many_requests(error_response: Option<String>) -> EmbedError {
+        Self {
+            kind: EmbedErrorKind::RestTooManyRequests(error_response),
+            fault: FaultSource::Runtime,
+        }
+    }
+
+    pub(crate) fn rest_bad_request(error_response: Option<String>) -> EmbedError {
+        Self { kind: EmbedErrorKind::RestBadRequest(error_response), fault: FaultSource::User }
+    }
+
+    pub(crate) fn rest_internal_server_error(
+        code: u16,
+        error_response: Option<String>,
+    ) -> EmbedError {
+        Self {
+            kind: EmbedErrorKind::RestInternalServerError(code, error_response),
+            fault: FaultSource::Runtime,
+        }
+    }
+
+    pub(crate) fn rest_other_status_code(code: u16, error_response: Option<String>) -> EmbedError {
+        Self {
+            kind: EmbedErrorKind::RestOtherStatusCode(code, error_response),
+            fault: FaultSource::Undecided,
+        }
+    }
+
+    pub(crate) fn rest_network(transport: ureq::Transport) -> EmbedError {
+        Self { kind: EmbedErrorKind::RestNetwork(transport), fault: FaultSource::Runtime }
+    }
+
+    pub(crate) fn rest_not_an_object(
+        query: serde_json::Value,
+        input_path: Vec<String>,
+    ) -> EmbedError {
+        Self { kind: EmbedErrorKind::RestNotAnObject(query, input_path), fault: FaultSource::User }
+    }
+
+    pub(crate) fn openai_unexpected_dimension(expected: usize, got: usize) -> EmbedError {
+        Self {
+            kind: EmbedErrorKind::OpenAiUnexpectedDimension(expected, got),
+            fault: FaultSource::Runtime,
+        }
+    }
+    pub(crate) fn missing_embedding() -> EmbedError {
+        Self { kind: EmbedErrorKind::MissingEmbedding, fault: FaultSource::Undecided }
     }
 }
 
@@ -188,16 +257,12 @@ impl NewEmbedderError {
         Self { kind: NewEmbedderErrorKind::LoadModel(inner), fault: FaultSource::Runtime }
     }
 
-    pub fn hf_could_not_determine_dimension(inner: EmbedError) -> NewEmbedderError {
+    pub fn could_not_determine_dimension(inner: EmbedError) -> NewEmbedderError {
         Self {
             kind: NewEmbedderErrorKind::CouldNotDetermineDimension(inner),
             fault: FaultSource::Runtime,
         }
     }
-
-    pub fn openai_invalid_api_key_format(inner: reqwest::header::InvalidHeaderValue) -> Self {
-        Self { kind: NewEmbedderErrorKind::InvalidApiKeyFormat(inner), fault: FaultSource::User }
-    }
 }
 
 #[derive(Debug, thiserror::Error)]
@@ -244,7 +309,4 @@ pub enum NewEmbedderErrorKind {
     CouldNotDetermineDimension(EmbedError),
     #[error("loading model failed: {0}")]
     LoadModel(candle_core::Error),
-    // openai
-    #[error("The API key passed to Authorization error was in an invalid format: {0}")]
-    InvalidApiKeyFormat(reqwest::header::InvalidHeaderValue),
 }

milli/src/vector/hf.rs

@@ -33,6 +33,7 @@ enum WeightSource {
 pub struct EmbedderOptions {
     pub model: String,
     pub revision: Option<String>,
+    pub distribution: Option<DistributionShift>,
 }
 
 impl EmbedderOptions {
@@ -40,6 +41,7 @@ impl EmbedderOptions {
         Self {
             model: "BAAI/bge-base-en-v1.5".to_string(),
             revision: Some("617ca489d9e86b49b8167676d8220688b99db36e".into()),
+            distribution: None,
         }
     }
 }
@@ -87,11 +89,11 @@ impl Embedder {
             let config = api.get("config.json").map_err(NewEmbedderError::api_get)?;
             let tokenizer = api.get("tokenizer.json").map_err(NewEmbedderError::api_get)?;
             let (weights, source) = {
-                api.get("pytorch_model.bin")
-                    .map(|filename| (filename, WeightSource::Pytorch))
+                api.get("model.safetensors")
+                    .map(|filename| (filename, WeightSource::Safetensors))
                     .or_else(|_| {
-                        api.get("model.safetensors")
-                            .map(|filename| (filename, WeightSource::Safetensors))
+                        api.get("pytorch_model.bin")
+                            .map(|filename| (filename, WeightSource::Pytorch))
                     })
                     .map_err(NewEmbedderError::api_get)?
             };
@@ -131,7 +133,7 @@ impl Embedder {
 
         let embeddings = this
             .embed(vec!["test".into()])
-            .map_err(NewEmbedderError::hf_could_not_determine_dimension)?;
+            .map_err(NewEmbedderError::could_not_determine_dimension)?;
         this.dimensions = embeddings.first().unwrap().dimension();
 
         Ok(this)
@@ -193,10 +195,15 @@ impl Embedder {
     }
 
     pub fn distribution(&self) -> Option<DistributionShift> {
-        if self.options.model == "BAAI/bge-base-en-v1.5" {
-            Some(DistributionShift { current_mean: 0.85, current_sigma: 0.1 })
-        } else {
-            None
-        }
+        self.options.distribution.or_else(|| {
+            if self.options.model == "BAAI/bge-base-en-v1.5" {
+                Some(DistributionShift {
+                    current_mean: ordered_float::OrderedFloat(0.85),
+                    current_sigma: ordered_float::OrderedFloat(0.1),
+                })
+            } else {
+                None
+            }
+        })
     }
 }

milli/src/vector/manual.rs

@@ -1,19 +1,21 @@
 use super::error::EmbedError;
-use super::Embeddings;
+use super::{DistributionShift, Embeddings};
 
 #[derive(Debug, Clone, Copy)]
 pub struct Embedder {
     dimensions: usize,
+    distribution: Option<DistributionShift>,
 }
 
 #[derive(Debug, Clone, Hash, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
 pub struct EmbedderOptions {
     pub dimensions: usize,
+    pub distribution: Option<DistributionShift>,
 }
 
 impl Embedder {
     pub fn new(options: EmbedderOptions) -> Self {
-        Self { dimensions: options.dimensions }
+        Self { dimensions: options.dimensions, distribution: options.distribution }
     }
 
     pub fn embed(&self, mut texts: Vec<String>) -> Result<Vec<Embeddings<f32>>, EmbedError> {
@@ -31,4 +33,8 @@ impl Embedder {
     ) -> Result<Vec<Vec<Embeddings<f32>>>, EmbedError> {
         text_chunks.into_iter().map(|prompts| self.embed(prompts)).collect()
     }
+
+    pub fn distribution(&self) -> Option<DistributionShift> {
+        self.distribution
+    }
 }

milli/src/vector/mod.rs

@@ -1,8 +1,13 @@
 use std::collections::HashMap;
 use std::sync::Arc;
 
+use deserr::{DeserializeError, Deserr};
+use ordered_float::OrderedFloat;
+use serde::{Deserialize, Serialize};
+
 use self::error::{EmbedError, NewEmbedderError};
 use crate::prompt::{Prompt, PromptData};
+use crate::ThreadPoolNoAbort;
 
 pub mod error;
 pub mod hf;
@@ -10,50 +15,71 @@ pub mod manual;
 pub mod openai;
 pub mod settings;
 
+pub mod ollama;
+pub mod rest;
+
 pub use self::error::Error;
 
 pub type Embedding = Vec<f32>;
 
+pub const REQUEST_PARALLELISM: usize = 40;
+
+/// One or multiple embeddings stored consecutively in a flat vector.
 pub struct Embeddings<F> {
     data: Vec<F>,
     dimension: usize,
 }
 
 impl<F> Embeddings<F> {
+    /// Declares an empty  vector of embeddings of the specified dimensions.
     pub fn new(dimension: usize) -> Self {
         Self { data: Default::default(), dimension }
     }
 
+    /// Declares a vector of embeddings containing a single element.
+    ///
+    /// The dimension is inferred from the length of the passed embedding.
     pub fn from_single_embedding(embedding: Vec<F>) -> Self {
         Self { dimension: embedding.len(), data: embedding }
     }
 
+    /// Declares a vector of embeddings from its components.
+    ///
+    /// `data.len()` must be a multiple of `dimension`, otherwise an error is returned.
     pub fn from_inner(data: Vec<F>, dimension: usize) -> Result<Self, Vec<F>> {
         let mut this = Self::new(dimension);
         this.append(data)?;
         Ok(this)
     }
 
+    /// Returns the number of embeddings in this vector of embeddings.
     pub fn embedding_count(&self) -> usize {
         self.data.len() / self.dimension
     }
 
+    /// Dimension of a single embedding.
     pub fn dimension(&self) -> usize {
         self.dimension
     }
 
+    /// Deconstructs self into the inner flat vector.
     pub fn into_inner(self) -> Vec<F> {
         self.data
     }
 
+    /// A reference to the inner flat vector.
     pub fn as_inner(&self) -> &[F] {
         &self.data
     }
 
+    /// Iterates over the embeddings contained in the flat vector.
     pub fn iter(&self) -> impl Iterator<Item = &'_ [F]> + '_ {
         self.data.as_slice().chunks_exact(self.dimension)
     }
 
+    /// Push an embedding at the end of the embeddings.
+    ///
+    /// If `embedding.len() != self.dimension`, then the push operation fails.
     pub fn push(&mut self, mut embedding: Vec<F>) -> Result<(), Vec<F>> {
         if embedding.len() != self.dimension {
             return Err(embedding);
@@ -62,6 +88,9 @@ impl<F> Embeddings<F> {
         Ok(())
     }
 
+    /// Append a flat vector of embeddings a the end of the embeddings.
+    ///
+    /// If `embeddings.len() % self.dimension != 0`, then the append operation fails.
     pub fn append(&mut self, mut embeddings: Vec<F>) -> Result<(), Vec<F>> {
         if embeddings.len() % self.dimension != 0 {
             return Err(embeddings);
@@ -71,44 +100,68 @@ impl<F> Embeddings<F> {
     }
 }
 
+/// An embedder can be used to transform text into embeddings.
 #[derive(Debug)]
 pub enum Embedder {
+    /// An embedder based on running local models, fetched from the Hugging Face Hub.
     HuggingFace(hf::Embedder),
+    /// An embedder based on making embedding queries against the OpenAI API.
     OpenAi(openai::Embedder),
+    /// An embedder based on the user providing the embeddings in the documents and queries.
     UserProvided(manual::Embedder),
+    /// An embedder based on making embedding queries against an <https://ollama.com> embedding server.
+    Ollama(ollama::Embedder),
+    /// An embedder based on making embedding queries against a generic JSON/REST embedding server.
+    Rest(rest::Embedder),
 }
 
+/// Configuration for an embedder.
 #[derive(Debug, Clone, Default, serde::Deserialize, serde::Serialize)]
 pub struct EmbeddingConfig {
+    /// Options of the embedder, specific to each kind of embedder
     pub embedder_options: EmbedderOptions,
+    /// Document template
     pub prompt: PromptData,
     // TODO: add metrics and anything needed
 }
 
+/// Map of embedder configurations.
+///
+/// Each configuration is mapped to a name.
 #[derive(Clone, Default)]
 pub struct EmbeddingConfigs(HashMap<String, (Arc<Embedder>, Arc<Prompt>)>);
 
 impl EmbeddingConfigs {
+    /// Create the map from its internal component.s
     pub fn new(data: HashMap<String, (Arc<Embedder>, Arc<Prompt>)>) -> Self {
         Self(data)
     }
 
+    /// Get an embedder configuration and template from its name.
     pub fn get(&self, name: &str) -> Option<(Arc<Embedder>, Arc<Prompt>)> {
         self.0.get(name).cloned()
     }
 
+    /// Get the default embedder configuration, if any.
     pub fn get_default(&self) -> Option<(Arc<Embedder>, Arc<Prompt>)> {
-        self.get_default_embedder_name().and_then(|default| self.get(&default))
+        self.get(self.get_default_embedder_name())
     }
 
-    pub fn get_default_embedder_name(&self) -> Option<String> {
+    /// Get the name of the default embedder configuration.
+    ///
+    /// The default embedder is determined as follows:
+    ///
+    /// - If there is only one embedder, it is always the default.
+    /// - If there are multiple embedders and one of them is called `default`, then that one is the default embedder.
+    /// - In all other cases, there is no default embedder.
+    pub fn get_default_embedder_name(&self) -> &str {
         let mut it = self.0.keys();
         let first_name = it.next();
         let second_name = it.next();
         match (first_name, second_name) {
-            (None, _) => None,
-            (Some(first), None) => Some(first.to_owned()),
-            (Some(_), Some(_)) => Some("default".to_owned()),
+            (None, _) => "default",
+            (Some(first), None) => first,
+            (Some(_), Some(_)) => "default",
         }
     }
 }
@@ -123,11 +176,14 @@ impl IntoIterator for EmbeddingConfigs {
     }
 }
 
+/// Options of an embedder, specific to each kind of embedder.
 #[derive(Debug, Clone, Hash, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
 pub enum EmbedderOptions {
     HuggingFace(hf::EmbedderOptions),
     OpenAi(openai::EmbedderOptions),
+    Ollama(ollama::EmbedderOptions),
     UserProvided(manual::EmbedderOptions),
+    Rest(rest::EmbedderOptions),
 }
 
 impl Default for EmbedderOptions {
@@ -137,91 +193,204 @@ impl Default for EmbedderOptions {
 }
 
 impl EmbedderOptions {
+    /// Default options for the Hugging Face embedder
     pub fn huggingface() -> Self {
         Self::HuggingFace(hf::EmbedderOptions::new())
     }
 
+    /// Default options for the OpenAI embedder
     pub fn openai(api_key: Option<String>) -> Self {
         Self::OpenAi(openai::EmbedderOptions::with_default_model(api_key))
     }
+
+    pub fn ollama(api_key: Option<String>, url: Option<String>) -> Self {
+        Self::Ollama(ollama::EmbedderOptions::with_default_model(api_key, url))
+    }
 }
 
 impl Embedder {
+    /// Spawns a new embedder built from its options.
     pub fn new(options: EmbedderOptions) -> std::result::Result<Self, NewEmbedderError> {
         Ok(match options {
             EmbedderOptions::HuggingFace(options) => Self::HuggingFace(hf::Embedder::new(options)?),
             EmbedderOptions::OpenAi(options) => Self::OpenAi(openai::Embedder::new(options)?),
+            EmbedderOptions::Ollama(options) => Self::Ollama(ollama::Embedder::new(options)?),
             EmbedderOptions::UserProvided(options) => {
                 Self::UserProvided(manual::Embedder::new(options))
             }
+            EmbedderOptions::Rest(options) => Self::Rest(rest::Embedder::new(options)?),
         })
     }
 
-    pub async fn embed(
+    /// Embed one or multiple texts.
+    ///
+    /// Each text can be embedded as one or multiple embeddings.
+    pub fn embed(
         &self,
         texts: Vec<String>,
     ) -> std::result::Result<Vec<Embeddings<f32>>, EmbedError> {
         match self {
             Embedder::HuggingFace(embedder) => embedder.embed(texts),
-            Embedder::OpenAi(embedder) => {
-                let client = embedder.new_client()?;
-                embedder.embed(texts, &client).await
-            }
+            Embedder::OpenAi(embedder) => embedder.embed(texts),
+            Embedder::Ollama(embedder) => embedder.embed(texts),
             Embedder::UserProvided(embedder) => embedder.embed(texts),
+            Embedder::Rest(embedder) => embedder.embed(texts),
         }
     }
 
-    /// # Panics
+    pub fn embed_one(&self, text: String) -> std::result::Result<Embedding, EmbedError> {
+        let mut embeddings = self.embed(vec![text])?;
+        let embeddings = embeddings.pop().ok_or_else(EmbedError::missing_embedding)?;
+        Ok(if embeddings.iter().nth(1).is_some() {
+            tracing::warn!("Ignoring embeddings past the first one in long search query");
+            embeddings.iter().next().unwrap().to_vec()
+        } else {
+            embeddings.into_inner()
+        })
+    }
+
+    /// Embed multiple chunks of texts.
     ///
-    /// - if called from an asynchronous context
+    /// Each chunk is composed of one or multiple texts.
     pub fn embed_chunks(
         &self,
         text_chunks: Vec<Vec<String>>,
+        threads: &ThreadPoolNoAbort,
     ) -> std::result::Result<Vec<Vec<Embeddings<f32>>>, EmbedError> {
         match self {
             Embedder::HuggingFace(embedder) => embedder.embed_chunks(text_chunks),
-            Embedder::OpenAi(embedder) => embedder.embed_chunks(text_chunks),
+            Embedder::OpenAi(embedder) => embedder.embed_chunks(text_chunks, threads),
+            Embedder::Ollama(embedder) => embedder.embed_chunks(text_chunks, threads),
             Embedder::UserProvided(embedder) => embedder.embed_chunks(text_chunks),
+            Embedder::Rest(embedder) => embedder.embed_chunks(text_chunks, threads),
         }
     }
 
+    /// Indicates the preferred number of chunks to pass to [`Self::embed_chunks`]
     pub fn chunk_count_hint(&self) -> usize {
         match self {
             Embedder::HuggingFace(embedder) => embedder.chunk_count_hint(),
             Embedder::OpenAi(embedder) => embedder.chunk_count_hint(),
+            Embedder::Ollama(embedder) => embedder.chunk_count_hint(),
             Embedder::UserProvided(_) => 1,
+            Embedder::Rest(embedder) => embedder.chunk_count_hint(),
         }
     }
 
+    /// Indicates the preferred number of texts in a single chunk passed to [`Self::embed`]
     pub fn prompt_count_in_chunk_hint(&self) -> usize {
         match self {
             Embedder::HuggingFace(embedder) => embedder.prompt_count_in_chunk_hint(),
             Embedder::OpenAi(embedder) => embedder.prompt_count_in_chunk_hint(),
+            Embedder::Ollama(embedder) => embedder.prompt_count_in_chunk_hint(),
             Embedder::UserProvided(_) => 1,
+            Embedder::Rest(embedder) => embedder.prompt_count_in_chunk_hint(),
         }
     }
 
+    /// Indicates the dimensions of a single embedding produced by the embedder.
     pub fn dimensions(&self) -> usize {
         match self {
             Embedder::HuggingFace(embedder) => embedder.dimensions(),
             Embedder::OpenAi(embedder) => embedder.dimensions(),
+            Embedder::Ollama(embedder) => embedder.dimensions(),
             Embedder::UserProvided(embedder) => embedder.dimensions(),
+            Embedder::Rest(embedder) => embedder.dimensions(),
         }
     }
 
+    /// An optional distribution used to apply an affine transformation to the similarity score of a document.
     pub fn distribution(&self) -> Option<DistributionShift> {
         match self {
             Embedder::HuggingFace(embedder) => embedder.distribution(),
             Embedder::OpenAi(embedder) => embedder.distribution(),
-            Embedder::UserProvided(_embedder) => None,
+            Embedder::Ollama(embedder) => embedder.distribution(),
+            Embedder::UserProvided(embedder) => embedder.distribution(),
+            Embedder::Rest(embedder) => embedder.distribution(),
         }
     }
 }
 
-#[derive(Debug, Clone, Copy)]
+/// Describes the mean and sigma of distribution of embedding similarity in the embedding space.
+///
+/// The intended use is to make the similarity score more comparable to the regular ranking score.
+/// This allows to correct effects where results are too "packed" around a certain value.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Deserialize, Serialize)]
+#[serde(from = "DistributionShiftSerializable")]
+#[serde(into = "DistributionShiftSerializable")]
 pub struct DistributionShift {
-    pub current_mean: f32,
-    pub current_sigma: f32,
+    /// Value where the results are "packed".
+    ///
+    /// Similarity scores are translated so that they are packed around 0.5 instead
+    pub current_mean: OrderedFloat<f32>,
+
+    /// standard deviation of a similarity score.
+    ///
+    /// Set below 0.4 to make the results less packed around the mean, and above 0.4 to make them more packed.
+    pub current_sigma: OrderedFloat<f32>,
+}
+
+impl<E> Deserr<E> for DistributionShift
+where
+    E: DeserializeError,
+{
+    fn deserialize_from_value<V: deserr::IntoValue>(
+        value: deserr::Value<V>,
+        location: deserr::ValuePointerRef,
+    ) -> Result<Self, E> {
+        let value = DistributionShiftSerializable::deserialize_from_value(value, location)?;
+        if value.mean < 0. || value.mean > 1. {
+            return Err(deserr::take_cf_content(E::error::<std::convert::Infallible>(
+                None,
+                deserr::ErrorKind::Unexpected {
+                    msg: format!(
+                        "the distribution mean must be in the range [0, 1], got {}",
+                        value.mean
+                    ),
+                },
+                location,
+            )));
+        }
+        if value.sigma <= 0. || value.sigma > 1. {
+            return Err(deserr::take_cf_content(E::error::<std::convert::Infallible>(
+                None,
+                deserr::ErrorKind::Unexpected {
+                    msg: format!(
+                        "the distribution sigma must be in the range ]0, 1], got {}",
+                        value.sigma
+                    ),
+                },
+                location,
+            )));
+        }
+
+        Ok(value.into())
+    }
+}
+
+#[derive(Serialize, Deserialize, Deserr)]
+#[serde(deny_unknown_fields)]
+#[deserr(deny_unknown_fields)]
+struct DistributionShiftSerializable {
+    mean: f32,
+    sigma: f32,
+}
+
+impl From<DistributionShift> for DistributionShiftSerializable {
+    fn from(
+        DistributionShift {
+            current_mean: OrderedFloat(current_mean),
+            current_sigma: OrderedFloat(current_sigma),
+        }: DistributionShift,
+    ) -> Self {
+        Self { mean: current_mean, sigma: current_sigma }
+    }
+}
+
+impl From<DistributionShiftSerializable> for DistributionShift {
+    fn from(DistributionShiftSerializable { mean, sigma }: DistributionShiftSerializable) -> Self {
+        Self { current_mean: OrderedFloat(mean), current_sigma: OrderedFloat(sigma) }
+    }
 }
 
 impl DistributionShift {
@@ -230,11 +399,13 @@ impl DistributionShift {
         if sigma <= 0.0 {
             None
         } else {
-            Some(Self { current_mean: mean, current_sigma: sigma })
+            Some(Self { current_mean: OrderedFloat(mean), current_sigma: OrderedFloat(sigma) })
         }
     }
 
     pub fn shift(&self, score: f32) -> f32 {
+        let current_mean = self.current_mean.0;
+        let current_sigma = self.current_sigma.0;
         // <https://math.stackexchange.com/a/2894689>
         // We're somewhat abusively mapping the distribution of distances to a gaussian.
         // The parameters we're given is the mean and sigma of the native result distribution.
@@ -244,9 +415,9 @@ impl DistributionShift {
         let target_sigma = 0.4;
 
         // a^2 sig1^2 = sig2^2 => a^2 = sig2^2 / sig1^2 => a = sig2 / sig1, assuming a, sig1, and sig2 positive.
-        let factor = target_sigma / self.current_sigma;
+        let factor = target_sigma / current_sigma;
         // a*mu1 + b = mu2 => b = mu2 - a*mu1
-        let offset = target_mean - (factor * self.current_mean);
+        let offset = target_mean - (factor * current_mean);
 
         let mut score = factor * score + offset;
 
@@ -262,6 +433,7 @@ impl DistributionShift {
     }
 }
 
+/// Whether CUDA is supported in this version of Meilisearch.
 pub const fn is_cuda_enabled() -> bool {
     cfg!(feature = "cuda")
 }

milli/src/vector/ollama.rs

@@ -0,0 +1,108 @@
+use rayon::iter::{IntoParallelIterator as _, ParallelIterator as _};
+
+use super::error::{EmbedError, EmbedErrorKind, NewEmbedderError, NewEmbedderErrorKind};
+use super::rest::{Embedder as RestEmbedder, EmbedderOptions as RestEmbedderOptions};
+use super::{DistributionShift, Embeddings};
+use crate::error::FaultSource;
+use crate::ThreadPoolNoAbort;
+
+#[derive(Debug)]
+pub struct Embedder {
+    rest_embedder: RestEmbedder,
+}
+
+#[derive(Debug, Clone, Hash, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
+pub struct EmbedderOptions {
+    pub embedding_model: String,
+    pub url: Option<String>,
+    pub api_key: Option<String>,
+    pub distribution: Option<DistributionShift>,
+}
+
+impl EmbedderOptions {
+    pub fn with_default_model(api_key: Option<String>, url: Option<String>) -> Self {
+        Self { embedding_model: "nomic-embed-text".into(), api_key, url, distribution: None }
+    }
+}
+
+impl Embedder {
+    pub fn new(options: EmbedderOptions) -> Result<Self, NewEmbedderError> {
+        let model = options.embedding_model.as_str();
+        let rest_embedder = match RestEmbedder::new(RestEmbedderOptions {
+            api_key: options.api_key,
+            dimensions: None,
+            distribution: options.distribution,
+            url: options.url.unwrap_or_else(get_ollama_path),
+            query: serde_json::json!({
+                "model": model,
+            }),
+            input_field: vec!["prompt".to_owned()],
+            path_to_embeddings: Default::default(),
+            embedding_object: vec!["embedding".to_owned()],
+            input_type: super::rest::InputType::Text,
+        }) {
+            Ok(embedder) => embedder,
+            Err(NewEmbedderError {
+                kind:
+                    NewEmbedderErrorKind::CouldNotDetermineDimension(EmbedError {
+                        kind: super::error::EmbedErrorKind::RestOtherStatusCode(404, error),
+                        fault: _,
+                    }),
+                fault: _,
+            }) => {
+                return Err(NewEmbedderError::could_not_determine_dimension(
+                    EmbedError::ollama_model_not_found(error),
+                ))
+            }
+            Err(error) => return Err(error),
+        };
+
+        Ok(Self { rest_embedder })
+    }
+
+    pub fn embed(&self, texts: Vec<String>) -> Result<Vec<Embeddings<f32>>, EmbedError> {
+        match self.rest_embedder.embed(texts) {
+            Ok(embeddings) => Ok(embeddings),
+            Err(EmbedError { kind: EmbedErrorKind::RestOtherStatusCode(404, error), fault: _ }) => {
+                Err(EmbedError::ollama_model_not_found(error))
+            }
+            Err(error) => Err(error),
+        }
+    }
+
+    pub fn embed_chunks(
+        &self,
+        text_chunks: Vec<Vec<String>>,
+        threads: &ThreadPoolNoAbort,
+    ) -> Result<Vec<Vec<Embeddings<f32>>>, EmbedError> {
+        threads
+            .install(move || {
+                text_chunks.into_par_iter().map(move |chunk| self.embed(chunk)).collect()
+            })
+            .map_err(|error| EmbedError {
+                kind: EmbedErrorKind::PanicInThreadPool(error),
+                fault: FaultSource::Bug,
+            })?
+    }
+
+    pub fn chunk_count_hint(&self) -> usize {
+        self.rest_embedder.chunk_count_hint()
+    }
+
+    pub fn prompt_count_in_chunk_hint(&self) -> usize {
+        self.rest_embedder.prompt_count_in_chunk_hint()
+    }
+
+    pub fn dimensions(&self) -> usize {
+        self.rest_embedder.dimensions()
+    }
+
+    pub fn distribution(&self) -> Option<DistributionShift> {
+        self.rest_embedder.distribution()
+    }
+}
+
+fn get_ollama_path() -> String {
+    // Important: Hostname not enough, has to be entire path to embeddings endpoint
+    std::env::var("MEILI_OLLAMA_URL").unwrap_or("http://localhost:11434/api/embeddings".to_string())
+}