WIP: Failing embedding no longer causes the whole search to fail

.cargo/config.toml

@@ -1,2 +1,2 @@
 [alias]
-xtask = "run --release --package xtask --"
+xtask = "run --package xtask --"

.github/ISSUE_TEMPLATE/new_feature_issue.md

@@ -1,56 +0,0 @@
----
-name: New feature issue
-about: ⚠️ Should only be used by the internal Meili team ⚠️
-title: ''
-labels: 'impacts docs, impacts integrations'
-assignees: ''
-
----
-
-Related product team resources: [PRD]() (_internal only_)
-
-## Usage
-
-<!---Link to the public part of the PRD, or to the related product discussion for experimental features-->
-
-TBD
-
-## TODO
-
-<!---If necessary, create a list with technical/product steps-->
-
-### Are you modifying a database?
-
-- [ ] If not, add the `no db change` label to your PR, and you're good to merge.
-- [ ] If yes, add the `db change` label to your PR. You'll receive a message explaining you what to do.
-
-### Reminders when modifying the API
-
-- [ ] Update the openAPI file with utoipa:
-  - [ ] If a new module has been introduced, create a new structure deriving [the OpenAPI proc-macro](https://docs.rs/utoipa/latest/utoipa/derive.OpenApi.html) and nest it in the main [openAPI structure](https://github.com/meilisearch/meilisearch/blob/f2185438eed60fa32d25b15480c5ee064f6fba4a/crates/meilisearch/src/routes/mod.rs#L64-L78).
-  - [ ] If a new route has been introduced, add the [path decorator](https://docs.rs/utoipa/latest/utoipa/attr.path.html) to it and add the route at the top of the file in its openAPI structure.
-  - [ ] If a structure which is deserialized or serialized in the API has been introduced or modified, it must derive the [`schema`](https://docs.rs/utoipa/latest/utoipa/macro.schema.html) or the [`IntoParams`](https://docs.rs/utoipa/latest/utoipa/derive.IntoParams.html) proc-macro.
-        If it's a **new** structure you must also add it to the big list of structures [in the main `OpenApi` structure](https://github.com/meilisearch/meilisearch/blob/f2185438eed60fa32d25b15480c5ee064f6fba4a/crates/meilisearch/src/routes/mod.rs#L88).
-  - [ ] Once everything is done, start Meilisearch with the swagger flag: `cargo run --features swagger`, open `http://localhost:7700/scalar` on your browser, and ensure everything works as expected.
-  - For more info, refer to [this presentation](https://pitch.com/v/generating-the-openapi-file-jrn3nh).
-
-### Reminders when modifying the Setting API
-
-<!--- Special steps to remind when adding a new index setting -->
-
-- [ ] Ensure the new setting route is at least tested by the [`test_setting_routes` macro](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/tests/settings/get_settings.rs#L276)
-- [ ] Ensure Analytics are fully implemented
-  - [ ] `/settings/my-new-setting` configurated in the [`make_setting_routes` macro](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/src/routes/indexes/settings.rs#L141-L165)
-  - [ ] global `/settings` route configurated in the [`update_all` function](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/src/routes/indexes/settings.rs#L655-L751)
-- [ ] Ensure the dump serializing is consistent with the `/settings` route serializing, e.g., enums case can be different (`camelCase` in route and `PascalCase` in the dump)
-
-#### Special cases when adding a setting for an experimental feature
-
-- [ ] ⚠️ API stability: The setting does not appear on the main settings route when the feature has never been enabled (e.g. mark it `Unset` when returned from the index in this situation. See [an example](https://github.com/meilisearch/meilisearch/blob/7a89abd2a025606a42f8b219e539117eb2eb029f/meilisearch-types/src/settings.rs#L608))
-- [ ] The setting cannot be set when the feature is disabled, either by the main settings route or the subroute (see [`validate_settings` function](https://github.com/meilisearch/meilisearch/blob/7a89abd2a025606a42f8b219e539117eb2eb029f/meilisearch/src/routes/indexes/settings.rs#L811))
-- [ ] If possible, the setting is reset when the feature is disabled (hard if it requires reindexing)
-
-## Impacted teams
-
-<!---Ping the related teams. Ask on Slack if any hesitation-->
-<!---@meilisearch/docs-team and @meilisearch/integration-team when there is any API change, e.g. settings addition-->

.github/ISSUE_TEMPLATE/sprint_issue.md

@@ -0,0 +1,49 @@
+---
+name: New sprint issue
+about: ⚠️ Should only be used by the engine team ⚠️
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+Related product team resources: [PRD]() (_internal only_)
+Related product discussion:
+Related spec: WIP
+
+## Motivation
+
+<!---Copy/paste the information in PRD or briefly detail the product motivation. Ask product team if any hesitation.-->
+
+## Usage
+
+<!---Link to the public part of the PRD, or to the related product discussion for experimental features-->
+
+## TODO
+
+<!---Feel free to adapt this list with more technical/product steps-->
+
+- [ ] Release a prototype
+- [ ] If prototype validated, merge changes into `main`
+- [ ] Update the spec
+
+### Reminders when modifying the Setting API
+
+<!--- Special steps to remind when adding a new index setting -->
+
+- [ ] Ensure the new setting route is at least tested by the [`test_setting_routes` macro](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/tests/settings/get_settings.rs#L276)
+- [ ] Ensure Analytics are fully implemented
+  - [ ] `/settings/my-new-setting` configurated in the [`make_setting_routes` macro](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/src/routes/indexes/settings.rs#L141-L165)
+  - [ ] global `/settings` route configurated in the [`update_all` function](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/src/routes/indexes/settings.rs#L655-L751)
+- [ ] Ensure the dump serializing is consistent with the `/settings` route serializing, e.g., enums case can be different (`camelCase` in route and `PascalCase` in the dump)
+
+#### Special cases when adding a setting for an experimental feature
+
+- [ ] ⚠️ API stability: The setting does not appear on the main settings route when the feature has never been enabled (e.g. mark it `Unset` when returned from the index in this situation. See [an example](https://github.com/meilisearch/meilisearch/blob/7a89abd2a025606a42f8b219e539117eb2eb029f/meilisearch-types/src/settings.rs#L608))
+- [ ] The setting cannot be set when the feature is disabled, either by the main settings route or the subroute (see [`validate_settings` function](https://github.com/meilisearch/meilisearch/blob/7a89abd2a025606a42f8b219e539117eb2eb029f/meilisearch/src/routes/indexes/settings.rs#L811))
+- [ ] If possible, the setting is reset when the feature is disabled (hard if it requires reindexing)
+
+## Impacted teams
+
+<!---Ping the related teams. Ask for the engine manager if any hesitation-->
+<!---@meilisearch/docs-team when there is any API change, e.g. settings addition-->

.github/dependabot.yml

@@ -7,5 +7,6 @@ updates:
     schedule:
       interval: "monthly"
     labels:
+      - 'skip changelog'
       - 'dependencies'
     rebase-strategy: disabled

.github/pull_request_template.md

@@ -1,16 +0,0 @@
-## Related issue
-
-Fixes #...
-
-## Requirements
-
-⚠️ Ensure the following requirements before merging ⚠️
-- [ ] Automated tests have been added.
-- [ ] If some tests cannot be automated, manual rigorous tests should be applied.
-- [ ] ⚠️ If there is any change in the DB: 
-    - [ ] Test that any impacted DB still works as expected after using `--experimental-dumpless-upgrade` on a DB created with the last released Meilisearch
-    - [ ] Test that during the upgrade, **search is still available** (artificially make the upgrade longer if needed)
-    - [ ] Set the `db change` label.
-- [ ] If necessary, the feature have been tested in the Cloud production environment (with [prototypes](./documentation/prototypes.md)) and the Cloud UI is ready.
-- [ ] If necessary, the [documentation](https://github.com/meilisearch/documentation) related to the implemented feature in the PR is ready.
-- [ ] If necessary, the [integrations](https://github.com/meilisearch/integration-guides) related to the implemented feature in the PR are ready.

.github/templates/dependency-issue.md

@@ -1,22 +0,0 @@
-This issue is about updating Meilisearch dependencies:
-  - [ ] Update Meilisearch dependencies with the help of `cargo +nightly udeps --all-targets` (remove unused dependencies) and `cargo upgrade` (upgrade dependencies versions) - ⚠️ Some repositories may contain subdirectories (like heed, charabia, or deserr). Take care of updating these in the main crate as well. This won't be done automatically by `cargo upgrade`.
-    - [ ] [deserr](https://github.com/meilisearch/deserr)
-    - [ ] [charabia](https://github.com/meilisearch/charabia/)
-    - [ ] [heed](https://github.com/meilisearch/heed/)
-    - [ ] [roaring-rs](https://github.com/RoaringBitmap/roaring-rs/)
-    - [ ] [obkv](https://github.com/meilisearch/obkv)
-    - [ ] [grenad](https://github.com/meilisearch/grenad/)
-    - [ ] [arroy](https://github.com/meilisearch/arroy/)
-    - [ ] [segment](https://github.com/meilisearch/segment)
-    - [ ] [bumparaw-collections](https://github.com/meilisearch/bumparaw-collections)
-    - [ ] [bbqueue](https://github.com/meilisearch/bbqueue)
-    - [ ] Finally, [Meilisearch](https://github.com/meilisearch/MeiliSearch)
-  - [ ] If new Rust versions have been released, update the minimal Rust version in use at Meilisearch:
-    - [ ] in this [GitHub Action file](https://github.com/meilisearch/meilisearch/blob/main/.github/workflows/test-suite.yml), by changing the `toolchain` field of the `rustfmt` job to the latest available nightly (of the day before or the current day).
-    - [ ] in every [GitHub Action files](https://github.com/meilisearch/meilisearch/blob/main/.github/workflows), by changing all the `dtolnay/rust-toolchain@` references to use the latest stable version.
-    - [ ] in this [`rust-toolchain.toml`](https://github.com/meilisearch/meilisearch/blob/main/rust-toolchain.toml), by changing the `channel` field to the latest stable version.
-    - [ ] in the [Dockerfile](https://github.com/meilisearch/meilisearch/blob/main/Dockerfile), by changing the base image to `rust:<target_rust_version>-alpine<alpine_version>`. Check that the image exists on [Dockerhub](https://hub.docker.com/_/rust/tags?page=1&name=alpine). Also, build and run the image to check everything still works!
-
-⚠️ This issue should be prioritized to avoid any deprecation and vulnerability issues.
-
-The GitHub action dependencies are managed by [Dependabot](https://github.com/meilisearch/meilisearch/blob/main/.github/dependabot.yml), so no need to update them when solving this issue.

.github/workflows/bench-manual.yml

@@ -1,27 +0,0 @@
-name: Bench (manual)
-
-on:
-  workflow_dispatch:
-    inputs:
-      workload:
-        description: "The path to the workloads to execute (workloads/...)"
-        required: true
-        default: "workloads/movies.json"
-
-env:
-  WORKLOAD_NAME: ${{ github.event.inputs.workload }}
-
-jobs:
-  benchmarks:
-    name: Run and upload benchmarks
-    runs-on: benchmarks
-    timeout-minutes: 180 # 3h
-    steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
-        with:
-          profile: minimal
-
-      - name: Run benchmarks - workload ${WORKLOAD_NAME} - branch ${{ github.ref }} - commit ${{ github.sha }}
-        run: |
-          cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" --reason "Manual [Run #${{ github.run_id }}](https://github.com/meilisearch/meilisearch/actions/runs/${{ github.run_id }})" -- ${WORKLOAD_NAME}

.github/workflows/bench-pr.yml

@@ -1,82 +0,0 @@
-name: Bench (PR)
-on:
-  issue_comment:
-    types: [created]
-
-permissions:
-  issues: write
-
-env:
-  GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-
-jobs:
-  run-benchmarks-on-comment:
-    if: startsWith(github.event.comment.body, '/bench')
-    name: Run and upload benchmarks
-    runs-on: benchmarks
-    timeout-minutes: 180 # 3h
-    steps:
-      - name: Check permissions
-        id: permission
-        env:
-          PR_AUTHOR: ${{github.event.issue.user.login }}
-          COMMENT_AUTHOR: ${{github.event.comment.user.login }}
-          REPOSITORY: ${{github.repository}}
-          PR_ID: ${{github.event.issue.number}}
-        run: |
-          PR_REPOSITORY=$(gh api /repos/"$REPOSITORY"/pulls/"$PR_ID" --jq .head.repo.full_name)
-          if $(gh api /repos/"$REPOSITORY"/collaborators/"$PR_AUTHOR"/permission --jq .user.permissions.push)
-          then
-            echo "::notice title=Authentication success::PR author authenticated"
-          else
-            echo "::error title=Authentication error::PR author doesn't have push permission on this repository"
-            exit 1
-          fi
-          if $(gh api /repos/"$REPOSITORY"/collaborators/"$COMMENT_AUTHOR"/permission --jq .user.permissions.push)
-          then
-            echo "::notice title=Authentication success::Comment author authenticated"
-          else
-            echo "::error title=Authentication error::Comment author doesn't have push permission on this repository"
-            exit 1
-          fi
-          if [ "$PR_REPOSITORY" = "$REPOSITORY" ]
-          then
-            echo "::notice title=Authentication success::PR started from main repository"
-          else
-            echo "::error title=Authentication error::PR started from a fork"
-            exit 1
-          fi
-
-      - name: Check for Command
-        id: command
-        uses: xt0rted/slash-command-action@v2
-        with:
-          command: bench
-          reaction-type: "rocket"
-          repo-token: ${{ env.GH_TOKEN }}
-
-      - uses: xt0rted/pull-request-comment-branch@v3
-        id: comment-branch
-        with:
-          repo_token: ${{ env.GH_TOKEN }}
-
-      - uses: actions/checkout@v5
-        if: success()
-        with:
-          fetch-depth: 0 # fetch full history to be able to get main commit sha
-          ref: ${{ steps.comment-branch.outputs.head_ref }}
-
-      - uses: dtolnay/rust-toolchain@1.89
-        with:
-          profile: minimal
-
-      - name: Run benchmarks on PR ${{ github.event.issue.id }}
-        run: |
-          cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" \
-             --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" \
-             --reason "[Comment](${{ github.event.comment.html_url }}) on [#${{ github.event.issue.number }}](${{ github.event.issue.html_url }})" \
-             -- ${{ steps.command.outputs.command-arguments }} > benchlinks.txt
-
-      - name: Send comment in PR
-        run: |
-          gh pr comment ${{github.event.issue.number}} --body-file benchlinks.txt

.github/workflows/bench-push-indexing.yml

@@ -1,22 +0,0 @@
-name: Indexing bench (push)
-
-on:
-  push:
-    branches:
-      - main
-
-jobs:
-  benchmarks:
-    name: Run and upload benchmarks
-    runs-on: benchmarks
-    timeout-minutes: 180 # 3h
-    steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
-        with:
-          profile: minimal
-
-      # Run benchmarks
-      - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch main - Commit ${{ github.sha }}
-        run: |
-          cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" --reason "Push on `main` [Run #${{ github.run_id }}](https://github.com/meilisearch/meilisearch/actions/runs/${{ github.run_id }})" -- workloads/*.json

.github/workflows/benchmarks-manual.yml

@@ -4,9 +4,9 @@ on:
   workflow_dispatch:
     inputs:
       dataset_name:
-        description: "The name of the dataset used to benchmark (search_songs, search_wiki, search_geo or indexing)"
+        description: 'The name of the dataset used to benchmark (search_songs, search_wiki, search_geo or indexing)'
         required: false
-        default: "search_songs"
+        default: 'search_songs'
 
 env:
   BENCH_NAME: ${{ github.event.inputs.dataset_name }}
@@ -17,10 +17,12 @@ jobs:
     runs-on: benchmarks
     timeout-minutes: 4320 # 72h
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -43,7 +45,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files
@@ -67,7 +69,7 @@ jobs:
           out_dir: critcmp_results
 
       # Helper
-      - name: "README: compare with another benchmark"
+      - name: 'README: compare with another benchmark'
         run: |
           echo "${{ steps.file.outputs.basename }}.json has just been pushed."
           echo 'How to compare this benchmark with another one?'

.github/workflows/benchmarks-pr.yml

@@ -13,40 +13,11 @@ jobs:
     runs-on: benchmarks
     timeout-minutes: 4320 # 72h
     steps:
-      - name: Check permissions
-        id: permission
-        env:
-          PR_AUTHOR: ${{github.event.issue.user.login }}
-          COMMENT_AUTHOR: ${{github.event.comment.user.login }}
-          REPOSITORY: ${{github.repository}}
-          PR_ID: ${{github.event.issue.number}}
-        run: |
-          PR_REPOSITORY=$(gh api /repos/"$REPOSITORY"/pulls/"$PR_ID" --jq .head.repo.full_name)
-          if $(gh api /repos/"$REPOSITORY"/collaborators/"$PR_AUTHOR"/permission --jq .user.permissions.push)
-          then
-            echo "::notice title=Authentication success::PR author authenticated"
-          else
-            echo "::error title=Authentication error::PR author doesn't have push permission on this repository"
-            exit 1
-          fi
-          if $(gh api /repos/"$REPOSITORY"/collaborators/"$COMMENT_AUTHOR"/permission --jq .user.permissions.push)
-          then
-            echo "::notice title=Authentication success::Comment author authenticated"
-          else
-            echo "::error title=Authentication error::Comment author doesn't have push permission on this repository"
-            exit 1
-          fi
-          if [ "$PR_REPOSITORY" = "$REPOSITORY" ]
-          then
-            echo "::notice title=Authentication success::PR started from main repository"
-          else
-            echo "::error title=Authentication error::PR started from a fork"
-            exit 1
-          fi
-
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       - name: Check for Command
         id: command
@@ -56,12 +27,12 @@ jobs:
           reaction-type: "eyes"
           repo-token: ${{ env.GH_TOKEN }}
 
-      - uses: xt0rted/pull-request-comment-branch@v3
+      - uses: xt0rted/pull-request-comment-branch@v2
         id: comment-branch
         with:
           repo_token: ${{ env.GH_TOKEN }}
 
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
         if: success()
         with:
           fetch-depth: 0 # fetch full history to be able to get main commit sha
@@ -88,7 +59,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${{ steps.command.outputs.command-arguments }} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${{ steps.command.outputs.command-arguments }} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files

.github/workflows/benchmarks-push-indexing.yml

@@ -15,10 +15,12 @@ jobs:
     runs-on: benchmarks
     timeout-minutes: 4320 # 72h
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -41,7 +43,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files
@@ -69,7 +71,7 @@ jobs:
         run: telegraf --config https://eu-central-1-1.aws.cloud2.influxdata.com/api/v2/telegrafs/08b52e34a370b000 --once --debug
 
       # Helper
-      - name: "README: compare with another benchmark"
+      - name: 'README: compare with another benchmark'
         run: |
           echo "${{ steps.file.outputs.basename }}.json has just been pushed."
           echo 'How to compare this benchmark with another one?'

.github/workflows/benchmarks-push-search-geo.yml

@@ -14,10 +14,12 @@ jobs:
     name: Run and upload benchmarks
     runs-on: benchmarks
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -40,7 +42,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files
@@ -68,7 +70,7 @@ jobs:
         run: telegraf --config https://eu-central-1-1.aws.cloud2.influxdata.com/api/v2/telegrafs/08b52e34a370b000 --once --debug
 
       # Helper
-      - name: "README: compare with another benchmark"
+      - name: 'README: compare with another benchmark'
         run: |
           echo "${{ steps.file.outputs.basename }}.json has just been pushed."
           echo 'How to compare this benchmark with another one?'

.github/workflows/benchmarks-push-search-songs.yml

@@ -14,10 +14,12 @@ jobs:
     name: Run and upload benchmarks
     runs-on: benchmarks
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -40,7 +42,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files
@@ -68,7 +70,7 @@ jobs:
         run: telegraf --config https://eu-central-1-1.aws.cloud2.influxdata.com/api/v2/telegrafs/08b52e34a370b000 --once --debug
 
       # Helper
-      - name: "README: compare with another benchmark"
+      - name: 'README: compare with another benchmark'
         run: |
           echo "${{ steps.file.outputs.basename }}.json has just been pushed."
           echo 'How to compare this benchmark with another one?'

.github/workflows/benchmarks-push-search-wiki.yml

@@ -14,10 +14,12 @@ jobs:
     name: Run and upload benchmarks
     runs-on: benchmarks
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -40,7 +42,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files
@@ -68,7 +70,7 @@ jobs:
         run: telegraf --config https://eu-central-1-1.aws.cloud2.influxdata.com/api/v2/telegrafs/08b52e34a370b000 --once --debug
 
       # Helper
-      - name: "README: compare with another benchmark"
+      - name: 'README: compare with another benchmark'
         run: |
           echo "${{ steps.file.outputs.basename }}.json has just been pushed."
           echo 'How to compare this benchmark with another one?'

.github/workflows/db-change-comments.yml

@@ -1,57 +0,0 @@
-name: Comment when db change labels are added
-
-on:
-  pull_request:
-    types: [labeled]
-
-env:
-  MESSAGE: |
-    ### Hello, I'm a bot 🤖 
-
-    You are receiving this message because you declared that this PR make changes to the Meilisearch database.
-    Depending on the nature of the change, additional actions might be required on your part. The following sections detail the additional actions depending on the nature of the change, please copy the relevant section in the description of your PR, and make sure to perform the required actions.
-
-    Thank you for contributing to Meilisearch :heart:
-
-    ## This PR makes forward-compatible changes
-
-    *Forward-compatible changes are changes to the database such that databases created in an older version of Meilisearch are still valid in the new version of Meilisearch. They usually represent additive changes, like adding a new optional attribute or setting.*
-
-    - [ ] Detail the change to the DB format and why they are forward compatible
-    - [ ] Forward-compatibility: A database created before this PR and using the features touched by this PR was able to be opened by a Meilisearch produced by the code of this PR.
-
-
-    ## This PR makes breaking changes
-
-    *Breaking changes are changes to the database such that databases created in an older version of Meilisearch need changes to remain valid in the new version of Meilisearch. This typically happens when the way to store the data changed (change of database, new required key, etc). This can also happen due to breaking changes in the API of an experimental feature. ⚠️ This kind of changes are more difficult to achieve safely, so proceed with caution and test dumpless upgrade right before merging the PR.*
-
-    - [ ] Detail the changes to the DB format,
-        - [ ] which are compatible, and why
-        - [ ] which are not compatible, why, and how they will be fixed up in the upgrade
-     - [ ] /!\ Ensure all the read operations still work!
-        - If the change happened in milli, you may need to check the version of the database before doing any read operation
-        - If the change happened in the index-scheduler, make sure the new code can immediately read the old database
-        - If the change happened in the meilisearch-auth database, reach out to the team; we don't know yet how to handle these changes
-      - [ ] Write the code to go from the old database to the new one
-        - If the change happened in milli, the upgrade function should be written and called [here](https://github.com/meilisearch/meilisearch/blob/3fd86e8d76d7d468b0095d679adb09211ca3b6c0/crates/milli/src/update/upgrade/mod.rs#L24-L47)
-        - If the change happened in the index-scheduler, we've never done it yet, but the right place to do it should be [here](https://github.com/meilisearch/meilisearch/blob/3fd86e8d76d7d468b0095d679adb09211ca3b6c0/crates/index-scheduler/src/scheduler/process_upgrade/mod.rs#L13)
-      - [ ] Write an integration test [here](https://github.com/meilisearch/meilisearch/blob/main/crates/meilisearch/tests/upgrade/mod.rs) ensuring you can read the old database, upgrade to the new database, and read the new database as expected
-    
-
-jobs:
-  add-comment:
-    runs-on: ubuntu-latest
-    if: github.event.label.name == 'db change'
-    steps:
-      - name: Add comment
-        uses: actions/github-script@v7
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            const message = process.env.MESSAGE;
-            github.rest.issues.createComment({
-              issue_number: context.issue.number,
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              body: message
-            })

.github/workflows/db-change-missing.yml

@@ -1,28 +0,0 @@
-name: Check db change labels
-
-on:
-  pull_request:
-    types: [opened, synchronize, reopened, labeled, unlabeled]
-
-jobs:
-  check-labels:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v5
-      - name: Check db change labels
-        id: check_labels
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: |
-          URL=/repos/meilisearch/meilisearch/pulls/${{ github.event.pull_request.number }}/labels
-          echo ${{ github.event.pull_request.number }}
-          echo $URL
-          LABELS=$(gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/labels -q .[].name)
-          echo "Labels: $LABELS"
-          if [[ ! "$LABELS" =~ "db change" && ! "$LABELS" =~ "no db change" ]]; then
-            echo "::error::Pull request must contain either the 'db change' or 'no db change' label."
-            exit 1
-          else
-            echo "The label is set"
-          fi

.github/workflows/dependency-issue.yml

@@ -13,9 +13,9 @@ jobs:
       ISSUE_TEMPLATE: issue-template.md
       GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
     steps:
-    - uses: actions/checkout@v5
+    - uses: actions/checkout@v3
     - name: Download the issue template
-      run: curl -s https://raw.githubusercontent.com/meilisearch/meilisearch/main/.github/templates/dependency-issue.md > $ISSUE_TEMPLATE
+      run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/dependency-issue.md > $ISSUE_TEMPLATE
     - name: Create issue
       run: |
         gh issue create \

.github/workflows/flaky-tests.yml

@@ -1,30 +1,32 @@
 name: Look for flaky tests
-
 on:
   workflow_dispatch:
   schedule:
-    - cron: '0 4 * * *' # Every day at 4:00AM
+    - cron: "0 12 * * FRI" # Every Friday at 12:00PM
 
 jobs:
   flaky:
     runs-on: ubuntu-latest
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
     steps:
-      - uses: actions/checkout@v5
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.89
-      - name: Install cargo-flaky
-        run: cargo install cargo-flaky
-      - name: Run cargo flaky in the dumps
-        run: cd crates/dump; cargo flaky -i 100 --release
-      - name: Run cargo flaky in the index-scheduler
-        run: cd crates/index-scheduler; cargo flaky -i 100 --release
-      - name: Run cargo flaky in the auth
-        run: cd crates/meilisearch-auth; cargo flaky -i 100 --release
-      - name: Run cargo flaky in meilisearch
-        run: cd crates/meilisearch; cargo flaky -i 100 --release
+    - uses: actions/checkout@v3
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Install cargo-flaky
+      run: cargo install cargo-flaky
+    - name: Run cargo flaky in the dumps
+      run: cd dump; cargo flaky -i 100 --release
+    - name: Run cargo flaky in the index-scheduler
+      run: cd index-scheduler; cargo flaky -i 100 --release
+    - name: Run cargo flaky in the auth
+      run: cd meilisearch-auth; cargo flaky -i 100 --release
+    - name: Run cargo flaky in meilisearch
+      run: cd meilisearch; cargo flaky -i 100 --release

.github/workflows/fuzzer-indexing.yml

@@ -11,10 +11,12 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 4320 # 72h
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Run benchmarks
       - name: Run the fuzzer

.github/workflows/latest-git-tag.yml

@@ -10,7 +10,7 @@ jobs:
     name: Check the version validity
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Check release validity
         if: github.event_name == 'release'
         run: bash .github/scripts/check-release.sh
@@ -19,7 +19,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: check-version
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - uses: rickstaa/action-create-tag@v1
         with:
           tag: "latest"

.github/workflows/milestone-workflow.yml

@@ -0,0 +1,154 @@
+name: Milestone's workflow
+
+# /!\ No git flow are handled here
+
+# For each Milestone created (not opened!), and if the release is NOT a patch release (only the patch changed)
+# - the roadmap issue is created, see https://github.com/meilisearch/engine-team/blob/main/issue-templates/roadmap-issue.md
+# - the changelog issue is created, see https://github.com/meilisearch/engine-team/blob/main/issue-templates/changelog-issue.md
+
+# For each Milestone closed
+# - the `release_version` label is created
+# - this label is applied to all issues/PRs in the Milestone
+
+on:
+  milestone:
+    types: [created, closed]
+
+env:
+  MILESTONE_VERSION: ${{ github.event.milestone.title }}
+  MILESTONE_URL: ${{ github.event.milestone.html_url }}
+  MILESTONE_DUE_ON: ${{ github.event.milestone.due_on }}
+  GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
+
+jobs:
+
+# -----------------
+# MILESTONE CREATED
+# -----------------
+
+  get-release-version:
+    if: github.event.action == 'created'
+    runs-on: ubuntu-latest
+    outputs:
+      is-patch: ${{ steps.check-patch.outputs.is-patch }}
+    steps:
+      - uses: actions/checkout@v3
+      - name: Check if this release is a patch release only
+        id: check-patch
+        run: |
+          echo version: $MILESTONE_VERSION
+          if [[ $MILESTONE_VERSION =~ ^v[0-9]+\.[0-9]+\.0$ ]]; then
+            echo 'This is NOT a patch release'
+            echo "is-patch=false" >> $GITHUB_OUTPUT
+          elif [[ $MILESTONE_VERSION =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+            echo 'This is a patch release'
+            echo "is-patch=true" >> $GITHUB_OUTPUT
+          else
+            echo "Not a valid format of release, check the Milestone's title."
+            echo 'Should be vX.Y.Z'
+            exit 1
+          fi
+
+  create-roadmap-issue:
+    needs: get-release-version
+    # Create the roadmap issue if the release is not only a patch release
+    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
+    runs-on: ubuntu-latest
+    env:
+      ISSUE_TEMPLATE: issue-template.md
+    steps:
+      - uses: actions/checkout@v3
+      - name: Download the issue template
+        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/roadmap-issue.md > $ISSUE_TEMPLATE
+      - name: Replace all empty occurrences in the templates
+        run: |
+          # Replace all <<version>> occurrences
+          sed -i "s/<<version>>/$MILESTONE_VERSION/g" $ISSUE_TEMPLATE
+
+          # Replace all <<milestone_id>> occurrences
+          milestone_id=$(echo $MILESTONE_URL | cut -d '/' -f 7)
+          sed -i "s/<<milestone_id>>/$milestone_id/g" $ISSUE_TEMPLATE
+
+          # Replace release date if exists
+          if [[ ! -z $MILESTONE_DUE_ON ]]; then
+            date=$(echo $MILESTONE_DUE_ON | cut -d 'T' -f 1)
+            sed -i "s/Release date\: 20XX-XX-XX/Release date\: $date/g" $ISSUE_TEMPLATE
+          fi
+      - name: Create the issue
+        run: |
+          gh issue create \
+            --title "$MILESTONE_VERSION ROADMAP" \
+            --label 'epic,impacts docs,impacts integrations,impacts cloud' \
+            --body-file $ISSUE_TEMPLATE \
+            --milestone $MILESTONE_VERSION
+
+  create-changelog-issue:
+    needs: get-release-version
+    # Create the changelog issue if the release is not only a patch release
+    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
+    runs-on: ubuntu-latest
+    env:
+      ISSUE_TEMPLATE: issue-template.md
+    steps:
+      - uses: actions/checkout@v3
+      - name: Download the issue template
+        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/changelog-issue.md > $ISSUE_TEMPLATE
+      - name: Replace all empty occurrences in the templates
+        run: |
+          # Replace all <<version>> occurrences
+          sed -i "s/<<version>>/$MILESTONE_VERSION/g" $ISSUE_TEMPLATE
+
+          # Replace all <<milestone_id>> occurrences
+          milestone_id=$(echo $MILESTONE_URL | cut -d '/' -f 7)
+          sed -i "s/<<milestone_id>>/$milestone_id/g" $ISSUE_TEMPLATE
+      - name: Create the issue
+        run: |
+          gh issue create \
+            --title "Create release changelogs for $MILESTONE_VERSION" \
+            --label 'impacts docs,documentation' \
+            --body-file $ISSUE_TEMPLATE \
+            --milestone $MILESTONE_VERSION \
+            --assignee curquiza
+
+# ----------------
+# MILESTONE CLOSED
+# ----------------
+
+  create-release-label:
+    if: github.event.action == 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Create the ${{ env.MILESTONE_VERSION }} label
+        run: |
+          label_description="PRs/issues solved in $MILESTONE_VERSION"
+          if [[ ! -z $MILESTONE_DUE_ON ]]; then
+            date=$(echo $MILESTONE_DUE_ON | cut -d 'T' -f 1)
+            label_description="$label_description released on $date"
+          fi
+
+          gh api repos/meilisearch/meilisearch/labels \
+            --method POST \
+            -H "Accept: application/vnd.github+json" \
+            -f name="$MILESTONE_VERSION" \
+            -f description="$label_description" \
+            -f color='ff5ba3'
+
+  labelize-all-milestone-content:
+    if: github.event.action == 'closed'
+    needs: create-release-label
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Add label ${{ env.MILESTONE_VERSION }} to all PRs in the Milestone
+        run: |
+          prs=$(gh pr list --search milestone:"$MILESTONE_VERSION" --limit 1000 --state all --json number --template '{{range .}}{{tablerow (printf "%v" .number)}}{{end}}')
+          for pr in $prs; do
+              gh pr edit $pr --add-label $MILESTONE_VERSION
+          done
+      - name: Add label ${{ env.MILESTONE_VERSION }} to all issues in the Milestone
+        run: |
+          issues=$(gh issue list --search milestone:"$MILESTONE_VERSION" --limit 1000 --state all --json number --template '{{range .}}{{tablerow (printf "%v" .number)}}{{end}}')
+          for issue in $issues; do
+              gh issue edit $issue --add-label $MILESTONE_VERSION
+          done

.github/workflows/publish-apt-brew-pkg.yml

@@ -9,7 +9,7 @@ jobs:
     name: Check the version validity
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Check release validity
         run: bash .github/scripts/check-release.sh
 
@@ -18,28 +18,31 @@ jobs:
     runs-on: ubuntu-latest
     needs: check-version
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27
+      image: ubuntu:18.04
     steps:
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.89
-      - name: Install cargo-deb
-        run: cargo install cargo-deb
-      - uses: actions/checkout@v5
-      - name: Build deb package
-        run: cargo deb -p meilisearch -o target/debian/meilisearch.deb
-      - name: Upload debian pkg to release
-        uses: svenstaro/upload-release-action@2.11.2
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: target/debian/meilisearch.deb
-          asset_name: meilisearch.deb
-          tag: ${{ github.ref }}
-      - name: Upload debian pkg to apt repository
-        run: curl -F package=@target/debian/meilisearch.deb https://${{ secrets.GEMFURY_PUSH_TOKEN }}@push.fury.io/meilisearch/
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Install cargo-deb
+      run: cargo install cargo-deb
+    - uses: actions/checkout@v3
+    - name: Build deb package
+      run: cargo deb -p meilisearch -o target/debian/meilisearch.deb
+    - name: Upload debian pkg to release
+      uses: svenstaro/upload-release-action@2.7.0
+      with:
+        repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        file: target/debian/meilisearch.deb
+        asset_name: meilisearch.deb
+        tag: ${{ github.ref }}
+    - name: Upload debian pkg to apt repository
+      run: curl -F package=@target/debian/meilisearch.deb https://${{ secrets.GEMFURY_PUSH_TOKEN }}@push.fury.io/meilisearch/
 
   homebrew:
     name: Bump Homebrew formula

.github/workflows/publish-binaries.yml

@@ -1,9 +1,9 @@
-name: Publish assets to GitHub release
+name: Publish binaries to GitHub release
 
 on:
   workflow_dispatch:
   schedule:
-    - cron: "0 2 * * *" # Every day at 2:00am
+    - cron: '0 2 * * *' # Every day at 2:00am
   release:
     types: [published]
 
@@ -11,9 +11,9 @@ jobs:
   check-version:
     name: Check the version validity
     runs-on: ubuntu-latest
-    # No need to check the version for dry run (cron or workflow_dispatch)
+    # No need to check the version for dry run (cron)
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       # Check if the tag has the v<nmumber>.<number>.<number> format.
       # If yes, it means we are publishing an official release.
       # If no, we are releasing a RC, so no need to check the version.
@@ -37,26 +37,29 @@ jobs:
     runs-on: ubuntu-latest
     needs: check-version
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27
+      image: ubuntu:18.04
     steps:
-      - uses: actions/checkout@v5
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.89
-      - name: Build
-        run: cargo build --release --locked
-      # No need to upload binaries for dry run (cron or workflow_dispatch)
-      - name: Upload binaries to release
-        if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.2
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: target/release/meilisearch
-          asset_name: meilisearch-linux-amd64
-          tag: ${{ github.ref }}
+    - uses: actions/checkout@v3
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Build
+      run: cargo build --release --locked
+    # No need to upload binaries for dry run (cron)
+    - name: Upload binaries to release
+      if: github.event_name == 'release'
+      uses: svenstaro/upload-release-action@2.7.0
+      with:
+        repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        file: target/release/meilisearch
+        asset_name: meilisearch-linux-amd64
+        tag: ${{ github.ref }}
 
   publish-macos-windows:
     name: Publish binary for ${{ matrix.os }}
@@ -65,32 +68,35 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-13, windows-2022]
+        os: [macos-12, windows-2022]
         include:
-          - os: macos-13
+          - os: macos-12
             artifact_name: meilisearch
             asset_name: meilisearch-macos-amd64
           - os: windows-2022
             artifact_name: meilisearch.exe
             asset_name: meilisearch-windows-amd64.exe
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
-      - name: Build
-        run: cargo build --release --locked
-      # No need to upload binaries for dry run (cron or workflow_dispatch)
-      - name: Upload binaries to release
-        if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.2
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: target/release/${{ matrix.artifact_name }}
-          asset_name: ${{ matrix.asset_name }}
-          tag: ${{ github.ref }}
+    - uses: actions/checkout@v3
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Build
+      run: cargo build --release --locked
+    # No need to upload binaries for dry run (cron)
+    - name: Upload binaries to release
+      if: github.event_name == 'release'
+      uses: svenstaro/upload-release-action@2.7.0
+      with:
+        repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        file: target/release/${{ matrix.artifact_name }}
+        asset_name: ${{ matrix.asset_name }}
+        tag: ${{ github.ref }}
 
   publish-macos-apple-silicon:
     name: Publish binary for macOS silicon
-    runs-on: macos-13
+    runs-on: macos-12
     needs: check-version
     strategy:
       matrix:
@@ -99,21 +105,23 @@ jobs:
             asset_name: meilisearch-macos-apple-silicon
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v3
       - name: Installing Rust toolchain
-        uses: dtolnay/rust-toolchain@1.89
+        uses: actions-rs/toolchain@v1
         with:
+          toolchain: stable
           profile: minimal
           target: ${{ matrix.target }}
+          override: true
       - name: Cargo build
         uses: actions-rs/cargo@v1
         with:
           command: build
           args: --release --target ${{ matrix.target }}
       - name: Upload the binary to release
-        # No need to upload binaries for dry run (cron or workflow_dispatch)
+        # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.2
+        uses: svenstaro/upload-release-action@2.7.0
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch
@@ -124,11 +132,9 @@ jobs:
     name: Publish binary for aarch64
     runs-on: ubuntu-latest
     needs: check-version
-    env:
-      DEBIAN_FRONTEND: noninteractive
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27
+      image: ubuntu:18.04
     strategy:
       matrix:
         include:
@@ -136,7 +142,7 @@ jobs:
             asset_name: meilisearch-linux-aarch64
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v5
+        uses: actions/checkout@v3
       - name: Install needed dependencies
         run: |
           apt-get update -y && apt upgrade -y
@@ -148,10 +154,12 @@ jobs:
           add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
           apt-get update -y && apt-get install -y docker-ce
       - name: Installing Rust toolchain
-        uses: dtolnay/rust-toolchain@1.89
+        uses: actions-rs/toolchain@v1
         with:
+          toolchain: stable
           profile: minimal
           target: ${{ matrix.target }}
+          override: true
       - name: Configure target aarch64 GNU
         ## Environment variable is not passed using env:
         ## LD gold won't work with MUSL
@@ -162,9 +170,6 @@ jobs:
           echo '[target.aarch64-unknown-linux-gnu]' >> ~/.cargo/config
           echo 'linker = "aarch64-linux-gnu-gcc"' >> ~/.cargo/config
           echo 'JEMALLOC_SYS_WITH_LG_PAGE=16' >> $GITHUB_ENV
-      - name: Install a default toolchain that will be used to build cargo cross
-        run: |
-          rustup default stable
       - name: Cargo build
         uses: actions-rs/cargo@v1
         with:
@@ -176,37 +181,11 @@ jobs:
       - name: List target output files
         run: ls -lR ./target
       - name: Upload the binary to release
-        # No need to upload binaries for dry run (cron or workflow_dispatch)
+        # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.2
+        uses: svenstaro/upload-release-action@2.7.0
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch
           asset_name: ${{ matrix.asset_name }}
           tag: ${{ github.ref }}
-
-  publish-openapi-file:
-    name: Publish OpenAPI file
-    needs: check-version
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v5
-      - name: Setup Rust
-        uses: actions-rs/toolchain@v1
-        with:
-          toolchain: stable
-          override: true
-      - name: Generate OpenAPI file
-        run: |
-          cd crates/openapi-generator
-          cargo run --release -- --pretty --output ../../meilisearch.json
-      - name: Upload OpenAPI to Release
-        # No need to upload for dry run (cron or workflow_dispatch)
-        if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.2
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: ./meilisearch.json
-          asset_name: meilisearch-openapi.json
-          tag: ${{ github.ref }}

.github/workflows/publish-docker-images.yml

@@ -13,95 +13,11 @@ on:
     - cron: '0 23 * * *' # Every day at 11:00pm
   workflow_dispatch:
 
-env:
-  REGISTRY_IMAGE: getmeili/meilisearch
-
 jobs:
-  build:
-    runs-on: ${{ matrix.runner }}
-
-    strategy:
-      matrix:
-        platform: [amd64, arm64]
-        include:
-          - platform: amd64
-            runner: ubuntu-24.04
-          - platform: arm64
-            runner: ubuntu-24.04-arm
-
-    permissions: {}
+  docker:
+    runs-on: docker
     steps:
-      - uses: actions/checkout@v5
-
-      - name: Prepare
-        run: |
-          platform=linux/${{ matrix.platform }}
-          echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-        with:
-          platforms: linux/${{ matrix.platform }}
-          install: true
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-
-      - name: Docker meta
-        id: meta
-        uses: docker/metadata-action@v5
-        with:
-          images: ${{ env.REGISTRY_IMAGE }}
-          # Prevent `latest` to be updated for each new tag pushed.
-          # We need latest and `vX.Y` tags to only be pushed for the stable Meilisearch releases.
-          flavor: latest=false
-          tags: |
-            type=ref,event=tag
-            type=raw,value=nightly,enable=${{ github.event_name != 'push' }}
-            type=semver,pattern=v{{major}}.{{minor}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
-            type=semver,pattern=v{{major}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
-            type=raw,value=latest,enable=${{ steps.check-tag-format.outputs.stable == 'true' && steps.check-tag-format.outputs.latest == 'true' }}
-
-      - name: Build and push by digest
-        uses: docker/build-push-action@v6
-        id: build-and-push
-        with:
-          platforms: linux/${{ matrix.platform }}
-          labels: ${{ steps.meta.outputs.labels }}
-          tags: ${{ env.REGISTRY_IMAGE }}
-          outputs: type=image,push-by-digest=true,name-canonical=true,push=true
-          build-args: |
-            COMMIT_SHA=${{ github.sha }}
-            COMMIT_DATE=${{ steps.build-metadata.outputs.date }}
-            GIT_TAG=${{ github.ref_name }}
-
-      - name: Export digest
-        run: |
-          mkdir -p ${{ runner.temp }}/digests
-          digest="${{ steps.build-and-push.outputs.digest }}"
-          touch "${{ runner.temp }}/digests/${digest#sha256:}"
-
-      - name: Upload digest
-        uses: actions/upload-artifact@v4
-        with:
-          name: digests-${{ env.PLATFORM_PAIR }}
-          path: ${{ runner.temp }}/digests/*
-          if-no-files-found: error
-          retention-days: 1
-
-  merge:
-    runs-on: ubuntu-latest
-    needs:
-      - build
-
-    permissions:
-      id-token: write # This is needed to use Cosign in keyless mode
-
-    steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
 
       # If we are running a cron or manual job ('schedule' or 'workflow_dispatch' event), it means we are publishing the `nightly` tag, so not considered stable.
       # If we have pushed a tag, and the tag has the v<nmumber>.<number>.<number> format, it means we are publishing an official release, so considered stable.
@@ -140,15 +56,11 @@ jobs:
 
           echo "date=$commit_date" >> $GITHUB_OUTPUT
 
-      - name: Install cosign
-        uses: sigstore/cosign-installer@d7543c93d881b35a8faa02e8e3605f69b7a1ce62 # tag=v3.10.0
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
 
-      - name: Download digests
-        uses: actions/download-artifact@v4
-        with:
-          path: ${{ runner.temp }}/digests
-          pattern: digests-*
-          merge-multiple: true
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
 
       - name: Login to Docker Hub
         uses: docker/login-action@v3
@@ -156,14 +68,11 @@ jobs:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-
       - name: Docker meta
         id: meta
         uses: docker/metadata-action@v5
         with:
-          images: ${{ env.REGISTRY_IMAGE }}
+          images: getmeili/meilisearch
           # Prevent `latest` to be updated for each new tag pushed.
           # We need latest and `vX.Y` tags to only be pushed for the stable Meilisearch releases.
           flavor: latest=false
@@ -171,29 +80,18 @@ jobs:
             type=ref,event=tag
             type=raw,value=nightly,enable=${{ github.event_name != 'push' }}
             type=semver,pattern=v{{major}}.{{minor}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
-            type=semver,pattern=v{{major}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
             type=raw,value=latest,enable=${{ steps.check-tag-format.outputs.stable == 'true' && steps.check-tag-format.outputs.latest == 'true' }}
 
-      - name: Create manifest list and push
-        working-directory: ${{ runner.temp }}/digests
-        run: |
-          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
-            $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)
-
-      - name: Inspect image
-        run: |
-          digest=$(docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }})
-          echo "DIGEST=${platform}" >> $GITHUB_ENV
-
-      - name: Sign the images with GitHub OIDC Token
-        env:
-          TAGS: ${{ steps.meta.outputs.tags }}
-        run: |
-          images=""
-          for tag in ${TAGS}; do
-            images+="${tag}@${{ env.DIGEST }} "
-          done
-          cosign sign --yes ${images}
+      - name: Build and push
+        uses: docker/build-push-action@v5
+        with:
+          push: true
+          platforms: linux/amd64,linux/arm64
+          tags: ${{ steps.meta.outputs.tags }}
+          build-args: |
+            COMMIT_SHA=${{ github.sha }}
+            COMMIT_DATE=${{ steps.build-metadata.outputs.date }}
+            GIT_TAG=${{ github.ref_name }}
 
       # /!\ Don't touch this without checking with Cloud team
       - name: Send CI information to Cloud team
@@ -205,22 +103,3 @@ jobs:
           repository: meilisearch/meilisearch-cloud
           event-type: cloud-docker-build
           client-payload: '{ "meilisearch_version": "${{ github.ref_name }}", "stable": "${{ steps.check-tag-format.outputs.stable }}" }'
-
-      # Send notification to Swarmia to notify of a deployment: https://app.swarmia.com
-      # - name: 'Setup jq'
-      #   uses: dcarbone/install-jq-action
-      # - name: Send deployment to Swarmia
-      #   if: github.event_name == 'push' && success()
-      #   run: |
-      #     JSON_STRING=$( jq --null-input --compact-output \
-      #     --arg version "${{ github.ref_name }}" \
-      #     --arg appName "meilisearch" \
-      #     --arg environment "production" \
-      #     --arg commitSha "${{ github.sha }}" \
-      #     --arg repositoryFullName "${{ github.repository }}" \
-      #     '{"version": $version, "appName": $appName, "environment": $environment, "commitSha": $commitSha, "repositoryFullName": $repositoryFullName}' )
-
-      #     curl -H "Authorization: ${{ secrets.SWARMIA_DEPLOYMENTS_AUTHORIZATION }}" \
-      #       -H "Content-Type: application/json" \
-      #       -d "$JSON_STRING" \
-      #       https://hook.swarmia.com/deployments

.github/workflows/sdks-tests.yml

@@ -9,7 +9,7 @@ on:
         required: false
         default: nightly
   schedule:
-    - cron: '0 6 * * *' # Every day at 6:00am
+    - cron: "0 6 * * MON" # Every Monday at 6:00AM
 
 env:
   MEILI_MASTER_KEY: 'masterKey'
@@ -22,7 +22,7 @@ jobs:
     outputs:
       docker-image: ${{ steps.define-image.outputs.docker-image }}
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
       - name: Define the Docker image we need to use
         id: define-image
         run: |
@@ -46,13 +46,13 @@ jobs:
       MEILISEARCH_VERSION: ${{ needs.define-docker-image.outputs.docker-image }}
 
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-dotnet
       - name: Setup .NET Core
-        uses: actions/setup-dotnet@v5
+        uses: actions/setup-dotnet@v4
         with:
-          dotnet-version: "8.0.x"
+          dotnet-version: "6.0.x"
       - name: Install dependencies
         run: dotnet restore
       - name: Build
@@ -75,7 +75,7 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-dart
       - uses: dart-lang/setup-dart@v1
@@ -100,10 +100,10 @@ jobs:
           - '7700:7700'
     steps:
       - name: Set up Go
-        uses: actions/setup-go@v6
+        uses: actions/setup-go@v5
         with:
           go-version: stable
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-go
       - name: Get dependencies
@@ -114,7 +114,7 @@ jobs:
             dep ensure
           fi
       - name: Run integration tests
-        run: go test --race -v ./integration
+        run: go test -v ./...
 
   meilisearch-java-tests:
     needs: define-docker-image
@@ -129,19 +129,19 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-java
       - name: Set up Java
-        uses: actions/setup-java@v5
+        uses: actions/setup-java@v4
         with:
-          java-version: 17
-          distribution: 'temurin'
+          java-version: 8
+          distribution: 'zulu'
           cache: gradle
       - name: Grant execute permission for gradlew
         run: chmod +x gradlew
       - name: Build and run unit and integration tests
-        run: ./gradlew build integrationTest --info
+        run: ./gradlew build integrationTest
 
   meilisearch-js-tests:
     needs: define-docker-image
@@ -156,11 +156,11 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-js
       - name: Setup node
-        uses: actions/setup-node@v5
+        uses: actions/setup-node@v4
         with:
           cache: 'yarn'
       - name: Install dependencies
@@ -191,7 +191,7 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-php
       - name: Install PHP
@@ -220,11 +220,11 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-python
       - name: Set up Python
-        uses: actions/setup-python@v6
+        uses: actions/setup-python@v5
       - name: Install pipenv
         uses: dschep/install-pipenv-action@v1
       - name: Install dependencies
@@ -245,7 +245,7 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-ruby
       - name: Set up Ruby 3
@@ -270,7 +270,7 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-rust
       - name: Build
@@ -291,7 +291,7 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-swift
       - name: Run tests
@@ -314,11 +314,11 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-js-plugins
       - name: Setup node
-        uses: actions/setup-node@v5
+        uses: actions/setup-node@v4
         with:
           cache: yarn
       - name: Install dependencies
@@ -344,23 +344,15 @@ jobs:
           MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
         ports:
           - '7700:7700'
-    env:
-      RAILS_VERSION: '7.0'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-rails
-      - name: Install SQLite dependencies
-        run: sudo apt-get update && sudo apt-get install -y libsqlite3-dev
-      - name: Set up Ruby
+      - name: Set up Ruby 3
         uses: ruby/setup-ruby@v1
         with:
           ruby-version: 3
           bundler-cache: true
-      - name: Start MongoDB
-        uses: supercharge/mongodb-github-action@1.12.0
-        with:
-          mongodb-version: 8.0
       - name: Run tests
         run: bundle exec rspec
 
@@ -377,7 +369,7 @@ jobs:
         ports:
           - '7700:7700'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v4
         with:
           repository: meilisearch/meilisearch-symfony
       - name: Install PHP

.github/workflows/test-suite.yml

@@ -3,10 +3,14 @@ name: Test suite
 on:
   workflow_dispatch:
   schedule:
-    # Every day at 5:00am
-    - cron: "0 5 * * *"
+    # Everyday at 5:00am
+    - cron: '0 5 * * *'
   pull_request:
-  merge_group:
+  push:
+    # trying and staging branches are for Bors config
+    branches:
+      - trying
+      - staging
 
 env:
   CARGO_TERM_COLOR: always
@@ -15,21 +19,24 @@ env:
 
 jobs:
   test-linux:
-    name: Tests on ubuntu-22.04
+    name: Tests on ubuntu-18.04
     runs-on: ubuntu-latest
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Install needed dependencies
         run: |
           apt-get update && apt-get install -y curl
           apt-get install build-essential -y
       - name: Setup test with Rust stable
-        uses: dtolnay/rust-toolchain@1.89
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
       - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.8.0
+        uses: Swatinem/rust-cache@v2.7.1
       - name: Run cargo check without any default features
         uses: actions-rs/cargo@v1
         with:
@@ -47,12 +54,15 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-13, windows-2022]
+        os: [macos-12, windows-2022]
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.8.0
-      - uses: dtolnay/rust-toolchain@1.89
+        uses: Swatinem/rust-cache@v2.7.1
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
       - name: Run cargo check without any default features
         uses: actions-rs/cargo@v1
         with:
@@ -68,71 +78,45 @@ jobs:
     name: Tests almost all features
     runs-on: ubuntu-latest
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
     if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Install needed dependencies
         run: |
           apt-get update
           apt-get install --assume-yes build-essential curl
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
       - name: Run cargo build with almost all features
         run: |
-          cargo build --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
+          cargo build --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda)"
       - name: Run cargo test with almost all features
         run: |
-          cargo test --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
-
-  ollama-ubuntu:
-    name: Test with Ollama
-    runs-on: ubuntu-latest
-    env:
-      MEILI_TEST_OLLAMA_SERVER: "http://localhost:11434"
-    steps:
-      - uses: actions/checkout@v5
-      - name: Install Ollama
-        run: |
-          curl -fsSL https://ollama.com/install.sh | sudo -E sh
-      - name: Start serving
-        run: |
-          # Run it in the background, there is no way to daemonise at the moment
-          ollama serve &
-
-          # A short pause is required before the HTTP port is opened
-          sleep 5
-
-          # This endpoint blocks until ready
-          time curl -i http://localhost:11434
-
-      - name: Pull nomic-embed-text & all-minilm
-        run: |
-          ollama pull nomic-embed-text
-          ollama pull all-minilm
-
-      - name: Run cargo test
-        uses: actions-rs/cargo@v1
-        with:
-          command: test
-          args: --locked --release --all --features test-ollama ollama
+          cargo test --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda)"
 
   test-disabled-tokenization:
     name: Test disabled tokenization
     runs-on: ubuntu-latest
     container:
-      image: ubuntu:22.04
+      image: ubuntu:18.04
     if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Install needed dependencies
         run: |
           apt-get update
           apt-get install --assume-yes build-essential curl
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
       - name: Run cargo tree without default features and check lindera is not present
         run: |
-          if cargo tree -f '{p} {f}' -e normal --no-default-features | grep -qz lindera; then
+          if cargo tree -f '{p} {f}' -e normal --no-default-features | grep -vqz lindera; then
             echo "lindera has been found in the sources and it shouldn't"
             exit 1
           fi
@@ -145,17 +129,20 @@ jobs:
     name: Run tests in debug
     runs-on: ubuntu-latest
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v3
       - name: Install needed dependencies
         run: |
           apt-get update && apt-get install -y curl
           apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
       - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.8.0
+        uses: Swatinem/rust-cache@v2.7.1
       - name: Run tests in debug
         uses: actions-rs/cargo@v1
         with:
@@ -166,13 +153,15 @@ jobs:
     name: Run Clippy
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: 1.75.0
+          override: true
           components: clippy
       - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.8.0
+        uses: Swatinem/rust-cache@v2.7.1
       - name: Run cargo clippy
         uses: actions-rs/cargo@v1
         with:
@@ -183,19 +172,19 @@ jobs:
     name: Run Rustfmt
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
-          toolchain: nightly-2024-07-09
+          toolchain: nightly
           override: true
           components: rustfmt
       - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.8.0
+        uses: Swatinem/rust-cache@v2.7.1
       - name: Run cargo fmt
         # Since we never ran the `build.rs` script in the benchmark directory we are missing one auto-generated import file.
         # Since we want to trigger (and fail) this action as fast as possible, instead of building the benchmark crate
         # we are going to create an empty file where rustfmt expects it.
         run: |
-          echo -ne "\n" > crates/benchmarks/benches/datasets_paths.rs
+          echo -ne "\n" > benchmarks/benches/datasets_paths.rs
           cargo fmt --all -- --check

.github/workflows/update-cargo-toml-version.yml

@@ -4,7 +4,7 @@ on:
   workflow_dispatch:
     inputs:
       new_version:
-        description: "The new version (vX.Y.Z)"
+        description: 'The new version (vX.Y.Z)'
         required: true
 
 env:
@@ -17,10 +17,12 @@ jobs:
     name: Update version in Cargo.toml
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v5
-      - uses: dtolnay/rust-toolchain@1.89
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
       - name: Install sd
         run: cargo install sd
       - name: Update Cargo.toml file
@@ -41,4 +43,5 @@ jobs:
             --title "Update version for the next release ($NEW_VERSION) in Cargo.toml" \
             --body '⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.' \
             --label 'skip changelog' \
+            --milestone $NEW_VERSION \
             --base $GITHUB_REF_NAME

.gitignore

@@ -5,27 +5,16 @@
 **/*.json_lines
 **/*.rs.bk
 /*.mdb
-/*.ms
+/query-history.txt
+/data.ms
 /snapshots
 /dumps
-/bench
-/_xtask_benchmark.ms
-/benchmarks
-.DS_Store
 
 # Snapshots
 ## ... large
 *.full.snap
-## ... unreviewed
+##  ... unreviewed
 *.snap.new
-## ... pending
-*.pending-snap
-
-# Tmp files
-.tmp*
-
-# Database snapshot
-crates/meilisearch/db.snapshot
 
 # Fuzzcheck data for the facet indexing fuzz test
-crates/milli/fuzz/update::facet::incremental::fuzz::fuzz/
+milli/fuzz/update::facet::incremental::fuzz::fuzz/

BENCHMARKS.md

@@ -1,392 +0,0 @@
-# Benchmarks
-
-Currently this repository hosts two kinds of benchmarks:
-
-1. The older "milli benchmarks", that use [criterion](https://github.com/bheisler/criterion.rs) and live in the "benchmarks" directory.
-2. The newer "bench" that are workload-based and so split between the [`workloads`](./workloads/) directory and the [`xtask::bench`](./xtask/src/bench/) module.
-
-This document describes the newer "bench" benchmarks. For more details on the "milli benchmarks", see [benchmarks/README.md](./benchmarks/README.md).
-
-## Design philosophy for the benchmarks
-
-The newer "bench" benchmarks are **integration** benchmarks, in the sense that they spawn an actual Meilisearch server and measure its performance end-to-end, including HTTP request overhead.
-
-Since this is prone to fluctuating, the benchmarks regain a bit of precision by measuring the runtime of the individual spans using the [logging machinery](./CONTRIBUTING.md#logging) of Meilisearch.
-
-A span roughly translates to a function call. The benchmark runner collects all the spans by name using the [logs route](https://github.com/orgs/meilisearch/discussions/721) and sums their runtime. The processed results are then sent to the [benchmark dashboard](https://bench.meilisearch.dev), which is in charge of storing and presenting the data.
-
-## Running the benchmarks
-
-Benchmarks can run locally or in CI.
-
-### Locally
-
-#### With a local benchmark dashboard
-
-The benchmarks dashboard lives in its [own repository](https://github.com/meilisearch/benchboard). We provide binaries for Ubuntu/Debian, but you can build from source for other platforms (MacOS should work as it was developed under that platform).
-
-Run the `benchboard` binary to create a fresh database of results. By default it will serve the results and the API to gather results on `http://localhost:9001`.
-
-From the Meilisearch repository, you can then run benchmarks with:
-
-```sh
-cargo xtask bench -- workloads/my_workload_1.json ..
-```
-
-This command will build and run Meilisearch locally on port 7700, so make sure that this port is available.
-To run benchmarks on a different commit, just use the usual git command to get back to the desired commit.
-
-#### Without a local benchmark dashboard
-
-To work with the raw results, you can also skip using a local benchmark dashboard.
-
-Run:
-
-```sh
-cargo xtask bench --no-dashboard -- workloads/my_workload_1.json workloads/my_workload_2.json ..
-```
-
-For processing the results, look at [Looking at benchmark results/Without dashboard](#without-dashboard).
-
-#### Sending a workload by hand
-
-Sometimes you want to visualize the metrics of a worlkoad that comes from a custom report.
-It is not quite easy to trick the benchboard in thinking that your report is legitimate but here are the commands you can run to upload your firefox report on a running benchboard.
-
-```bash
-# Name this hostname whatever you want
-echo '{ "hostname": "the-best-place" }' | xh PUT 'http://127.0.0.1:9001/api/v1/machine'
-
-# You'll receive an UUID from this command that we will call $invocation_uuid
-echo '{ "commit": { "sha1": "1234567", "commit_date": "2024-09-05 12:00:12.0 +00:00:00", "message": "A cool message" }, "machine_hostname": "the-best-place", "max_workloads": 1 }' | xh PUT 'http://127.0.0.1:9001/api/v1/invocation'
-
-# Just use UUID from the previous command
-# and you'll receive another UUID that we will call $workload_uuid
-echo '{ "invocation_uuid": "$invocation_uuid", "name": "toto", "max_runs": 1 }' | xh PUT 'http://127.0.0.1:9001/api/v1/workload'
-
-# And now use your $workload_uuid and the content of your firefox report
-# but don't forget to convert your firefox report from JSONLines into an object
-echo '{ "workload_uuid": "$workload_uuid", "data": $REPORT_JSON_DATA }' | xh PUT 'http://127.0.0.1:9001/api/v1/run'
-```
-
-### In CI
-
-We have dedicated runners to run workloads on CI. Currently, there are three ways of running the CI:
-
-1. Automatically, on every push to `main`.
-2. Manually, by clicking the [`Run workflow`](https://github.com/meilisearch/meilisearch/actions/workflows/bench-manual.yml) button and specifying the target reference (tag, commit or branch) as well as one or multiple workloads to run. The workloads must exist in the Meilisearch repository (conventionally, in the [`workloads`](./workloads/) directory) on the target reference. Globbing (e.g., `workloads/*.json`) works.
-3. Manually on a PR, by posting a comment containing a `/bench` command, followed by one or multiple workloads to run. Globbing works. The workloads must exist in the Meilisearch repository in the branch of the PR.
-  ```
-  /bench workloads/movies*.json /hackernews_1M.json
-  ```
-
-## Looking at benchmark results
-
-### On the dashboard
-
-Results are available on the global dashboard used by CI at <https://bench.meilisearch.dev> or on your [local dashboard](#with-a-local-benchmark-dashboard).
-
-The dashboard homepage presents three sections:
-
-1. The latest invocations (a call to `cargo xtask bench`, either local or by CI) with their reason (generally set to some helpful link in CI) and their status.
-2. The latest workloads ran on `main`.
-3. The latest workloads ran on other references.
-
-By default, the workload shows the total runtime delta with the latest applicable commit on `main`. The latest applicable commit is the latest commit for workload invocations that do not originate on `main`, and the latest previous commit for workload invocations that originate on `main`.
-
-You can explicitly request a detailed comparison by span with the `main` branch, the branch or origin, or any previous commit, by clicking the links at the bottom of the workload invocation.
-
-In the detailed comparison view, the spans are sorted by improvements, regressions, stable (no statistically significant change) and unstable (the span runtime is comparable to its standard deviation).
-
-You can click on the name of any span to get a box plot comparing the target commit with multiple commits of the selected branch.
-
-### Without dashboard
-
-After the workloads are done running, the reports will live in the Meilisearch repository, in the `bench/reports` directory (by default).
-
-You can then convert these reports into other formats.
-
-- To [Firefox profiler](https://profiler.firefox.com) format. Run:
-  ```sh
-  cd bench/reports
-  cargo run --release --bin trace-to-firefox -- my_workload_1-0-trace.json
-  ```
-  You can then upload the resulting `firefox-my_workload_1-0-trace.json` file to the online profiler.
-
-
-## Designing benchmark workloads
-
-Benchmark workloads conventionally live in the `workloads` directory of the Meilisearch repository.
-
-They are JSON files with the following structure (comments are not actually supported, to make your own, remove them or copy some existing workload file):
-
-```jsonc
-{
-  // Name of the workload. Must be unique to the workload, as it will be used to group results on the dashboard.
-  "name": "hackernews.ndjson_1M,no-threads",
-  // Number of consecutive runs of the commands that should be performed.
-  // Each run uses a fresh instance of Meilisearch and a fresh database.
-  // Each run produces its own report file.
-  "run_count": 3,
-  // List of arguments to add to the Meilisearch command line.
-  "extra_cli_args": ["--max-indexing-threads=1"],
-  // An expression that can be parsed as a comma-separated list of targets and levels
-  // as described in [tracing_subscriber's documentation](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/targets/struct.Targets.html#examples).
-  // The expression is used to filter the spans that are measured for profiling purposes.
-  // Optional, defaults to "indexing::=trace" (for indexing workloads), common other values is
-  // "search::=trace"
-  "target": "indexing::=trace",
-  // List of named assets that can be used in the commands.
-  "assets": {
-    // name of the asset.
-    // Must be unique at the workload level.
-    // For better results, the same asset (same sha256) should have the same name accross workloads.
-    // Having multiple assets with the same name and distinct hashes is supported accross workloads,
-    // but will lead to superfluous downloads.
-    //
-    // Assets are stored in the `bench/assets/` directory by default.
-    "hackernews-100_000.ndjson": {
-      // If the assets exists in the local filesystem (Meilisearch repository or for your local workloads)
-      // Its file path can be specified here.
-      // `null` if the asset should be downloaded from a remote location.
-      "local_location": null,
-      // URL of the remote location where the asset can be downloaded.
-      // Use the `--assets-key` of the runner to pass an API key in the `Authorization: Bearer` header of the download requests.
-      // `null` if the asset should be imported from a local location.
-      // if both local and remote locations are specified, then the local one is tried first, then the remote one
-      // if the file is locally missing or its hash differs.
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-100_000.ndjson",
-      // SHA256 of the asset.
-      // Optional, the `sha256` of the asset will be displayed during a run of the workload if it is missing.
-      // If present, the hash of the asset in the `bench/assets/` directory will be compared against this hash before
-      // running the workload. If the hashes differ, the asset will be downloaded anew.
-      "sha256": "60ecd23485d560edbd90d9ca31f0e6dba1455422f2a44e402600fbb5f7f1b213",
-      // Optional, one of "Auto", "Json", "NdJson" or "Raw".
-      // If missing, assumed to be "Auto".
-      // If "Auto", the format will be determined from the extension in the asset name.
-      "format": "NdJson"
-    },
-    "hackernews-200_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-200_000.ndjson",
-      "sha256": "785b0271fdb47cba574fab617d5d332276b835c05dd86e4a95251cf7892a1685"
-    },
-    "hackernews-300_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-300_000.ndjson",
-      "sha256": "de73c7154652eddfaf69cdc3b2f824d5c452f095f40a20a1c97bb1b5c4d80ab2"
-    },
-    "hackernews-400_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-400_000.ndjson",
-      "sha256": "c1b00a24689110f366447e434c201c086d6f456d54ed1c4995894102794d8fe7"
-    },
-    "hackernews-500_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-500_000.ndjson",
-      "sha256": "ae98f9dbef8193d750e3e2dbb6a91648941a1edca5f6e82c143e7996f4840083"
-    },
-    "hackernews-600_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-600_000.ndjson",
-      "sha256": "b495fdc72c4a944801f786400f22076ab99186bee9699f67cbab2f21f5b74dbe"
-    },
-    "hackernews-700_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-700_000.ndjson",
-      "sha256": "4b2c63974f3dabaa4954e3d4598b48324d03c522321ac05b0d583f36cb78a28b"
-    },
-    "hackernews-800_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-800_000.ndjson",
-      "sha256": "cb7b6afe0e6caa1be111be256821bc63b0771b2a0e1fad95af7aaeeffd7ba546"
-    },
-    "hackernews-900_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-900_000.ndjson",
-      "sha256": "e1154ddcd398f1c867758a93db5bcb21a07b9e55530c188a2917fdef332d3ba9"
-    },
-    "hackernews-1_000_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-1_000_000.ndjson",
-      "sha256": "27e25efd0b68b159b8b21350d9af76938710cb29ce0393fa71b41c4f3c630ffe"
-    }
-  },
-  // Core of the workload.
-  // A list of commands to run sequentially.
-  // Optional: A precommand is a request to the Meilisearch instance that is executed before the profiling runs.
-  "precommands": [
-    {
-      // Meilisearch route to call. `http://localhost:7700/` will be prepended.
-      "route": "indexes/movies/settings",
-      // HTTP method to call.
-      "method": "PATCH",
-      // If applicable, body of the request.
-      // Optional, if missing, the body will be empty.
-      "body": {
-        // One of "empty", "inline" or "asset".
-        // If using "empty", you can skip the entire "body" key.
-        "inline": {
-          // when "inline" is used, the body is the JSON object that is the value of the `"inline"` key.
-          "displayedAttributes": [
-            "title",
-            "by",
-            "score",
-            "time"
-          ],
-          "searchableAttributes": [
-            "title"
-          ],
-          "filterableAttributes": [
-            "by"
-          ],
-          "sortableAttributes": [
-            "score",
-            "time"
-          ]
-        }
-      },
-      // Whether to wait before running the next request.
-      // One of:
-      // - DontWait: run the next command without waiting the response to this one.
-      // - WaitForResponse: run the next command as soon as the response from the server is received.
-      // - WaitForTask: run the next command once **all** the Meilisearch tasks created up to now have finished processing.
-      "synchronous": "WaitForTask"
-    }
-  ],
-  // A command is a request to the Meilisearch instance that is executed while the profiling runs.
-  "commands": [
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        // When using "asset", use the name of an asset as value to use the content of that asset as body.
-        // the content type is derived of the format of the asset:
-        // "NdJson" => "application/x-ndjson"
-        // "Json" => "application/json"
-        // "Raw" => "application/octet-stream"
-        // See [AssetFormat::to_content_type](https://github.com/meilisearch/meilisearch/blob/7b670a4afadb132ac4a01b6403108700501a391d/xtask/src/bench/assets.rs#L30)
-        // for details and up-to-date list.
-        "asset": "hackernews-100_000.ndjson"
-      },
-      "synchronous": "WaitForTask"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-200_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-300_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-400_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-500_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-600_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-700_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-800_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-900_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-1_000_000.ndjson"
-      },
-      "synchronous": "WaitForTask"
-    }
-  ]
-}
-```
-
-### Adding new assets
-
-Assets reside in our DigitalOcean S3 space. Assuming you have team access to the DigitalOcean S3 space:
-
-1. go to <https://cloud.digitalocean.com/spaces/milli-benchmarks?i=d1c552&path=bench%2Fdatasets%2F>
-2. upload your dataset:
-   1. if your dataset is a single file, upload that single file using the "upload" button,
-   2. otherwise, create a folder using the "create folder" button, then inside that folder upload your individual files.
-
-## Upgrading `https://bench.meilisearch.dev`
-
-The URL of the server is in our password manager (look for "benchboard").
-
-1. Make the needed modifications on the [benchboard repository](https://github.com/meilisearch/benchboard) and merge them to main.
-2. Publish a new release to produce the Ubuntu/Debian binary.
-3. Download the binary locally, send it to the server:
-  ```
-  scp -6 ~/Downloads/benchboard root@\[<ipv6-address>\]:/bench/new-benchboard
-  ```
-  Note that the ipv6 must be between escaped square brackets for SCP.
-4. SSH to the server:
-  ```
-  ssh root@<ipv6-address>
-  ```
-  Note the ipv6 must **NOT** be between escaped square brackets for SSH 🥲
-5. On the server, set the correct permissions for the new binary:
-   ```
-   chown bench:bench /bench/new-benchboard
-   chmod 700 /bench/new-benchboard
-   ```
-6. On the server, move the new binary to the location of the running binary (if unsure, start by making a backup of the running binary):
-  ```
-  mv /bench/{new-,}benchboard
-  ```
-7. Restart the benchboard service.
-  ```
-  systemctl restart benchboard
-  ```
-8. Check that the service runs correctly.
-  ```
-  systemctl status benchboard
-  ```
-9. Check the availability of the service by going to <https://bench.meilisearch.dev> on your browser.

CONTRIBUTING.md

@@ -4,7 +4,7 @@ First, thank you for contributing to Meilisearch! The goal of this document is t
 
 Remember that there are many ways to contribute other than writing code: writing [tutorials or blog posts](https://github.com/meilisearch/awesome-meilisearch), improving [the documentation](https://github.com/meilisearch/documentation), submitting [bug reports](https://github.com/meilisearch/meilisearch/issues/new?assignees=&labels=&template=bug_report.md&title=) and [feature requests](https://github.com/meilisearch/product/discussions/categories/feedback-feature-proposal)...
 
-Meilisearch can manage multiple indexes, handle the update store, and expose an HTTP API. Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/meilisearch/tree/main/milli), while tokenization is handled by [our `charabia` library](https://github.com/meilisearch/charabia/).
+The code in this repository is only concerned with managing multiple indexes, handling the update store, and exposing an HTTP API. Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/milli), while tokenization is handled by [our `charabia` library](https://github.com/meilisearch/charabia/).
 
 If Meilisearch does not offer optimized support for your language, please consider contributing to `charabia` by following the [CONTRIBUTING.md file](https://github.com/meilisearch/charabia/blob/main/CONTRIBUTING.md) and integrating your intended normalizer/segmenter.
 
@@ -52,28 +52,6 @@ cargo test
 
 This command will be triggered to each PR as a requirement for merging it.
 
-#### Faster build
-
-You can set the `LINDERA_CACHE` environment variable to speed up your successive builds by up to 2 minutes.
-It'll store some built artifacts in the directory of your choice.
-
-We recommend using the `$HOME/.cache/meili/lindera` directory:
-```sh
-export LINDERA_CACHE=$HOME/.cache/meili/lindera
-```
-
-You can set the `MILLI_BENCH_DATASETS_PATH` environment variable to further speed up your builds.
-It'll store some big files used for the benchmarks in the directory of your choice.
-
-We recommend using the `$HOME/.cache/meili/benches` directory:
-```sh
-export MILLI_BENCH_DATASETS_PATH=$HOME/.cache/meili/benches
-```
-
-Furthermore, you can improve incremental compilation by setting the `MEILI_NO_VERGEN` environment variable.
-Setting this variable will prevent the Meilisearch binary from being rebuilt each time the directory that hosts the Meilisearch repository changes.
-Do not enable this environment variable for production builds (as it will break the `version` route, among other things).
-
 #### Snapshot-based tests
 
 We are using [insta](https://insta.rs) to perform snapshot-based testing.
@@ -85,7 +63,7 @@ Furthermore, we provide some macros on top of insta, notably a way to use snapsh
 
 To effectively debug snapshot-based hashes, we recommend you export the `MEILI_TEST_FULL_SNAPS` environment variable so that snapshot are fully created locally:
 
-```sh
+```
 export MEILI_TEST_FULL_SNAPS=true # add this to your .bashrc, .zshrc, ...
 ```
 
@@ -103,47 +81,6 @@ Meilisearch follows the [cargo xtask](https://github.com/matklad/cargo-xtask) wo
 
 Run `cargo xtask --help` from the root of the repository to find out what is available.
 
-#### Update the openAPI file if the API changed
-
-To update the openAPI file in the code, see [sprint_issue.md](https://github.com/meilisearch/meilisearch/blob/main/.github/ISSUE_TEMPLATE/sprint_issue.md#reminders-when-modifying-the-api).
-
-If you want to generate OpenAPI file manually:
-
-With swagger:
-- Starts Meilisearch with the `swagger` feature flag: `cargo run --features swagger`
-- On a browser, open the following URL: http://localhost:7700/scalar
-- Click the « Download openAPI file »
-
-With the internal crate:
-```bash
-cd crates/openapi-generator
-cargo run --release -- --pretty --output meilisearch.json
-```
-
-### Logging
-
-Meilisearch uses [`tracing`](https://lib.rs/crates/tracing) for logging purposes. Tracing logs are structured and can be displayed as JSON to the end user, so prefer passing arguments as fields rather than interpolating them in the message.
-
-Refer to the [documentation](https://docs.rs/tracing/0.1.40/tracing/index.html#using-the-macros) for the syntax of the spans and events.
-
-Logging spans are used for 3 distinct purposes:
-
-1. Regular logging
-2. Profiling
-3. Benchmarking
-
-As a result, the spans should follow some rules:
-
-- They should not be put on functions that are called too often. That is because opening and closing a span causes some overhead. For regular logging, avoid putting spans on functions that are taking less than a few hundred nanoseconds. For profiling or benchmarking, avoid putting spans on functions that are taking less than a few microseconds.
-- For profiling and benchmarking, use the `TRACE` level.
-- For profiling and benchmarking, use the following `target` prefixes:
-  - `indexing::` for spans meant when profiling the indexing operations.
-  - `search::` for spans meant when profiling the search operations.
-
-### Benchmarking
-
-See [BENCHMARKS.md](./BENCHMARKS.md)
-
 ## Git Guidelines
 
 ### Git Branches
@@ -170,39 +107,28 @@ Some notes on GitHub PRs:
 - The PR title should be accurate and descriptive of the changes.
 - [Convert your PR as a draft](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/changing-the-stage-of-a-pull-request) if your changes are a work in progress: no one will review it until you pass your PR as ready for review.<br>
   The draft PRs are recommended when you want to show that you are working on something and make your work visible.
-- The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [GitHub Merge Queues](https://github.blog/news-insights/product-news/github-merge-queue-is-generally-available/) to automatically enforce this requirement without the PR author having to rebase manually.
+- The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [Bors](https://github.com/bors-ng/bors-ng) to automatically enforce this requirement without the PR author having to rebase manually.
 
-## Merging PRs
-
-This project uses GitHub Merge Queues that helps us manage pull requests merging.
-
-Before merging a PR, the maintainer should ensure the following requirements are met
-- Automated tests have been added.
-- If some tests cannot be automated, manual rigorous tests should be applied.
-- ⚠️ If there is an change in the DB: it's mandatory to manually test the `--experimental-dumpless-upgrade` on a DB of the previous Meilisearch minor version (e.g. v1.13 for the v1.14 release).
-- If necessary, the feature have been tested in the Cloud production environment (with [prototypes](./documentation/prototypes.md)) and the Cloud UI is ready.
-- If necessary, the [documentation](https://github.com/meilisearch/documentation) related to the implemented feature in the PR is ready.
-- If necessary, the [integrations](https://github.com/meilisearch/integration-guides) related to the implemented feature in the PR are ready.
-
-## Publish Process (for internal team only)
+## Release Process (for internal team only)
 
 Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org/).
 
-### How to publish a new release
+### Automation to rebase and Merge the PRs
 
-The full Meilisearch release process is described in [this guide](./documentation/release.md).
+This project integrates a bot that helps us manage pull requests merging.<br>
+_[Read more about this](https://github.com/meilisearch/integration-guides/blob/main/resources/bors.md)._
+
+### How to Publish a new Release
+
+The full Meilisearch release process is described in [this guide](https://github.com/meilisearch/engine-team/blob/main/resources/meilisearch-release.md). Please follow it carefully before doing any release.
 
 ### How to publish a prototype
 
 Depending on the developed feature, you might need to provide a prototyped version of Meilisearch to make it easier to test by the users.
 
 This happens in two steps:
-- [Release the prototype](./documentation/prototypes.md#how-to-publish-a-prototype)
-- [Communicate about it](./documentation/prototypes.md#communication)
-
-### How to implement and publish an experimental feature
-
-Here is our [guidelines and process](./documentation/experimental-features.md) to implement and publish an experimental feature.
+- [Release the prototype](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#how-to-publish-a-prototype)
+- [Communicate about it](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#communication)
 
 ### Release assets
 

Cargo.toml

@@ -1,29 +1,27 @@
 [workspace]
 resolver = "2"
 members = [
-    "crates/meilisearch",
-    "crates/meilitool",
-    "crates/meilisearch-types",
-    "crates/meilisearch-auth",
-    "crates/meili-snap",
-    "crates/index-scheduler",
-    "crates/dump",
-    "crates/file-store",
-    "crates/permissive-json-pointer",
-    "crates/milli",
-    "crates/filter-parser",
-    "crates/flatten-serde-json",
-    "crates/json-depth-checker",
-    "crates/benchmarks",
-    "crates/fuzzers",
-    "crates/tracing-trace",
-    "crates/xtask",
-    "crates/build-info",
-    "crates/openapi-generator",
+    "meilisearch",
+    "meilitool",
+    "meilisearch-types",
+    "meilisearch-auth",
+    "meili-snap",
+    "index-scheduler",
+    "dump",
+    "file-store",
+    "permissive-json-pointer",
+    "milli",
+    "filter-parser",
+    "flatten-serde-json",
+    "json-depth-checker",
+    "benchmarks",
+    "fuzzers",
+    "tracing-trace",
+    "xtask",
 ]
 
 [workspace.package]
-version = "1.26.0"
+version = "1.7.0"
 authors = [
     "Quentin de Quelen <quentin@dequelen.me>",
     "Clément Renault <clement@meilisearch.com>",
@@ -37,12 +35,6 @@ license = "MIT"
 [profile.release]
 codegen-units = 1
 
-# We now compile heed without the NDEBUG define for better performance.
-# However, we still enable debug assertions for a better detection of
-# disk corruption on the cloud or in OSS.
-[profile.release.package.heed]
-debug-assertions = true
-
 [profile.dev.package.flate2]
 opt-level = 3
 
@@ -50,3 +42,24 @@ opt-level = 3
 opt-level = 3
 [profile.dev.package.roaring]
 opt-level = 3
+
+[profile.dev.package.lindera-ipadic-builder]
+opt-level = 3
+[profile.dev.package.encoding]
+opt-level = 3
+[profile.dev.package.yada]
+opt-level = 3
+
+[profile.release.package.lindera-ipadic-builder]
+opt-level = 3
+[profile.release.package.encoding]
+opt-level = 3
+[profile.release.package.yada]
+opt-level = 3
+
+[profile.bench.package.lindera-ipadic-builder]
+opt-level = 3
+[profile.bench.package.encoding]
+opt-level = 3
+[profile.bench.package.yada]
+opt-level = 3

Dockerfile

@@ -1,14 +1,14 @@
 # Compile
-FROM    rust:1.89-alpine3.22 AS compiler
+FROM    rust:1.75.0-alpine3.18 AS compiler
 
-RUN     apk add -q --no-cache build-base openssl-dev
+RUN     apk add -q --update-cache --no-cache build-base openssl-dev
 
 WORKDIR /
 
 ARG     COMMIT_SHA
 ARG     COMMIT_DATE
 ARG     GIT_TAG
-ENV     VERGEN_GIT_SHA=${COMMIT_SHA} VERGEN_GIT_COMMIT_TIMESTAMP=${COMMIT_DATE} VERGEN_GIT_DESCRIBE=${GIT_TAG}
+ENV     VERGEN_GIT_SHA=${COMMIT_SHA} VERGEN_GIT_COMMIT_TIMESTAMP=${COMMIT_DATE} VERGEN_GIT_SEMVER_LIGHTWEIGHT=${GIT_TAG}
 ENV     RUSTFLAGS="-C target-feature=-crt-static"
 
 COPY    . .
@@ -20,13 +20,13 @@ RUN     set -eux; \
         cargo build --release -p meilisearch -p meilitool
 
 # Run
-FROM    alpine:3.22
-LABEL   org.opencontainers.image.source="https://github.com/meilisearch/meilisearch"
+FROM    alpine:3.16
 
 ENV     MEILI_HTTP_ADDR 0.0.0.0:7700
 ENV     MEILI_SERVER_PROVIDER docker
 
-RUN     apk add -q --no-cache libgcc tini curl
+RUN     apk update --quiet \
+        && apk add -q --no-cache libgcc tini curl
 
 # add meilisearch and meilitool to the `/bin` so you can run it from anywhere
 # and it's easy to find.

LICENSE

@@ -1,9 +1,21 @@
-# License
+MIT License
 
-Copyright (c) 2019-2025 Meili SAS
+Copyright (c) 2019-2024 Meili SAS
 
-Part of this work fall under the Meilisearch Enterprise Edition (EE) and are licensed under the Business Source License 1.1, please refer to [LICENSE-EE](./LICENSE-EE) for details.
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-The other parts of this work are licensed under the [MIT license](./LICENSE-MIT).
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
 
-`SPDX-License-Identifier: MIT AND BUSL-1.1`
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

LICENSE-EE

@@ -1,67 +0,0 @@
-Business Source License 1.1 – Adapted for Meili SAS
-This license is based on the Business Source License version 1.1, as published by MariaDB Corporation Ab.
-
-Parameters
-
-Licensor: Meili SAS
-
-Licensed Work: Any file explicitly marked as “Enterprise Edition (EE)” or “governed by the Business Source License” residing in enterprise_editions modules/folders.
-
-Additional Use Grant:
-You may use, modify, and distribute the Licensed Work for non-production purposes only, such as testing, development, or evaluation.
-
-Production use of the Licensed Work requires a commercial license agreement with Meilisearch. Contact bonjour@meilisearch.com for licensing.
-
-Change License: MIT
-
-Change Date: Four years from the date the Licensed Work is published.
-
-This License does not apply to any code outside of the Licensed Work, which remains under the MIT license.
-
-For information about alternative licensing arrangements for the Licensed Work,
-please contact bonjour@meilisearch.com or sales@meilisearch.com.
-
-Notice
-
-Business Source License 1.1
-
-Terms
-
-The Licensor hereby grants you the right to copy, modify, create derivative
-works, redistribute, and make non-production use of the Licensed Work. The
-Licensor may make an Additional Use Grant, above, permitting limited production use.
-
-Effective on the Change Date, or the fourth anniversary of the first publicly
-available distribution of a specific version of the Licensed Work under this
-License, whichever comes first, the Licensor hereby grants you rights under
-the terms of the Change License, and the rights granted in the paragraph
-above terminate.
-
-If your use of the Licensed Work does not comply with the requirements
-currently in effect as described in this License, you must purchase a
-commercial license from the Licensor, its affiliated entities, or authorized
-resellers, or you must refrain from using the Licensed Work.
-
-All copies of the original and modified Licensed Work, and derivative works
-of the Licensed Work, are subject to this License. This License applies
-separately for each version of the Licensed Work and the Change Date may vary
-for each version of the Licensed Work released by Licensor.
-
-You must conspicuously display this License on each original or modified copy
-of the Licensed Work. If you receive the Licensed Work in original or
-modified form from a third party, the terms and conditions set forth in this
-License apply to your use of that work.
-
-Any use of the Licensed Work in violation of this License will automatically
-terminate your rights under this License for the current and all other
-versions of the Licensed Work.
-
-This License does not grant you any right in any trademark or logo of
-Licensor or its affiliates (provided that you may use a trademark or logo of
-Licensor as expressly required by this License).
-
-TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
-AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
-EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
-TITLE.

LICENSE-MIT

@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2019-2025 Meili SAS
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

README.md

@@ -20,12 +20,12 @@
 <p align="center">
   <a href="https://deps.rs/repo/github/meilisearch/meilisearch"><img src="https://deps.rs/repo/github/meilisearch/meilisearch/status.svg" alt="Dependency status"></a>
   <a href="https://github.com/meilisearch/meilisearch/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-informational" alt="License"></a>
-  <a href="https://github.com/meilisearch/meilisearch/queue"><img alt="Merge Queues enabled" src="https://img.shields.io/badge/Merge_Queues-enabled-%2357cf60?logo=github"></a>
+  <a href="https://ms-bors.herokuapp.com/repositories/52"><img src="https://bors.tech/images/badge_small.svg" alt="Bors enabled"></a>
 </p>
 
 <p align="center">⚡ A lightning-fast search engine that fits effortlessly into your apps, websites, and workflow 🔍</p>
 
-[Meilisearch](https://www.meilisearch.com?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=intro) helps you shape a delightful search experience in a snap, offering features that work out of the box to speed up your workflow.
+Meilisearch helps you shape a delightful search experience in a snap, offering features that work out-of-the-box to speed up your workflow.
 
 <p align="center" name="demo">
   <a href="https://where2watch.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demo-gif#gh-light-mode-only" target="_blank">
@@ -36,43 +36,34 @@
   </a>
 </p>
 
-## 🖥 Examples
-
-- [**Movies**](https://where2watch.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=organization) — An application to help you find streaming platforms to watch movies using [hybrid search](https://www.meilisearch.com/solutions/hybrid-search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos).
-- [**Flickr**](https://flickr.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=organization) — Search and explore one hundred million Flickr images with semantic search.
-- [**Ecommerce**](https://ecommerce.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) — Ecommerce website using disjunctive [facets](https://www.meilisearch.com/docs/learn/fine_tuning_results/faceted_search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos), range and rating filtering, and pagination.
-- [**Songs**](https://music.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) — Search through 47 million of songs.
-- [**SaaS**](https://saas.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) — Search for contacts, deals, and companies in this [multi-tenant](https://www.meilisearch.com/docs/learn/security/multitenancy_tenant_tokens?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) CRM application.
-
-See the list of all our example apps in our [demos repository](https://github.com/meilisearch/demos).
+🔥 [**Try it!**](https://where2watch.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demo-link) 🔥
 
 ## ✨ Features
-- **Hybrid search:** Combine the best of both [semantic](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features) & full-text search to get the most relevant results
-- **Search-as-you-type:** Find & display results in less than 50 milliseconds to provide an intuitive experience
-- **[Typo tolerance](https://www.meilisearch.com/docs/learn/relevancy/typo_tolerance_settings?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** get relevant matches even when queries contain typos and misspellings
+
+- **Search-as-you-type:** find search results in less than 50 milliseconds
+- **[Typo tolerance](https://www.meilisearch.com/docs/learn/configuration/typo_tolerance?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** get relevant matches even when queries contain typos and misspellings
 - **[Filtering](https://www.meilisearch.com/docs/learn/fine_tuning_results/filtering?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features) and [faceted search](https://www.meilisearch.com/docs/learn/fine_tuning_results/faceted_search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** enhance your users' search experience with custom filters and build a faceted search interface in a few lines of code
 - **[Sorting](https://www.meilisearch.com/docs/learn/fine_tuning_results/sorting?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** sort results based on price, date, or pretty much anything else your users need
-- **[Synonym support](https://www.meilisearch.com/docs/learn/relevancy/synonyms?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** configure synonyms to include more relevant content in your search results
+- **[Synonym support](https://www.meilisearch.com/docs/learn/configuration/synonyms?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** configure synonyms to include more relevant content in your search results
 - **[Geosearch](https://www.meilisearch.com/docs/learn/fine_tuning_results/geosearch?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** filter and sort documents based on geographic data
 - **[Extensive language support](https://www.meilisearch.com/docs/learn/what_is_meilisearch/language?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** search datasets in any language, with optimized support for Chinese, Japanese, Hebrew, and languages using the Latin alphabet
 - **[Security management](https://www.meilisearch.com/docs/learn/security/master_api_keys?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** control which users can access what data with API keys that allow fine-grained permissions handling
-- **[Multi-Tenancy](https://www.meilisearch.com/docs/learn/security/multitenancy_tenant_tokens?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** personalize search results for any number of application tenants
+- **[Multi-Tenancy](https://www.meilisearch.com/docs/learn/security/tenant_tokens?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** personalize search results for any number of application tenants
 - **Highly Customizable:** customize Meilisearch to your specific needs or use our out-of-the-box and hassle-free presets
 - **[RESTful API](https://www.meilisearch.com/docs/reference/api/overview?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** integrate Meilisearch in your technical stack with our plugins and SDKs
-- **AI-ready:** works out of the box with [langchain](https://www.meilisearch.com/with/langchain) and the [model context protocol](https://github.com/meilisearch/meilisearch-mcp)
 - **Easy to install, deploy, and maintain**
 
 ## 📖 Documentation
 
-You can consult Meilisearch's documentation at [meilisearch.com/docs](https://www.meilisearch.com/docs/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=docs).
+You can consult Meilisearch's documentation at [https://www.meilisearch.com/docs](https://www.meilisearch.com/docs/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=docs).
 
 ## 🚀 Getting started
 
-For basic instructions on how to set up Meilisearch, add documents to an index, and search for documents, take a look at our [documentation](https://www.meilisearch.com/docs?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=get-started) guide.
+For basic instructions on how to set up Meilisearch, add documents to an index, and search for documents, take a look at our [Quick Start](https://www.meilisearch.com/docs/learn/getting_started/quick_start?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=get-started) guide.
 
-## 🌍 Supercharge your Meilisearch experience
+## ⚡ Supercharge your Meilisearch experience
 
-Say goodbye to server deployment and manual updates with [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=github&utm_medium=meilisearch). Additional features include analytics & monitoring in many regions around the world. No credit card is required.
+Say goodbye to server deployment and manual updates with [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=github&utm_medium=meilisearch). No credit card required.
 
 ## 🧰 SDKs & integration tools
 
@@ -90,39 +81,19 @@ We also offer a wide range of dedicated guides to all Meilisearch features, such
 
 Finally, for more in-depth information, refer to our articles explaining fundamental Meilisearch concepts such as [documents](https://www.meilisearch.com/docs/learn/core_concepts/documents?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced) and [indexes](https://www.meilisearch.com/docs/learn/core_concepts/indexes?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced).
 
-## 🧾 Editions & Licensing
-
-Meilisearch is available in two editions:
-
-### 🧪 Community Edition (CE)
-
-- Fully open source under the [MIT license](./LICENSE)
-- Core search engine with fast and relevant full-text, semantic or hybrid search
-- Free to use for anyone, including commercial usage
-
-### 🏢 Enterprise Edition (EE)
-
-- Includes advanced features such as:
-  - Sharding
-- Governed by a [commercial license](./LICENSE-EE) or the [Business Source License 1.1](https://mariadb.com/bsl11)
-- Not allowed in production without a commercial agreement with Meilisearch.
-  - You may use, modify, and distribute the Licensed Work for non-production purposes only, such as testing, development, or evaluation.
-
-Want access to Enterprise features? → Contact us at [sales@meilisearch.com](maito:sales@meilisearch.com).
-
 ## 📊 Telemetry
 
-Meilisearch collects **anonymized** user data to help us improve our product. You can [deactivate this](https://www.meilisearch.com/docs/learn/what_is_meilisearch/telemetry?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=telemetry#how-to-disable-data-collection) whenever you want.
+Meilisearch collects **anonymized** data from users to help us improve our product. You can [deactivate this](https://www.meilisearch.com/docs/learn/what_is_meilisearch/telemetry?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=telemetry#how-to-disable-data-collection) whenever you want.
 
-To request deletion of collected data, please write to us at [privacy@meilisearch.com](mailto:privacy@meilisearch.com). Remember to include your `Instance UID` in the message, as this helps us quickly find and delete your data.
+To request deletion of collected data, please write to us at [privacy@meilisearch.com](mailto:privacy@meilisearch.com). Don't forget to include your `Instance UID` in the message, as this helps us quickly find and delete your data.
 
 If you want to know more about the kind of data we collect and what we use it for, check the [telemetry section](https://www.meilisearch.com/docs/learn/what_is_meilisearch/telemetry?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=telemetry#how-to-disable-data-collection) of our documentation.
 
 ## 📫 Get in touch!
 
-Meilisearch is a search engine created by [Meili](https://www.meilisearch.com/careers), a software development company headquartered in France and with team members all over the world. Want to know more about us? [Check out our blog!](https://blog.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=contact)
+Meilisearch is a search engine created by [Meili](https://www.welcometothejungle.com/en/companies/meilisearch), a software development company based in France and with team members all over the world. Want to know more about us? [Check out our blog!](https://blog.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=contact)
 
-🗞 [Subscribe to our newsletter](https://share-eu1.hsforms.com/1LN5N0x_GQgq7ss7tXmSykwfg3aq) if you don't want to miss any updates! We promise we won't clutter your mailbox: we only send one edition every two months.
+🗞 [Subscribe to our newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e) if you don't want to miss any updates! We promise we won't clutter your mailbox: we only send one edition every two months.
 
 💌 Want to make a suggestion or give feedback? Here are some of the channels where you can reach us:
 
@@ -134,12 +105,12 @@ Thank you for your support!
 
 ## 👩‍💻 Contributing
 
-Meilisearch is, and will always be, open-source! If you want to contribute to the project, please look at [our contribution guidelines](CONTRIBUTING.md).
+Meilisearch is, and will always be, open-source! If you want to contribute to the project, please take a look at [our contribution guidelines](CONTRIBUTING.md).
 
 ## 📦 Versioning
 
-Meilisearch releases and their associated binaries are available on the project's [releases page](https://github.com/meilisearch/meilisearch/releases).
+Meilisearch releases and their associated binaries are available [in this GitHub page](https://github.com/meilisearch/meilisearch/releases).
 
-The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](./documentation/versioning-policy.md).
+The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](https://github.com/meilisearch/engine-team/blob/main/resources/versioning-policy.md).
 
 Differently from the binaries, crates in this repository are not currently available on [crates.io](https://crates.io/) and do not follow [SemVer conventions](https://semver.org).

benchmarks/Cargo.toml

@@ -11,27 +11,24 @@ edition.workspace = true
 license.workspace = true
 
 [dependencies]
-anyhow = "1.0.98"
-bumpalo = "3.18.1"
-csv = "1.3.1"
-memmap2 = "0.9.7"
+anyhow = "1.0.79"
+csv = "1.3.0"
 milli = { path = "../milli" }
-mimalloc = { version = "0.1.47", default-features = false }
-serde_json = { version = "1.0.140", features = ["preserve_order"] }
-tempfile = "3.20.0"
+mimalloc = { version = "0.1.39", default-features = false }
+serde_json = { version = "1.0.111", features = ["preserve_order"] }
 
 [dev-dependencies]
-criterion = { version = "0.6.0", features = ["html_reports"] }
+criterion = { version = "0.5.1", features = ["html_reports"] }
 rand = "0.8.5"
 rand_chacha = "0.3.1"
-roaring = "0.10.12"
+roaring = "0.10.2"
 
 [build-dependencies]
-anyhow = "1.0.98"
-bytes = "1.10.1"
-convert_case = "0.8.0"
-flate2 = "1.1.2"
-reqwest = { version = "0.12.20", features = ["blocking", "rustls-tls"], default-features = false }
+anyhow = "1.0.79"
+bytes = "1.5.0"
+convert_case = "0.6.0"
+flate2 = "1.0.28"
+reqwest = { version = "0.11.23", features = ["blocking", "rustls-tls"], default-features = false }
 
 [features]
 default = ["milli/all-tokenizations"]
@@ -51,11 +48,3 @@ harness = false
 [[bench]]
 name = "indexing"
 harness = false
-
-[[bench]]
-name = "sort"
-harness = false
-
-[[bench]]
-name = "filter_starts_with"
-harness = false

benchmarks/benches/search_geo.rs

@@ -3,10 +3,8 @@ mod utils;
 
 use criterion::{criterion_group, criterion_main};
 use milli::update::Settings;
-use milli::FilterableAttributesRule;
 use utils::Conf;
 
-#[cfg(not(windows))]
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
@@ -22,10 +20,8 @@ fn base_conf(builder: &mut Settings) {
         ["name", "alternatenames", "elevation"].iter().map(|s| s.to_string()).collect();
     builder.set_searchable_fields(searchable_fields);
 
-    let filterable_fields = ["_geo", "population", "elevation"]
-        .iter()
-        .map(|s| FilterableAttributesRule::Field(s.to_string()))
-        .collect();
+    let filterable_fields =
+        ["_geo", "population", "elevation"].iter().map(|s| s.to_string()).collect();
     builder.set_filterable_fields(filterable_fields);
 
     let sortable_fields =

benchmarks/benches/search_songs.rs

@@ -3,10 +3,8 @@ mod utils;
 
 use criterion::{criterion_group, criterion_main};
 use milli::update::Settings;
-use milli::FilterableAttributesRule;
 use utils::Conf;
 
-#[cfg(not(windows))]
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
@@ -23,7 +21,7 @@ fn base_conf(builder: &mut Settings) {
 
     let faceted_fields = ["released-timestamp", "duration-float", "genre", "country", "artist"]
         .iter()
-        .map(|s| FilterableAttributesRule::Field(s.to_string()))
+        .map(|s| s.to_string())
         .collect();
     builder.set_filterable_fields(faceted_fields);
 }

benchmarks/benches/search_wiki.rs

@@ -5,7 +5,6 @@ use criterion::{criterion_group, criterion_main};
 use milli::update::Settings;
 use utils::Conf;
 
-#[cfg(not(windows))]
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 

benchmarks/benches/utils.rs

@@ -0,0 +1,256 @@
+#![allow(dead_code)]
+
+use std::fs::{create_dir_all, remove_dir_all, File};
+use std::io::{self, BufRead, BufReader, Cursor, Read, Seek};
+use std::num::ParseFloatError;
+use std::path::Path;
+use std::str::FromStr;
+
+use criterion::BenchmarkId;
+use milli::documents::{DocumentsBatchBuilder, DocumentsBatchReader};
+use milli::heed::EnvOpenOptions;
+use milli::update::{
+    IndexDocuments, IndexDocumentsConfig, IndexDocumentsMethod, IndexerConfig, Settings,
+};
+use milli::{Criterion, Filter, Index, Object, TermsMatchingStrategy};
+use serde_json::Value;
+
+pub struct Conf<'a> {
+    /// where we are going to create our database.mmdb directory
+    /// each benchmark will first try to delete it and then recreate it
+    pub database_name: &'a str,
+    /// the dataset to be used, it must be an uncompressed csv
+    pub dataset: &'a str,
+    /// The format of the dataset
+    pub dataset_format: &'a str,
+    pub group_name: &'a str,
+    pub queries: &'a [&'a str],
+    /// here you can change which criterion are used and in which order.
+    /// - if you specify something all the base configuration will be thrown out
+    /// - if you don't specify anything (None) the default configuration will be kept
+    pub criterion: Option<&'a [&'a str]>,
+    /// the last chance to configure your database as you want
+    pub configure: fn(&mut Settings),
+    pub filter: Option<&'a str>,
+    pub sort: Option<Vec<&'a str>>,
+    /// enable or disable the optional words on the query
+    pub optional_words: bool,
+    /// primary key, if there is None we'll auto-generate docids for every documents
+    pub primary_key: Option<&'a str>,
+}
+
+impl Conf<'_> {
+    pub const BASE: Self = Conf {
+        database_name: "benches.mmdb",
+        dataset_format: "csv",
+        dataset: "",
+        group_name: "",
+        queries: &[],
+        criterion: None,
+        configure: |_| (),
+        filter: None,
+        sort: None,
+        optional_words: true,
+        primary_key: None,
+    };
+}
+
+pub fn base_setup(conf: &Conf) -> Index {
+    match remove_dir_all(conf.database_name) {
+        Ok(_) => (),
+        Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
+        Err(e) => panic!("{}", e),
+    }
+    create_dir_all(conf.database_name).unwrap();
+
+    let mut options = EnvOpenOptions::new();
+    options.map_size(100 * 1024 * 1024 * 1024); // 100 GB
+    options.max_readers(10);
+    let index = Index::new(options, conf.database_name).unwrap();
+
+    let config = IndexerConfig::default();
+    let mut wtxn = index.write_txn().unwrap();
+    let mut builder = Settings::new(&mut wtxn, &index, &config);
+
+    if let Some(primary_key) = conf.primary_key {
+        builder.set_primary_key(primary_key.to_string());
+    }
+
+    if let Some(criterion) = conf.criterion {
+        builder.reset_filterable_fields();
+        builder.reset_criteria();
+        builder.reset_stop_words();
+
+        let criterion = criterion.iter().map(|s| Criterion::from_str(s).unwrap()).collect();
+        builder.set_criteria(criterion);
+    }
+
+    (conf.configure)(&mut builder);
+
+    builder.execute(|_| (), || false).unwrap();
+    wtxn.commit().unwrap();
+
+    let config = IndexerConfig::default();
+    let mut wtxn = index.write_txn().unwrap();
+    let indexing_config = IndexDocumentsConfig {
+        autogenerate_docids: conf.primary_key.is_none(),
+        update_method: IndexDocumentsMethod::ReplaceDocuments,
+        ..Default::default()
+    };
+    let builder =
+        IndexDocuments::new(&mut wtxn, &index, &config, indexing_config, |_| (), || false).unwrap();
+    let documents = documents_from(conf.dataset, conf.dataset_format);
+    let (builder, user_error) = builder.add_documents(documents).unwrap();
+    user_error.unwrap();
+    builder.execute().unwrap();
+    wtxn.commit().unwrap();
+
+    index
+}
+
+pub fn run_benches(c: &mut criterion::Criterion, confs: &[Conf]) {
+    for conf in confs {
+        let index = base_setup(conf);
+
+        let file_name = Path::new(conf.dataset).file_name().and_then(|f| f.to_str()).unwrap();
+        let name = format!("{}: {}", file_name, conf.group_name);
+        let mut group = c.benchmark_group(&name);
+
+        for &query in conf.queries {
+            group.bench_with_input(BenchmarkId::from_parameter(query), &query, |b, &query| {
+                b.iter(|| {
+                    let rtxn = index.read_txn().unwrap();
+                    let mut search = index.search(&rtxn);
+                    search.query(query).terms_matching_strategy(TermsMatchingStrategy::default());
+                    if let Some(filter) = conf.filter {
+                        let filter = Filter::from_str(filter).unwrap().unwrap();
+                        search.filter(filter);
+                    }
+                    if let Some(sort) = &conf.sort {
+                        let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
+                        search.sort_criteria(sort);
+                    }
+                    let _ids = search.execute().unwrap();
+                });
+            });
+        }
+        group.finish();
+
+        index.prepare_for_closing().wait();
+    }
+}
+
+pub fn documents_from(filename: &str, filetype: &str) -> DocumentsBatchReader<impl BufRead + Seek> {
+    let reader = File::open(filename)
+        .unwrap_or_else(|_| panic!("could not find the dataset in: {}", filename));
+    let reader = BufReader::new(reader);
+    let documents = match filetype {
+        "csv" => documents_from_csv(reader).unwrap(),
+        "json" => documents_from_json(reader).unwrap(),
+        "jsonl" => documents_from_jsonl(reader).unwrap(),
+        otherwise => panic!("invalid update format {:?}", otherwise),
+    };
+    DocumentsBatchReader::from_reader(Cursor::new(documents)).unwrap()
+}
+
+fn documents_from_jsonl(reader: impl BufRead) -> anyhow::Result<Vec<u8>> {
+    let mut documents = DocumentsBatchBuilder::new(Vec::new());
+
+    for result in serde_json::Deserializer::from_reader(reader).into_iter::<Object>() {
+        let object = result?;
+        documents.append_json_object(&object)?;
+    }
+
+    documents.into_inner().map_err(Into::into)
+}
+
+fn documents_from_json(reader: impl BufRead) -> anyhow::Result<Vec<u8>> {
+    let mut documents = DocumentsBatchBuilder::new(Vec::new());
+
+    documents.append_json_array(reader)?;
+
+    documents.into_inner().map_err(Into::into)
+}
+
+fn documents_from_csv(reader: impl BufRead) -> anyhow::Result<Vec<u8>> {
+    let csv = csv::Reader::from_reader(reader);
+
+    let mut documents = DocumentsBatchBuilder::new(Vec::new());
+    documents.append_csv(csv)?;
+
+    documents.into_inner().map_err(Into::into)
+}
+
+enum AllowedType {
+    String,
+    Number,
+}
+
+fn parse_csv_header(header: &str) -> (String, AllowedType) {
+    // if there are several separators we only split on the last one.
+    match header.rsplit_once(':') {
+        Some((field_name, field_type)) => match field_type {
+            "string" => (field_name.to_string(), AllowedType::String),
+            "number" => (field_name.to_string(), AllowedType::Number),
+            // we may return an error in this case.
+            _otherwise => (header.to_string(), AllowedType::String),
+        },
+        None => (header.to_string(), AllowedType::String),
+    }
+}
+
+struct CSVDocumentDeserializer<R>
+where
+    R: Read,
+{
+    documents: csv::StringRecordsIntoIter<R>,
+    headers: Vec<(String, AllowedType)>,
+}
+
+impl<R: Read> CSVDocumentDeserializer<R> {
+    fn from_reader(reader: R) -> io::Result<Self> {
+        let mut records = csv::Reader::from_reader(reader);
+
+        let headers = records.headers()?.into_iter().map(parse_csv_header).collect();
+
+        Ok(Self { documents: records.into_records(), headers })
+    }
+}
+
+impl<R: Read> Iterator for CSVDocumentDeserializer<R> {
+    type Item = anyhow::Result<Object>;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        let csv_document = self.documents.next()?;
+
+        match csv_document {
+            Ok(csv_document) => {
+                let mut document = Object::new();
+
+                for ((field_name, field_type), value) in
+                    self.headers.iter().zip(csv_document.into_iter())
+                {
+                    let parsed_value: Result<Value, ParseFloatError> = match field_type {
+                        AllowedType::Number => {
+                            value.parse::<f64>().map(Value::from).map_err(Into::into)
+                        }
+                        AllowedType::String => Ok(Value::String(value.to_string())),
+                    };
+
+                    match parsed_value {
+                        Ok(value) => drop(document.insert(field_name.to_string(), value)),
+                        Err(_e) => {
+                            return Some(Err(anyhow::anyhow!(
+                                "Value '{}' is not a valid number",
+                                value
+                            )))
+                        }
+                    }
+                }
+
+                Some(Ok(document))
+            }
+            Err(e) => Some(Err(anyhow::anyhow!("Error parsing csv document: {}", e))),
+        }
+    }
+}

benchmarks/build.rs

@@ -67,7 +67,7 @@ fn main() -> anyhow::Result<()> {
         writeln!(
             &mut manifest_paths_file,
             r#"pub const {}: &str = {:?};"#,
-            dataset.to_case(Case::UpperSnake),
+            dataset.to_case(Case::ScreamingSnake),
             out_file.display(),
         )?;
 

bors.toml

@@ -0,0 +1,11 @@
+status = [
+    'Tests on ubuntu-18.04',
+    'Tests on macos-12',
+    'Tests on windows-2022',
+    'Run Clippy',
+    'Run Rustfmt',
+    'Run tests in debug',
+]
+pr_status = ['Milestone Check']
+# 3 hours timeout
+timeout-sec = 10800

crates/benchmarks/benches/filter_starts_with.rs

@@ -1,66 +0,0 @@
-mod datasets_paths;
-mod utils;
-
-use criterion::{criterion_group, criterion_main};
-use milli::update::Settings;
-use milli::FilterableAttributesRule;
-use utils::Conf;
-
-#[cfg(not(windows))]
-#[global_allocator]
-static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
-
-fn base_conf(builder: &mut Settings) {
-    let displayed_fields = ["geonameid", "name"].iter().map(|s| s.to_string()).collect();
-    builder.set_displayed_fields(displayed_fields);
-
-    let filterable_fields =
-        ["name"].iter().map(|s| FilterableAttributesRule::Field(s.to_string())).collect();
-    builder.set_filterable_fields(filterable_fields);
-}
-
-#[rustfmt::skip]
-const BASE_CONF: Conf = Conf {
-    dataset: datasets_paths::SMOL_ALL_COUNTRIES,
-    dataset_format: "jsonl",
-    queries: &[
-        "",
-    ],
-    configure: base_conf,
-    primary_key: Some("geonameid"),
-    ..Conf::BASE
-};
-
-fn filter_starts_with(c: &mut criterion::Criterion) {
-    #[rustfmt::skip]
-    let confs = &[
-        utils::Conf {
-            group_name: "1 letter",
-            filter: Some("name STARTS WITH e"),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "2 letters",
-            filter: Some("name STARTS WITH es"),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "3 letters",
-            filter: Some("name STARTS WITH est"),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "6 letters",
-            filter: Some("name STARTS WITH estoni"),
-            ..BASE_CONF
-        }
-    ];
-
-    utils::run_benches(c, confs);
-}
-
-criterion_group!(benches, filter_starts_with);
-criterion_main!(benches);

crates/benchmarks/benches/sort.rs

@@ -1,114 +0,0 @@
-//! This benchmark module is used to compare the performance of sorting documents in /search VS /documents
-//!
-//! The tests/benchmarks were designed in the context of a query returning only 20 documents.
-
-mod datasets_paths;
-mod utils;
-
-use criterion::{criterion_group, criterion_main};
-use milli::update::Settings;
-use utils::Conf;
-
-#[cfg(not(windows))]
-#[global_allocator]
-static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
-
-fn base_conf(builder: &mut Settings) {
-    let displayed_fields =
-        ["geonameid", "name", "asciiname", "alternatenames", "_geo", "population"]
-            .iter()
-            .map(|s| s.to_string())
-            .collect();
-    builder.set_displayed_fields(displayed_fields);
-
-    let sortable_fields =
-        ["_geo", "name", "population", "elevation", "timezone", "modification-date"]
-            .iter()
-            .map(|s| s.to_string())
-            .collect();
-    builder.set_sortable_fields(sortable_fields);
-}
-
-#[rustfmt::skip]
-const BASE_CONF: Conf = Conf {
-    dataset: datasets_paths::SMOL_ALL_COUNTRIES,
-    dataset_format: "jsonl",
-    configure: base_conf,
-    primary_key: Some("geonameid"),
-    queries: &[""],
-    offsets: &[
-        Some((0, 20)), // The most common query in the real world
-        Some((0, 500)), // A query that ranges over many documents
-        Some((980, 20)), // The worst query that could happen in the real world
-        Some((800_000, 20)) // The worst query
-    ],
-    get_documents: true,
-    ..Conf::BASE
-};
-
-fn bench_sort(c: &mut criterion::Criterion) {
-    #[rustfmt::skip]
-    let confs = &[
-        utils::Conf {
-            group_name: "without sort",
-            sort: None,
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many different values",
-            sort: Some(vec!["name:asc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many similar values",
-            sort: Some(vec!["timezone:desc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many similar then different values",
-            sort: Some(vec!["timezone:desc", "name:asc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many different then similar values",
-            sort: Some(vec!["timezone:desc", "name:asc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "geo sort",
-            sample_size: Some(10),
-            sort: Some(vec!["_geoPoint(45.4777599, 9.1967508):asc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many similar values then geo sort",
-            sample_size: Some(50),
-            sort: Some(vec!["timezone:desc", "_geoPoint(45.4777599, 9.1967508):asc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many different values then geo sort",
-            sample_size: Some(50),
-            sort: Some(vec!["name:desc", "_geoPoint(45.4777599, 9.1967508):asc"]),
-            ..BASE_CONF
-        },
-
-        utils::Conf {
-            group_name: "sort on many fields",
-            sort: Some(vec!["population:asc", "name:asc", "elevation:asc", "timezone:asc"]),
-            ..BASE_CONF
-        },
-    ];
-
-    utils::run_benches(c, confs);
-}
-
-criterion_group!(benches, bench_sort);
-criterion_main!(benches);

crates/benchmarks/benches/utils.rs

@@ -1,402 +0,0 @@
-#![allow(dead_code)]
-
-use std::fs::{create_dir_all, remove_dir_all, File};
-use std::io::{self, BufReader, BufWriter, Read};
-use std::path::Path;
-use std::str::FromStr as _;
-
-use anyhow::Context;
-use bumpalo::Bump;
-use criterion::BenchmarkId;
-use memmap2::Mmap;
-use milli::documents::sort::recursive_sort;
-use milli::heed::EnvOpenOptions;
-use milli::progress::Progress;
-use milli::update::new::indexer;
-use milli::update::{IndexerConfig, Settings};
-use milli::vector::RuntimeEmbedders;
-use milli::{Criterion, Filter, Index, Object, TermsMatchingStrategy};
-use serde_json::Value;
-
-pub struct Conf<'a> {
-    /// where we are going to create our database.mmdb directory
-    /// each benchmark will first try to delete it and then recreate it
-    pub database_name: &'a str,
-    /// the dataset to be used, it must be an uncompressed csv
-    pub dataset: &'a str,
-    /// The format of the dataset
-    pub dataset_format: &'a str,
-    pub group_name: &'a str,
-    pub queries: &'a [&'a str],
-    /// here you can change which criterion are used and in which order.
-    /// - if you specify something all the base configuration will be thrown out
-    /// - if you don't specify anything (None) the default configuration will be kept
-    pub criterion: Option<&'a [&'a str]>,
-    /// the last chance to configure your database as you want
-    pub configure: fn(&mut Settings),
-    pub filter: Option<&'a str>,
-    pub sort: Option<Vec<&'a str>>,
-    /// set to skip documents (offset, limit)
-    pub offsets: &'a [Option<(usize, usize)>],
-    /// enable if you want to bench getting documents without querying
-    pub get_documents: bool,
-    /// configure the benchmark sample size
-    pub sample_size: Option<usize>,
-    /// enable or disable the optional words on the query
-    pub optional_words: bool,
-    /// primary key, if there is None we'll auto-generate docids for every documents
-    pub primary_key: Option<&'a str>,
-}
-
-impl Conf<'_> {
-    pub const BASE: Self = Conf {
-        database_name: "benches.mmdb",
-        dataset_format: "csv",
-        dataset: "",
-        group_name: "",
-        queries: &[],
-        criterion: None,
-        configure: |_| (),
-        filter: None,
-        sort: None,
-        offsets: &[None],
-        get_documents: false,
-        sample_size: None,
-        optional_words: true,
-        primary_key: None,
-    };
-}
-
-pub fn base_setup(conf: &Conf) -> Index {
-    match remove_dir_all(conf.database_name) {
-        Ok(_) => (),
-        Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
-        Err(e) => panic!("{}", e),
-    }
-    create_dir_all(conf.database_name).unwrap();
-
-    let options = EnvOpenOptions::new();
-    let mut options = options.read_txn_without_tls();
-    options.map_size(100 * 1024 * 1024 * 1024); // 100 GB
-    options.max_readers(100);
-    let index = Index::new(options, conf.database_name, true).unwrap();
-
-    let config = IndexerConfig::default();
-    let mut wtxn = index.write_txn().unwrap();
-    let mut builder = Settings::new(&mut wtxn, &index, &config);
-
-    if let Some(primary_key) = conf.primary_key {
-        builder.set_primary_key(primary_key.to_string());
-    }
-
-    if let Some(criterion) = conf.criterion {
-        builder.reset_filterable_fields();
-        builder.reset_criteria();
-        builder.reset_stop_words();
-
-        let criterion = criterion.iter().map(|s| Criterion::from_str(s).unwrap()).collect();
-        builder.set_criteria(criterion);
-    }
-
-    (conf.configure)(&mut builder);
-
-    builder.execute(&|| false, &Progress::default(), Default::default()).unwrap();
-    wtxn.commit().unwrap();
-
-    let config = IndexerConfig::default();
-    let mut wtxn = index.write_txn().unwrap();
-    let rtxn = index.read_txn().unwrap();
-    let db_fields_ids_map = index.fields_ids_map(&rtxn).unwrap();
-    let mut new_fields_ids_map = db_fields_ids_map.clone();
-
-    let documents = documents_from(conf.dataset, conf.dataset_format);
-    let mut indexer = indexer::DocumentOperation::new();
-    indexer.replace_documents(&documents).unwrap();
-
-    let indexer_alloc = Bump::new();
-    let (document_changes, _operation_stats, primary_key) = indexer
-        .into_changes(
-            &indexer_alloc,
-            &index,
-            &rtxn,
-            None,
-            &mut new_fields_ids_map,
-            &|| false,
-            Progress::default(),
-            None,
-        )
-        .unwrap();
-
-    indexer::index(
-        &mut wtxn,
-        &index,
-        &milli::ThreadPoolNoAbortBuilder::new().build().unwrap(),
-        config.grenad_parameters(),
-        &db_fields_ids_map,
-        new_fields_ids_map,
-        primary_key,
-        &document_changes,
-        RuntimeEmbedders::default(),
-        &|| false,
-        &Progress::default(),
-        &Default::default(),
-    )
-    .unwrap();
-
-    wtxn.commit().unwrap();
-    drop(rtxn);
-
-    index
-}
-
-pub fn run_benches(c: &mut criterion::Criterion, confs: &[Conf]) {
-    for conf in confs {
-        let index = base_setup(conf);
-
-        let file_name = Path::new(conf.dataset).file_name().and_then(|f| f.to_str()).unwrap();
-        let name = format!("{}: {}", file_name, conf.group_name);
-        let mut group = c.benchmark_group(&name);
-        if let Some(sample_size) = conf.sample_size {
-            group.sample_size(sample_size);
-        }
-
-        for &query in conf.queries {
-            for offset in conf.offsets {
-                let parameter = match offset {
-                    None => query.to_string(),
-                    Some((offset, limit)) => format!("{query}[{offset}:{limit}]"),
-                };
-                group.bench_with_input(
-                    BenchmarkId::from_parameter(parameter),
-                    &query,
-                    |b, &query| {
-                        b.iter(|| {
-                            let rtxn = index.read_txn().unwrap();
-                            let mut search = index.search(&rtxn);
-                            search
-                                .query(query)
-                                .terms_matching_strategy(TermsMatchingStrategy::default());
-                            if let Some(filter) = conf.filter {
-                                let filter = Filter::from_str(filter).unwrap().unwrap();
-                                search.filter(filter);
-                            }
-                            if let Some(sort) = &conf.sort {
-                                let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
-                                search.sort_criteria(sort);
-                            }
-                            if let Some((offset, limit)) = offset {
-                                search.offset(*offset).limit(*limit);
-                            }
-
-                            let _ids = search.execute().unwrap();
-                        });
-                    },
-                );
-            }
-        }
-
-        if conf.get_documents {
-            for offset in conf.offsets {
-                let parameter = match offset {
-                    None => String::from("get_documents"),
-                    Some((offset, limit)) => format!("get_documents[{offset}:{limit}]"),
-                };
-                group.bench_with_input(BenchmarkId::from_parameter(parameter), &(), |b, &()| {
-                    b.iter(|| {
-                        let rtxn = index.read_txn().unwrap();
-                        if let Some(sort) = &conf.sort {
-                            let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
-                            let all_docs = index.documents_ids(&rtxn).unwrap();
-                            let facet_sort =
-                                recursive_sort(&index, &rtxn, sort, &all_docs).unwrap();
-                            let iter = facet_sort.iter().unwrap();
-                            if let Some((offset, limit)) = offset {
-                                let _results = iter.skip(*offset).take(*limit).collect::<Vec<_>>();
-                            } else {
-                                let _results = iter.collect::<Vec<_>>();
-                            }
-                        } else {
-                            let all_docs = index.documents_ids(&rtxn).unwrap();
-                            if let Some((offset, limit)) = offset {
-                                let _results =
-                                    all_docs.iter().skip(*offset).take(*limit).collect::<Vec<_>>();
-                            } else {
-                                let _results = all_docs.iter().collect::<Vec<_>>();
-                            }
-                        }
-                    });
-                });
-            }
-        }
-
-        group.finish();
-
-        index.prepare_for_closing().wait();
-    }
-}
-
-pub fn documents_from(filename: &str, filetype: &str) -> Mmap {
-    let file = File::open(filename)
-        .unwrap_or_else(|_| panic!("could not find the dataset in: {filename}"));
-    match filetype {
-        "csv" => documents_from_csv(file).unwrap(),
-        "json" => documents_from_json(file).unwrap(),
-        "jsonl" => documents_from_jsonl(file).unwrap(),
-        otherwise => panic!("invalid update format {otherwise:?}"),
-    }
-}
-
-fn documents_from_jsonl(file: File) -> anyhow::Result<Mmap> {
-    unsafe { Mmap::map(&file).map_err(Into::into) }
-}
-
-fn documents_from_json(file: File) -> anyhow::Result<Mmap> {
-    let reader = BufReader::new(file);
-    let documents: Vec<milli::Object> = serde_json::from_reader(reader)?;
-    let mut output = tempfile::tempfile().map(BufWriter::new)?;
-
-    for document in documents {
-        serde_json::to_writer(&mut output, &document)?;
-    }
-
-    let file = output.into_inner()?;
-    unsafe { Mmap::map(&file).map_err(Into::into) }
-}
-
-fn documents_from_csv(file: File) -> anyhow::Result<Mmap> {
-    let output = tempfile::tempfile()?;
-    let mut output = BufWriter::new(output);
-    let mut reader = csv::ReaderBuilder::new().from_reader(file);
-
-    let headers = reader.headers().context("while retrieving headers")?.clone();
-    let typed_fields: Vec<_> = headers.iter().map(parse_csv_header).collect();
-    let mut object: serde_json::Map<_, _> =
-        typed_fields.iter().map(|(k, _)| (k.to_string(), Value::Null)).collect();
-
-    let mut line = 0;
-    let mut record = csv::StringRecord::new();
-    while reader.read_record(&mut record).context("while reading a record")? {
-        // We increment here and not at the end of the loop
-        // to take the header offset into account.
-        line += 1;
-
-        // Reset the document values
-        object.iter_mut().for_each(|(_, v)| *v = Value::Null);
-
-        for (i, (name, atype)) in typed_fields.iter().enumerate() {
-            let value = &record[i];
-            let trimmed_value = value.trim();
-            let value = match atype {
-                AllowedType::Number if trimmed_value.is_empty() => Value::Null,
-                AllowedType::Number => {
-                    match trimmed_value.parse::<i64>() {
-                        Ok(integer) => Value::from(integer),
-                        Err(_) => match trimmed_value.parse::<f64>() {
-                            Ok(float) => Value::from(float),
-                            Err(error) => {
-                                anyhow::bail!("document format error on line {line}: {error}. For value: {value}")
-                            }
-                        },
-                    }
-                }
-                AllowedType::Boolean if trimmed_value.is_empty() => Value::Null,
-                AllowedType::Boolean => match trimmed_value.parse::<bool>() {
-                    Ok(bool) => Value::from(bool),
-                    Err(error) => {
-                        anyhow::bail!(
-                            "document format error on line {line}: {error}. For value: {value}"
-                        )
-                    }
-                },
-                AllowedType::String if value.is_empty() => Value::Null,
-                AllowedType::String => Value::from(value),
-            };
-
-            *object.get_mut(name).expect("encountered an unknown field") = value;
-        }
-
-        serde_json::to_writer(&mut output, &object).context("while writing to disk")?;
-    }
-
-    let output = output.into_inner()?;
-    unsafe { Mmap::map(&output).map_err(Into::into) }
-}
-
-enum AllowedType {
-    String,
-    Boolean,
-    Number,
-}
-
-fn parse_csv_header(header: &str) -> (String, AllowedType) {
-    // if there are several separators we only split on the last one.
-    match header.rsplit_once(':') {
-        Some((field_name, field_type)) => match field_type {
-            "string" => (field_name.to_string(), AllowedType::String),
-            "boolean" => (field_name.to_string(), AllowedType::Boolean),
-            "number" => (field_name.to_string(), AllowedType::Number),
-            // if the pattern isn't recognized, we keep the whole field.
-            _otherwise => (header.to_string(), AllowedType::String),
-        },
-        None => (header.to_string(), AllowedType::String),
-    }
-}
-
-struct CSVDocumentDeserializer<R>
-where
-    R: Read,
-{
-    documents: csv::StringRecordsIntoIter<R>,
-    headers: Vec<(String, AllowedType)>,
-}
-
-impl<R: Read> CSVDocumentDeserializer<R> {
-    fn from_reader(reader: R) -> io::Result<Self> {
-        let mut records = csv::Reader::from_reader(reader);
-
-        let headers = records.headers()?.into_iter().map(parse_csv_header).collect();
-
-        Ok(Self { documents: records.into_records(), headers })
-    }
-}
-
-impl<R: Read> Iterator for CSVDocumentDeserializer<R> {
-    type Item = anyhow::Result<Object>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let csv_document = self.documents.next()?;
-
-        match csv_document {
-            Ok(csv_document) => {
-                let mut document = Object::new();
-
-                for ((field_name, field_type), value) in
-                    self.headers.iter().zip(csv_document.into_iter())
-                {
-                    let parsed_value: anyhow::Result<Value> = match field_type {
-                        AllowedType::Number => {
-                            value.parse::<f64>().map(Value::from).map_err(Into::into)
-                        }
-                        AllowedType::Boolean => {
-                            value.parse::<bool>().map(Value::from).map_err(Into::into)
-                        }
-                        AllowedType::String => Ok(Value::String(value.to_string())),
-                    };
-
-                    match parsed_value {
-                        Ok(value) => drop(document.insert(field_name.to_string(), value)),
-                        Err(_e) => {
-                            return Some(Err(anyhow::anyhow!(
-                                "Value '{}' is not a valid number",
-                                value
-                            )))
-                        }
-                    }
-                }
-
-                Some(Ok(document))
-            }
-            Err(e) => Some(Err(anyhow::anyhow!("Error parsing csv document: {}", e))),
-        }
-    }
-}

crates/build-info/build.rs

@@ -1,29 +0,0 @@
-fn main() {
-    if let Err(err) = emit_git_variables() {
-        println!("cargo:warning=vergen: {}", err);
-    }
-}
-
-fn emit_git_variables() -> anyhow::Result<()> {
-    println!("cargo::rerun-if-env-changed=MEILI_NO_VERGEN");
-
-    let has_vergen =
-        !matches!(std::env::var_os("MEILI_NO_VERGEN"), Some(x) if x != "false" && x != "0");
-
-    anyhow::ensure!(has_vergen, "disabled via `MEILI_NO_VERGEN`");
-
-    // Note: any code that needs VERGEN_ environment variables should take care to define them manually in the Dockerfile and pass them
-    // in the corresponding GitHub workflow (publish_docker.yml).
-    // This is due to the Dockerfile building the binary outside of the git directory.
-    let mut builder = vergen_git2::Git2Builder::default();
-
-    builder.branch(true);
-    builder.commit_timestamp(true);
-    builder.commit_message(true);
-    builder.describe(true, true, None);
-    builder.sha(false);
-
-    let git2 = builder.build()?;
-
-    vergen_git2::Emitter::default().fail_on_error().add_instructions(&git2)?.emit()
-}

crates/build-info/src/lib.rs

@@ -1,203 +0,0 @@
-use time::format_description::well_known::Iso8601;
-
-#[derive(Debug, Clone)]
-pub struct BuildInfo {
-    pub branch: Option<&'static str>,
-    pub describe: Option<DescribeResult>,
-    pub commit_sha1: Option<&'static str>,
-    pub commit_msg: Option<&'static str>,
-    pub commit_timestamp: Option<time::OffsetDateTime>,
-}
-
-impl BuildInfo {
-    pub fn from_build() -> Self {
-        let branch: Option<&'static str> = option_env!("VERGEN_GIT_BRANCH");
-        let describe = DescribeResult::from_build();
-        let commit_sha1 = option_env!("VERGEN_GIT_SHA");
-        let commit_msg = option_env!("VERGEN_GIT_COMMIT_MESSAGE");
-        let commit_timestamp = option_env!("VERGEN_GIT_COMMIT_TIMESTAMP");
-
-        let commit_timestamp = commit_timestamp.and_then(|commit_timestamp| {
-            time::OffsetDateTime::parse(commit_timestamp, &Iso8601::DEFAULT).ok()
-        });
-
-        Self { branch, describe, commit_sha1, commit_msg, commit_timestamp }
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
-pub enum DescribeResult {
-    Prototype { name: &'static str },
-    Release { version: &'static str, major: u64, minor: u64, patch: u64 },
-    Prerelease { version: &'static str, major: u64, minor: u64, patch: u64, rc: u64 },
-    NotATag { describe: &'static str },
-}
-
-impl DescribeResult {
-    pub fn new(describe: &'static str) -> Self {
-        if let Some(name) = prototype_name(describe) {
-            Self::Prototype { name }
-        } else if let Some(release) = release_version(describe) {
-            release
-        } else if let Some(prerelease) = prerelease_version(describe) {
-            prerelease
-        } else {
-            Self::NotATag { describe }
-        }
-    }
-
-    pub fn from_build() -> Option<Self> {
-        let describe: &'static str = option_env!("VERGEN_GIT_DESCRIBE")?;
-        Some(Self::new(describe))
-    }
-
-    pub fn as_tag(&self) -> Option<&'static str> {
-        match self {
-            DescribeResult::Prototype { name } => Some(name),
-            DescribeResult::Release { version, .. } => Some(version),
-            DescribeResult::Prerelease { version, .. } => Some(version),
-            DescribeResult::NotATag { describe: _ } => None,
-        }
-    }
-
-    pub fn as_prototype(&self) -> Option<&'static str> {
-        match self {
-            DescribeResult::Prototype { name } => Some(name),
-            DescribeResult::Release { .. }
-            | DescribeResult::Prerelease { .. }
-            | DescribeResult::NotATag { .. } => None,
-        }
-    }
-}
-
-/// Parses the input as a prototype name.
-///
-/// Returns `Some(prototype_name)` if the following conditions are met on this value:
-///
-/// 1. starts with `prototype-`,
-/// 2. ends with `-<some_number>`,
-/// 3. does not end with `<some_number>-<some_number>`.
-///
-/// Otherwise, returns `None`.
-fn prototype_name(describe: &'static str) -> Option<&'static str> {
-    if !describe.starts_with("prototype-") {
-        return None;
-    }
-
-    let mut rsplit_prototype = describe.rsplit('-');
-    // last component MUST be a number
-    rsplit_prototype.next()?.parse::<u64>().ok()?;
-    // before than last component SHALL NOT be a number
-    rsplit_prototype.next()?.parse::<u64>().err()?;
-
-    Some(describe)
-}
-
-fn release_version(describe: &'static str) -> Option<DescribeResult> {
-    if !describe.starts_with('v') {
-        return None;
-    }
-
-    // full release version don't contain a `-`
-    if describe.contains('-') {
-        return None;
-    }
-
-    // full release version parse as vX.Y.Z, with X, Y, Z numbers.
-    let mut dots = describe[1..].split('.');
-    let major: u64 = dots.next()?.parse().ok()?;
-    let minor: u64 = dots.next()?.parse().ok()?;
-    let patch: u64 = dots.next()?.parse().ok()?;
-
-    if dots.next().is_some() {
-        return None;
-    }
-
-    Some(DescribeResult::Release { version: describe, major, minor, patch })
-}
-
-fn prerelease_version(describe: &'static str) -> Option<DescribeResult> {
-    // prerelease version is in the shape vM.N.P-rc.C
-    let mut hyphen = describe.rsplit('-');
-    let prerelease = hyphen.next()?;
-    if !prerelease.starts_with("rc.") {
-        return None;
-    }
-
-    let rc: u64 = prerelease[3..].parse().ok()?;
-
-    let release = hyphen.next()?;
-
-    let DescribeResult::Release { version: _, major, minor, patch } = release_version(release)?
-    else {
-        return None;
-    };
-
-    Some(DescribeResult::Prerelease { version: describe, major, minor, patch, rc })
-}
-
-#[cfg(test)]
-mod test {
-    use super::DescribeResult;
-
-    fn assert_not_a_tag(describe: &'static str) {
-        assert_eq!(DescribeResult::NotATag { describe }, DescribeResult::new(describe))
-    }
-
-    fn assert_proto(describe: &'static str) {
-        assert_eq!(DescribeResult::Prototype { name: describe }, DescribeResult::new(describe))
-    }
-
-    fn assert_release(describe: &'static str, major: u64, minor: u64, patch: u64) {
-        assert_eq!(
-            DescribeResult::Release { version: describe, major, minor, patch },
-            DescribeResult::new(describe)
-        )
-    }
-
-    fn assert_prerelease(describe: &'static str, major: u64, minor: u64, patch: u64, rc: u64) {
-        assert_eq!(
-            DescribeResult::Prerelease { version: describe, major, minor, patch, rc },
-            DescribeResult::new(describe)
-        )
-    }
-
-    #[test]
-    fn not_a_tag() {
-        assert_not_a_tag("whatever-fuzzy");
-        assert_not_a_tag("whatever-fuzzy-5-ggg-dirty");
-        assert_not_a_tag("whatever-fuzzy-120-ggg-dirty");
-
-        // technically a tag, but not a proto nor a version, so not parsed as a tag
-        assert_not_a_tag("whatever");
-
-        // dirty version
-        assert_not_a_tag("v1.7.0-1-ggga-dirty");
-        assert_not_a_tag("v1.7.0-rc.1-1-ggga-dirty");
-
-        // after version
-        assert_not_a_tag("v1.7.0-1-ggga");
-        assert_not_a_tag("v1.7.0-rc.1-1-ggga");
-
-        // after proto
-        assert_not_a_tag("protoype-tag-0-1-ggga");
-        assert_not_a_tag("protoype-tag-0-1-ggga-dirty");
-    }
-
-    #[test]
-    fn prototype() {
-        assert_proto("prototype-tag-0");
-        assert_proto("prototype-tag-10");
-        assert_proto("prototype-long-name-tag-10");
-    }
-
-    #[test]
-    fn release() {
-        assert_release("v1.7.2", 1, 7, 2);
-    }
-
-    #[test]
-    fn prerelease() {
-        assert_prerelease("v1.7.2-rc.3", 1, 7, 2, 3);
-    }
-}

crates/dump/Cargo.toml

@@ -1,34 +0,0 @@
-[package]
-name = "dump"
-publish = false
-
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-edition.workspace = true
-homepage.workspace = true
-readme.workspace = true
-license.workspace = true
-
-[dependencies]
-anyhow = "1.0.98"
-flate2 = "1.1.2"
-http = "1.3.1"
-meilisearch-types = { path = "../meilisearch-types" }
-once_cell = "1.21.3"
-regex = "1.11.1"
-roaring = { version = "0.10.12", features = ["serde"] }
-serde = { version = "1.0.219", features = ["derive"] }
-serde_json = { version = "1.0.140", features = ["preserve_order"] }
-tar = "0.4.44"
-tempfile = "3.20.0"
-thiserror = "2.0.12"
-time = { version = "0.3.41", features = ["serde-well-known", "formatting", "parsing", "macros"] }
-tracing = "0.1.41"
-uuid = { version = "1.17.0", features = ["serde", "v4"] }
-
-[dev-dependencies]
-big_s = "1.0.2"
-maplit = "1.0.2"
-meili-snap = { path = "../meili-snap" }
-meilisearch-types = { path = "../meilisearch-types" }

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-5.snap

@@ -1,57 +0,0 @@
----
-source: crates/dump/src/reader/mod.rs
-expression: vector_index.settings().unwrap()
----
-{
-  "displayedAttributes": [
-    "*"
-  ],
-  "searchableAttributes": [
-    "*"
-  ],
-  "filterableAttributes": [],
-  "sortableAttributes": [],
-  "rankingRules": [
-    "words",
-    "typo",
-    "proximity",
-    "attribute",
-    "sort",
-    "exactness"
-  ],
-  "stopWords": [],
-  "nonSeparatorTokens": [],
-  "separatorTokens": [],
-  "dictionary": [],
-  "synonyms": {},
-  "distinctAttribute": null,
-  "proximityPrecision": "byWord",
-  "typoTolerance": {
-    "enabled": true,
-    "minWordSizeForTypos": {
-      "oneTypo": 5,
-      "twoTypos": 9
-    },
-    "disableOnWords": [],
-    "disableOnAttributes": []
-  },
-  "faceting": {
-    "maxValuesPerFacet": 100,
-    "sortFacetValuesBy": {
-      "*": "alpha"
-    }
-  },
-  "pagination": {
-    "maxTotalHits": 1000
-  },
-  "embedders": {
-    "default": {
-      "source": "huggingFace",
-      "model": "BAAI/bge-base-en-v1.5",
-      "revision": "617ca489d9e86b49b8167676d8220688b99db36e",
-      "pooling": "forceMean",
-      "documentTemplate": "{% for field in fields %} {{ field.name }}: {{ field.value }}\n{% endfor %}"
-    }
-  },
-  "searchCutoffMs": null
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-6.snap

@@ -1,783 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e3",
-  "desc": "overriden vector + map",
-  "_vectors": {
-    "default": [
-      0.2,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1
-    ],
-    "toto": [
-      0.1
-    ]
-  }
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-7.snap

@@ -1,786 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e2",
-  "desc": "natural vector + map",
-  "_vectors": {
-    "toto": [],
-    "default": {
-      "embeddings": [
-        [
-          -0.05189208313822746,
-          -0.9273212552070618,
-          0.1443813145160675,
-          0.0932632014155388,
-          0.2665371894836426,
-          0.36266782879829407,
-          0.6402910947799683,
-          0.32014018297195435,
-          0.030915971845388412,
-          -0.9312191605567932,
-          -0.3718109726905823,
-          -0.2700554132461548,
-          -1.1014580726623535,
-          0.9154956936836244,
-          -0.3406888246536255,
-          1.0077725648880005,
-          0.6577560901641846,
-          -0.3955195546150207,
-          -0.4148270785808563,
-          0.1855088472366333,
-          0.5062315464019775,
-          -0.3632686734199524,
-          -0.2277890294790268,
-          0.2560805082321167,
-          -0.3853609561920166,
-          -0.1604762226343155,
-          -0.13947471976280212,
-          -0.20147813856601715,
-          -0.4466346800327301,
-          -0.3761846721172333,
-          0.1443382054567337,
-          0.18205296993255615,
-          0.49359792470932007,
-          -0.22538000345230105,
-          -0.4996317625045776,
-          -0.22734887897968292,
-          -0.6034309267997742,
-          -0.7857939600944519,
-          -0.34923747181892395,
-          -0.3466345965862274,
-          0.21176661550998688,
-          -0.5101462006568909,
-          -0.3403083384037018,
-          0.000315118464641273,
-          0.236465722322464,
-          -0.10246097296476364,
-          -1.3013339042663574,
-          0.3419138789176941,
-          -0.32963496446609497,
-          -0.0901619717478752,
-          -0.5426247119903564,
-          0.22656650841236117,
-          -0.44758284091949463,
-          0.14151698350906372,
-          -0.1089438870549202,
-          0.5500766634941101,
-          -0.670711100101471,
-          -0.6227269768714905,
-          0.3894464075565338,
-          -0.27609574794769287,
-          0.7028202414512634,
-          -0.19697771966457367,
-          0.328511506319046,
-          0.5063360929489136,
-          0.4065195322036743,
-          0.2614171802997589,
-          -0.30274391174316406,
-          1.0393824577331543,
-          -0.7742937207221985,
-          -0.7874112129211426,
-          -0.6749666929244995,
-          0.5190866589546204,
-          0.004123548045754433,
-          -0.28312963247299194,
-          -0.038731709122657776,
-          -1.0142987966537476,
-          -0.09519586712121964,
-          0.8755272626876831,
-          0.4876938760280609,
-          0.7811151742935181,
-          0.85174959897995,
-          0.11826585978269576,
-          0.5373436808586121,
-          0.3649002015590668,
-          0.19064077734947205,
-          -0.00287026260048151,
-          -0.7305403351783752,
-          -0.015206154435873032,
-          -0.7899249196052551,
-          0.19407285749912265,
-          0.08596625179052353,
-          -0.28976231813430786,
-          -0.1525907665491104,
-          0.3798313438892365,
-          0.050306469202041626,
-          -0.5697937607765198,
-          0.4219021201133728,
-          0.276252806186676,
-          0.1559903472661972,
-          0.10030482709407806,
-          -0.4043720066547394,
-          -0.1969818025827408,
-          0.5739826560020447,
-          0.2116064727306366,
-          -1.4620544910430908,
-          -0.7802462577819824,
-          -0.24739810824394223,
-          -0.09791352599859238,
-          -0.4413802027702331,
-          0.21549351513385773,
-          -0.9520436525344848,
-          -0.08762510865926743,
-          0.08154498040676117,
-          -0.6154940724372864,
-          -1.01079523563385,
-          0.885427713394165,
-          0.6967288851737976,
-          0.27186504006385803,
-          -0.43194177746772766,
-          -0.11248451471328735,
-          0.7576630711555481,
-          0.4998855590820313,
-          0.0264343973249197,
-          0.9872855544090272,
-          0.5634694695472717,
-          0.053698331117630005,
-          0.19410227239131927,
-          0.3570743501186371,
-          -0.23670297861099243,
-          -0.9114483594894408,
-          0.07884842902421951,
-          0.7318344116210938,
-          0.44630110263824463,
-          0.08745364099740982,
-          -0.347101628780365,
-          -0.4314247667789459,
-          -0.5060274004936218,
-          0.003706763498485088,
-          0.44320008158683777,
-          -0.00788921769708395,
-          -0.1368623524904251,
-          -0.17391923069953918,
-          0.14473655819892883,
-          0.10927865654230118,
-          0.6974599361419678,
-          0.005052129738032818,
-          -0.016953065991401672,
-          -0.1256176233291626,
-          -0.036742497235536575,
-          0.5591985583305359,
-          -0.37619709968566895,
-          0.22429119050502777,
-          0.5403043031692505,
-          -0.8603790998458862,
-          -0.3456307053565979,
-          0.9292937517166138,
-          0.5074859261512756,
-          0.6310645937919617,
-          -0.3091641068458557,
-          0.46902573108673096,
-          0.7891915440559387,
-          0.4499550759792328,
-          0.2744995653629303,
-          0.2712305784225464,
-          -0.04349074140191078,
-          -0.3638863265514374,
-          0.7839881777763367,
-          0.7352104783058167,
-          -0.19457511603832245,
-          -0.5957832932472229,
-          -0.43704694509506226,
-          -1.084769368171692,
-          0.4904985725879669,
-          0.5385226011276245,
-          0.1891629993915558,
-          0.12338479608297348,
-          0.8315675258636475,
-          -0.07830192148685455,
-          1.0916285514831543,
-          -0.28066861629486084,
-          -1.3585069179534912,
-          0.5203898549079895,
-          0.08678033947944641,
-          -0.2566044330596924,
-          0.09484415501356123,
-          -0.0180208683013916,
-          1.0264745950698853,
-          -0.023572135716676712,
-          0.5864979028701782,
-          0.7625196576118469,
-          -0.2543414533138275,
-          -0.8877770900726318,
-          0.7611982822418213,
-          -0.06220436468720436,
-          0.937336564064026,
-          0.2704363465309143,
-          -0.37733694911003113,
-          0.5076137781143188,
-          -0.30641937255859375,
-          0.6252772808074951,
-          -0.0823579877614975,
-          -0.03736555948853493,
-          0.4131673276424408,
-          -0.6514252424240112,
-          0.12918265163898468,
-          -0.4483584463596344,
-          0.6750786304473877,
-          -0.37008383870124817,
-          -0.02324833907186985,
-          0.38027650117874146,
-          -0.26374951004981995,
-          0.4346931278705597,
-          0.42882832884788513,
-          -0.48798441886901855,
-          1.1882442235946655,
-          0.5132288336753845,
-          0.5284568667411804,
-          -0.03538886830210686,
-          0.29620853066444397,
-          -1.0683696269989014,
-          0.25936177372932434,
-          0.10404160618782043,
-          -0.25796034932136536,
-          0.027896970510482788,
-          -0.09225251525640488,
-          1.4811025857925415,
-          0.641173779964447,
-          -0.13838383555412292,
-          -0.3437179923057556,
-          0.5667019486427307,
-          -0.5400741696357727,
-          0.31090837717056274,
-          0.6470608115196228,
-          -0.3747067153453827,
-          -0.7364534735679626,
-          -0.07431528717279434,
-          0.5173454880714417,
-          -0.6578747034072876,
-          0.7107478976249695,
-          -0.7918999791145325,
-          -0.0648345872759819,
-          0.609937846660614,
-          -0.7329513430595398,
-          0.9741371870040894,
-          0.17912346124649048,
-          -0.02658769302070141,
-          0.5162150859832764,
-          -0.3978803157806397,
-          -0.7833885550498962,
-          -0.6497276425361633,
-          -0.3898126780986786,
-          -0.0952848568558693,
-          0.2663288116455078,
-          -0.1604052186012268,
-          0.373076468706131,
-          -0.8357769250869751,
-          -0.05217683315277099,
-          -0.2680160701274872,
-          0.8389158248901367,
-          0.6833611130714417,
-          -0.6712407469749451,
-          0.7406917214393616,
-          -0.44522786140441895,
-          -0.34645363688468933,
-          -0.27384576201438904,
-          -0.9878405928611756,
-          -0.8166060447692871,
-          0.06268279999494553,
-          0.38567957282066345,
-          -0.3274703919887543,
-          0.5296315550804138,
-          -0.11810623109340668,
-          0.23029841482639313,
-          0.08616159111261368,
-          -0.2195747196674347,
-          0.09430307894945145,
-          0.4057176411151886,
-          0.4892159104347229,
-          -0.1636916548013687,
-          -0.6071445345878601,
-          0.41256585717201233,
-          0.622254490852356,
-          -0.41223976016044617,
-          -0.6686707139015198,
-          -0.7474371790885925,
-          -0.8509522080421448,
-          -0.16754287481307983,
-          -0.9078601002693176,
-          -0.29653599858283997,
-          -0.5020652413368225,
-          0.4692700505256653,
-          0.01281109917908907,
-          -0.16071580350399017,
-          0.03388889133930206,
-          -0.020511148497462273,
-          0.5027827024459839,
-          -0.20729811489582065,
-          0.48107290267944336,
-          0.33669769763946533,
-          -0.5275911688804626,
-          0.48271527886390686,
-          0.2738940715789795,
-          -0.033152539283037186,
-          -0.13629786670207977,
-          -0.05965912342071533,
-          -0.26200807094573975,
-          0.04002794995903969,
-          -0.34095603227615356,
-          -3.986898899078369,
-          -0.46819332242012024,
-          -0.422744482755661,
-          -0.169097900390625,
-          0.6008929014205933,
-          0.058016058057546616,
-          -0.11401277780532836,
-          -0.3077819049358368,
-          -0.09595538675785063,
-          0.6723822355270386,
-          0.19367831945419312,
-          0.28304359316825867,
-          0.1609862744808197,
-          0.7567598819732666,
-          0.6889985799789429,
-          0.06907720118761063,
-          -0.04188092052936554,
-          -0.7434936165809631,
-          0.13321782648563385,
-          0.8456063270568848,
-          -0.10364038497209548,
-          -0.45084846019744873,
-          -0.4758241474628449,
-          0.43882066011428833,
-          -0.6432598829269409,
-          0.7217311859130859,
-          -0.24189773201942444,
-          0.12737572193145752,
-          -1.1008601188659668,
-          -0.3305315673351288,
-          0.14614742994308472,
-          -0.7819333076477051,
-          0.5287120342254639,
-          -0.055538054555654526,
-          0.1877404749393463,
-          -0.6907662153244019,
-          0.5616975426673889,
-          -0.4611121714115143,
-          -0.26109233498573303,
-          -0.12898315489292145,
-          -0.3724522292613983,
-          -0.7191406488418579,
-          -0.4425233602523804,
-          -0.644108235836029,
-          0.8424481153488159,
-          0.17532426118850708,
-          -0.5121750235557556,
-          -0.6467239260673523,
-          -0.0008507720194756985,
-          0.7866212129592896,
-          -0.02644744887948036,
-          -0.005045140627771616,
-          0.015782782807946205,
-          0.16334445774555206,
-          -0.1913367658853531,
-          -0.13697923719882965,
-          -0.6684983372688293,
-          0.18346354365348816,
-          -0.341105580329895,
-          0.5427411198616028,
-          0.3779832422733307,
-          -0.6778115034103394,
-          -0.2931850254535675,
-          -0.8805161714553833,
-          -0.4212774932384491,
-          -0.5368952751159668,
-          -1.3937891721725464,
-          -1.225494146347046,
-          0.4276703894138336,
-          1.1205668449401855,
-          -0.6005299687385559,
-          0.15732505917549133,
-          -0.3914784789085388,
-          -1.357046604156494,
-          -0.4707142114639282,
-          -0.1497287154197693,
-          -0.25035548210144043,
-          -0.34328439831733704,
-          0.39083412289619446,
-          0.1623048633337021,
-          -0.9275814294815063,
-          -0.6430015563964844,
-          0.2973862886428833,
-          0.5580436587333679,
-          -0.6232585310935974,
-          -0.6611042022705078,
-          0.4015969038009643,
-          -1.0232892036437988,
-          -0.2585645020008087,
-          -0.5431421399116516,
-          0.5021264553070068,
-          -0.48601630330085754,
-          -0.010242084041237833,
-          0.5862035155296326,
-          0.7316920161247253,
-          0.4036808013916016,
-          0.4269520044326782,
-          -0.705938458442688,
-          0.7747307419776917,
-          0.10164368897676468,
-          0.7887958884239197,
-          -0.9612497091293336,
-          0.12755516171455383,
-          0.06812842190265656,
-          -0.022603651508688927,
-          0.14722754061222076,
-          -0.5588505268096924,
-          -0.20689940452575684,
-          0.3557641804218292,
-          -0.6812759637832642,
-          0.2860803008079529,
-          -0.38954633474349976,
-          0.1759403496980667,
-          -0.5678874850273132,
-          -0.1692986786365509,
-          -0.14578519761562347,
-          0.5711379051208496,
-          1.0208125114440918,
-          0.7759483456611633,
-          -0.372348427772522,
-          -0.5460885763168335,
-          0.7190321683883667,
-          -0.6914990544319153,
-          0.13365162909030914,
-          -0.4854792356491089,
-          0.4054908752441406,
-          0.4502798914909363,
-          -0.3041122555732727,
-          -0.06726965308189392,
-          -0.05570871382951737,
-          -0.0455719493329525,
-          0.4785125255584717,
-          0.8867972493171692,
-          0.4107886850833893,
-          0.6121342182159424,
-          -0.20477132499217987,
-          -0.5598517656326294,
-          -0.6443566679954529,
-          -0.5905212759971619,
-          -0.5571200251579285,
-          0.17573799192905426,
-          -0.28621870279312134,
-          0.1685224026441574,
-          0.09719007462263109,
-          -0.04223639518022537,
-          -0.28623101115226746,
-          -0.1449810117483139,
-          -0.3789580464363098,
-          -0.5227636098861694,
-          -0.049728814512491226,
-          0.7849089503288269,
-          0.16792525351047516,
-          0.9849340915679932,
-          -0.6559549570083618,
-          0.35723909735679626,
-          -0.6822739243507385,
-          1.2873116731643677,
-          0.19993330538272855,
-          0.03512010723352432,
-          -0.6972134113311768,
-          0.18453484773635864,
-          -0.2437680810689926,
-          0.2156416028738022,
-          0.5230382680892944,
-          0.22020135819911957,
-          0.8314080238342285,
-          0.15627102553844452,
-          -0.7330264449119568,
-          0.3888184726238251,
-          -0.22034703195095065,
-          0.5457669496536255,
-          -0.48084837198257446,
-          -0.45576658844947815,
-          -0.09287727624177931,
-          -0.06968110054731369,
-          0.35125672817230225,
-          -0.4278119504451752,
-          0.2038476765155792,
-          0.11392722278833388,
-          0.9433983564376832,
-          -0.4097744226455689,
-          0.035297419875860214,
-          -0.4274404048919678,
-          -0.25100165605545044,
-          1.0943366289138794,
-          -0.07634022831916809,
-          -0.2925529479980469,
-          -0.7512530088424683,
-          0.2649727463722229,
-          -0.4078235328197479,
-          -0.3372223973274231,
-          0.05190162733197212,
-          0.005654910113662481,
-          -0.0001571219472680241,
-          -0.35445958375930786,
-          -0.7837416529655457,
-          0.1500556766986847,
-          0.4383024573326111,
-          0.6099548935890198,
-          0.05951934307813645,
-          -0.21325334906578064,
-          0.0199207104742527,
-          -0.22704418003559113,
-          -0.6481077671051025,
-          0.37442275881767273,
-          -1.015955924987793,
-          0.38637226819992065,
-          -0.06489371508359909,
-          -0.494120329618454,
-          0.3469836115837097,
-          0.15402406454086304,
-          -0.7660972476005554,
-          -0.7053225040435791,
-          -0.25964751839637756,
-          0.014004424214363098,
-          -0.2860170006752014,
-          -0.17565494775772095,
-          -0.45117494463920593,
-          -0.0031954257283359766,
-          0.09676837921142578,
-          -0.514464259147644,
-          0.41698193550109863,
-          -0.21642713248729703,
-          -0.5398141145706177,
-          -0.3647628426551819,
-          0.37005379796028137,
-          0.239425927400589,
-          -0.08833975344896317,
-          0.934946596622467,
-          -0.48340797424316406,
-          0.6241437792778015,
-          -0.7253676652908325,
-          -0.04303571209311485,
-          1.1125205755233765,
-          -0.15692919492721558,
-          -0.2914651036262512,
-          -0.5117168426513672,
-          0.21365483105182648,
-          0.4924402534961701,
-          0.5269662141799927,
-          0.0352792888879776,
-          -0.149167999625206,
-          -0.6019760370254517,
-          0.08245442807674408,
-          0.4900692105293274,
-          0.518824577331543,
-          -0.00005570516441366635,
-          -0.553304135799408,
-          0.22217543423175812,
-          0.5047767758369446,
-          0.135724738240242,
-          1.1511540412902832,
-          -0.3541218340396881,
-          -0.9712511897087096,
-          0.8353699445724487,
-          -0.39227569103240967,
-          -0.9117669463157654,
-          -0.26349931955337524,
-          0.05597023293375969,
-          0.20695461332798004,
-          0.3178807199001312,
-          1.0663238763809204,
-          0.5062212347984314,
-          0.7288597822189331,
-          0.09899299591779707,
-          0.553720235824585,
-          0.675009548664093,
-          -0.20067055523395536,
-          0.3138423264026642,
-          -0.6886593103408813,
-          -0.2910398542881012,
-          -1.3186300992965698,
-          -0.4684459865093231,
-          -0.095743365585804,
-          -0.1257995069026947,
-          -0.4858281314373016,
-          -0.4935407340526581,
-          -0.3266896903514862,
-          -0.3928797245025635,
-          -0.40803104639053345,
-          -0.9975396394729614,
-          0.4229583740234375,
-          0.37309643626213074,
-          0.4431034922599793,
-          0.30364808440208435,
-          -0.3765178918838501,
-          0.5616499185562134,
-          0.16904796659946442,
-          -0.7343707084655762,
-          0.2560209631919861,
-          0.6166825294494629,
-          0.3200829327106476,
-          -0.4483652710914612,
-          0.16224201023578644,
-          -0.31495288014411926,
-          -0.42713335156440735,
-          0.7270734906196594,
-          0.7049484848976135,
-          -0.0571461021900177,
-          0.04477125033736229,
-          -0.6647796034812927,
-          1.183672308921814,
-          0.36199676990509033,
-          0.046881116926670074,
-          0.4515796303749085,
-          0.9278061985969543,
-          0.31471705436706543,
-          -0.7073333859443665,
-          -0.3443860113620758,
-          0.5440067052841187,
-          -0.15020819008350372,
-          -0.541202962398529,
-          0.5203295946121216,
-          1.2192286252975464,
-          -0.9983593225479126,
-          -0.18758884072303772,
-          0.2758221924304962,
-          -0.6511523723602295,
-          -0.1584404855966568,
-          -0.236241415143013,
-          0.2692437767982483,
-          -0.4941152036190033,
-          0.4987454116344452,
-          -0.3331359028816223,
-          0.3163745701313019,
-          0.745529294013977,
-          -0.2905873656272888,
-          0.13602906465530396,
-          0.4679684340953827,
-          1.0555986166000366,
-          1.075700044631958,
-          0.5368486046791077,
-          -0.5118206739425659,
-          0.8668332099914551,
-          -0.5726966857910156,
-          -0.7811751961708069,
-          0.1938626915216446,
-          -0.1929349899291992,
-          0.1757766306400299,
-          0.6384295225143433,
-          0.26462844014167786,
-          0.9542630314826964,
-          0.19313029944896695,
-          1.264248013496399,
-          -0.6304428577423096,
-          0.0487106591463089,
-          -0.16211535036563873,
-          -0.7894763350486755,
-          0.3582514822483063,
-          -0.04153040423989296,
-          0.635784387588501,
-          0.6554391980171204,
-          -0.47010496258735657,
-          -0.8302040696144104,
-          -0.1350124627351761,
-          0.2568812072277069,
-          0.13614831864833832,
-          -0.2563649117946625,
-          -1.0434694290161133,
-          0.3232482671737671,
-          0.47882452607154846,
-          0.4298652410507202,
-          1.0563770532608032,
-          -0.28917592763900757,
-          -0.8533256649971008,
-          0.10648339986801147,
-          0.6376127004623413,
-          -0.20832888782024384,
-          0.2370245456695557,
-          0.0018312990432605147,
-          -0.2034837007522583,
-          0.01051164511591196,
-          -1.105310082435608,
-          0.29724350571632385,
-          0.15604574978351593,
-          0.1973688006401062,
-          0.44394731521606445,
-          0.3974513411521912,
-          -0.13625948131084442,
-          0.9571986198425292,
-          0.2257384955883026,
-          0.2323588728904724,
-          -0.5583669543266296,
-          -0.7854922413825989,
-          0.1647188365459442,
-          -1.6098142862319946,
-          0.318587988615036,
-          -0.13399995863437653,
-          -0.2172701060771942,
-          -0.767514705657959,
-          -0.5813586711883545,
-          -0.3195130527019501,
-          -0.04894036799669266,
-          0.2929930090904236,
-          -0.8213384747505188,
-          0.07181350141763687,
-          0.7469993829727173,
-          0.6407455801963806,
-          0.16365697979927063,
-          0.7870153188705444,
-          0.6524736881256104,
-          0.6399973630905151,
-          -0.04992736503481865,
-          -0.03959266096353531,
-          -0.2512352466583252,
-          0.8448855876922607,
-          -0.1422702670097351,
-          0.1216789186000824,
-          -1.2647287845611572,
-          0.5931149125099182,
-          0.7186052203178406,
-          -0.06118432432413101,
-          -1.1942816972732544,
-          -0.17677085101604462,
-          0.31543800234794617,
-          -0.32252824306488037,
-          0.8255583047866821,
-          -0.14529970288276672,
-          -0.2695446312427521,
-          -0.33378756046295166,
-          -0.1653425395488739,
-          0.1454019844532013,
-          -0.3920115828514099,
-          0.912214994430542,
-          -0.7279734015464783,
-          0.7374742031097412,
-          0.933980405330658,
-          0.13429680466651917,
-          -0.514870285987854,
-          0.3989711999893189,
-          -0.11613689363002776,
-          0.4022413492202759,
-          -0.9990655779838562,
-          -0.33749932050704956,
-          -0.4334589838981629,
-          -1.376373291015625,
-          -0.2993924915790558,
-          -0.09454808384180068,
-          -0.01314175222069025,
-          -0.001090060803107917,
-          0.2137461006641388,
-          0.2938512861728668,
-          0.17508235573768616,
-          0.8260607123374939,
-          -0.7218498587608337,
-          0.2414487451314926,
-          -0.47296759486198425,
-          -0.3002610504627228,
-          -1.238540768623352,
-          0.08663805574178696,
-          0.6805586218833923,
-          0.5909030437469482,
-          -0.42807504534721375,
-          -0.22887496650218964,
-          0.47537800669670105,
-          -1.0474627017974854,
-          0.6338009238243103,
-          0.06548397243022919,
-          0.4971011281013489,
-          1.3484878540039063
-        ]
-      ],
-      "regenerate": true
-    }
-  }
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-8.snap

@@ -1,785 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e1",
-  "desc": "natural vector",
-  "_vectors": {
-    "default": {
-      "embeddings": [
-        [
-          -0.2979458272457123,
-          -0.5288640856742859,
-          -0.019957859069108963,
-          -0.18495318293571472,
-          0.7429973483085632,
-          0.5238497257232666,
-          0.432366281747818,
-          0.32744166254997253,
-          0.0020762972999364138,
-          -0.9507834911346436,
-          -0.35097137093544006,
-          0.08469701558351517,
-          -1.4176613092422483,
-          0.4647577106952667,
-          -0.69340580701828,
-          1.0372896194458008,
-          0.3716741800308227,
-          0.06031008064746857,
-          -0.6152024269104004,
-          0.007914665155112743,
-          0.7954924702644348,
-          -0.20773003995418549,
-          0.09376765787601472,
-          0.04508133605122566,
-          -0.2084471583366394,
-          -0.1518009901046753,
-          0.018195509910583496,
-          -0.07044368237257004,
-          -0.18119366466999057,
-          -0.4480230510234833,
-          0.3822529911994934,
-          0.1911812424659729,
-          0.4674372375011444,
-          0.06963984668254852,
-          -0.09341949224472046,
-          0.005675444379448891,
-          -0.6774799227714539,
-          -0.7066726684570313,
-          -0.39256376028060913,
-          0.04005039855837822,
-          0.2084812968969345,
-          -0.7872875928878784,
-          -0.8205880522727966,
-          0.2919981777667999,
-          -0.06004738807678223,
-          -0.4907574355602264,
-          -1.5937862396240234,
-          0.24249385297298431,
-          -0.14709846675395966,
-          -0.11860740929841997,
-          -0.8299489617347717,
-          0.472964346408844,
-          -0.497518390417099,
-          -0.22205302119255063,
-          -0.4196169078350067,
-          0.32697558403015137,
-          -0.360930860042572,
-          -0.9789686799049376,
-          0.1887447088956833,
-          -0.403737336397171,
-          0.18524253368377688,
-          0.3768732249736786,
-          0.3666233420372009,
-          0.3511938452720642,
-          0.6985810995101929,
-          0.41721710562705994,
-          0.09754953533411026,
-          0.6204307079315186,
-          -1.0762996673583984,
-          -0.06263761967420578,
-          -0.7376511693000793,
-          0.6849768161773682,
-          -0.1745152473449707,
-          -0.40449759364128113,
-          0.20757411420345304,
-          -0.8424443006515503,
-          0.330015629529953,
-          0.3489064872264862,
-          1.0954371690750122,
-          0.8487558960914612,
-          1.1076823472976685,
-          0.61430823802948,
-          0.4155903458595276,
-          0.4111340939998626,
-          0.05753209814429283,
-          -0.06429877132177353,
-          -0.765606164932251,
-          -0.41703930497169495,
-          -0.508820652961731,
-          0.19859947264194489,
-          -0.16607828438282013,
-          -0.28112146258354187,
-          0.11032675206661224,
-          0.38809511065483093,
-          -0.36498191952705383,
-          -0.48671194911003113,
-          0.6755134463310242,
-          0.03958442434668541,
-          0.4478721618652344,
-          -0.10335399955511092,
-          -0.9546685814857484,
-          -0.6087718605995178,
-          0.17498846352100372,
-          0.08320838958024979,
-          -1.4478336572647097,
-          -0.605027437210083,
-          -0.5867993235588074,
-          -0.14711688458919525,
-          -0.5447602272033691,
-          -0.026259321719408035,
-          -0.6997418403625488,
-          -0.07349082082509995,
-          0.10638900846242905,
-          -0.7133527398109436,
-          -0.9396815299987792,
-          1.087092399597168,
-          1.1885089874267578,
-          0.4011896848678589,
-          -0.4089202582836151,
-          -0.10938972979784012,
-          0.6726722121238708,
-          0.24576938152313232,
-          -0.24247920513153076,
-          1.1499971151351929,
-          0.47813335061073303,
-          -0.05331678315997124,
-          0.32338133454322815,
-          0.4870913326740265,
-          -0.23144258558750153,
-          -1.2023426294326782,
-          0.2349330335855484,
-          1.080536961555481,
-          0.29334118962287903,
-          0.391574501991272,
-          -0.15818795561790466,
-          -0.2948290705680847,
-          -0.024689948186278343,
-          0.06602869182825089,
-          0.5937030911445618,
-          -0.047901444137096405,
-          -0.512734591960907,
-          -0.35780075192451477,
-          0.28751692175865173,
-          0.4298716187477112,
-          0.9242428541183472,
-          -0.17208744585514069,
-          0.11515070497989656,
-          -0.0335976779460907,
-          -0.3422986567020416,
-          0.5344581604003906,
-          0.19895796477794647,
-          0.33001241087913513,
-          0.6390730142593384,
-          -0.6074934005737305,
-          -0.2553696632385254,
-          0.9644920229911804,
-          0.2699219584465027,
-          0.6403993368148804,
-          -0.6380003690719604,
-          -0.027310986071825027,
-          0.638815701007843,
-          0.27719101309776306,
-          -0.13553589582443237,
-          0.750195324420929,
-          0.1224869191646576,
-          -0.20613941550254825,
-          0.8444448709487915,
-          0.16200250387191772,
-          -0.24750925600528717,
-          -0.739950954914093,
-          -0.28443849086761475,
-          -1.176282525062561,
-          0.516107976436615,
-          0.3774825632572174,
-          0.10906043648719788,
-          0.07962015271186829,
-          0.7384604215621948,
-          -0.051241904497146606,
-          1.1730090379714966,
-          -0.4828610122203827,
-          -1.404372215270996,
-          0.8811132311820984,
-          -0.3839482367038727,
-          0.022516896948218346,
-          -0.0491158664226532,
-          -0.43027013540267944,
-          1.2049334049224854,
-          -0.27309560775756836,
-          0.6883630752563477,
-          0.8264574408531189,
-          -0.5020735263824463,
-          -0.4874092042446137,
-          0.6007202863693237,
-          -0.4965405762195587,
-          1.1302915811538696,
-          0.032572727650403976,
-          -0.3731859028339386,
-          0.658271849155426,
-          -0.9023059010505676,
-          0.7400162220001221,
-          0.014550759457051754,
-          -0.19699542224407196,
-          0.2319706380367279,
-          -0.789058268070221,
-          -0.14905710518360138,
-          -0.5826214551925659,
-          0.207652747631073,
-          -0.4507439732551574,
-          -0.3163885474205017,
-          0.3604124188423157,
-          -0.45119962096214294,
-          0.3428427278995514,
-          0.3005594313144684,
-          -0.36026081442832947,
-          1.1014249324798584,
-          0.40884315967559814,
-          0.34991952776908875,
-          -0.1806638240814209,
-          0.27440476417541504,
-          -0.7118373513221741,
-          0.4645499587059021,
-          0.214790478348732,
-          -0.2343102991580963,
-          0.10500429570674896,
-          -0.28034430742263794,
-          1.2267805337905884,
-          1.0561333894729614,
-          -0.497364342212677,
-          -0.6143305897712708,
-          0.24963727593421936,
-          -0.33136463165283203,
-          -0.01473914459347725,
-          0.495918869972229,
-          -0.6985538005828857,
-          -1.0033197402954102,
-          0.35937801003456116,
-          0.6325868368148804,
-          -0.6808838844299316,
-          1.0354058742523191,
-          -0.7214401960372925,
-          -0.33318862318992615,
-          0.874398410320282,
-          -0.6594992280006409,
-          0.6830640435218811,
-          -0.18534131348133087,
-          0.024834271520376205,
-          0.19901277124881744,
-          -0.5992477536201477,
-          -1.2126628160476685,
-          -0.9245557188987732,
-          -0.3898217976093292,
-          -0.1286519467830658,
-          0.4217943847179413,
-          -0.1143646091222763,
-          0.5630772709846497,
-          -0.5240639448165894,
-          0.21152715384960177,
-          -0.3792001008987427,
-          0.8266305327415466,
-          1.170984387397766,
-          -0.8072142004966736,
-          0.11382893472909927,
-          -0.17953898012638092,
-          -0.1789460331201553,
-          -0.15078622102737427,
-          -1.2082908153533936,
-          -0.7812382578849792,
-          -0.10903695970773696,
-          0.7303897142410278,
-          -0.39054441452026367,
-          0.19511254131793976,
-          -0.09121843427419662,
-          0.22400228679180145,
-          0.30143046379089355,
-          0.1141919493675232,
-          0.48112115263938904,
-          0.7307931780815125,
-          0.09701362252235413,
-          -0.2795647978782654,
-          -0.3997688889503479,
-          0.5540812611579895,
-          0.564578115940094,
-          -0.40065160393714905,
-          -0.3629159033298493,
-          -0.3789091110229492,
-          -0.7298538088798523,
-          -0.6996853351593018,
-          -0.4477842152118683,
-          -0.289089560508728,
-          -0.6430277824401855,
-          0.2344944179058075,
-          0.3742927014827728,
-          -0.5079357028007507,
-          0.28841453790664673,
-          0.06515737622976303,
-          0.707315981388092,
-          0.09498685598373412,
-          0.8365515470504761,
-          0.10002726316452026,
-          -0.7695478200912476,
-          0.6264724135398865,
-          0.7562043070793152,
-          -0.23112858831882477,
-          -0.2871039807796478,
-          -0.25010058283805847,
-          0.2783474028110504,
-          -0.03224996477365494,
-          -0.9119359850883484,
-          -3.6940200328826904,
-          -0.5099936127662659,
-          -0.1604711413383484,
-          0.17453284561634064,
-          0.41759559512138367,
-          0.1419190913438797,
-          -0.11362407356500626,
-          -0.33312007784843445,
-          0.11511333286762238,
-          0.4667884409427643,
-          -0.0031647447030991316,
-          0.15879854559898376,
-          0.3042248487472534,
-          0.5404849052429199,
-          0.8515422344207764,
-          0.06286454200744629,
-          0.43790125846862793,
-          -0.8682025074958801,
-          -0.06363756954669952,
-          0.5547921657562256,
-          -0.01483887154608965,
-          -0.07361344993114471,
-          -0.929947018623352,
-          0.3502565622329712,
-          -0.5080993175506592,
-          1.0380364656448364,
-          -0.2017953395843506,
-          0.21319580078125,
-          -1.0763001441955566,
-          -0.556368887424469,
-          0.1949922740459442,
-          -0.6445739269256592,
-          0.6791343688964844,
-          0.21188358962535855,
-          0.3736183941364288,
-          -0.21800459921360016,
-          0.7597446441650391,
-          -0.3732394874095917,
-          -0.4710160195827484,
-          0.025146087631583217,
-          0.05341297015547752,
-          -0.9522109627723694,
-          -0.6000866889953613,
-          -0.08469046652317047,
-          0.5966026186943054,
-          0.3444081246852875,
-          -0.461188405752182,
-          -0.5279349088668823,
-          0.10296865552663804,
-          0.5175143480300903,
-          -0.20671147108078003,
-          0.13392412662506104,
-          0.4812754988670349,
-          0.2993808686733246,
-          -0.3005635440349579,
-          0.5141698122024536,
-          -0.6239235401153564,
-          0.2877119481563568,
-          -0.4452739953994751,
-          0.5621107816696167,
-          0.5047508478164673,
-          -0.4226335883140564,
-          -0.18578553199768064,
-          -1.1967322826385498,
-          0.28178197145462036,
-          -0.8692031502723694,
-          -1.1812998056411743,
-          -1.4526212215423584,
-          0.4645712077617645,
-          0.9327932000160216,
-          -0.6560136675834656,
-          0.461549699306488,
-          -0.5621527433395386,
-          -1.328449010848999,
-          -0.08676894754171371,
-          0.00021918353741057217,
-          -0.18864136934280396,
-          0.1259666532278061,
-          0.18240638077259064,
-          -0.14919660985469818,
-          -0.8965857625007629,
-          -0.7539900541305542,
-          0.013973715715110302,
-          0.504276692867279,
-          -0.704748272895813,
-          -0.6428424119949341,
-          0.6303996443748474,
-          -0.5404738187789917,
-          -0.31176653504371643,
-          -0.21262824535369873,
-          0.18736739456653595,
-          -0.7998970746994019,
-          0.039946746081113815,
-          0.7390344738960266,
-          0.4283199906349182,
-          0.3795057237148285,
-          0.07204607129096985,
-          -0.9230587482452391,
-          0.9440426230430604,
-          0.26272690296173096,
-          0.5598306655883789,
-          -1.0520871877670288,
-          -0.2677186131477356,
-          -0.1888762265443802,
-          0.30426350235939026,
-          0.4746131896972656,
-          -0.5746733546257019,
-          -0.4197768568992615,
-          0.8565112948417664,
-          -0.6767723560333252,
-          0.23448683321475983,
-          -0.2010004222393036,
-          0.4112907350063324,
-          -0.6497949957847595,
-          -0.418667733669281,
-          -0.4950824975967407,
-          0.44438859820365906,
-          1.026281714439392,
-          0.482397586107254,
-          -0.26220494508743286,
-          -0.3640787005424499,
-          0.5907743573188782,
-          -0.8771642446517944,
-          0.09708411991596222,
-          -0.3671700060367584,
-          0.4331349730491638,
-          0.619417667388916,
-          -0.2684665620326996,
-          -0.5123821496963501,
-          -0.1502324342727661,
-          -0.012190685607492924,
-          0.3580845892429352,
-          0.8617186546325684,
-          0.3493645489215851,
-          1.0270192623138428,
-          0.18297909200191495,
-          -0.5881339311599731,
-          -0.1733516901731491,
-          -0.5040576457977295,
-          -0.340370237827301,
-          -0.26767754554748535,
-          -0.28570041060447693,
-          -0.032928116619586945,
-          0.6029254794120789,
-          0.17397655546665192,
-          0.09346921741962431,
-          0.27815181016921997,
-          -0.46699589490890503,
-          -0.8148876428604126,
-          -0.3964351713657379,
-          0.3812595009803772,
-          0.13547226786613464,
-          0.7126688361167908,
-          -0.3473474085330963,
-          -0.06573959439992905,
-          -0.6483767032623291,
-          1.4808889627456665,
-          0.30924928188323975,
-          -0.5085946917533875,
-          -0.8613000512123108,
-          0.3048902451992035,
-          -0.4241599142551422,
-          0.15909206867218018,
-          0.5764641761779785,
-          -0.07879110425710678,
-          1.015336513519287,
-          0.07599356025457382,
-          -0.7025855779647827,
-          0.30047643184661865,
-          -0.35094937682151794,
-          0.2522146999835968,
-          -0.2338722199201584,
-          -0.8326804637908936,
-          -0.13695412874221802,
-          -0.03452421352267265,
-          0.47974953055381775,
-          -0.18385636806488037,
-          0.32438594102859497,
-          0.1797013282775879,
-          0.787494957447052,
-          -0.12579888105392456,
-          -0.07507286965847015,
-          -0.4389670491218567,
-          0.2720070779323578,
-          0.8138866424560547,
-          0.01974171027541161,
-          -0.3057698905467987,
-          -0.6709924936294556,
-          0.0885881632566452,
-          -0.2862754464149475,
-          0.03475658595561981,
-          -0.1285519152879715,
-          0.3838353455066681,
-          -0.2944154739379883,
-          -0.4204859137535095,
-          -0.4416137933731079,
-          0.13426260650157928,
-          0.36733248829841614,
-          0.573428750038147,
-          -0.14928072690963745,
-          -0.026076916605234143,
-          0.33286052942276,
-          -0.5340145826339722,
-          -0.17279052734375,
-          -0.01154550164937973,
-          -0.6620771884918213,
-          0.18390542268753052,
-          -0.08265615254640579,
-          -0.2489682286977768,
-          0.2429984211921692,
-          -0.044153645634651184,
-          -0.986578404903412,
-          -0.33574509620666504,
-          -0.5387663841247559,
-          0.19767941534519196,
-          0.12540718913078308,
-          -0.3403128981590271,
-          -0.4154576361179352,
-          0.17275673151016235,
-          0.09407442808151244,
-          -0.5414086580276489,
-          0.4393929839134216,
-          0.1725579798221588,
-          -0.4998118281364441,
-          -0.6926208138465881,
-          0.16552448272705078,
-          0.6659538149833679,
-          -0.10949844866991044,
-          0.986426830291748,
-          0.01748848147690296,
-          0.4003709554672241,
-          -0.5430638194084167,
-          0.35347291827201843,
-          0.6887399554252625,
-          0.08274628221988678,
-          0.13407137989997864,
-          -0.591465950012207,
-          0.3446292281150818,
-          0.6069018244743347,
-          0.1935492902994156,
-          -0.0989871397614479,
-          0.07008486241102219,
-          -0.8503749370574951,
-          -0.09507356584072112,
-          0.6259510517120361,
-          0.13934025168418884,
-          0.06392545253038406,
-          -0.4112265408039093,
-          -0.08475656062364578,
-          0.4974113404750824,
-          -0.30606114864349365,
-          1.111435890197754,
-          -0.018766529858112335,
-          -0.8422622680664063,
-          0.4325508773326874,
-          -0.2832120656967163,
-          -0.4859798848628998,
-          -0.41498348116874695,
-          0.015977520495653152,
-          0.5292825698852539,
-          0.4538311660289765,
-          1.1328668594360352,
-          0.22632671892642975,
-          0.7918671369552612,
-          0.33401933312416077,
-          0.7306135296821594,
-          0.3548600673675537,
-          0.12506209313869476,
-          0.8573207855224609,
-          -0.5818327069282532,
-          -0.6953738927841187,
-          -1.6171947717666626,
-          -0.1699674427509308,
-          0.6318262815475464,
-          -0.05671752244234085,
-          -0.28145185112953186,
-          -0.3976689279079437,
-          -0.2041076272726059,
-          -0.5495951175689697,
-          -0.5152917504310608,
-          -0.9309796094894408,
-          0.101932130753994,
-          0.1367802917957306,
-          0.1490798443555832,
-          0.5304336547851563,
-          -0.5082434415817261,
-          0.06688683480024338,
-          0.14657628536224365,
-          -0.782435953617096,
-          0.2962816655635834,
-          0.6965363621711731,
-          0.8496337532997131,
-          -0.3042965829372406,
-          0.04343798756599426,
-          0.0330701619386673,
-          -0.5662598013877869,
-          1.1086925268173218,
-          0.756072998046875,
-          -0.204134538769722,
-          0.2404300570487976,
-          -0.47848284244537354,
-          1.3659011125564575,
-          0.5645433068275452,
-          -0.15836156904697418,
-          0.43395575881004333,
-          0.5944653749465942,
-          1.0043466091156006,
-          -0.49446743726730347,
-          -0.5954391360282898,
-          0.5341240763664246,
-          0.020598189905285835,
-          -0.4036853015422821,
-          0.4473709762096405,
-          1.1998231410980225,
-          -0.9317775368690492,
-          -0.23321466147899628,
-          0.2052552700042725,
-          -0.7423108816146851,
-          -0.19917210936546328,
-          -0.1722569614648819,
-          -0.034072667360305786,
-          -0.00671181408688426,
-          0.46396249532699585,
-          -0.1372445821762085,
-          0.053376372903585434,
-          0.7392690777778625,
-          -0.38447609543800354,
-          0.07497968524694443,
-          0.5197252631187439,
-          1.3746477365493774,
-          0.9060075879096984,
-          0.20000585913658145,
-          -0.4053704142570496,
-          0.7497360110282898,
-          -0.34087055921554565,
-          -1.101803183555603,
-          0.273650586605072,
-          -0.5125769376754761,
-          0.22472351789474487,
-          0.480757474899292,
-          -0.19845178723335263,
-          0.8857700824737549,
-          0.30752456188201904,
-          1.1109285354614258,
-          -0.6768012642860413,
-          0.524367094039917,
-          -0.22495046257972717,
-          -0.4224412739276886,
-          0.40753406286239624,
-          -0.23133376240730288,
-          0.3297771215438843,
-          0.4905449151992798,
-          -0.6813114285469055,
-          -0.7543983459472656,
-          -0.5599071383476257,
-          0.14351597428321838,
-          -0.029278717935085297,
-          -0.3970443606376648,
-          -0.303079217672348,
-          0.24161772429943085,
-          0.008353390730917454,
-          -0.0062365154735744,
-          1.0824860334396362,
-          -0.3704061508178711,
-          -1.0337258577346802,
-          0.04638749733567238,
-          1.163011074066162,
-          -0.31737643480300903,
-          0.013986887410283089,
-          0.19223114848136905,
-          -0.2260770797729492,
-          -0.210910826921463,
-          -1.0191949605941772,
-          0.22356095910072327,
-          0.09353553503751756,
-          0.18096882104873657,
-          0.14867214858531952,
-          0.43408671021461487,
-          -0.33312076330184937,
-          0.8173948526382446,
-          0.6428242921829224,
-          0.20215003192424777,
-          -0.6634518504142761,
-          -0.4132290482521057,
-          0.29815030097961426,
-          -1.579406976699829,
-          -0.0981958732008934,
-          -0.03941014781594277,
-          0.1709178239107132,
-          -0.5481140613555908,
-          -0.5338194966316223,
-          -0.3528362512588501,
-          -0.11561278253793716,
-          -0.21793591976165771,
-          -1.1570470333099363,
-          0.2157980799674988,
-          0.42083489894866943,
-          0.9639263153076172,
-          0.09747201204299928,
-          0.15671424567699432,
-          0.4034591615200043,
-          0.6728067994117737,
-          -0.5216875672340393,
-          0.09657668322324751,
-          -0.2416689097881317,
-          0.747975766658783,
-          0.1021689772605896,
-          0.11652665585279463,
-          -1.0484966039657593,
-          0.8489304780960083,
-          0.7169828414916992,
-          -0.09012343734502792,
-          -1.3173753023147583,
-          0.057890523225069046,
-          -0.006231260951608419,
-          -0.1018214002251625,
-          0.936040461063385,
-          -0.0502331368625164,
-          -0.4284322261810303,
-          -0.38209280371665955,
-          -0.22668412327766416,
-          0.0782942995429039,
-          -0.4881664514541626,
-          0.9268959760665894,
-          0.001867273123934865,
-          0.42261114716529846,
-          0.8283362984657288,
-          0.4256294071674347,
-          -0.7965338826179504,
-          0.4840078353881836,
-          -0.19861412048339844,
-          0.33977967500686646,
-          -0.4604192078113556,
-          -0.3107339143753052,
-          -0.2839638590812683,
-          -1.5734281539916992,
-          0.005220232997089624,
-          0.09239906817674635,
-          -0.7828494906425476,
-          -0.1397123783826828,
-          0.2576255202293396,
-          0.21372435986995697,
-          -0.23169949650764465,
-          0.4016408920288086,
-          -0.462497353553772,
-          -0.2186472862958908,
-          -0.5617868900299072,
-          -0.3649831712245941,
-          -1.1585862636566162,
-          -0.08222806453704834,
-          0.931126832962036,
-          0.4327389597892761,
-          -0.46451422572135925,
-          -0.5430706143379211,
-          -0.27434298396110535,
-          -0.9479129314422609,
-          0.1845661848783493,
-          0.3972720205783844,
-          0.4883299469947815,
-          1.04031240940094
-        ]
-      ],
-      "regenerate": true
-    }
-  }
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-9.snap

@@ -1,780 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e0",
-  "desc": "overriden vector",
-  "_vectors": {
-    "default": [
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1
-    ]
-  }
-}

crates/filter-parser/src/condition.rs

@@ -1,257 +0,0 @@
-//! BNF grammar:
-//!
-//! ```text
-//! condition      = value ("==" | ">" ...) value
-//! to             = value value TO value
-//! ```
-
-use nom::branch::alt;
-use nom::bytes::complete::tag;
-use nom::character::complete::{char, multispace0, multispace1};
-use nom::combinator::{cut, map, value};
-use nom::sequence::{preceded, terminated, tuple};
-use Condition::*;
-
-use crate::error::IResultExt;
-use crate::value::{parse_vector_value, parse_vector_value_cut};
-use crate::{parse_value, Error, ErrorKind, FilterCondition, IResult, Span, Token, VectorFilter};
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub enum Condition<'a> {
-    GreaterThan(Token<'a>),
-    GreaterThanOrEqual(Token<'a>),
-    Equal(Token<'a>),
-    NotEqual(Token<'a>),
-    Null,
-    Empty,
-    Exists,
-    LowerThan(Token<'a>),
-    LowerThanOrEqual(Token<'a>),
-    Between { from: Token<'a>, to: Token<'a> },
-    Contains { keyword: Token<'a>, word: Token<'a> },
-    StartsWith { keyword: Token<'a>, word: Token<'a> },
-}
-
-impl Condition<'_> {
-    pub fn operator(&self) -> &str {
-        match self {
-            Condition::GreaterThan(_) => ">",
-            Condition::GreaterThanOrEqual(_) => ">=",
-            Condition::Equal(_) => "=",
-            Condition::NotEqual(_) => "!=",
-            Condition::Null => "IS NULL",
-            Condition::Empty => "IS EMPTY",
-            Condition::Exists => "EXISTS",
-            Condition::LowerThan(_) => "<",
-            Condition::LowerThanOrEqual(_) => "<=",
-            Condition::Between { .. } => "TO",
-            Condition::Contains { .. } => "CONTAINS",
-            Condition::StartsWith { .. } => "STARTS WITH",
-        }
-    }
-}
-
-/// condition      = value ("==" | ">" ...) value
-pub fn parse_condition(input: Span) -> IResult<FilterCondition> {
-    let operator = alt((tag("<="), tag(">="), tag("!="), tag("<"), tag(">"), tag("=")));
-    let (input, (fid, op, value)) = tuple((parse_value, operator, cut(parse_value)))(input)?;
-
-    let condition = match *op.fragment() {
-        "<=" => FilterCondition::Condition { fid, op: LowerThanOrEqual(value) },
-        ">=" => FilterCondition::Condition { fid, op: GreaterThanOrEqual(value) },
-        "!=" => FilterCondition::Condition { fid, op: NotEqual(value) },
-        "<" => FilterCondition::Condition { fid, op: LowerThan(value) },
-        ">" => FilterCondition::Condition { fid, op: GreaterThan(value) },
-        "=" => FilterCondition::Condition { fid, op: Equal(value) },
-        _ => unreachable!(),
-    };
-
-    Ok((input, condition))
-}
-
-/// null          = value "IS" WS+ "NULL"
-pub fn parse_is_null(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("NULL")))(input)?;
-    Ok((input, FilterCondition::Condition { fid: key, op: Null }))
-}
-
-/// null          = value "IS" WS+ "NOT" WS+ "NULL"
-pub fn parse_is_not_null(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("NOT"), multispace1, tag("NULL")))(input)?;
-    Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Null }))))
-}
-
-/// empty          = value "IS" WS+ "EMPTY"
-pub fn parse_is_empty(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("EMPTY")))(input)?;
-    Ok((input, FilterCondition::Condition { fid: key, op: Empty }))
-}
-
-/// empty          = value "IS" WS+ "NOT" WS+ "EMPTY"
-pub fn parse_is_not_empty(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("NOT"), multispace1, tag("EMPTY")))(input)?;
-    Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Empty }))))
-}
-
-/// exist          = value "EXISTS"
-pub fn parse_exists(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = terminated(parse_value, tag("EXISTS"))(input)?;
-
-    Ok((input, FilterCondition::Condition { fid: key, op: Exists }))
-}
-/// exist          = value "NOT" WS+ "EXISTS"
-pub fn parse_not_exists(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("NOT"), multispace1, tag("EXISTS")))(input)?;
-    Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Exists }))))
-}
-
-fn parse_vectors(input: Span) -> IResult<(Token, Option<Token>, VectorFilter)> {
-    let (input, _) = multispace0(input)?;
-    let (input, fid) = tag("_vectors")(input)?;
-
-    if let Ok((input, _)) = multispace1::<_, crate::Error>(input) {
-        return Ok((input, (Token::from(fid), None, VectorFilter::None)));
-    }
-
-    let (input, _) = char('.')(input)?;
-
-    // From this point, we are certain this is a vector filter, so our errors must be final.
-    // We could use nom's `cut` but it's better to be explicit about the errors
-
-    if let Ok((_, space)) = tag::<_, _, ()>(" ")(input) {
-        return Err(crate::Error::failure_from_kind(space, ErrorKind::VectorFilterMissingEmbedder));
-    }
-
-    let (input, embedder_name) =
-        parse_vector_value_cut(input, ErrorKind::VectorFilterInvalidEmbedder)?;
-
-    let (input, filter) = alt((
-        map(
-            preceded(tag(".fragments"), |input| {
-                let (input, _) = tag(".")(input).map_cut(ErrorKind::VectorFilterMissingFragment)?;
-                parse_vector_value_cut(input, ErrorKind::VectorFilterInvalidFragment)
-            }),
-            VectorFilter::Fragment,
-        ),
-        value(VectorFilter::UserProvided, tag(".userProvided")),
-        value(VectorFilter::DocumentTemplate, tag(".documentTemplate")),
-        value(VectorFilter::Regenerate, tag(".regenerate")),
-        value(VectorFilter::None, nom::combinator::success("")),
-    ))(input)?;
-
-    if let Ok((input, point)) = tag::<_, _, ()>(".")(input) {
-        let opt_value = parse_vector_value(input).ok().map(|(_, v)| v);
-        let value =
-            opt_value.as_ref().map(|v| v.value().to_owned()).unwrap_or_else(|| point.to_string());
-        let context = opt_value.map(|v| v.original_span()).unwrap_or(point);
-        let previous_kind = match filter {
-            VectorFilter::Fragment(_) => Some("fragments"),
-            VectorFilter::DocumentTemplate => Some("documentTemplate"),
-            VectorFilter::UserProvided => Some("userProvided"),
-            VectorFilter::Regenerate => Some("regenerate"),
-            VectorFilter::None => None,
-        };
-        return Err(Error::failure_from_kind(
-            context,
-            ErrorKind::VectorFilterUnknownSuffix(previous_kind, value),
-        ));
-    }
-
-    let (input, _) = multispace1(input).map_cut(ErrorKind::VectorFilterLeftover)?;
-
-    Ok((input, (Token::from(fid), Some(embedder_name), filter)))
-}
-
-/// vectors_exists          = vectors ("EXISTS" | ("NOT" WS+ "EXISTS"))
-pub fn parse_vectors_exists(input: Span) -> IResult<FilterCondition> {
-    let (input, (fid, embedder, filter)) = parse_vectors(input)?;
-
-    // Try parsing "EXISTS" first
-    if let Ok((input, _)) = tag::<_, _, ()>("EXISTS")(input) {
-        return Ok((input, FilterCondition::VectorExists { fid, embedder, filter }));
-    }
-
-    // Try parsing "NOT EXISTS"
-    if let Ok((input, _)) = tuple::<_, _, (), _>((tag("NOT"), multispace1, tag("EXISTS")))(input) {
-        return Ok((
-            input,
-            FilterCondition::Not(Box::new(FilterCondition::VectorExists { fid, embedder, filter })),
-        ));
-    }
-
-    Err(crate::Error::failure_from_kind(input, ErrorKind::VectorFilterOperation))
-}
-
-/// contains        = value "CONTAINS" value
-pub fn parse_contains(input: Span) -> IResult<FilterCondition> {
-    let (input, (fid, contains, value)) =
-        tuple((parse_value, tag("CONTAINS"), cut(parse_value)))(input)?;
-    Ok((
-        input,
-        FilterCondition::Condition {
-            fid,
-            op: Contains { keyword: Token { span: contains, value: None }, word: value },
-        },
-    ))
-}
-
-/// contains        = value "NOT" WS+ "CONTAINS" value
-pub fn parse_not_contains(input: Span) -> IResult<FilterCondition> {
-    let keyword = tuple((tag("NOT"), multispace1, tag("CONTAINS")));
-    let (input, (fid, (_not, _spaces, contains), value)) =
-        tuple((parse_value, keyword, cut(parse_value)))(input)?;
-
-    Ok((
-        input,
-        FilterCondition::Not(Box::new(FilterCondition::Condition {
-            fid,
-            op: Contains { keyword: Token { span: contains, value: None }, word: value },
-        })),
-    ))
-}
-
-/// starts with        = value "CONTAINS" value
-pub fn parse_starts_with(input: Span) -> IResult<FilterCondition> {
-    let (input, (fid, starts_with, value)) =
-        tuple((parse_value, tag("STARTS WITH"), cut(parse_value)))(input)?;
-    Ok((
-        input,
-        FilterCondition::Condition {
-            fid,
-            op: StartsWith { keyword: Token { span: starts_with, value: None }, word: value },
-        },
-    ))
-}
-
-/// starts with        = value "NOT" WS+ "CONTAINS" value
-pub fn parse_not_starts_with(input: Span) -> IResult<FilterCondition> {
-    let keyword = tuple((tag("NOT"), multispace1, tag("STARTS WITH")));
-    let (input, (fid, (_not, _spaces, starts_with), value)) =
-        tuple((parse_value, keyword, cut(parse_value)))(input)?;
-
-    Ok((
-        input,
-        FilterCondition::Not(Box::new(FilterCondition::Condition {
-            fid,
-            op: StartsWith { keyword: Token { span: starts_with, value: None }, word: value },
-        })),
-    ))
-}
-
-/// to             = value value "TO" WS+ value
-pub fn parse_to(input: Span) -> IResult<FilterCondition> {
-    let (input, (key, from, _, _, to)) =
-        tuple((parse_value, parse_value, tag("TO"), multispace1, cut(parse_value)))(input)?;
-
-    Ok((input, FilterCondition::Condition { fid: key, op: Between { from, to } }))
-}

crates/fuzzers/Cargo.toml

@@ -1,22 +0,0 @@
-[package]
-name = "fuzzers"
-publish = false
-
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-homepage.workspace = true
-readme.workspace = true
-edition.workspace = true
-license.workspace = true
-
-[dependencies]
-arbitrary = { version = "1.4.1", features = ["derive"] }
-bumpalo = "3.18.1"
-clap = { version = "4.5.40", features = ["derive"] }
-either = "1.15.0"
-fastrand = "2.3.0"
-milli = { path = "../milli" }
-serde = { version = "1.0.219", features = ["derive"] }
-serde_json = { version = "1.0.140", features = ["preserve_order"] }
-tempfile = "3.20.0"

crates/index-scheduler/Cargo.toml

@@ -1,60 +0,0 @@
-[package]
-name = "index-scheduler"
-publish = false
-
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-homepage.workspace = true
-readme.workspace = true
-edition.workspace = true
-license.workspace = true
-
-[dependencies]
-anyhow = "1.0.98"
-bincode = "1.3.3"
-byte-unit = "5.1.6"
-bytes = "1.10.1"
-bumpalo = "3.18.1"
-bumparaw-collections = "0.1.4"
-convert_case = "0.8.0"
-csv = "1.3.1"
-derive_builder = "0.20.2"
-dump = { path = "../dump" }
-enum-iterator = "2.1.0"
-file-store = { path = "../file-store" }
-flate2 = "1.1.2"
-indexmap = "2.9.0"
-meilisearch-auth = { path = "../meilisearch-auth" }
-meilisearch-types = { path = "../meilisearch-types" }
-memmap2 = "0.9.7"
-page_size = "0.6.0"
-rayon = "1.10.0"
-roaring = { version = "0.10.12", features = ["serde"] }
-serde = { version = "1.0.219", features = ["derive"] }
-serde_json = { version = "1.0.140", features = ["preserve_order"] }
-tar = "0.4.44"
-synchronoise = "1.0.1"
-tempfile = "3.20.0"
-thiserror = "2.0.12"
-time = { version = "0.3.41", features = [
-    "serde-well-known",
-    "formatting",
-    "parsing",
-    "macros",
-] }
-tracing = "0.1.41"
-ureq = "2.12.1"
-uuid = { version = "1.17.0", features = ["serde", "v4"] }
-backoff = "0.4.0"
-reqwest = { version = "0.12.23", features = ["rustls-tls", "http2"], default-features = false }
-rusty-s3 = "0.8.1"
-tokio = { version = "1.47.1", features = ["full"] }
-
-[dev-dependencies]
-big_s = "1.0.2"
-crossbeam-channel = "0.5.15"
-# fixed version due to format breakages in v1.40
-insta = { version = "=1.39.0", features = ["json", "redactions"] }
-maplit = "1.0.2"
-meili-snap = { path = "../meili-snap" }

crates/index-scheduler/src/dump.rs

@@ -1,330 +0,0 @@
-#![allow(clippy::result_large_err)]
-
-use std::collections::HashMap;
-use std::io;
-
-use dump::{KindDump, TaskDump, UpdateFile};
-use meilisearch_types::batches::{Batch, BatchId};
-use meilisearch_types::heed::RwTxn;
-use meilisearch_types::index_uid_pattern::IndexUidPattern;
-use meilisearch_types::milli;
-use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
-use roaring::RoaringBitmap;
-use uuid::Uuid;
-
-use crate::{utils, Error, IndexScheduler, Result};
-
-pub struct Dump<'a> {
-    index_scheduler: &'a IndexScheduler,
-    wtxn: RwTxn<'a>,
-
-    batch_to_task_mapping: HashMap<BatchId, RoaringBitmap>,
-
-    indexes: HashMap<String, RoaringBitmap>,
-    statuses: HashMap<Status, RoaringBitmap>,
-    kinds: HashMap<Kind, RoaringBitmap>,
-
-    batch_indexes: HashMap<String, RoaringBitmap>,
-    batch_statuses: HashMap<Status, RoaringBitmap>,
-    batch_kinds: HashMap<Kind, RoaringBitmap>,
-}
-
-impl<'a> Dump<'a> {
-    pub(crate) fn new(index_scheduler: &'a mut IndexScheduler) -> Result<Self> {
-        // While loading a dump no one should be able to access the scheduler thus I can block everything.
-        let wtxn = index_scheduler.env.write_txn()?;
-
-        Ok(Dump {
-            index_scheduler,
-            wtxn,
-            batch_to_task_mapping: HashMap::new(),
-            indexes: HashMap::new(),
-            statuses: HashMap::new(),
-            kinds: HashMap::new(),
-            batch_indexes: HashMap::new(),
-            batch_statuses: HashMap::new(),
-            batch_kinds: HashMap::new(),
-        })
-    }
-
-    /// Register a new batch coming from a dump in the scheduler.
-    /// By taking a mutable ref we're pretty sure no one will ever import a dump while actix is running.
-    pub fn register_dumped_batch(&mut self, batch: Batch) -> Result<()> {
-        self.index_scheduler.queue.batches.all_batches.put(&mut self.wtxn, &batch.uid, &batch)?;
-        if let Some(enqueued_at) = batch.enqueued_at {
-            utils::insert_task_datetime(
-                &mut self.wtxn,
-                self.index_scheduler.queue.batches.enqueued_at,
-                enqueued_at.earliest,
-                batch.uid,
-            )?;
-            utils::insert_task_datetime(
-                &mut self.wtxn,
-                self.index_scheduler.queue.batches.enqueued_at,
-                enqueued_at.oldest,
-                batch.uid,
-            )?;
-        }
-        utils::insert_task_datetime(
-            &mut self.wtxn,
-            self.index_scheduler.queue.batches.started_at,
-            batch.started_at,
-            batch.uid,
-        )?;
-        if let Some(finished_at) = batch.finished_at {
-            utils::insert_task_datetime(
-                &mut self.wtxn,
-                self.index_scheduler.queue.batches.finished_at,
-                finished_at,
-                batch.uid,
-            )?;
-        }
-
-        for index in batch.stats.index_uids.keys() {
-            match self.batch_indexes.get_mut(index) {
-                Some(bitmap) => {
-                    bitmap.insert(batch.uid);
-                }
-                None => {
-                    let mut bitmap = RoaringBitmap::new();
-                    bitmap.insert(batch.uid);
-                    self.batch_indexes.insert(index.to_string(), bitmap);
-                }
-            };
-        }
-
-        for status in batch.stats.status.keys() {
-            self.batch_statuses.entry(*status).or_default().insert(batch.uid);
-        }
-        for kind in batch.stats.types.keys() {
-            self.batch_kinds.entry(*kind).or_default().insert(batch.uid);
-        }
-
-        Ok(())
-    }
-
-    /// Register a new task coming from a dump in the scheduler.
-    /// By taking a mutable ref we're pretty sure no one will ever import a dump while actix is running.
-    pub fn register_dumped_task(
-        &mut self,
-        task: TaskDump,
-        content_file: Option<Box<UpdateFile>>,
-    ) -> Result<Task> {
-        let task_has_no_docs = matches!(task.kind, KindDump::DocumentImport { documents_count, .. } if documents_count == 0);
-
-        let content_uuid = match content_file {
-            Some(content_file) if task.status == Status::Enqueued => {
-                let (uuid, file) = self.index_scheduler.queue.create_update_file(false)?;
-                let mut writer = io::BufWriter::new(file);
-                for doc in content_file {
-                    let doc = doc?;
-                    serde_json::to_writer(&mut writer, &doc).map_err(|e| {
-                        Error::from_milli(milli::InternalError::SerdeJson(e).into(), None)
-                    })?;
-                }
-                let file = writer.into_inner().map_err(|e| e.into_error())?;
-                file.persist()?;
-
-                Some(uuid)
-            }
-            // If the task isn't `Enqueued` then just generate a recognisable `Uuid`
-            // in case we try to open it later.
-            _ if task.status != Status::Enqueued => Some(Uuid::nil()),
-            None if task.status == Status::Enqueued && task_has_no_docs => {
-                let (uuid, file) = self.index_scheduler.queue.create_update_file(false)?;
-                file.persist()?;
-
-                Some(uuid)
-            }
-            _ => None,
-        };
-
-        let task = Task {
-            uid: task.uid,
-            batch_uid: task.batch_uid,
-            enqueued_at: task.enqueued_at,
-            started_at: task.started_at,
-            finished_at: task.finished_at,
-            error: task.error,
-            canceled_by: task.canceled_by,
-            details: task.details,
-            status: task.status,
-            network: task.network,
-            custom_metadata: task.custom_metadata,
-            kind: match task.kind {
-                KindDump::DocumentImport {
-                    primary_key,
-                    method,
-                    documents_count,
-                    allow_index_creation,
-                } => KindWithContent::DocumentAdditionOrUpdate {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    primary_key,
-                    method,
-                    content_file: content_uuid.ok_or(Error::CorruptedDump)?,
-                    documents_count,
-                    allow_index_creation,
-                },
-                KindDump::DocumentDeletion { documents_ids } => KindWithContent::DocumentDeletion {
-                    documents_ids,
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                },
-                KindDump::DocumentDeletionByFilter { filter } => {
-                    KindWithContent::DocumentDeletionByFilter {
-                        filter_expr: filter,
-                        index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    }
-                }
-                KindDump::DocumentEdition { filter, context, function } => {
-                    KindWithContent::DocumentEdition {
-                        index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                        filter_expr: filter,
-                        context,
-                        function,
-                    }
-                }
-                KindDump::DocumentClear => KindWithContent::DocumentClear {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                },
-                KindDump::Settings { settings, is_deletion, allow_index_creation } => {
-                    KindWithContent::SettingsUpdate {
-                        index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                        new_settings: settings,
-                        is_deletion,
-                        allow_index_creation,
-                    }
-                }
-                KindDump::IndexDeletion => KindWithContent::IndexDeletion {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                },
-                KindDump::IndexCreation { primary_key } => KindWithContent::IndexCreation {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    primary_key,
-                },
-                KindDump::IndexUpdate { primary_key, uid } => KindWithContent::IndexUpdate {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    primary_key,
-                    new_index_uid: uid,
-                },
-                KindDump::IndexSwap { swaps } => KindWithContent::IndexSwap { swaps },
-                KindDump::TaskCancelation { query, tasks } => {
-                    KindWithContent::TaskCancelation { query, tasks }
-                }
-                KindDump::TasksDeletion { query, tasks } => {
-                    KindWithContent::TaskDeletion { query, tasks }
-                }
-                KindDump::DumpCreation { keys, instance_uid } => {
-                    KindWithContent::DumpCreation { keys, instance_uid }
-                }
-                KindDump::SnapshotCreation => KindWithContent::SnapshotCreation,
-                KindDump::Export { url, api_key, payload_size, indexes } => {
-                    KindWithContent::Export {
-                        url,
-                        api_key,
-                        payload_size,
-                        indexes: indexes
-                            .into_iter()
-                            .map(|(pattern, settings)| {
-                                Ok((
-                                    IndexUidPattern::try_from(pattern)
-                                        .map_err(|_| Error::CorruptedDump)?,
-                                    settings,
-                                ))
-                            })
-                            .collect::<Result<_, Error>>()?,
-                    }
-                }
-                KindDump::UpgradeDatabase { from } => KindWithContent::UpgradeDatabase { from },
-                KindDump::IndexCompaction { index_uid } => {
-                    KindWithContent::IndexCompaction { index_uid }
-                }
-            },
-        };
-
-        self.index_scheduler.queue.tasks.all_tasks.put(&mut self.wtxn, &task.uid, &task)?;
-        if let Some(batch_id) = task.batch_uid {
-            self.batch_to_task_mapping.entry(batch_id).or_default().insert(task.uid);
-        }
-
-        for index in task.indexes() {
-            match self.indexes.get_mut(index) {
-                Some(bitmap) => {
-                    bitmap.insert(task.uid);
-                }
-                None => {
-                    let mut bitmap = RoaringBitmap::new();
-                    bitmap.insert(task.uid);
-                    self.indexes.insert(index.to_string(), bitmap);
-                }
-            };
-        }
-
-        utils::insert_task_datetime(
-            &mut self.wtxn,
-            self.index_scheduler.queue.tasks.enqueued_at,
-            task.enqueued_at,
-            task.uid,
-        )?;
-
-        // we can't override the started_at & finished_at, so we must only set it if the tasks is finished and won't change
-        if matches!(task.status, Status::Succeeded | Status::Failed | Status::Canceled) {
-            if let Some(started_at) = task.started_at {
-                utils::insert_task_datetime(
-                    &mut self.wtxn,
-                    self.index_scheduler.queue.tasks.started_at,
-                    started_at,
-                    task.uid,
-                )?;
-            }
-            if let Some(finished_at) = task.finished_at {
-                utils::insert_task_datetime(
-                    &mut self.wtxn,
-                    self.index_scheduler.queue.tasks.finished_at,
-                    finished_at,
-                    task.uid,
-                )?;
-            }
-        }
-
-        self.statuses.entry(task.status).or_default().insert(task.uid);
-        self.kinds.entry(task.kind.as_kind()).or_default().insert(task.uid);
-
-        Ok(task)
-    }
-
-    /// Commit all the changes and exit the importing dump state
-    pub fn finish(mut self) -> Result<()> {
-        for (batch_id, task_ids) in self.batch_to_task_mapping {
-            self.index_scheduler.queue.batch_to_tasks_mapping.put(
-                &mut self.wtxn,
-                &batch_id,
-                &task_ids,
-            )?;
-        }
-
-        for (index, bitmap) in self.indexes {
-            self.index_scheduler.queue.tasks.index_tasks.put(&mut self.wtxn, &index, &bitmap)?;
-        }
-        for (status, bitmap) in self.statuses {
-            self.index_scheduler.queue.tasks.put_status(&mut self.wtxn, status, &bitmap)?;
-        }
-        for (kind, bitmap) in self.kinds {
-            self.index_scheduler.queue.tasks.put_kind(&mut self.wtxn, kind, &bitmap)?;
-        }
-
-        for (index, bitmap) in self.batch_indexes {
-            self.index_scheduler.queue.batches.index_tasks.put(&mut self.wtxn, &index, &bitmap)?;
-        }
-        for (status, bitmap) in self.batch_statuses {
-            self.index_scheduler.queue.batches.put_status(&mut self.wtxn, status, &bitmap)?;
-        }
-        for (kind, bitmap) in self.batch_kinds {
-            self.index_scheduler.queue.batches.put_kind(&mut self.wtxn, kind, &bitmap)?;
-        }
-
-        self.wtxn.commit()?;
-        self.index_scheduler.scheduler.wake_up.signal();
-
-        Ok(())
-    }
-}

crates/index-scheduler/src/features.rs

@@ -1,253 +0,0 @@
-use std::sync::{Arc, RwLock};
-
-use meilisearch_types::enterprise_edition::network::Network;
-use meilisearch_types::features::{InstanceTogglableFeatures, RuntimeTogglableFeatures};
-use meilisearch_types::heed::types::{SerdeJson, Str};
-use meilisearch_types::heed::{Database, Env, RwTxn, WithoutTls};
-
-use crate::error::FeatureNotEnabledError;
-use crate::Result;
-
-/// The number of database used by features
-const NUMBER_OF_DATABASES: u32 = 1;
-/// Database const names for the `FeatureData`.
-mod db_name {
-    pub const EXPERIMENTAL_FEATURES: &str = "experimental-features";
-}
-
-mod db_keys {
-    pub const EXPERIMENTAL_FEATURES: &str = "experimental-features";
-    pub const NETWORK: &str = "network";
-}
-
-#[derive(Clone)]
-pub(crate) struct FeatureData {
-    persisted: Database<Str, SerdeJson<RuntimeTogglableFeatures>>,
-    runtime: Arc<RwLock<RuntimeTogglableFeatures>>,
-    network: Arc<RwLock<Network>>,
-}
-
-#[derive(Debug, Clone, Copy)]
-pub struct RoFeatures {
-    runtime: RuntimeTogglableFeatures,
-}
-
-impl RoFeatures {
-    fn new(data: &FeatureData) -> Self {
-        let runtime = data.runtime_features();
-        Self { runtime }
-    }
-
-    pub fn runtime_features(&self) -> RuntimeTogglableFeatures {
-        self.runtime
-    }
-
-    pub fn check_metrics(&self) -> Result<()> {
-        if self.runtime.metrics {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Getting metrics",
-                feature: "metrics",
-                issue_link: "https://github.com/meilisearch/product/discussions/625",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_logs_route(&self) -> Result<()> {
-        if self.runtime.logs_route {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Modifying logs through the `/logs/*` routes",
-                feature: "logs route",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/721",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_edit_documents_by_function(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.edit_documents_by_function {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "edit documents by function",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/762",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_contains_filter(&self) -> Result<()> {
-        if self.runtime.contains_filter {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Using `CONTAINS` in a filter",
-                feature: "contains filter",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/763",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_network(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.network {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "network",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/805",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_get_task_documents_route(&self) -> Result<()> {
-        if self.runtime.get_task_documents_route {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Getting the documents of an enqueued task",
-                feature: "get task documents route",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/808",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_composite_embedders(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.composite_embedders {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "composite embedders",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/816",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_chat_completions(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.chat_completions {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "chat completions",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/835",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_multimodal(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.multimodal {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "multimodal",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/846",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_vector_store_setting(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.vector_store_setting {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "vector_store_setting",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/860",
-            }
-            .into())
-        }
-    }
-}
-
-impl FeatureData {
-    pub(crate) const fn nb_db() -> u32 {
-        NUMBER_OF_DATABASES
-    }
-
-    pub fn new(
-        env: &Env<WithoutTls>,
-        wtxn: &mut RwTxn,
-        instance_features: InstanceTogglableFeatures,
-    ) -> Result<Self> {
-        let runtime_features_db =
-            env.create_database(wtxn, Some(db_name::EXPERIMENTAL_FEATURES))?;
-
-        let persisted_features: RuntimeTogglableFeatures =
-            runtime_features_db.get(wtxn, db_keys::EXPERIMENTAL_FEATURES)?.unwrap_or_default();
-        let InstanceTogglableFeatures { metrics, logs_route, contains_filter } = instance_features;
-        let runtime = Arc::new(RwLock::new(RuntimeTogglableFeatures {
-            metrics: metrics || persisted_features.metrics,
-            logs_route: logs_route || persisted_features.logs_route,
-            contains_filter: contains_filter || persisted_features.contains_filter,
-            ..persisted_features
-        }));
-
-        // Once this is stabilized, network should be stored along with webhooks in index-scheduler's persisted database
-        let network_db = runtime_features_db.remap_data_type::<SerdeJson<Network>>();
-        let network: Network = network_db.get(wtxn, db_keys::NETWORK)?.unwrap_or_default();
-
-        Ok(Self {
-            persisted: runtime_features_db,
-            runtime,
-            network: Arc::new(RwLock::new(network)),
-        })
-    }
-
-    pub fn put_runtime_features(
-        &self,
-        mut wtxn: RwTxn,
-        features: RuntimeTogglableFeatures,
-    ) -> Result<()> {
-        self.persisted.put(&mut wtxn, db_keys::EXPERIMENTAL_FEATURES, &features)?;
-        wtxn.commit()?;
-
-        // safe to unwrap, the lock will only fail if:
-        // 1. requested by the same thread concurrently -> it is called and released in methods that don't call each other
-        // 2. there's a panic while the thread is held -> it is only used for an assignment here.
-        let mut toggled_features = self.runtime.write().unwrap();
-        *toggled_features = features;
-        Ok(())
-    }
-
-    fn runtime_features(&self) -> RuntimeTogglableFeatures {
-        // sound to unwrap, the lock will only fail if:
-        // 1. requested by the same thread concurrently -> it is called and released in methods that don't call each other
-        // 2. there's a panic while the thread is held -> it is only used for copying the data here
-        *self.runtime.read().unwrap()
-    }
-
-    pub fn features(&self) -> RoFeatures {
-        RoFeatures::new(self)
-    }
-
-    pub fn put_network(&self, mut wtxn: RwTxn, new_network: Network) -> Result<()> {
-        self.persisted.remap_data_type::<SerdeJson<Network>>().put(
-            &mut wtxn,
-            db_keys::NETWORK,
-            &new_network,
-        )?;
-        wtxn.commit()?;
-
-        let mut network = self.network.write().unwrap();
-        *network = new_network;
-        Ok(())
-    }
-
-    pub fn network(&self) -> Network {
-        Network::clone(&*self.network.read().unwrap())
-    }
-}

crates/index-scheduler/src/insta_snapshot.rs

@@ -1,430 +0,0 @@
-use std::collections::BTreeSet;
-use std::fmt::Write;
-
-use meilisearch_types::batches::{Batch, BatchEnqueuedAt, BatchStats};
-use meilisearch_types::heed::types::{SerdeBincode, SerdeJson, Str};
-use meilisearch_types::heed::{Database, RoTxn};
-use meilisearch_types::milli::{CboRoaringBitmapCodec, RoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Details, Kind, Status, Task};
-use meilisearch_types::versioning;
-use roaring::RoaringBitmap;
-
-use crate::index_mapper::IndexMapper;
-use crate::{IndexScheduler, BEI128};
-
-pub fn snapshot_index_scheduler(scheduler: &IndexScheduler) -> String {
-    // Since we'll snapshot the index right afterward, we don't need to ensure it's internally consistent for every run.
-    // We can only do it for the release run, where the function runs way faster.
-    #[cfg(not(debug_assertions))]
-    scheduler.assert_internally_consistent();
-
-    let IndexScheduler {
-        cleanup_enabled: _,
-        experimental_no_edition_2024_for_dumps: _,
-        processing_tasks,
-        env,
-        version,
-        queue,
-        scheduler,
-        persisted,
-
-        index_mapper,
-        features: _,
-        webhooks: _,
-        test_breakpoint_sdr: _,
-        planned_failures: _,
-        run_loop_iteration: _,
-        embedders: _,
-        chat_settings: _,
-        runtime: _,
-    } = scheduler;
-
-    let rtxn = env.read_txn().unwrap();
-
-    let mut snap = String::new();
-
-    let indx_sched_version = version.get_version(&rtxn).unwrap();
-    let latest_version =
-        (versioning::VERSION_MAJOR, versioning::VERSION_MINOR, versioning::VERSION_PATCH);
-    if indx_sched_version != Some(latest_version) {
-        snap.push_str(&format!("index scheduler running on version {indx_sched_version:?}\n"));
-    }
-
-    let processing = processing_tasks.read().unwrap().clone();
-    snap.push_str(&format!("### Autobatching Enabled = {}\n", scheduler.autobatching_enabled));
-    snap.push_str(&format!(
-        "### Processing batch {:?}:\n",
-        processing.batch.as_ref().map(|batch| batch.uid)
-    ));
-    snap.push_str(&snapshot_bitmap(&processing.processing));
-    if let Some(ref batch) = processing.batch {
-        snap.push('\n');
-        snap.push_str(&snapshot_batch(&batch.to_batch()));
-    }
-    snap.push_str("\n----------------------------------------------------------------------\n");
-
-    let persisted_db_snapshot = snapshot_persisted_db(&rtxn, persisted);
-    if !persisted_db_snapshot.is_empty() {
-        snap.push_str("### Persisted:\n");
-        snap.push_str(&persisted_db_snapshot);
-        snap.push_str("----------------------------------------------------------------------\n");
-    }
-
-    snap.push_str("### All Tasks:\n");
-    snap.push_str(&snapshot_all_tasks(&rtxn, queue.tasks.all_tasks));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Status:\n");
-    snap.push_str(&snapshot_status(&rtxn, queue.tasks.status));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Kind:\n");
-    snap.push_str(&snapshot_kind(&rtxn, queue.tasks.kind));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Index Tasks:\n");
-    snap.push_str(&snapshot_index_tasks(&rtxn, queue.tasks.index_tasks));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Index Mapper:\n");
-    snap.push_str(&snapshot_index_mapper(&rtxn, index_mapper));
-    snap.push_str("\n----------------------------------------------------------------------\n");
-
-    snap.push_str("### Canceled By:\n");
-    snap.push_str(&snapshot_canceled_by(&rtxn, queue.tasks.canceled_by));
-    snap.push_str("\n----------------------------------------------------------------------\n");
-
-    snap.push_str("### Enqueued At:\n");
-    snap.push_str(&snapshot_date_db(&rtxn, queue.tasks.enqueued_at));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Started At:\n");
-    snap.push_str(&snapshot_date_db(&rtxn, queue.tasks.started_at));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Finished At:\n");
-    snap.push_str(&snapshot_date_db(&rtxn, queue.tasks.finished_at));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### All Batches:\n");
-    snap.push_str(&snapshot_all_batches(&rtxn, queue.batches.all_batches));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batch to tasks mapping:\n");
-    snap.push_str(&snapshot_batches_to_tasks_mappings(&rtxn, queue.batch_to_tasks_mapping));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batches Status:\n");
-    snap.push_str(&snapshot_status(&rtxn, queue.batches.status));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batches Kind:\n");
-    snap.push_str(&snapshot_kind(&rtxn, queue.batches.kind));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batches Index Tasks:\n");
-    snap.push_str(&snapshot_index_tasks(&rtxn, queue.batches.index_tasks));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batches Enqueued At:\n");
-    snap.push_str(&snapshot_date_db(&rtxn, queue.batches.enqueued_at));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batches Started At:\n");
-    snap.push_str(&snapshot_date_db(&rtxn, queue.batches.started_at));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### Batches Finished At:\n");
-    snap.push_str(&snapshot_date_db(&rtxn, queue.batches.finished_at));
-    snap.push_str("----------------------------------------------------------------------\n");
-
-    snap.push_str("### File Store:\n");
-    snap.push_str(&snapshot_file_store(&queue.file_store));
-    snap.push_str("\n----------------------------------------------------------------------\n");
-
-    snap
-}
-
-pub fn snapshot_file_store(file_store: &file_store::FileStore) -> String {
-    let mut snap = String::new();
-    // we store the uuid in a `BTreeSet` to keep them ordered.
-    let all_uuids = file_store.all_uuids().unwrap().collect::<Result<BTreeSet<_>, _>>().unwrap();
-    for uuid in all_uuids {
-        snap.push_str(&format!("{uuid}\n"));
-    }
-    snap
-}
-
-pub fn snapshot_bitmap(r: &RoaringBitmap) -> String {
-    let mut snap = String::new();
-    snap.push('[');
-    for x in r {
-        snap.push_str(&format!("{x},"));
-    }
-    snap.push(']');
-    snap
-}
-
-pub fn snapshot_all_tasks(rtxn: &RoTxn, db: Database<BEU32, SerdeJson<Task>>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (task_id, task) = next.unwrap();
-        snap.push_str(&format!("{task_id} {}\n", snapshot_task(&task)));
-    }
-    snap
-}
-
-pub fn snapshot_all_batches(rtxn: &RoTxn, db: Database<BEU32, SerdeJson<Batch>>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (batch_id, batch) = next.unwrap();
-        snap.push_str(&format!("{batch_id} {}\n", snapshot_batch(&batch)));
-    }
-    snap
-}
-
-pub fn snapshot_batches_to_tasks_mappings(
-    rtxn: &RoTxn,
-    db: Database<BEU32, CboRoaringBitmapCodec>,
-) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (batch_id, tasks) = next.unwrap();
-        snap.push_str(&format!("{batch_id} {}\n", snapshot_bitmap(&tasks)));
-    }
-    snap
-}
-
-pub fn snapshot_date_db(rtxn: &RoTxn, db: Database<BEI128, CboRoaringBitmapCodec>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (_timestamp, task_ids) = next.unwrap();
-        snap.push_str(&format!("[timestamp] {}\n", snapshot_bitmap(&task_ids)));
-    }
-    snap
-}
-
-pub fn snapshot_persisted_db(rtxn: &RoTxn, db: &Database<Str, Str>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (key, value) = next.unwrap();
-        snap.push_str(&format!("{key}: {value}\n"));
-    }
-    snap
-}
-
-pub fn snapshot_task(task: &Task) -> String {
-    let mut snap = String::new();
-    let Task {
-        uid,
-        batch_uid,
-        enqueued_at: _,
-        started_at: _,
-        finished_at: _,
-        error,
-        canceled_by,
-        details,
-        status,
-        kind,
-        network,
-        custom_metadata,
-    } = task;
-    snap.push('{');
-    snap.push_str(&format!("uid: {uid}, "));
-    if let Some(batch_uid) = batch_uid {
-        snap.push_str(&format!("batch_uid: {batch_uid}, "));
-    }
-    snap.push_str(&format!("status: {status}, "));
-    if let Some(canceled_by) = canceled_by {
-        snap.push_str(&format!("canceled_by: {canceled_by}, "));
-    }
-    if let Some(error) = error {
-        snap.push_str(&format!("error: {error:?}, "));
-    }
-    if let Some(details) = details {
-        snap.push_str(&format!("details: {}, ", &snapshot_details(details)));
-    }
-    snap.push_str(&format!("kind: {kind:?}"));
-    if let Some(network) = network {
-        snap.push_str(&format!("network: {network:?}, "))
-    }
-    if let Some(custom_metadata) = custom_metadata {
-        snap.push_str(&format!("custom_metadata: {custom_metadata:?}"))
-    }
-
-    snap.push('}');
-    snap
-}
-
-fn snapshot_details(d: &Details) -> String {
-    match d {
-        Details::DocumentAdditionOrUpdate {
-            received_documents,
-            indexed_documents,
-        } => {
-            format!("{{ received_documents: {received_documents}, indexed_documents: {indexed_documents:?} }}")
-        }
-        Details::DocumentEdition {
-            deleted_documents,
-            edited_documents,
-            original_filter,
-            context,
-            function,
-        } => {
-            format!(
-                "{{ deleted_documents: {deleted_documents:?}, edited_documents: {edited_documents:?}, context: {context:?}, function: {function:?}, original_filter: {original_filter:?} }}"
-            )
-        }
-        Details::SettingsUpdate { settings } => {
-            format!("{{ settings: {settings:?} }}")
-        }
-        Details::IndexInfo { primary_key, new_index_uid, old_index_uid } => {
-            format!("{{ primary_key: {primary_key:?}, old_new_uid: {old_index_uid:?}, new_index_uid: {new_index_uid:?} }}")
-        }
-        Details::DocumentDeletion {
-            provided_ids: received_document_ids,
-            deleted_documents,
-        } => format!("{{ received_document_ids: {received_document_ids}, deleted_documents: {deleted_documents:?} }}"),
-        Details::DocumentDeletionByFilter { original_filter, deleted_documents } => format!(
-           "{{ original_filter: {original_filter}, deleted_documents: {deleted_documents:?} }}"
-        ),
-        Details::ClearAll { deleted_documents } => {
-            format!("{{ deleted_documents: {deleted_documents:?} }}")
-        },
-        Details::TaskCancelation {
-            matched_tasks,
-            canceled_tasks,
-            original_filter,
-        } => {
-            format!("{{ matched_tasks: {matched_tasks:?}, canceled_tasks: {canceled_tasks:?}, original_filter: {original_filter:?} }}")
-        }
-        Details::TaskDeletion {
-            matched_tasks,
-            deleted_tasks,
-            original_filter,
-        } => {
-            format!("{{ matched_tasks: {matched_tasks:?}, deleted_tasks: {deleted_tasks:?}, original_filter: {original_filter:?} }}")
-        },
-        Details::Dump { dump_uid } => {
-            format!("{{ dump_uid: {dump_uid:?} }}")
-        },
-        Details::IndexSwap { swaps } => {
-            format!("{{ swaps: {swaps:?} }}")
-        }
-        Details::Export { url, api_key, payload_size, indexes } => {
-            format!("{{ url: {url:?}, api_key: {api_key:?}, payload_size: {payload_size:?}, indexes: {indexes:?} }}")
-        }
-        Details::UpgradeDatabase { from, to } => {
-            format!("{{ from: {from:?}, to: {to:?} }}")
-        }
-        Details::IndexCompaction { index_uid, pre_compaction_size, post_compaction_size } => {
-            format!("{{ index_uid: {index_uid:?}, pre_compaction_size: {pre_compaction_size:?}, post_compaction_size: {post_compaction_size:?} }}")
-        }
-    }
-}
-
-pub fn snapshot_status(
-    rtxn: &RoTxn,
-    db: Database<SerdeBincode<Status>, RoaringBitmapCodec>,
-) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (status, task_ids) = next.unwrap();
-        writeln!(snap, "{status} {}", snapshot_bitmap(&task_ids)).unwrap();
-    }
-    snap
-}
-
-pub fn snapshot_kind(rtxn: &RoTxn, db: Database<SerdeBincode<Kind>, RoaringBitmapCodec>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (kind, task_ids) = next.unwrap();
-        let kind = serde_json::to_string(&kind).unwrap();
-        writeln!(snap, "{kind} {}", snapshot_bitmap(&task_ids)).unwrap();
-    }
-    snap
-}
-
-pub fn snapshot_index_tasks(rtxn: &RoTxn, db: Database<Str, RoaringBitmapCodec>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (index, task_ids) = next.unwrap();
-        writeln!(snap, "{index} {}", snapshot_bitmap(&task_ids)).unwrap();
-    }
-    snap
-}
-
-pub fn snapshot_canceled_by(rtxn: &RoTxn, db: Database<BEU32, RoaringBitmapCodec>) -> String {
-    let mut snap = String::new();
-    let iter = db.iter(rtxn).unwrap();
-    for next in iter {
-        let (kind, task_ids) = next.unwrap();
-        writeln!(snap, "{kind} {}", snapshot_bitmap(&task_ids)).unwrap();
-    }
-    snap
-}
-
-pub fn snapshot_batch(batch: &Batch) -> String {
-    let mut snap = String::new();
-    let Batch {
-        uid,
-        details,
-        stats,
-        embedder_stats,
-        started_at,
-        finished_at,
-        progress: _,
-        enqueued_at,
-        stop_reason,
-    } = batch;
-    let stats = BatchStats {
-        progress_trace: Default::default(),
-        internal_database_sizes: Default::default(),
-        write_channel_congestion: None,
-        ..stats.clone()
-    };
-    if let Some(finished_at) = finished_at {
-        assert!(finished_at > started_at);
-    }
-    let BatchEnqueuedAt { earliest, oldest } = enqueued_at.unwrap();
-    assert!(*started_at > earliest);
-    assert!(earliest >= oldest);
-
-    snap.push('{');
-    snap.push_str(&format!("uid: {uid}, "));
-    snap.push_str(&format!("details: {}, ", serde_json::to_string(details).unwrap()));
-    snap.push_str(&format!("stats: {}, ", serde_json::to_string(&stats).unwrap()));
-    if !embedder_stats.skip_serializing() {
-        snap.push_str(&format!(
-            "embedder stats: {}, ",
-            serde_json::to_string(&embedder_stats).unwrap()
-        ));
-    }
-    snap.push_str(&format!("stop reason: {}, ", serde_json::to_string(&stop_reason).unwrap()));
-    snap.push('}');
-    snap
-}
-
-pub fn snapshot_index_mapper(rtxn: &RoTxn, mapper: &IndexMapper) -> String {
-    let mut s = String::new();
-    let names = mapper.index_names(rtxn).unwrap();
-
-    for name in names {
-        let stats = mapper.stats_of(rtxn, &name).unwrap();
-        s.push_str(&format!(
-            "{name}: {{ number_of_documents: {}, field_distribution: {:?} }}\n",
-            stats.documents_database_stats.number_of_entries(),
-            stats.field_distribution
-        ));
-    }
-
-    s
-}

crates/index-scheduler/src/processing.rs

@@ -1,320 +0,0 @@
-use std::sync::Arc;
-
-use meilisearch_types::milli::progress::{AtomicSubStep, NamedStep, Progress, ProgressView};
-use meilisearch_types::milli::{make_atomic_progress, make_enum_progress};
-use roaring::RoaringBitmap;
-
-use crate::utils::ProcessingBatch;
-
-#[derive(Clone, Default)]
-pub struct ProcessingTasks {
-    pub batch: Option<Arc<ProcessingBatch>>,
-    /// The list of tasks ids that are currently running.
-    pub processing: Arc<RoaringBitmap>,
-    /// The progress on processing tasks
-    pub progress: Option<Progress>,
-}
-
-impl ProcessingTasks {
-    /// Creates an empty `ProcessingAt` struct.
-    pub fn new() -> ProcessingTasks {
-        ProcessingTasks::default()
-    }
-
-    pub fn get_progress_view(&self) -> Option<ProgressView> {
-        Some(self.progress.as_ref()?.as_progress_view())
-    }
-
-    /// Stores the currently processing tasks, and the date time at which it started.
-    pub fn start_processing(
-        &mut self,
-        processing_batch: ProcessingBatch,
-        processing: RoaringBitmap,
-    ) -> Progress {
-        self.batch = Some(Arc::new(processing_batch));
-        self.processing = Arc::new(processing);
-        let progress = Progress::default();
-        progress.update_progress(BatchProgress::ProcessingTasks);
-        self.progress = Some(progress.clone());
-
-        progress
-    }
-
-    /// Set the processing tasks to an empty list
-    pub fn stop_processing(&mut self) -> Self {
-        self.progress = None;
-
-        Self {
-            batch: std::mem::take(&mut self.batch),
-            processing: std::mem::take(&mut self.processing),
-            progress: None,
-        }
-    }
-
-    /// Returns `true` if there, at least, is one task that is currently processing that we must stop.
-    pub fn must_cancel_processing_tasks(&self, canceled_tasks: &RoaringBitmap) -> bool {
-        !self.processing.is_disjoint(canceled_tasks)
-    }
-}
-
-make_enum_progress! {
-    pub enum BatchProgress {
-        ProcessingTasks,
-        WritingTasksToDisk,
-    }
-}
-
-make_enum_progress! {
-    pub enum FinalizingIndexStep {
-        Committing,
-        ComputingStats,
-    }
-}
-
-make_enum_progress! {
-    pub enum TaskCancelationProgress {
-        RetrievingTasks,
-        CancelingUpgrade,
-        CleaningCompactionLeftover,
-        UpdatingTasks,
-    }
-}
-
-make_enum_progress! {
-    pub enum TaskDeletionProgress {
-        DeletingTasksDateTime,
-        DeletingTasksMetadata,
-        DeletingTasks,
-        DeletingBatches,
-    }
-}
-
-make_enum_progress! {
-    pub enum SnapshotCreationProgress {
-        StartTheSnapshotCreation,
-        SnapshotTheIndexScheduler,
-        SnapshotTheUpdateFiles,
-        SnapshotTheIndexes,
-        SnapshotTheApiKeys,
-        CreateTheTarball,
-    }
-}
-
-make_enum_progress! {
-    pub enum DumpCreationProgress {
-        StartTheDumpCreation,
-        DumpTheApiKeys,
-        DumpTheChatCompletionSettings,
-        DumpTheTasks,
-        DumpTheBatches,
-        DumpTheIndexes,
-        DumpTheExperimentalFeatures,
-        DumpTheWebhooks,
-        CompressTheDump,
-    }
-}
-
-make_enum_progress! {
-    pub enum CreateIndexProgress {
-        CreatingTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum UpdateIndexProgress {
-        UpdatingTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum DeleteIndexProgress {
-        DeletingTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum SwappingTheIndexes {
-        EnsuringCorrectnessOfTheSwap,
-        SwappingTheIndexes,
-    }
-}
-
-make_enum_progress! {
-    pub enum IndexCompaction {
-        RetrieveTheIndex,
-        CreateTemporaryFile,
-        CopyAndCompactTheIndex,
-        PersistTheCompactedIndex,
-        CloseTheIndex,
-        ReopenTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum InnerSwappingTwoIndexes {
-        RetrieveTheTasks,
-        UpdateTheTasks,
-        UpdateTheIndexesMetadata,
-    }
-}
-
-make_enum_progress! {
-    pub enum DocumentOperationProgress {
-        RetrievingConfig,
-        ComputingDocumentChanges,
-        Indexing,
-    }
-}
-
-make_enum_progress! {
-    pub enum DocumentEditionProgress {
-        RetrievingConfig,
-        ComputingDocumentChanges,
-        Indexing,
-    }
-}
-
-make_enum_progress! {
-    pub enum DocumentDeletionProgress {
-        RetrievingConfig,
-        DeleteDocuments,
-        Indexing,
-    }
-}
-
-make_enum_progress! {
-    pub enum SettingsProgress {
-        RetrievingAndMergingTheSettings,
-        ApplyTheSettings,
-    }
-}
-
-make_enum_progress! {
-    pub enum Export {
-        EnsuringCorrectnessOfTheTarget,
-        ExportingTheSettings,
-        ExportingTheDocuments,
-    }
-}
-
-make_atomic_progress!(Task alias AtomicTaskStep => "task" );
-make_atomic_progress!(Document alias AtomicDocumentStep => "document" );
-make_atomic_progress!(Index alias AtomicIndexStep => "index" );
-make_atomic_progress!(Batch alias AtomicBatchStep => "batch" );
-make_atomic_progress!(UpdateFile alias AtomicUpdateFileStep => "update file" );
-
-#[cfg(test)]
-mod test {
-    use std::sync::atomic::Ordering;
-
-    use meili_snap::{json_string, snapshot};
-
-    use super::*;
-
-    #[test]
-    fn one_level() {
-        let mut processing = ProcessingTasks::new();
-        processing.start_processing(ProcessingBatch::new(0), RoaringBitmap::new());
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "processing tasks",
-              "finished": 0,
-              "total": 2
-            }
-          ],
-          "percentage": 0.0
-        }
-        "#);
-        processing.progress.as_ref().unwrap().update_progress(BatchProgress::WritingTasksToDisk);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "writing tasks to disk",
-              "finished": 1,
-              "total": 2
-            }
-          ],
-          "percentage": 50.0
-        }
-        "#);
-    }
-
-    #[test]
-    fn task_progress() {
-        let mut processing = ProcessingTasks::new();
-        processing.start_processing(ProcessingBatch::new(0), RoaringBitmap::new());
-        let (atomic, tasks) = AtomicTaskStep::new(10);
-        processing.progress.as_ref().unwrap().update_progress(tasks);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "processing tasks",
-              "finished": 0,
-              "total": 2
-            },
-            {
-              "currentStep": "task",
-              "finished": 0,
-              "total": 10
-            }
-          ],
-          "percentage": 0.0
-        }
-        "#);
-        atomic.fetch_add(6, Ordering::Relaxed);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "processing tasks",
-              "finished": 0,
-              "total": 2
-            },
-            {
-              "currentStep": "task",
-              "finished": 6,
-              "total": 10
-            }
-          ],
-          "percentage": 30.000002
-        }
-        "#);
-        processing.progress.as_ref().unwrap().update_progress(BatchProgress::WritingTasksToDisk);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "writing tasks to disk",
-              "finished": 1,
-              "total": 2
-            }
-          ],
-          "percentage": 50.0
-        }
-        "#);
-        let (atomic, tasks) = AtomicTaskStep::new(5);
-        processing.progress.as_ref().unwrap().update_progress(tasks);
-        atomic.fetch_add(4, Ordering::Relaxed);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "writing tasks to disk",
-              "finished": 1,
-              "total": 2
-            },
-            {
-              "currentStep": "task",
-              "finished": 4,
-              "total": 5
-            }
-          ],
-          "percentage": 90.0
-        }
-        "#);
-    }
-}

crates/index-scheduler/src/queue/batches.rs

@@ -1,613 +0,0 @@
-use std::collections::HashSet;
-use std::ops::{Bound, RangeBounds};
-
-use meilisearch_types::batches::{Batch, BatchId};
-use meilisearch_types::heed::types::{DecodeIgnore, SerdeBincode, SerdeJson, Str};
-use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
-use meilisearch_types::milli::{CboRoaringBitmapCodec, RoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Kind, Status};
-use roaring::{MultiOps, RoaringBitmap};
-use time::OffsetDateTime;
-
-use super::{Query, Queue};
-use crate::processing::ProcessingTasks;
-use crate::utils::{
-    insert_task_datetime, keep_ids_within_datetimes, map_bound,
-    remove_n_tasks_datetime_earlier_than, remove_task_datetime, ProcessingBatch,
-};
-use crate::{Error, Result, BEI128};
-
-/// The number of database used by the batch queue
-const NUMBER_OF_DATABASES: u32 = 7;
-/// Database const names for the `IndexScheduler`.
-mod db_name {
-    pub const ALL_BATCHES: &str = "all-batches";
-
-    pub const BATCH_STATUS: &str = "batch-status";
-    pub const BATCH_KIND: &str = "batch-kind";
-    pub const BATCH_INDEX_TASKS: &str = "batch-index-tasks";
-    pub const BATCH_ENQUEUED_AT: &str = "batch-enqueued-at";
-    pub const BATCH_STARTED_AT: &str = "batch-started-at";
-    pub const BATCH_FINISHED_AT: &str = "batch-finished-at";
-}
-
-pub struct BatchQueue {
-    /// Contains all the batches accessible by their Id.
-    pub(crate) all_batches: Database<BEU32, SerdeJson<Batch>>,
-
-    /// All the batches containing a task matching the selected status.
-    pub(crate) status: Database<SerdeBincode<Status>, RoaringBitmapCodec>,
-    /// All the batches ids grouped by the kind of their task.
-    pub(crate) kind: Database<SerdeBincode<Kind>, RoaringBitmapCodec>,
-    /// Store the batches associated to an index.
-    pub(crate) index_tasks: Database<Str, RoaringBitmapCodec>,
-    /// Store the batches containing tasks which were enqueued at a specific date
-    pub(crate) enqueued_at: Database<BEI128, CboRoaringBitmapCodec>,
-    /// Store the batches containing finished tasks started at a specific date
-    pub(crate) started_at: Database<BEI128, CboRoaringBitmapCodec>,
-    /// Store the batches containing tasks finished at a specific date
-    pub(crate) finished_at: Database<BEI128, CboRoaringBitmapCodec>,
-}
-
-impl BatchQueue {
-    pub(crate) fn private_clone(&self) -> BatchQueue {
-        BatchQueue {
-            all_batches: self.all_batches,
-            status: self.status,
-            kind: self.kind,
-            index_tasks: self.index_tasks,
-            enqueued_at: self.enqueued_at,
-            started_at: self.started_at,
-            finished_at: self.finished_at,
-        }
-    }
-
-    pub(crate) const fn nb_db() -> u32 {
-        NUMBER_OF_DATABASES
-    }
-
-    pub(super) fn new(env: &Env<WithoutTls>, wtxn: &mut RwTxn) -> Result<Self> {
-        Ok(Self {
-            all_batches: env.create_database(wtxn, Some(db_name::ALL_BATCHES))?,
-            status: env.create_database(wtxn, Some(db_name::BATCH_STATUS))?,
-            kind: env.create_database(wtxn, Some(db_name::BATCH_KIND))?,
-            index_tasks: env.create_database(wtxn, Some(db_name::BATCH_INDEX_TASKS))?,
-            enqueued_at: env.create_database(wtxn, Some(db_name::BATCH_ENQUEUED_AT))?,
-            started_at: env.create_database(wtxn, Some(db_name::BATCH_STARTED_AT))?,
-            finished_at: env.create_database(wtxn, Some(db_name::BATCH_FINISHED_AT))?,
-        })
-    }
-
-    pub(crate) fn all_batch_ids(&self, rtxn: &RoTxn) -> Result<RoaringBitmap> {
-        enum_iterator::all().map(|s| self.get_status(rtxn, s)).union()
-    }
-
-    pub(crate) fn next_batch_id(&self, rtxn: &RoTxn) -> Result<BatchId> {
-        Ok(self
-            .all_batches
-            .remap_data_type::<DecodeIgnore>()
-            .last(rtxn)?
-            .map(|(k, _)| k + 1)
-            .unwrap_or_default())
-    }
-
-    pub(crate) fn get_batch(&self, rtxn: &RoTxn, batch_id: BatchId) -> Result<Option<Batch>> {
-        Ok(self.all_batches.get(rtxn, &batch_id)?)
-    }
-
-    /// Returns the whole set of batches that belongs to this index.
-    pub(crate) fn index_batches(&self, rtxn: &RoTxn, index: &str) -> Result<RoaringBitmap> {
-        Ok(self.index_tasks.get(rtxn, index)?.unwrap_or_default())
-    }
-
-    pub(crate) fn update_index(
-        &self,
-        wtxn: &mut RwTxn,
-        index: &str,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut batches = self.index_batches(wtxn, index)?;
-        f(&mut batches);
-        if batches.is_empty() {
-            self.index_tasks.delete(wtxn, index)?;
-        } else {
-            self.index_tasks.put(wtxn, index, &batches)?;
-        }
-
-        Ok(())
-    }
-
-    pub(crate) fn get_status(&self, rtxn: &RoTxn, status: Status) -> Result<RoaringBitmap> {
-        Ok(self.status.get(rtxn, &status)?.unwrap_or_default())
-    }
-
-    pub(crate) fn put_status(
-        &self,
-        wtxn: &mut RwTxn,
-        status: Status,
-        bitmap: &RoaringBitmap,
-    ) -> Result<()> {
-        Ok(self.status.put(wtxn, &status, bitmap)?)
-    }
-
-    pub(crate) fn update_status(
-        &self,
-        wtxn: &mut RwTxn,
-        status: Status,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.get_status(wtxn, status)?;
-        f(&mut tasks);
-        self.put_status(wtxn, status, &tasks)?;
-
-        Ok(())
-    }
-
-    pub(crate) fn get_kind(&self, rtxn: &RoTxn, kind: Kind) -> Result<RoaringBitmap> {
-        Ok(self.kind.get(rtxn, &kind)?.unwrap_or_default())
-    }
-
-    pub(crate) fn put_kind(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: Kind,
-        bitmap: &RoaringBitmap,
-    ) -> Result<()> {
-        Ok(self.kind.put(wtxn, &kind, bitmap)?)
-    }
-
-    pub(crate) fn update_kind(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: Kind,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.get_kind(wtxn, kind)?;
-        f(&mut tasks);
-        self.put_kind(wtxn, kind, &tasks)?;
-        Ok(())
-    }
-
-    pub(crate) fn write_batch(&self, wtxn: &mut RwTxn, batch: ProcessingBatch) -> Result<()> {
-        let old_batch = self.all_batches.get(wtxn, &batch.uid)?;
-
-        self.all_batches.put(
-            wtxn,
-            &batch.uid,
-            &Batch {
-                uid: batch.uid,
-                progress: None,
-                details: batch.details,
-                stats: batch.stats,
-                embedder_stats: batch.embedder_stats.as_ref().into(),
-                started_at: batch.started_at,
-                finished_at: batch.finished_at,
-                enqueued_at: batch.enqueued_at,
-                stop_reason: batch.reason.to_string(),
-            },
-        )?;
-
-        // Update the statuses
-        if let Some(ref old_batch) = old_batch {
-            for status in old_batch.stats.status.keys() {
-                self.update_status(wtxn, *status, |bitmap| {
-                    bitmap.remove(batch.uid);
-                })?;
-            }
-        }
-        for status in batch.statuses {
-            self.update_status(wtxn, status, |bitmap| {
-                bitmap.insert(batch.uid);
-            })?;
-        }
-
-        // Update the kinds / types
-        if let Some(ref old_batch) = old_batch {
-            let kinds: HashSet<_> = old_batch.stats.types.keys().cloned().collect();
-            for kind in kinds.difference(&batch.kinds) {
-                self.update_kind(wtxn, *kind, |bitmap| {
-                    bitmap.remove(batch.uid);
-                })?;
-            }
-        }
-        for kind in batch.kinds {
-            self.update_kind(wtxn, kind, |bitmap| {
-                bitmap.insert(batch.uid);
-            })?;
-        }
-
-        // Update the indexes
-        if let Some(ref old_batch) = old_batch {
-            let indexes: HashSet<_> = old_batch.stats.index_uids.keys().cloned().collect();
-            for index in indexes.difference(&batch.indexes) {
-                self.update_index(wtxn, index, |bitmap| {
-                    bitmap.remove(batch.uid);
-                })?;
-            }
-        }
-        for index in batch.indexes {
-            self.update_index(wtxn, &index, |bitmap| {
-                bitmap.insert(batch.uid);
-            })?;
-        }
-
-        // Update the enqueued_at: we cannot retrieve the previous enqueued at from the previous batch, and
-        // must instead go through the db looking for it. We cannot look at the task contained in this batch either
-        // because they may have been removed.
-        // What we know, though, is that the task date is from before the enqueued_at, and max two timestamps have been written
-        // to the DB per batches.
-        if let Some(ref old_batch) = old_batch {
-            if let Some(enqueued_at) = old_batch.enqueued_at {
-                remove_task_datetime(wtxn, self.enqueued_at, enqueued_at.earliest, old_batch.uid)?;
-                remove_task_datetime(wtxn, self.enqueued_at, enqueued_at.oldest, old_batch.uid)?;
-            } else {
-                // If we don't have the enqueued at in the batch it means the database comes from the v1.12
-                // and we still need to find the date by scrolling the database
-                remove_n_tasks_datetime_earlier_than(
-                    wtxn,
-                    self.enqueued_at,
-                    old_batch.started_at,
-                    old_batch.stats.total_nb_tasks.clamp(1, 2) as usize,
-                    old_batch.uid,
-                )?;
-            }
-        }
-        // A finished batch MUST contains at least one task and have an enqueued_at
-        let enqueued_at = batch.enqueued_at.as_ref().unwrap();
-        insert_task_datetime(wtxn, self.enqueued_at, enqueued_at.earliest, batch.uid)?;
-        insert_task_datetime(wtxn, self.enqueued_at, enqueued_at.oldest, batch.uid)?;
-
-        // Update the started at and finished at
-        if let Some(ref old_batch) = old_batch {
-            remove_task_datetime(wtxn, self.started_at, old_batch.started_at, old_batch.uid)?;
-            if let Some(finished_at) = old_batch.finished_at {
-                remove_task_datetime(wtxn, self.finished_at, finished_at, old_batch.uid)?;
-            }
-        }
-        insert_task_datetime(wtxn, self.started_at, batch.started_at, batch.uid)?;
-        insert_task_datetime(wtxn, self.finished_at, batch.finished_at.unwrap(), batch.uid)?;
-
-        Ok(())
-    }
-
-    /// Convert an iterator to a `Vec` of batches. The batches MUST exist or a
-    /// `CorruptedTaskQueue` error will be thrown.
-    pub(crate) fn get_existing_batches(
-        &self,
-        rtxn: &RoTxn,
-        batches: impl IntoIterator<Item = BatchId>,
-        processing: &ProcessingTasks,
-    ) -> Result<Vec<Batch>> {
-        batches
-            .into_iter()
-            .map(|batch_id| {
-                if Some(batch_id) == processing.batch.as_ref().map(|batch| batch.uid) {
-                    let mut batch = processing.batch.as_ref().unwrap().to_batch();
-                    batch.progress = processing.get_progress_view();
-                    // Add progress_trace from the current progress state
-                    if let Some(progress) = &processing.progress {
-                        batch.stats.progress_trace = progress
-                            .accumulated_durations()
-                            .into_iter()
-                            .map(|(k, v)| (k, v.into()))
-                            .collect();
-                    }
-                    Ok(batch)
-                } else {
-                    self.get_batch(rtxn, batch_id)
-                        .and_then(|batch| batch.ok_or(Error::CorruptedTaskQueue))
-                }
-            })
-            .collect::<Result<_>>()
-    }
-}
-
-impl Queue {
-    /// Return the batch ids matched by the given query from the index scheduler's point of view.
-    pub(crate) fn get_batch_ids(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        processing: &ProcessingTasks,
-    ) -> Result<RoaringBitmap> {
-        let Query {
-            limit,
-            from,
-            reverse,
-            uids,
-            batch_uids,
-            statuses,
-            types,
-            index_uids,
-            canceled_by,
-            before_enqueued_at,
-            after_enqueued_at,
-            before_started_at,
-            after_started_at,
-            before_finished_at,
-            after_finished_at,
-        } = query;
-
-        let mut batches = self.batches.all_batch_ids(rtxn)?;
-        if let Some(batch_id) = processing.batch.as_ref().map(|batch| batch.uid) {
-            batches.insert(batch_id);
-        }
-
-        if let Some(from) = from {
-            let range = if reverse.unwrap_or_default() {
-                u32::MIN..*from
-            } else {
-                from.saturating_add(1)..u32::MAX
-            };
-            batches.remove_range(range);
-        }
-
-        if let Some(batch_uids) = &batch_uids {
-            let batches_uids = RoaringBitmap::from_iter(batch_uids);
-            batches &= batches_uids;
-        }
-
-        if let Some(status) = &statuses {
-            let mut status_batches = RoaringBitmap::new();
-            for status in status {
-                match status {
-                    // special case for Processing batches
-                    Status::Processing => {
-                        if let Some(batch_id) = processing.batch.as_ref().map(|batch| batch.uid) {
-                            status_batches.insert(batch_id);
-                        }
-                    }
-                    // Enqueued tasks are not stored in batches
-                    Status::Enqueued => (),
-                    status => status_batches |= &self.batches.get_status(rtxn, *status)?,
-                };
-            }
-            if !status.contains(&Status::Processing) {
-                if let Some(ref batch) = processing.batch {
-                    batches.remove(batch.uid);
-                }
-            }
-            batches &= status_batches;
-        }
-
-        if let Some(task_uids) = &uids {
-            let mut batches_by_task_uids = RoaringBitmap::new();
-            for task_uid in task_uids {
-                if let Some(task) = self.tasks.get_task(rtxn, *task_uid)? {
-                    if let Some(batch_uid) = task.batch_uid {
-                        batches_by_task_uids.insert(batch_uid);
-                    }
-                }
-            }
-            batches &= batches_by_task_uids;
-        }
-
-        // There is no database for this query, we must retrieve the task queried by the client and ensure it's valid
-        if let Some(canceled_by) = &canceled_by {
-            let mut all_canceled_batches = RoaringBitmap::new();
-            for cancel_uid in canceled_by {
-                if let Some(task) = self.tasks.get_task(rtxn, *cancel_uid)? {
-                    if task.kind.as_kind() == Kind::TaskCancelation
-                        && task.status == Status::Succeeded
-                    {
-                        if let Some(batch_uid) = task.batch_uid {
-                            all_canceled_batches.insert(batch_uid);
-                        }
-                    }
-                }
-            }
-
-            // if the canceled_by has been specified but no batch
-            // matches then we prefer matching zero than all batches.
-            if all_canceled_batches.is_empty() {
-                return Ok(RoaringBitmap::new());
-            } else {
-                batches &= all_canceled_batches;
-            }
-        }
-
-        if let Some(kind) = &types {
-            let mut kind_batches = RoaringBitmap::new();
-            for kind in kind {
-                kind_batches |= self.batches.get_kind(rtxn, *kind)?;
-                if let Some(uid) = processing
-                    .batch
-                    .as_ref()
-                    .and_then(|batch| batch.kinds.contains(kind).then_some(batch.uid))
-                {
-                    kind_batches.insert(uid);
-                }
-            }
-            batches &= &kind_batches;
-        }
-
-        if let Some(index) = &index_uids {
-            let mut index_batches = RoaringBitmap::new();
-            for index in index {
-                index_batches |= self.batches.index_batches(rtxn, index)?;
-                if let Some(uid) = processing
-                    .batch
-                    .as_ref()
-                    .and_then(|batch| batch.indexes.contains(index).then_some(batch.uid))
-                {
-                    index_batches.insert(uid);
-                }
-            }
-            batches &= &index_batches;
-        }
-
-        // For the started_at filter, we need to treat the part of the batches that are processing from the part of the
-        // batches that are not processing. The non-processing ones are filtered normally while the processing ones
-        // are entirely removed unless the in-memory startedAt variable falls within the date filter.
-        // Once we have filtered the two subsets, we put them back together and assign it back to `batches`.
-        batches = {
-            let (mut filtered_non_processing_batches, mut filtered_processing_batches) =
-                (&batches - &*processing.processing, &batches & &*processing.processing);
-
-            // special case for Processing batches
-            // A closure that clears the filtered_processing_batches if their started_at date falls outside the given bounds
-            let mut clear_filtered_processing_batches =
-                |start: Bound<OffsetDateTime>, end: Bound<OffsetDateTime>| {
-                    let start = map_bound(start, |b| b.unix_timestamp_nanos());
-                    let end = map_bound(end, |b| b.unix_timestamp_nanos());
-                    let is_within_dates = RangeBounds::contains(
-                        &(start, end),
-                        &processing
-                            .batch
-                            .as_ref()
-                            .map_or_else(OffsetDateTime::now_utc, |batch| batch.started_at)
-                            .unix_timestamp_nanos(),
-                    );
-                    if !is_within_dates {
-                        filtered_processing_batches.clear();
-                    }
-                };
-            match (after_started_at, before_started_at) {
-                (None, None) => (),
-                (None, Some(before)) => {
-                    clear_filtered_processing_batches(Bound::Unbounded, Bound::Excluded(*before))
-                }
-                (Some(after), None) => {
-                    clear_filtered_processing_batches(Bound::Excluded(*after), Bound::Unbounded)
-                }
-                (Some(after), Some(before)) => clear_filtered_processing_batches(
-                    Bound::Excluded(*after),
-                    Bound::Excluded(*before),
-                ),
-            };
-
-            keep_ids_within_datetimes(
-                rtxn,
-                &mut filtered_non_processing_batches,
-                self.batches.started_at,
-                *after_started_at,
-                *before_started_at,
-            )?;
-            filtered_non_processing_batches | filtered_processing_batches
-        };
-
-        keep_ids_within_datetimes(
-            rtxn,
-            &mut batches,
-            self.batches.enqueued_at,
-            *after_enqueued_at,
-            *before_enqueued_at,
-        )?;
-
-        keep_ids_within_datetimes(
-            rtxn,
-            &mut batches,
-            self.batches.finished_at,
-            *after_finished_at,
-            *before_finished_at,
-        )?;
-
-        if let Some(limit) = limit {
-            batches = if query.reverse.unwrap_or_default() {
-                batches.into_iter().take(*limit as usize).collect()
-            } else {
-                batches.into_iter().rev().take(*limit as usize).collect()
-            };
-        }
-
-        Ok(batches)
-    }
-
-    /// Return the batch ids matching the query along with the total number of batches
-    /// by ignoring the from and limit parameters from the user's point of view.
-    ///
-    /// There are two differences between an internal query and a query executed by
-    /// the user.
-    ///
-    /// 1. IndexSwap tasks are not publicly associated with any index, but they are associated
-    ///    with many indexes internally.
-    /// 2. The user may not have the rights to access the tasks (internally) associated with all indexes.
-    pub(crate) fn get_batch_ids_from_authorized_indexes(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-        processing: &ProcessingTasks,
-    ) -> Result<(RoaringBitmap, u64)> {
-        // compute all batches matching the filter by ignoring the limits, to find the number of batches matching
-        // the filter.
-        // As this causes us to compute the filter twice it is slightly inefficient, but doing it this way spares
-        // us from modifying the underlying implementation, and the performance remains sufficient.
-        // Should this change, we would modify `get_batch_ids` to directly return the number of matching batches.
-        let total_batches =
-            self.get_batch_ids(rtxn, &query.clone().without_limits(), processing)?;
-        let mut batches = self.get_batch_ids(rtxn, query, processing)?;
-
-        // If the query contains a list of index uid or there is a finite list of authorized indexes,
-        // then we must exclude all the batches that only contains tasks associated to multiple indexes.
-        // This works because we don't autobatch tasks associated to multiple indexes with tasks associated
-        // to a single index. e.g: IndexSwap cannot be batched with IndexCreation.
-        if query.index_uids.is_some() || !filters.all_indexes_authorized() {
-            for kind in enum_iterator::all::<Kind>().filter(|kind| !kind.related_to_one_index()) {
-                batches -= self.tasks.get_kind(rtxn, kind)?;
-                if let Some(batch) = processing.batch.as_ref() {
-                    if batch.kinds.contains(&kind) {
-                        batches.remove(batch.uid);
-                    }
-                }
-            }
-        }
-
-        // Any batch that is internally associated with at least one authorized index
-        // must be returned.
-        if !filters.all_indexes_authorized() {
-            let mut valid_indexes = RoaringBitmap::new();
-            let mut forbidden_indexes = RoaringBitmap::new();
-
-            let all_indexes_iter = self.batches.index_tasks.iter(rtxn)?;
-            for result in all_indexes_iter {
-                let (index, index_tasks) = result?;
-                if filters.is_index_authorized(index) {
-                    valid_indexes |= index_tasks;
-                } else {
-                    forbidden_indexes |= index_tasks;
-                }
-            }
-            if let Some(batch) = processing.batch.as_ref() {
-                for index in &batch.indexes {
-                    if filters.is_index_authorized(index) {
-                        valid_indexes.insert(batch.uid);
-                    } else {
-                        forbidden_indexes.insert(batch.uid);
-                    }
-                }
-            }
-
-            // If a batch had ONE valid task then it should be returned
-            let invalid_batches = forbidden_indexes - valid_indexes;
-
-            batches -= invalid_batches;
-        }
-
-        Ok((batches, total_batches.len()))
-    }
-
-    pub(crate) fn get_batches_from_authorized_indexes(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-        processing: &ProcessingTasks,
-    ) -> Result<(Vec<Batch>, u64)> {
-        let (batches, total) =
-            self.get_batch_ids_from_authorized_indexes(rtxn, query, filters, processing)?;
-        let batches = if query.reverse.unwrap_or_default() {
-            Box::new(batches.into_iter()) as Box<dyn Iterator<Item = u32>>
-        } else {
-            Box::new(batches.into_iter().rev()) as Box<dyn Iterator<Item = u32>>
-        };
-
-        let batches = self.batches.get_existing_batches(
-            rtxn,
-            batches.take(query.limit.unwrap_or(u32::MAX) as usize),
-            processing,
-        )?;
-
-        Ok((batches, total))
-    }
-}

crates/index-scheduler/src/queue/batches_test.rs

@@ -1,489 +0,0 @@
-use meili_snap::snapshot;
-use meilisearch_auth::AuthFilter;
-use meilisearch_types::index_uid_pattern::IndexUidPattern;
-use meilisearch_types::tasks::{IndexSwap, KindWithContent, Status};
-use time::{Duration, OffsetDateTime};
-
-use crate::insta_snapshot::{snapshot_bitmap, snapshot_index_scheduler};
-use crate::test_utils::Breakpoint::*;
-use crate::test_utils::{index_creation_task, FailureLocation};
-use crate::{IndexScheduler, Query};
-
-#[test]
-fn query_batches_from_and_limit() {
-    let (index_scheduler, mut handle) = IndexScheduler::test(true, vec![]);
-
-    let kind = index_creation_task("doggo", "bone");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_first_task");
-    let kind = index_creation_task("whalo", "plankton");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_second_task");
-    let kind = index_creation_task("catto", "his_own_vomit");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_third_task");
-
-    handle.advance_n_successful_batches(3);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "processed_all_tasks");
-
-    let proc = index_scheduler.processing_tasks.read().unwrap().clone();
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let query = Query { limit: Some(0), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query { limit: Some(1), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query { limit: Some(2), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[1,2,]");
-
-    let query = Query { from: Some(1), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-
-    let query = Query { from: Some(2), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,2,]");
-
-    let query = Query { from: Some(1), limit: Some(1), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-
-    let query = Query { from: Some(1), limit: Some(2), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-}
-
-#[test]
-fn query_batches_simple() {
-    let start_time = OffsetDateTime::now_utc();
-
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("whalo", "fish");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    handle.advance_till([Start, BatchCreated]);
-
-    let query = Query { statuses: Some(vec![Status::Processing]), ..Default::default() };
-    let (mut batches, _) = index_scheduler
-        .get_batches_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    assert_eq!(batches.len(), 1);
-    batches[0].started_at = OffsetDateTime::UNIX_EPOCH;
-    assert!(batches[0].enqueued_at.is_some());
-    batches[0].enqueued_at = None;
-
-    if !batches[0].stats.progress_trace.is_empty() {
-        batches[0].stats.progress_trace.clear();
-        batches[0]
-            .stats
-            .progress_trace
-            .insert("processing tasks".to_string(), "deterministic_duration".into());
-    }
-
-    // Insta cannot snapshot our batches because the batch stats contains an enum as key: https://github.com/mitsuhiko/insta/issues/689
-    let batch = serde_json::to_string_pretty(&batches[0]).unwrap();
-    snapshot!(batch, @r###"
-    {
-      "uid": 0,
-      "details": {
-        "primaryKey": "mouse"
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "processing": 1
-        },
-        "types": {
-          "indexCreation": 1
-        },
-        "indexUids": {
-          "catto": 1
-        },
-        "progressTrace": {
-          "processing tasks": "deterministic_duration"
-        }
-      },
-      "startedAt": "1970-01-01T00:00:00Z",
-      "finishedAt": null,
-      "enqueuedAt": null,
-      "stopReason": "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task."
-    }
-    "###);
-
-    let query = Query { statuses: Some(vec![Status::Enqueued]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[]"); // The batches don't contains any enqueued tasks
-
-    let query =
-        Query { statuses: Some(vec![Status::Enqueued, Status::Processing]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,]"); // both enqueued and processing tasks in the first tick
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        after_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test, which should excludes the enqueued tasks
-    snapshot!(snapshot_bitmap(&batches), @"[0,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        before_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes before the start of the test, which should excludes all of them
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        after_started_at: Some(start_time),
-        before_started_at: Some(start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test and before one minute after the start of the test,
-    // which should exclude the enqueued tasks and include the only processing task
-    snapshot!(snapshot_bitmap(&batches), @"[0,]");
-
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-        Start,
-        BatchCreated,
-    ]);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "after-advancing-a-bit");
-
-    let second_start_time = OffsetDateTime::now_utc();
-
-    let query = Query {
-        statuses: Some(vec![Status::Succeeded, Status::Processing]),
-        after_started_at: Some(start_time),
-        before_started_at: Some(start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test and before one minute after the start of the test,
-    // which should include all tasks
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Succeeded, Status::Processing]),
-        before_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes before the start of the test, which should exclude all tasks
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Succeeded, Status::Processing]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the second part of the test and before one minute after the
-    // second start of the test, which should exclude all tasks
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    // now we make one more batch, the started_at field of the new tasks will be past `second_start_time`
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-        Start,
-        BatchCreated,
-    ]);
-
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // we run the same query to verify that, and indeed find that the last task is matched
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Succeeded, Status::Processing]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // enqueued, succeeded, or processing tasks started after the second part of the test, should
-    // again only return the last task
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    handle.advance_till([ProcessBatchFailed, AfterProcessing]);
-
-    // now the last task should have failed
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "end");
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // so running the last query should return nothing
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // but the same query on failed tasks should return the last task
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // but the same query on failed tasks should return the last task
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        uids: Some(vec![1]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // same query but with an invalid uid
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        uids: Some(vec![2]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // same query but with a valid uid
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-}
-
-#[test]
-fn query_batches_special_rules() {
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()), rename: false }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    handle.advance_till([Start, BatchCreated]);
-
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap().clone();
-
-    let query = Query { index_uids: Some(vec!["catto".to_owned()]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // only the first task associated with catto is returned, the indexSwap tasks are excluded!
-    snapshot!(snapshot_bitmap(&batches), @"[0,]");
-
-    let query = Query { index_uids: Some(vec!["catto".to_owned()]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we have asked for only the tasks associated with catto, but are only authorized to retrieve the tasks
-    // associated with doggo -> empty result
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    drop(rtxn);
-    // We're going to advance and process all the batches for the next query to actually hit the db
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-    ]);
-    handle.advance_one_successful_batch();
-    handle.advance_n_failed_batches(2);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "after-processing-everything");
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-
-    let query = Query::default();
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we asked for all the tasks, but we are only authorized to retrieve the doggo tasks
-    // -> only the index creation of doggo should be returned
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-
-    let query = Query::default();
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![
-                    IndexUidPattern::new_unchecked("catto"),
-                    IndexUidPattern::new_unchecked("doggo"),
-                ]
-                .into_iter()
-                .collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we asked for all the tasks, but we are only authorized to retrieve the doggo and catto tasks
-    // -> all tasks except the swap of catto with whalo are returned
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-
-    let query = Query::default();
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // we asked for all the tasks with all index authorized -> all tasks returned
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,2,3,]");
-}
-
-#[test]
-fn query_batches_canceled_by() {
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _ = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _ = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    handle.advance_n_successful_batches(1);
-    let kind = KindWithContent::TaskCancelation {
-        query: "test_query".to_string(),
-        tasks: [0, 1, 2, 3].into_iter().collect(),
-    };
-    let task_cancelation = index_scheduler.register(kind, None, false).unwrap();
-    handle.advance_n_successful_batches(1);
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    let query = Query { canceled_by: Some(vec![task_cancelation.uid]), ..Query::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // The batch zero was the index creation task, the 1 is the task cancellation
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-
-    let query = Query { canceled_by: Some(vec![task_cancelation.uid]), ..Query::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-        )
-        .unwrap();
-    // Return only 1 because the user is not authorized to see task 2
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-}

crates/index-scheduler/src/queue/mod.rs

@@ -1,394 +0,0 @@
-mod batches;
-#[cfg(test)]
-mod batches_test;
-mod tasks;
-#[cfg(test)]
-mod tasks_test;
-#[cfg(test)]
-mod test;
-
-use std::collections::BTreeMap;
-use std::fs::File as StdFile;
-use std::time::Duration;
-
-use file_store::FileStore;
-use meilisearch_types::batches::BatchId;
-use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
-use meilisearch_types::milli::{CboRoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
-use roaring::RoaringBitmap;
-use time::format_description::well_known::Rfc3339;
-use time::OffsetDateTime;
-use uuid::Uuid;
-
-pub(crate) use self::batches::BatchQueue;
-pub(crate) use self::tasks::TaskQueue;
-use crate::processing::ProcessingTasks;
-use crate::utils::{
-    check_index_swap_validity, filter_out_references_to_newer_tasks, ProcessingBatch,
-};
-use crate::{Error, IndexSchedulerOptions, Result, TaskId};
-
-/// The number of database used by queue itself
-const NUMBER_OF_DATABASES: u32 = 1;
-/// Database const names for the `IndexScheduler`.
-mod db_name {
-    pub const BATCH_TO_TASKS_MAPPING: &str = "batch-to-tasks-mapping";
-}
-
-/// Defines a subset of tasks to be retrieved from the [`IndexScheduler`].
-///
-/// An empty/default query (where each field is set to `None`) matches all tasks.
-/// Each non-null field restricts the set of tasks further.
-#[derive(Default, Debug, Clone, PartialEq, Eq)]
-pub struct Query {
-    /// The maximum number of tasks to be matched
-    pub limit: Option<u32>,
-    /// The minimum [task id](`meilisearch_types::tasks::Task::uid`) to be matched
-    pub from: Option<u32>,
-    /// The order used to return the tasks. By default the newest tasks are returned first and the boolean is `false`.
-    pub reverse: Option<bool>,
-    /// The [task ids](`meilisearch_types::tasks::Task::uid`) to be matched
-    pub uids: Option<Vec<TaskId>>,
-    /// The [batch ids](`meilisearch_types::batches::Batch::uid`) to be matched
-    pub batch_uids: Option<Vec<BatchId>>,
-    /// The allowed [statuses](`meilisearch_types::tasks::Task::status`) of the matched tasls
-    pub statuses: Option<Vec<Status>>,
-    /// The allowed [kinds](meilisearch_types::tasks::Kind) of the matched tasks.
-    ///
-    /// The kind of a task is given by:
-    /// ```
-    /// # use meilisearch_types::tasks::{Task, Kind};
-    /// # fn doc_func(task: Task) -> Kind {
-    /// task.kind.as_kind()
-    /// # }
-    /// ```
-    pub types: Option<Vec<Kind>>,
-    /// The allowed [index ids](meilisearch_types::tasks::Task::index_uid) of the matched tasks
-    pub index_uids: Option<Vec<String>>,
-    /// The [task ids](`meilisearch_types::tasks::Task::uid`) of the [`TaskCancelation`](meilisearch_types::tasks::Task::Kind::TaskCancelation) tasks
-    /// that canceled the matched tasks.
-    pub canceled_by: Option<Vec<TaskId>>,
-    /// Exclusive upper bound of the matched tasks' [`enqueued_at`](meilisearch_types::tasks::Task::enqueued_at) field.
-    pub before_enqueued_at: Option<OffsetDateTime>,
-    /// Exclusive lower bound of the matched tasks' [`enqueued_at`](meilisearch_types::tasks::Task::enqueued_at) field.
-    pub after_enqueued_at: Option<OffsetDateTime>,
-    /// Exclusive upper bound of the matched tasks' [`started_at`](meilisearch_types::tasks::Task::started_at) field.
-    pub before_started_at: Option<OffsetDateTime>,
-    /// Exclusive lower bound of the matched tasks' [`started_at`](meilisearch_types::tasks::Task::started_at) field.
-    pub after_started_at: Option<OffsetDateTime>,
-    /// Exclusive upper bound of the matched tasks' [`finished_at`](meilisearch_types::tasks::Task::finished_at) field.
-    pub before_finished_at: Option<OffsetDateTime>,
-    /// Exclusive lower bound of the matched tasks' [`finished_at`](meilisearch_types::tasks::Task::finished_at) field.
-    pub after_finished_at: Option<OffsetDateTime>,
-}
-
-impl Query {
-    /// Return `true` if every field of the query is set to `None`, such that the query
-    /// matches all tasks.
-    pub fn is_empty(&self) -> bool {
-        matches!(
-            self,
-            Query {
-                limit: None,
-                from: None,
-                reverse: None,
-                uids: None,
-                batch_uids: None,
-                statuses: None,
-                types: None,
-                index_uids: None,
-                canceled_by: None,
-                before_enqueued_at: None,
-                after_enqueued_at: None,
-                before_started_at: None,
-                after_started_at: None,
-                before_finished_at: None,
-                after_finished_at: None,
-            }
-        )
-    }
-
-    /// Add an [index id](meilisearch_types::tasks::Task::index_uid) to the list of permitted indexes.
-    pub fn with_index(self, index_uid: String) -> Self {
-        let mut index_vec = self.index_uids.unwrap_or_default();
-        index_vec.push(index_uid);
-        Self { index_uids: Some(index_vec), ..self }
-    }
-
-    // Removes the `from` and `limit` restrictions from the query.
-    // Useful to get the total number of tasks matching a filter.
-    pub fn without_limits(self) -> Self {
-        Query { limit: None, from: None, ..self }
-    }
-}
-
-/// Structure which holds meilisearch's indexes and schedules the tasks
-/// to be performed on them.
-pub struct Queue {
-    pub(crate) tasks: tasks::TaskQueue,
-    pub(crate) batches: batches::BatchQueue,
-
-    /// Matches a batch id with the associated task ids.
-    pub(crate) batch_to_tasks_mapping: Database<BEU32, CboRoaringBitmapCodec>,
-
-    /// The list of files referenced by the tasks.
-    pub(crate) file_store: FileStore,
-
-    /// The max number of tasks allowed before the scheduler starts to delete
-    /// the finished tasks automatically.
-    pub(crate) max_number_of_tasks: usize,
-}
-
-impl Queue {
-    pub(crate) fn private_clone(&self) -> Queue {
-        Queue {
-            tasks: self.tasks.private_clone(),
-            batches: self.batches.private_clone(),
-            batch_to_tasks_mapping: self.batch_to_tasks_mapping,
-            file_store: self.file_store.clone(),
-            max_number_of_tasks: self.max_number_of_tasks,
-        }
-    }
-
-    pub(crate) const fn nb_db() -> u32 {
-        tasks::TaskQueue::nb_db() + batches::BatchQueue::nb_db() + NUMBER_OF_DATABASES
-    }
-
-    /// Create an index scheduler and start its run loop.
-    pub(crate) fn new(
-        env: &Env<WithoutTls>,
-        wtxn: &mut RwTxn,
-        options: &IndexSchedulerOptions,
-    ) -> Result<Self> {
-        // allow unreachable_code to get rids of the warning in the case of a test build.
-        Ok(Self {
-            file_store: FileStore::new(&options.update_file_path)?,
-            batch_to_tasks_mapping: env
-                .create_database(wtxn, Some(db_name::BATCH_TO_TASKS_MAPPING))?,
-            tasks: TaskQueue::new(env, wtxn)?,
-            batches: BatchQueue::new(env, wtxn)?,
-            max_number_of_tasks: options.max_number_of_tasks,
-        })
-    }
-
-    /// Returns the whole set of tasks that belongs to this batch.
-    pub(crate) fn tasks_in_batch(&self, rtxn: &RoTxn, batch_id: BatchId) -> Result<RoaringBitmap> {
-        Ok(self.batch_to_tasks_mapping.get(rtxn, &batch_id)?.unwrap_or_default())
-    }
-
-    /// Convert an iterator to a `Vec` of tasks and edit the `ProcessingBatch` to add the given tasks.
-    ///
-    /// The tasks MUST exist, or a `CorruptedTaskQueue` error will be thrown.
-    pub(crate) fn get_existing_tasks_for_processing_batch(
-        &self,
-        rtxn: &RoTxn,
-        processing_batch: &mut ProcessingBatch,
-        tasks: impl IntoIterator<Item = TaskId>,
-    ) -> Result<Vec<Task>> {
-        tasks
-            .into_iter()
-            .map(|task_id| {
-                let mut task = self
-                    .tasks
-                    .get_task(rtxn, task_id)
-                    .and_then(|task| task.ok_or(Error::CorruptedTaskQueue));
-                processing_batch.processing(&mut task);
-                task
-            })
-            .collect::<Result<_>>()
-    }
-
-    pub(crate) fn write_batch(
-        &self,
-        wtxn: &mut RwTxn,
-        batch: ProcessingBatch,
-        tasks: &RoaringBitmap,
-    ) -> Result<()> {
-        self.batch_to_tasks_mapping.put(wtxn, &batch.uid, tasks)?;
-        self.batches.write_batch(wtxn, batch)?;
-        Ok(())
-    }
-
-    pub(crate) fn delete_persisted_task_data(&self, task: &Task) -> Result<()> {
-        match task.content_uuid() {
-            Some(content_file) => self.delete_update_file(content_file),
-            None => Ok(()),
-        }
-    }
-
-    /// Open and returns the task's content File.
-    pub fn update_file(&self, uuid: Uuid) -> file_store::Result<StdFile> {
-        self.file_store.get_update(uuid)
-    }
-
-    /// Delete a file from the index scheduler.
-    ///
-    /// Counterpart to the [`create_update_file`](IndexScheduler::create_update_file) method.
-    pub fn delete_update_file(&self, uuid: Uuid) -> Result<()> {
-        Ok(self.file_store.delete(uuid)?)
-    }
-
-    /// Create a file and register it in the index scheduler.
-    ///
-    /// The returned file and uuid can be used to associate
-    /// some data to a task. The file will be kept until
-    /// the task has been fully processed.
-    pub fn create_update_file(&self, dry_run: bool) -> Result<(Uuid, file_store::File)> {
-        if dry_run {
-            Ok((Uuid::nil(), file_store::File::dry_file()?))
-        } else {
-            Ok(self.file_store.new_update()?)
-        }
-    }
-
-    #[cfg(test)]
-    pub fn create_update_file_with_uuid(&self, uuid: u128) -> Result<(Uuid, file_store::File)> {
-        Ok(self.file_store.new_update_with_uuid(uuid)?)
-    }
-
-    /// The size on disk taken by all the updates files contained in the `IndexScheduler`, in bytes.
-    pub fn compute_update_file_size(&self) -> Result<u64> {
-        Ok(self.file_store.compute_total_size()?)
-    }
-
-    pub fn register(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: &KindWithContent,
-        task_id: Option<TaskId>,
-        custom_metadata: Option<String>,
-        dry_run: bool,
-    ) -> Result<Task> {
-        let next_task_id = self.tasks.next_task_id(wtxn)?;
-
-        if let Some(uid) = task_id {
-            if uid < next_task_id {
-                return Err(Error::BadTaskId { received: uid, expected: next_task_id });
-            }
-        }
-
-        let mut task = Task {
-            uid: task_id.unwrap_or(next_task_id),
-            // The batch is defined once we starts processing the task
-            batch_uid: None,
-            enqueued_at: OffsetDateTime::now_utc(),
-            started_at: None,
-            finished_at: None,
-            error: None,
-            canceled_by: None,
-            details: kind.default_details(),
-            status: Status::Enqueued,
-            kind: kind.clone(),
-            network: None,
-            custom_metadata,
-        };
-        // For deletion and cancelation tasks, we want to make extra sure that they
-        // don't attempt to delete/cancel tasks that are newer than themselves.
-        filter_out_references_to_newer_tasks(&mut task);
-        // If the register task is an index swap task, verify that it is well-formed
-        // (that it does not contain duplicate indexes).
-        check_index_swap_validity(&task)?;
-
-        // At this point the task is going to be registered and no further checks will be done
-        if dry_run {
-            return Ok(task);
-        }
-
-        self.tasks.register(wtxn, &task)?;
-
-        Ok(task)
-    }
-
-    /// Register a task to cleanup the task queue if needed
-    pub fn cleanup_task_queue(&self, wtxn: &mut RwTxn) -> Result<()> {
-        let nb_tasks = self.tasks.all_task_ids(wtxn)?.len();
-        // if we have less than 1M tasks everything is fine
-        if nb_tasks < self.max_number_of_tasks as u64 {
-            return Ok(());
-        }
-
-        let finished = self.tasks.status.get(wtxn, &Status::Succeeded)?.unwrap_or_default()
-            | self.tasks.status.get(wtxn, &Status::Failed)?.unwrap_or_default()
-            | self.tasks.status.get(wtxn, &Status::Canceled)?.unwrap_or_default();
-
-        let to_delete =
-            RoaringBitmap::from_sorted_iter(finished.into_iter().take(100_000)).unwrap();
-
-        // /!\ the len must be at least 2 or else we might enter an infinite loop where we only delete
-        //     the deletion tasks we enqueued ourselves.
-        if to_delete.len() < 2 {
-            tracing::warn!("The task queue is almost full, but no task can be deleted yet.");
-            // the only thing we can do is hope that the user tasks are going to finish
-            return Ok(());
-        }
-
-        tracing::info!(
-            "The task queue is almost full. Deleting the oldest {} finished tasks.",
-            to_delete.len()
-        );
-
-        // it's safe to unwrap here because we checked the len above
-        let newest_task_id = to_delete.iter().next_back().unwrap();
-        let last_task_to_delete =
-            self.tasks.get_task(wtxn, newest_task_id)?.ok_or(Error::CorruptedTaskQueue)?;
-
-        // increase time by one nanosecond so that the enqueuedAt of the last task to delete is also lower than that date.
-        let delete_before = last_task_to_delete.enqueued_at + Duration::from_nanos(1);
-
-        self.register(
-            wtxn,
-            &KindWithContent::TaskDeletion {
-                query: format!(
-                    "?beforeEnqueuedAt={}&statuses=succeeded,failed,canceled",
-                    delete_before.format(&Rfc3339).map_err(|_| Error::CorruptedTaskQueue)?,
-                ),
-                tasks: to_delete,
-            },
-            None,
-            None,
-            false,
-        )?;
-
-        Ok(())
-    }
-
-    pub fn get_stats(
-        &self,
-        rtxn: &RoTxn,
-        processing: &ProcessingTasks,
-    ) -> Result<BTreeMap<String, BTreeMap<String, u64>>> {
-        let mut res = BTreeMap::new();
-        let processing_tasks = processing.processing.len();
-
-        res.insert(
-            "statuses".to_string(),
-            enum_iterator::all::<Status>()
-                .map(|s| {
-                    let tasks = self.tasks.get_status(rtxn, s)?.len();
-                    match s {
-                        Status::Enqueued => Ok((s.to_string(), tasks - processing_tasks)),
-                        Status::Processing => Ok((s.to_string(), processing_tasks)),
-                        s => Ok((s.to_string(), tasks)),
-                    }
-                })
-                .collect::<Result<BTreeMap<String, u64>>>()?,
-        );
-        res.insert(
-            "types".to_string(),
-            enum_iterator::all::<Kind>()
-                .map(|s| Ok((s.to_string(), self.tasks.get_kind(rtxn, s)?.len())))
-                .collect::<Result<BTreeMap<String, u64>>>()?,
-        );
-        res.insert(
-            "indexes".to_string(),
-            self.tasks
-                .index_tasks
-                .iter(rtxn)?
-                .map(|res| Ok(res.map(|(name, bitmap)| (name.to_string(), bitmap.len()))?))
-                .collect::<Result<BTreeMap<String, u64>>>()?,
-        );
-
-        Ok(res)
-    }
-}

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_canceled_by/start.snap

@@ -1,86 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
-3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,3,]
-canceled [1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,]
-"taskCancelation" [3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,]
-doggo [1,2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-3 [1,2,]
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"],"rename":false}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 3 of type `taskCancelation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,2,3,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-canceled [1,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,]
-"indexSwap" [1,]
-"taskCancelation" [1,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,1,]
-doggo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/processed_all_tasks.snap

@@ -1,87 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-whalo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"bone"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"plankton"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"whalo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-2 {uid: 2, details: {"primaryKey":"his_own_vomit"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 2 of type `indexCreation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,2,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_second_task.snap

@@ -1,54 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-----------------------------------------------------------------------
-### Index Tasks:
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_third_task.snap

@@ -1,57 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/after-advancing-a-bit.snap

@@ -1,70 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch Some(1):
-[1,]
-{uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [1,2,]
-succeeded [0,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/end.snap

@@ -1,88 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,]
-failed [2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-2 {uid: 2, details: {"primaryKey":"fish"}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexCreation":1},"indexUids":{"whalo":1}}, stop reason: "created batch containing only task with id 2 of type `indexCreation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-failed [2,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/start.snap

@@ -1,57 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_special_rules/after-processing-everything.snap

@@ -1,99 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
-3 {uid: 3, batch_uid: 3, status: failed, error: ResponseError { code: 200, message: "Index `whalo` not found.", error_code: "index_not_found", error_type: "invalid_request", error_link: "https://docs.meilisearch.com/errors#index_not_found" }, details: { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,]
-failed [2,3,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,3,]
-doggo [1,2,]
-whalo [3,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-2 {uid: 2, details: {"swaps":[{"indexes":["catto","doggo"],"rename":false}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, stop reason: "created batch containing only task with id 2 of type `indexSwap` that cannot be batched with any other task.", }
-3 {uid: 3, details: {"swaps":[{"indexes":["catto","whalo"],"rename":false}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, stop reason: "created batch containing only task with id 3 of type `indexSwap` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-3 [3,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-failed [2,3,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,3,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,2,3,]
-doggo [1,2,]
-whalo [3,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_special_rules/start.snap

@@ -1,60 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
-3 {uid: 3, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,3,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,3,]
-doggo [1,2,]
-whalo [3,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_canceled_by/start.snap

@@ -1,86 +0,0 @@
----
-source: crates/index-scheduler/src/queue/tasks_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
-3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,3,]
-canceled [1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,]
-"taskCancelation" [3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,]
-doggo [1,2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-3 [1,2,]
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"],"rename":false}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 3 of type `taskCancelation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,2,3,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-canceled [1,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,]
-"indexSwap" [1,]
-"taskCancelation" [1,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,1,]
-doggo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_from_and_limit/processed_all_tasks.snap

@@ -1,87 +0,0 @@
----
-source: crates/index-scheduler/src/queue/tasks_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-whalo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"bone"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"plankton"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"whalo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-2 {uid: 2, details: {"primaryKey":"his_own_vomit"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 2 of type `indexCreation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,2,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_from_and_limit/registered_the_third_task.snap

@@ -1,57 +0,0 @@
----
-source: crates/index-scheduler/src/queue/tasks_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_simple/end.snap

@@ -1,88 +0,0 @@
----
-source: crates/index-scheduler/src/queue/tasks_test.rs
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,]
-failed [2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-2 {uid: 2, details: {"primaryKey":"fish"}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexCreation":1},"indexUids":{"whalo":1}}, stop reason: "created batch containing only task with id 2 of type `indexCreation` that cannot be batched with any other task.", }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-failed [2,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/tasks.rs

@@ -1,562 +0,0 @@
-use std::ops::{Bound, RangeBounds};
-
-use meilisearch_types::heed::types::{DecodeIgnore, SerdeBincode, SerdeJson, Str};
-use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
-use meilisearch_types::milli::{CboRoaringBitmapCodec, RoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Kind, Status, Task};
-use roaring::{MultiOps, RoaringBitmap};
-use time::OffsetDateTime;
-
-use super::{Query, Queue};
-use crate::processing::ProcessingTasks;
-use crate::utils::{
-    self, insert_task_datetime, keep_ids_within_datetimes, map_bound, remove_task_datetime,
-};
-use crate::{Error, Result, TaskId, BEI128};
-
-/// The number of database used by the task queue
-const NUMBER_OF_DATABASES: u32 = 8;
-/// Database const names for the `IndexScheduler`.
-mod db_name {
-    pub const ALL_TASKS: &str = "all-tasks";
-
-    pub const STATUS: &str = "status";
-    pub const KIND: &str = "kind";
-    pub const INDEX_TASKS: &str = "index-tasks";
-    pub const CANCELED_BY: &str = "canceled_by";
-    pub const ENQUEUED_AT: &str = "enqueued-at";
-    pub const STARTED_AT: &str = "started-at";
-    pub const FINISHED_AT: &str = "finished-at";
-}
-
-pub struct TaskQueue {
-    /// The main database, it contains all the tasks accessible by their Id.
-    pub(crate) all_tasks: Database<BEU32, SerdeJson<Task>>,
-
-    /// All the tasks ids grouped by their status.
-    // TODO we should not be able to serialize a `Status::Processing` in this database.
-    pub(crate) status: Database<SerdeBincode<Status>, RoaringBitmapCodec>,
-    /// All the tasks ids grouped by their kind.
-    pub(crate) kind: Database<SerdeBincode<Kind>, RoaringBitmapCodec>,
-    /// Store the tasks associated to an index.
-    pub(crate) index_tasks: Database<Str, RoaringBitmapCodec>,
-    /// Store the tasks that were canceled by a task uid
-    pub(crate) canceled_by: Database<BEU32, RoaringBitmapCodec>,
-    /// Store the task ids of tasks which were enqueued at a specific date
-    pub(crate) enqueued_at: Database<BEI128, CboRoaringBitmapCodec>,
-    /// Store the task ids of finished tasks which started being processed at a specific date
-    pub(crate) started_at: Database<BEI128, CboRoaringBitmapCodec>,
-    /// Store the task ids of tasks which finished at a specific date
-    pub(crate) finished_at: Database<BEI128, CboRoaringBitmapCodec>,
-}
-
-impl TaskQueue {
-    pub(crate) fn private_clone(&self) -> TaskQueue {
-        TaskQueue {
-            all_tasks: self.all_tasks,
-            status: self.status,
-            kind: self.kind,
-            index_tasks: self.index_tasks,
-            canceled_by: self.canceled_by,
-            enqueued_at: self.enqueued_at,
-            started_at: self.started_at,
-            finished_at: self.finished_at,
-        }
-    }
-
-    pub(crate) const fn nb_db() -> u32 {
-        NUMBER_OF_DATABASES
-    }
-
-    pub(crate) fn new(env: &Env<WithoutTls>, wtxn: &mut RwTxn) -> Result<Self> {
-        Ok(Self {
-            all_tasks: env.create_database(wtxn, Some(db_name::ALL_TASKS))?,
-            status: env.create_database(wtxn, Some(db_name::STATUS))?,
-            kind: env.create_database(wtxn, Some(db_name::KIND))?,
-            index_tasks: env.create_database(wtxn, Some(db_name::INDEX_TASKS))?,
-            canceled_by: env.create_database(wtxn, Some(db_name::CANCELED_BY))?,
-            enqueued_at: env.create_database(wtxn, Some(db_name::ENQUEUED_AT))?,
-            started_at: env.create_database(wtxn, Some(db_name::STARTED_AT))?,
-            finished_at: env.create_database(wtxn, Some(db_name::FINISHED_AT))?,
-        })
-    }
-
-    pub(crate) fn last_task_id(&self, rtxn: &RoTxn) -> Result<Option<TaskId>> {
-        Ok(self.all_tasks.remap_data_type::<DecodeIgnore>().last(rtxn)?.map(|(k, _)| k + 1))
-    }
-
-    pub(crate) fn next_task_id(&self, rtxn: &RoTxn) -> Result<TaskId> {
-        Ok(self.last_task_id(rtxn)?.unwrap_or_default())
-    }
-
-    pub(crate) fn all_task_ids(&self, rtxn: &RoTxn) -> Result<RoaringBitmap> {
-        enum_iterator::all().map(|s| self.get_status(rtxn, s)).union()
-    }
-
-    pub(crate) fn get_task(&self, rtxn: &RoTxn, task_id: TaskId) -> Result<Option<Task>> {
-        Ok(self.all_tasks.get(rtxn, &task_id)?)
-    }
-
-    /// Update the inverted task indexes and write the new value of the task.
-    ///
-    /// The passed `task` object typically comes from a previous transaction, so two kinds of modification might have occurred:
-    /// 1. Modification to the `task` object after loading it from the DB (the purpose of this method is to persist these changes)
-    /// 2. Modification to the task committed by another transaction in the DB (an annoying consequence of having lost the original
-    ///    transaction from which the `task` instance was deserialized)
-    ///
-    /// When calling this function, this `task` is modified to take into account any existing `network`
-    /// that can have been added since the task was loaded into memory.
-    ///
-    /// Any other modification to the task that was committed from the DB since the parameter was pulled from the DB will be overwritten.
-    ///
-    /// # Errors
-    ///
-    /// - CorruptedTaskQueue: The task doesn't exist in the database
-    pub(crate) fn update_task(&self, wtxn: &mut RwTxn, task: &mut Task) -> Result<()> {
-        let old_task = self.get_task(wtxn, task.uid)?.ok_or(Error::CorruptedTaskQueue)?;
-        let reprocessing = old_task.status != Status::Enqueued;
-
-        debug_assert!(old_task != *task);
-        debug_assert_eq!(old_task.uid, task.uid);
-
-        // If we're processing a task that failed it may already contains a batch_uid
-        debug_assert!(
-            reprocessing || (old_task.batch_uid.is_none() && task.batch_uid.is_some()),
-            "\n==> old: {old_task:?}\n==> new: {task:?}"
-        );
-
-        if old_task.status != task.status {
-            self.update_status(wtxn, old_task.status, |bitmap| {
-                bitmap.remove(task.uid);
-            })?;
-            self.update_status(wtxn, task.status, |bitmap| {
-                bitmap.insert(task.uid);
-            })?;
-        }
-
-        if old_task.kind.as_kind() != task.kind.as_kind() {
-            self.update_kind(wtxn, old_task.kind.as_kind(), |bitmap| {
-                bitmap.remove(task.uid);
-            })?;
-            self.update_kind(wtxn, task.kind.as_kind(), |bitmap| {
-                bitmap.insert(task.uid);
-            })?;
-        }
-
-        assert_eq!(
-            old_task.enqueued_at, task.enqueued_at,
-            "Cannot update a task's enqueued_at time"
-        );
-        if old_task.started_at != task.started_at {
-            assert!(
-                reprocessing || old_task.started_at.is_none(),
-                "Cannot update a task's started_at time"
-            );
-            if let Some(started_at) = old_task.started_at {
-                remove_task_datetime(wtxn, self.started_at, started_at, task.uid)?;
-            }
-            if let Some(started_at) = task.started_at {
-                insert_task_datetime(wtxn, self.started_at, started_at, task.uid)?;
-            }
-        }
-        if old_task.finished_at != task.finished_at {
-            assert!(
-                reprocessing || old_task.finished_at.is_none(),
-                "Cannot update a task's finished_at time"
-            );
-            if let Some(finished_at) = old_task.finished_at {
-                remove_task_datetime(wtxn, self.finished_at, finished_at, task.uid)?;
-            }
-            if let Some(finished_at) = task.finished_at {
-                insert_task_datetime(wtxn, self.finished_at, finished_at, task.uid)?;
-            }
-        }
-
-        task.network = match (old_task.network, task.network.take()) {
-            (None, None) => None,
-            (None, Some(network)) | (Some(network), None) => Some(network),
-            (Some(_), Some(network)) => Some(network),
-        };
-
-        self.all_tasks.put(wtxn, &task.uid, task)?;
-        Ok(())
-    }
-
-    /// Returns the whole set of tasks that belongs to this index.
-    pub(crate) fn index_tasks(&self, rtxn: &RoTxn, index: &str) -> Result<RoaringBitmap> {
-        Ok(self.index_tasks.get(rtxn, index)?.unwrap_or_default())
-    }
-
-    pub(crate) fn update_index(
-        &self,
-        wtxn: &mut RwTxn,
-        index: &str,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.index_tasks(wtxn, index)?;
-        f(&mut tasks);
-        if tasks.is_empty() {
-            self.index_tasks.delete(wtxn, index)?;
-        } else {
-            self.index_tasks.put(wtxn, index, &tasks)?;
-        }
-
-        Ok(())
-    }
-
-    pub(crate) fn get_status(&self, rtxn: &RoTxn, status: Status) -> Result<RoaringBitmap> {
-        Ok(self.status.get(rtxn, &status)?.unwrap_or_default())
-    }
-
-    pub(crate) fn put_status(
-        &self,
-        wtxn: &mut RwTxn,
-        status: Status,
-        bitmap: &RoaringBitmap,
-    ) -> Result<()> {
-        Ok(self.status.put(wtxn, &status, bitmap)?)
-    }
-
-    pub(crate) fn update_status(
-        &self,
-        wtxn: &mut RwTxn,
-        status: Status,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.get_status(wtxn, status)?;
-        f(&mut tasks);
-        self.put_status(wtxn, status, &tasks)?;
-
-        Ok(())
-    }
-
-    pub(crate) fn get_kind(&self, rtxn: &RoTxn, kind: Kind) -> Result<RoaringBitmap> {
-        Ok(self.kind.get(rtxn, &kind)?.unwrap_or_default())
-    }
-
-    pub(crate) fn put_kind(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: Kind,
-        bitmap: &RoaringBitmap,
-    ) -> Result<()> {
-        Ok(self.kind.put(wtxn, &kind, bitmap)?)
-    }
-
-    pub(crate) fn update_kind(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: Kind,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.get_kind(wtxn, kind)?;
-        f(&mut tasks);
-        self.put_kind(wtxn, kind, &tasks)?;
-
-        Ok(())
-    }
-
-    /// Convert an iterator to a `Vec` of tasks. The tasks MUST exist or a
-    /// `CorruptedTaskQueue` error will be thrown.
-    pub(crate) fn get_existing_tasks(
-        &self,
-        rtxn: &RoTxn,
-        tasks: impl IntoIterator<Item = TaskId>,
-    ) -> Result<Vec<Task>> {
-        tasks
-            .into_iter()
-            .map(|task_id| {
-                self.get_task(rtxn, task_id).and_then(|task| task.ok_or(Error::CorruptedTaskQueue))
-            })
-            .collect::<Result<_>>()
-    }
-
-    pub(crate) fn register(&self, wtxn: &mut RwTxn, task: &Task) -> Result<()> {
-        self.all_tasks.put(wtxn, &task.uid, task)?;
-
-        for index in task.indexes() {
-            self.update_index(wtxn, index, |bitmap| {
-                bitmap.insert(task.uid);
-            })?;
-        }
-
-        self.update_status(wtxn, Status::Enqueued, |bitmap| {
-            bitmap.insert(task.uid);
-        })?;
-
-        self.update_kind(wtxn, task.kind.as_kind(), |bitmap| {
-            bitmap.insert(task.uid);
-        })?;
-
-        utils::insert_task_datetime(wtxn, self.enqueued_at, task.enqueued_at, task.uid)?;
-
-        Ok(())
-    }
-}
-
-impl Queue {
-    /// Return the task ids matched by the given query from the index scheduler's point of view.
-    pub(crate) fn get_task_ids(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        processing_tasks: &ProcessingTasks,
-    ) -> Result<RoaringBitmap> {
-        let ProcessingTasks { batch: processing_batch, processing: processing_tasks, progress: _ } =
-            processing_tasks;
-        let Query {
-            limit,
-            from,
-            reverse,
-            uids,
-            batch_uids,
-            statuses,
-            types,
-            index_uids,
-            canceled_by,
-            before_enqueued_at,
-            after_enqueued_at,
-            before_started_at,
-            after_started_at,
-            before_finished_at,
-            after_finished_at,
-        } = query;
-
-        let mut tasks = self.tasks.all_task_ids(rtxn)?;
-
-        if let Some(from) = from {
-            let range = if reverse.unwrap_or_default() {
-                u32::MIN..*from
-            } else {
-                from.saturating_add(1)..u32::MAX
-            };
-            tasks.remove_range(range);
-        }
-
-        if let Some(batch_uids) = batch_uids {
-            let mut batch_tasks = RoaringBitmap::new();
-            for batch_uid in batch_uids {
-                if processing_batch.as_ref().is_some_and(|batch| batch.uid == *batch_uid) {
-                    batch_tasks |= &**processing_tasks;
-                } else {
-                    batch_tasks |= self.tasks_in_batch(rtxn, *batch_uid)?;
-                }
-            }
-            tasks &= batch_tasks;
-        }
-
-        if let Some(status) = statuses {
-            let mut status_tasks = RoaringBitmap::new();
-            for status in status {
-                match status {
-                    // special case for Processing tasks
-                    Status::Processing => {
-                        status_tasks |= &**processing_tasks;
-                    }
-                    status => status_tasks |= &self.tasks.get_status(rtxn, *status)?,
-                };
-            }
-            if !status.contains(&Status::Processing) {
-                tasks -= &**processing_tasks;
-            }
-            tasks &= status_tasks;
-        }
-
-        if let Some(uids) = uids {
-            let uids = RoaringBitmap::from_iter(uids);
-            tasks &= &uids;
-        }
-
-        if let Some(canceled_by) = canceled_by {
-            let mut all_canceled_tasks = RoaringBitmap::new();
-            for cancel_task_uid in canceled_by {
-                if let Some(canceled_by_uid) = self.tasks.canceled_by.get(rtxn, cancel_task_uid)? {
-                    all_canceled_tasks |= canceled_by_uid;
-                }
-            }
-
-            // if the canceled_by has been specified but no task
-            // matches then we prefer matching zero than all tasks.
-            if all_canceled_tasks.is_empty() {
-                return Ok(RoaringBitmap::new());
-            } else {
-                tasks &= all_canceled_tasks;
-            }
-        }
-
-        if let Some(kind) = types {
-            let mut kind_tasks = RoaringBitmap::new();
-            for kind in kind {
-                kind_tasks |= self.tasks.get_kind(rtxn, *kind)?;
-            }
-            tasks &= &kind_tasks;
-        }
-
-        if let Some(index) = index_uids {
-            let mut index_tasks = RoaringBitmap::new();
-            for index in index {
-                index_tasks |= self.tasks.index_tasks(rtxn, index)?;
-            }
-            tasks &= &index_tasks;
-        }
-
-        // For the started_at filter, we need to treat the part of the tasks that are processing from the part of the
-        // tasks that are not processing. The non-processing ones are filtered normally while the processing ones
-        // are entirely removed unless the in-memory startedAt variable falls within the date filter.
-        // Once we have filtered the two subsets, we put them back together and assign it back to `tasks`.
-        tasks = {
-            let (mut filtered_non_processing_tasks, mut filtered_processing_tasks) =
-                (&tasks - &**processing_tasks, &tasks & &**processing_tasks);
-
-            // special case for Processing tasks
-            // A closure that clears the filtered_processing_tasks if their started_at date falls outside the given bounds
-            let mut clear_filtered_processing_tasks =
-                |start: Bound<OffsetDateTime>, end: Bound<OffsetDateTime>| {
-                    let start = map_bound(start, |b| b.unix_timestamp_nanos());
-                    let end = map_bound(end, |b| b.unix_timestamp_nanos());
-                    let is_within_dates = RangeBounds::contains(
-                        &(start, end),
-                        &processing_batch
-                            .as_ref()
-                            .map_or_else(OffsetDateTime::now_utc, |batch| batch.started_at)
-                            .unix_timestamp_nanos(),
-                    );
-                    if !is_within_dates {
-                        filtered_processing_tasks.clear();
-                    }
-                };
-            match (after_started_at, before_started_at) {
-                (None, None) => (),
-                (None, Some(before)) => {
-                    clear_filtered_processing_tasks(Bound::Unbounded, Bound::Excluded(*before))
-                }
-                (Some(after), None) => {
-                    clear_filtered_processing_tasks(Bound::Excluded(*after), Bound::Unbounded)
-                }
-                (Some(after), Some(before)) => clear_filtered_processing_tasks(
-                    Bound::Excluded(*after),
-                    Bound::Excluded(*before),
-                ),
-            };
-
-            keep_ids_within_datetimes(
-                rtxn,
-                &mut filtered_non_processing_tasks,
-                self.tasks.started_at,
-                *after_started_at,
-                *before_started_at,
-            )?;
-            filtered_non_processing_tasks | filtered_processing_tasks
-        };
-
-        keep_ids_within_datetimes(
-            rtxn,
-            &mut tasks,
-            self.tasks.enqueued_at,
-            *after_enqueued_at,
-            *before_enqueued_at,
-        )?;
-
-        keep_ids_within_datetimes(
-            rtxn,
-            &mut tasks,
-            self.tasks.finished_at,
-            *after_finished_at,
-            *before_finished_at,
-        )?;
-
-        if let Some(limit) = limit {
-            tasks = if query.reverse.unwrap_or_default() {
-                tasks.into_iter().take(*limit as usize).collect()
-            } else {
-                tasks.into_iter().rev().take(*limit as usize).collect()
-            };
-        }
-
-        Ok(tasks)
-    }
-
-    pub(crate) fn get_task_ids_from_authorized_indexes(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-        processing_tasks: &ProcessingTasks,
-    ) -> Result<(RoaringBitmap, u64)> {
-        // compute all tasks matching the filter by ignoring the limits, to find the number of tasks matching
-        // the filter.
-        // As this causes us to compute the filter twice it is slightly inefficient, but doing it this way spares
-        // us from modifying the underlying implementation, and the performance remains sufficient.
-        // Should this change, we would modify `get_task_ids` to directly return the number of matching tasks.
-        let total_tasks =
-            self.get_task_ids(rtxn, &query.clone().without_limits(), processing_tasks)?;
-        let mut tasks = self.get_task_ids(rtxn, query, processing_tasks)?;
-
-        // If the query contains a list of index uid or there is a finite list of authorized indexes,
-        // then we must exclude all the kinds that aren't associated to one and only one index.
-        if query.index_uids.is_some() || !filters.all_indexes_authorized() {
-            for kind in enum_iterator::all::<Kind>().filter(|kind| !kind.related_to_one_index()) {
-                tasks -= self.tasks.get_kind(rtxn, kind)?;
-            }
-        }
-
-        // Any task that is internally associated with a non-authorized index
-        // must be discarded.
-        if !filters.all_indexes_authorized() {
-            let all_indexes_iter = self.tasks.index_tasks.iter(rtxn)?;
-            for result in all_indexes_iter {
-                let (index, index_tasks) = result?;
-                if !filters.is_index_authorized(index) {
-                    tasks -= index_tasks;
-                }
-            }
-        }
-
-        Ok((tasks, total_tasks.len()))
-    }
-
-    pub(crate) fn get_tasks_from_authorized_indexes(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-        processing_tasks: &ProcessingTasks,
-    ) -> Result<(Vec<Task>, u64)> {
-        let (tasks, total) =
-            self.get_task_ids_from_authorized_indexes(rtxn, query, filters, processing_tasks)?;
-        let tasks = if query.reverse.unwrap_or_default() {
-            Box::new(tasks.into_iter()) as Box<dyn Iterator<Item = u32>>
-        } else {
-            Box::new(tasks.into_iter().rev()) as Box<dyn Iterator<Item = u32>>
-        };
-        let tasks = self
-            .tasks
-            .get_existing_tasks(rtxn, tasks.take(query.limit.unwrap_or(u32::MAX) as usize))?;
-
-        let ProcessingTasks { batch, processing, progress: _ } = processing_tasks;
-
-        let ret = tasks.into_iter();
-        if processing.is_empty() || batch.is_none() {
-            Ok((ret.collect(), total))
-        } else {
-            // Safe because we ensured there was a batch in the previous branch
-            let batch = batch.as_ref().unwrap();
-            Ok((
-                ret.map(|task| {
-                    if processing.contains(task.uid) {
-                        Task {
-                            status: Status::Processing,
-                            batch_uid: Some(batch.uid),
-                            started_at: Some(batch.started_at),
-                            ..task
-                        }
-                    } else {
-                        task
-                    }
-                })
-                .collect(),
-                total,
-            ))
-        }
-    }
-}

crates/index-scheduler/src/queue/tasks_test.rs

@@ -1,441 +0,0 @@
-use meili_snap::snapshot;
-use meilisearch_auth::AuthFilter;
-use meilisearch_types::index_uid_pattern::IndexUidPattern;
-use meilisearch_types::tasks::{IndexSwap, KindWithContent, Status};
-use time::{Duration, OffsetDateTime};
-
-use crate::insta_snapshot::{snapshot_bitmap, snapshot_index_scheduler};
-use crate::test_utils::Breakpoint::*;
-use crate::test_utils::{index_creation_task, FailureLocation};
-use crate::{IndexScheduler, Query};
-
-#[test]
-fn query_tasks_from_and_limit() {
-    let (index_scheduler, mut handle) = IndexScheduler::test(true, vec![]);
-
-    let kind = index_creation_task("doggo", "bone");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_first_task");
-    let kind = index_creation_task("whalo", "plankton");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_second_task");
-    let kind = index_creation_task("catto", "his_own_vomit");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_third_task");
-
-    handle.advance_n_successful_batches(3);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "processed_all_tasks");
-
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let processing = index_scheduler.processing_tasks.read().unwrap();
-    let query = Query { limit: Some(0), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    let query = Query { limit: Some(1), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[2,]");
-
-    let query = Query { limit: Some(2), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[1,2,]");
-
-    let query = Query { from: Some(1), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,]");
-
-    let query = Query { from: Some(2), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,2,]");
-
-    let query = Query { from: Some(1), limit: Some(1), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[1,]");
-
-    let query = Query { from: Some(1), limit: Some(2), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &processing)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,]");
-}
-
-#[test]
-fn query_tasks_simple() {
-    let start_time = OffsetDateTime::now_utc();
-
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("whalo", "fish");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    handle.advance_till([Start, BatchCreated]);
-
-    let query = Query { statuses: Some(vec![Status::Processing]), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[0,]"); // only the processing tasks in the first tick
-
-    let query = Query { statuses: Some(vec![Status::Enqueued]), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[1,2,]"); // only the enqueued tasks in the first tick
-
-    let query =
-        Query { statuses: Some(vec![Status::Enqueued, Status::Processing]), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,2,]"); // both enqueued and processing tasks in the first tick
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        after_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test, which should excludes the enqueued tasks
-    snapshot!(snapshot_bitmap(&tasks), @"[0,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        before_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes before the start of the test, which should excludes all of them
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        after_started_at: Some(start_time),
-        before_started_at: Some(start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test and before one minute after the start of the test,
-    // which should exclude the enqueued tasks and include the only processing task
-    snapshot!(snapshot_bitmap(&tasks), @"[0,]");
-
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-        Start,
-        BatchCreated,
-    ]);
-
-    let second_start_time = OffsetDateTime::now_utc();
-
-    let query = Query {
-        statuses: Some(vec![Status::Succeeded, Status::Processing]),
-        after_started_at: Some(start_time),
-        before_started_at: Some(start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test and before one minute after the start of the test,
-    // which should include all tasks
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Succeeded, Status::Processing]),
-        before_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes before the start of the test, which should exclude all tasks
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Succeeded, Status::Processing]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the second part of the test and before one minute after the
-    // second start of the test, which should exclude all tasks
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    // now we make one more batch, the started_at field of the new tasks will be past `second_start_time`
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-        Start,
-        BatchCreated,
-    ]);
-
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // we run the same query to verify that, and indeed find that the last task is matched
-    snapshot!(snapshot_bitmap(&tasks), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Succeeded, Status::Processing]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // enqueued, succeeded, or processing tasks started after the second part of the test, should
-    // again only return the last task
-    snapshot!(snapshot_bitmap(&tasks), @"[2,]");
-
-    handle.advance_till([ProcessBatchFailed, AfterProcessing]);
-
-    // now the last task should have failed
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "end");
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // so running the last query should return nothing
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // but the same query on failed tasks should return the last task
-    snapshot!(snapshot_bitmap(&tasks), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // but the same query on failed tasks should return the last task
-    snapshot!(snapshot_bitmap(&tasks), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        uids: Some(vec![1]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // same query but with an invalid uid
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        uids: Some(vec![2]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (tasks, _) = index_scheduler
-        .get_task_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // same query but with a valid uid
-    snapshot!(snapshot_bitmap(&tasks), @"[2,]");
-}
-
-#[test]
-fn query_tasks_special_rules() {
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()), rename: false }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    handle.advance_till([Start, BatchCreated]);
-
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-
-    let query = Query { index_uids: Some(vec!["catto".to_owned()]), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // only the first task associated with catto is returned, the indexSwap tasks are excluded!
-    snapshot!(snapshot_bitmap(&tasks), @"[0,]");
-
-    let query = Query { index_uids: Some(vec!["catto".to_owned()]), ..Default::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we have asked for only the tasks associated with catto, but are only authorized to retrieve the tasks
-    // associated with doggo -> empty result
-    snapshot!(snapshot_bitmap(&tasks), @"[]");
-
-    let query = Query::default();
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we asked for all the tasks, but we are only authorized to retrieve the doggo tasks
-    // -> only the index creation of doggo should be returned
-    snapshot!(snapshot_bitmap(&tasks), @"[1,]");
-
-    let query = Query::default();
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![
-                    IndexUidPattern::new_unchecked("catto"),
-                    IndexUidPattern::new_unchecked("doggo"),
-                ]
-                .into_iter()
-                .collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we asked for all the tasks, but we are only authorized to retrieve the doggo and catto tasks
-    // -> all tasks except the swap of catto with whalo are returned
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,]");
-
-    let query = Query::default();
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // we asked for all the tasks with all index authorized -> all tasks returned
-    snapshot!(snapshot_bitmap(&tasks), @"[0,1,2,3,]");
-}
-
-#[test]
-fn query_tasks_canceled_by() {
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _ = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _ = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    handle.advance_n_successful_batches(1);
-    let kind = KindWithContent::TaskCancelation {
-        query: "test_query".to_string(),
-        tasks: [0, 1, 2, 3].into_iter().collect(),
-    };
-    let task_cancelation = index_scheduler.register(kind, None, false).unwrap();
-    handle.advance_n_successful_batches(1);
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    let rtxn = index_scheduler.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let query = Query { canceled_by: Some(vec![task_cancelation.uid]), ..Query::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // 0 is not returned because it was not canceled, 3 is not returned because it is the uid of the
-    // taskCancelation itself
-    snapshot!(snapshot_bitmap(&tasks), @"[1,2,]");
-
-    let query = Query { canceled_by: Some(vec![task_cancelation.uid]), ..Query::default() };
-    let (tasks, _) = index_scheduler
-        .queue
-        .get_task_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // Return only 1 because the user is not authorized to see task 2
-    snapshot!(snapshot_bitmap(&tasks), @"[1,]");
-}

crates/index-scheduler/src/queue/test.rs

@@ -1,427 +0,0 @@
-use big_s::S;
-use meili_snap::{json_string, snapshot};
-use meilisearch_types::error::ErrorCode;
-use meilisearch_types::tasks::{KindWithContent, Status};
-use roaring::RoaringBitmap;
-
-use crate::insta_snapshot::snapshot_index_scheduler;
-use crate::test_utils::Breakpoint::*;
-use crate::test_utils::{index_creation_task, replace_document_import_task};
-use crate::{IndexScheduler, Query};
-
-#[test]
-fn register() {
-    // In this test, the handle doesn't make any progress, we only check that the tasks are registered
-    let (index_scheduler, mut _handle) = IndexScheduler::test(true, vec![]);
-
-    let kinds = [
-        index_creation_task("catto", "mouse"),
-        replace_document_import_task("catto", None, 0, 12),
-        replace_document_import_task("catto", None, 1, 50),
-        replace_document_import_task("doggo", Some("bone"), 2, 5000),
-    ];
-    let (_, file) = index_scheduler.queue.create_update_file_with_uuid(0).unwrap();
-    file.persist().unwrap();
-    let (_, file) = index_scheduler.queue.create_update_file_with_uuid(1).unwrap();
-    file.persist().unwrap();
-    let (_, file) = index_scheduler.queue.create_update_file_with_uuid(2).unwrap();
-    file.persist().unwrap();
-
-    for (idx, kind) in kinds.into_iter().enumerate() {
-        let k = kind.as_kind();
-        let task = index_scheduler.register(kind, None, false).unwrap();
-        index_scheduler.assert_internally_consistent();
-
-        assert_eq!(task.uid, idx as u32);
-        assert_eq!(task.status, Status::Enqueued);
-        assert_eq!(task.kind.as_kind(), k);
-    }
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "everything_is_successfully_registered");
-}
-
-#[test]
-fn dry_run() {
-    let (index_scheduler, _handle) = IndexScheduler::test(true, vec![]);
-
-    let kind = KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None };
-    let task = index_scheduler.register(kind, None, true).unwrap();
-    snapshot!(task.uid, @"0");
-    snapshot!(snapshot_index_scheduler(&index_scheduler), @r"
-        ### Autobatching Enabled = true
-        ### Processing batch None:
-        []
-        ----------------------------------------------------------------------
-        ### All Tasks:
-        ----------------------------------------------------------------------
-        ### Status:
-        ----------------------------------------------------------------------
-        ### Kind:
-        ----------------------------------------------------------------------
-        ### Index Tasks:
-        ----------------------------------------------------------------------
-        ### Index Mapper:
-
-        ----------------------------------------------------------------------
-        ### Canceled By:
-
-        ----------------------------------------------------------------------
-        ### Enqueued At:
-        ----------------------------------------------------------------------
-        ### Started At:
-        ----------------------------------------------------------------------
-        ### Finished At:
-        ----------------------------------------------------------------------
-        ### All Batches:
-        ----------------------------------------------------------------------
-        ### Batch to tasks mapping:
-        ----------------------------------------------------------------------
-        ### Batches Status:
-        ----------------------------------------------------------------------
-        ### Batches Kind:
-        ----------------------------------------------------------------------
-        ### Batches Index Tasks:
-        ----------------------------------------------------------------------
-        ### Batches Enqueued At:
-        ----------------------------------------------------------------------
-        ### Batches Started At:
-        ----------------------------------------------------------------------
-        ### Batches Finished At:
-        ----------------------------------------------------------------------
-        ### File Store:
-
-        ----------------------------------------------------------------------
-        ");
-
-    let kind = KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None };
-    let task = index_scheduler.register(kind, Some(12), true).unwrap();
-    snapshot!(task.uid, @"12");
-    snapshot!(snapshot_index_scheduler(&index_scheduler), @r"
-        ### Autobatching Enabled = true
-        ### Processing batch None:
-        []
-        ----------------------------------------------------------------------
-        ### All Tasks:
-        ----------------------------------------------------------------------
-        ### Status:
-        ----------------------------------------------------------------------
-        ### Kind:
-        ----------------------------------------------------------------------
-        ### Index Tasks:
-        ----------------------------------------------------------------------
-        ### Index Mapper:
-
-        ----------------------------------------------------------------------
-        ### Canceled By:
-
-        ----------------------------------------------------------------------
-        ### Enqueued At:
-        ----------------------------------------------------------------------
-        ### Started At:
-        ----------------------------------------------------------------------
-        ### Finished At:
-        ----------------------------------------------------------------------
-        ### All Batches:
-        ----------------------------------------------------------------------
-        ### Batch to tasks mapping:
-        ----------------------------------------------------------------------
-        ### Batches Status:
-        ----------------------------------------------------------------------
-        ### Batches Kind:
-        ----------------------------------------------------------------------
-        ### Batches Index Tasks:
-        ----------------------------------------------------------------------
-        ### Batches Enqueued At:
-        ----------------------------------------------------------------------
-        ### Batches Started At:
-        ----------------------------------------------------------------------
-        ### Batches Finished At:
-        ----------------------------------------------------------------------
-        ### File Store:
-
-        ----------------------------------------------------------------------
-        ");
-}
-
-#[test]
-fn basic_set_taskid() {
-    let (index_scheduler, _handle) = IndexScheduler::test(true, vec![]);
-
-    let kind = KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None };
-    let task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(task.uid, @"0");
-
-    let kind = KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None };
-    let task = index_scheduler.register(kind, Some(12), false).unwrap();
-    snapshot!(task.uid, @"12");
-
-    let kind = KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None };
-    let error = index_scheduler.register(kind, Some(5), false).unwrap_err();
-    snapshot!(error, @"Received bad task id: 5 should be >= to 13.");
-}
-
-#[test]
-fn test_disable_auto_deletion_of_tasks() {
-    let (index_scheduler, mut handle) = IndexScheduler::test_with_custom_config(vec![], |config| {
-        config.cleanup_enabled = false;
-        config.max_number_of_tasks = 2;
-        None
-    });
-
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_successful_batch();
-
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_failed_batch();
-
-    // at this point the max number of tasks is reached
-    // we can still enqueue multiple tasks
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]" }), name: "task_queue_is_full");
-    drop(rtxn);
-    drop(proc);
-
-    // now we're above the max number of tasks
-    // and if we try to advance in the tick function no new task deletion should be enqueued
-    handle.advance_till([Start, BatchCreated]);
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_not_been_enqueued");
-    drop(rtxn);
-    drop(proc);
-}
-
-#[test]
-fn test_auto_deletion_of_tasks() {
-    let (index_scheduler, mut handle) = IndexScheduler::test_with_custom_config(vec![], |config| {
-        config.max_number_of_tasks = 2;
-        None
-    });
-
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_successful_batch();
-
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_failed_batch();
-
-    // at this point the max number of tasks is reached
-    // we can still enqueue multiple tasks
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]" }), name: "task_queue_is_full");
-    drop(rtxn);
-    drop(proc);
-
-    // now we're above the max number of tasks
-    // and if we try to advance in the tick function a new task deletion should be enqueued
-    handle.advance_till([Start, BatchCreated]);
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_been_enqueued");
-    drop(rtxn);
-    drop(proc);
-
-    handle.advance_till([InsideProcessBatch, ProcessBatchSucceeded, AfterProcessing]);
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "task_deletion_have_been_processed");
-    drop(rtxn);
-    drop(proc);
-
-    handle.advance_one_failed_batch();
-    // a new task deletion has been enqueued
-    handle.advance_one_successful_batch();
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "after_the_second_task_deletion");
-    drop(rtxn);
-    drop(proc);
-
-    handle.advance_one_failed_batch();
-    handle.advance_one_successful_batch();
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap();
-    let tasks =
-        index_scheduler.queue.get_task_ids(&rtxn, &Query { ..Default::default() }, &proc).unwrap();
-    let tasks = index_scheduler.queue.tasks.get_existing_tasks(&rtxn, tasks).unwrap();
-    snapshot!(json_string!(tasks, { "[].enqueuedAt" => "[date]", "[].startedAt" => "[date]", "[].finishedAt" => "[date]", ".**.original_filter" => "[filter]", ".**.query" => "[query]" }), name: "everything_has_been_processed");
-    drop(rtxn);
-    drop(proc);
-}
-
-#[test]
-fn test_task_queue_is_full() {
-    let (index_scheduler, mut handle) = IndexScheduler::test_with_custom_config(vec![], |config| {
-        // that's the minimum map size possible
-        config.task_db_size = 1048576 * 3;
-        None
-    });
-
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_successful_batch();
-    // on average this task takes ~600 bytes
-    loop {
-        let result = index_scheduler.register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        );
-        if result.is_err() {
-            break;
-        }
-        handle.advance_one_failed_batch();
-    }
-    index_scheduler.assert_internally_consistent();
-
-    // at this point the task DB shoud have reached its limit and we should not be able to register new tasks
-    let result = index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap_err();
-    snapshot!(result, @"Meilisearch cannot receive write operations because the limit of the task database has been reached. Please delete tasks to continue performing write operations.");
-    // we won't be able to test this error in an integration test thus as a best effort test I still ensure the error return the expected error code
-    snapshot!(format!("{:?}", result.error_code()), @"NoSpaceLeftOnDevice");
-
-    // Even the task deletion and cancelation that don't delete anything should be refused
-    let result = index_scheduler
-        .register(
-            KindWithContent::TaskDeletion { query: S("test"), tasks: RoaringBitmap::new() },
-            None,
-            false,
-        )
-        .unwrap_err();
-    snapshot!(result, @"Meilisearch cannot receive write operations because the limit of the task database has been reached. Please delete tasks to continue performing write operations.");
-    let result = index_scheduler
-        .register(
-            KindWithContent::TaskCancelation { query: S("test"), tasks: RoaringBitmap::new() },
-            None,
-            false,
-        )
-        .unwrap_err();
-    snapshot!(result, @"Meilisearch cannot receive write operations because the limit of the task database has been reached. Please delete tasks to continue performing write operations.");
-
-    // we won't be able to test this error in an integration test thus as a best effort test I still ensure the error return the expected error code
-    snapshot!(format!("{:?}", result.error_code()), @"NoSpaceLeftOnDevice");
-
-    // But a task cancelation that cancel something should work
-    index_scheduler
-        .register(
-            KindWithContent::TaskCancelation { query: S("test"), tasks: (0..100).collect() },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_successful_batch();
-
-    // But we should still be forbidden from enqueuing new tasks
-    let result = index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap_err();
-    snapshot!(result, @"Meilisearch cannot receive write operations because the limit of the task database has been reached. Please delete tasks to continue performing write operations.");
-
-    // And a task deletion that delete something should works
-    index_scheduler
-        .register(
-            KindWithContent::TaskDeletion { query: S("test"), tasks: (0..100).collect() },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_successful_batch();
-
-    // Now we should be able to enqueue a few tasks again
-    index_scheduler
-        .register(
-            KindWithContent::IndexCreation { index_uid: S("doggo"), primary_key: None },
-            None,
-            false,
-        )
-        .unwrap();
-    handle.advance_one_failed_batch();
-}

crates/index-scheduler/src/scheduler/autobatcher.rs

@@ -1,539 +0,0 @@
-/*!
-The autobatcher is responsible for combining the next enqueued
-tasks affecting a single index into a [batch](crate::batch::Batch).
-
-The main function of the autobatcher is [`next_autobatch`].
-*/
-
-use std::ops::ControlFlow::{self, Break, Continue};
-
-use meilisearch_types::tasks::{BatchStopReason, PrimaryKeyMismatchReason, TaskId};
-
-use crate::KindWithContent;
-
-/// Succinctly describes a task's [`Kind`](meilisearch_types::tasks::Kind)
-/// for the purpose of simplifying the implementation of the autobatcher.
-///
-/// Only the non-prioritised tasks that can be grouped in a batch have a corresponding [`AutobatchKind`]
-enum AutobatchKind {
-    DocumentImport { allow_index_creation: bool, primary_key: Option<String> },
-    DocumentEdition,
-    DocumentDeletion { by_filter: bool },
-    DocumentClear,
-    Settings { allow_index_creation: bool },
-    IndexCreation,
-    IndexDeletion,
-    IndexUpdate,
-    IndexSwap,
-}
-
-impl AutobatchKind {
-    #[rustfmt::skip]
-    fn allow_index_creation(&self) -> Option<bool> {
-        match self {
-            AutobatchKind::DocumentImport { allow_index_creation, .. }
-            | AutobatchKind::Settings { allow_index_creation, .. } => Some(*allow_index_creation),
-            _ => None,
-        }
-    }
-
-    fn primary_key(&self) -> Option<Option<&str>> {
-        match self {
-            AutobatchKind::DocumentImport { primary_key, .. } => Some(primary_key.as_deref()),
-            _ => None,
-        }
-    }
-}
-
-impl From<KindWithContent> for AutobatchKind {
-    fn from(kind: KindWithContent) -> Self {
-        match kind {
-            KindWithContent::DocumentAdditionOrUpdate {
-                allow_index_creation, primary_key, ..
-            } => AutobatchKind::DocumentImport { allow_index_creation, primary_key },
-            KindWithContent::DocumentEdition { .. } => AutobatchKind::DocumentEdition,
-            KindWithContent::DocumentDeletion { .. } => {
-                AutobatchKind::DocumentDeletion { by_filter: false }
-            }
-            KindWithContent::DocumentClear { .. } => AutobatchKind::DocumentClear,
-            KindWithContent::DocumentDeletionByFilter { .. } => {
-                AutobatchKind::DocumentDeletion { by_filter: true }
-            }
-            KindWithContent::SettingsUpdate { allow_index_creation, is_deletion, .. } => {
-                AutobatchKind::Settings {
-                    allow_index_creation: allow_index_creation && !is_deletion,
-                }
-            }
-            KindWithContent::IndexDeletion { .. } => AutobatchKind::IndexDeletion,
-            KindWithContent::IndexCreation { .. } => AutobatchKind::IndexCreation,
-            KindWithContent::IndexUpdate { .. } => AutobatchKind::IndexUpdate,
-            KindWithContent::IndexSwap { .. } => AutobatchKind::IndexSwap,
-            KindWithContent::IndexCompaction { .. }
-            | KindWithContent::TaskCancelation { .. }
-            | KindWithContent::TaskDeletion { .. }
-            | KindWithContent::DumpCreation { .. }
-            | KindWithContent::Export { .. }
-            | KindWithContent::UpgradeDatabase { .. }
-            | KindWithContent::SnapshotCreation => {
-                panic!("The autobatcher should never be called with tasks with special priority or that don't apply to an index.")
-            }
-        }
-    }
-}
-
-#[derive(Debug)]
-pub enum BatchKind {
-    DocumentClear {
-        ids: Vec<TaskId>,
-    },
-    DocumentOperation {
-        allow_index_creation: bool,
-        primary_key: Option<String>,
-        operation_ids: Vec<TaskId>,
-    },
-    DocumentEdition {
-        id: TaskId,
-    },
-    DocumentDeletion {
-        deletion_ids: Vec<TaskId>,
-        includes_by_filter: bool,
-    },
-    ClearAndSettings {
-        other: Vec<TaskId>,
-        allow_index_creation: bool,
-        settings_ids: Vec<TaskId>,
-    },
-    Settings {
-        allow_index_creation: bool,
-        settings_ids: Vec<TaskId>,
-    },
-    IndexDeletion {
-        ids: Vec<TaskId>,
-    },
-    IndexCreation {
-        id: TaskId,
-    },
-    IndexUpdate {
-        id: TaskId,
-    },
-    IndexSwap {
-        id: TaskId,
-    },
-}
-
-impl BatchKind {
-    #[rustfmt::skip]
-    fn allow_index_creation(&self) -> Option<bool> {
-        match self {
-            BatchKind::DocumentOperation { allow_index_creation, .. }
-            | BatchKind::ClearAndSettings { allow_index_creation, .. }
-            | BatchKind::Settings { allow_index_creation, .. } => Some(*allow_index_creation),
-            _ => None,
-        }
-    }
-
-    fn primary_key(&self) -> Option<Option<&str>> {
-        match self {
-            BatchKind::DocumentOperation { primary_key, .. } => Some(primary_key.as_deref()),
-            _ => None,
-        }
-    }
-}
-
-impl BatchKind {
-    /// Returns a `ControlFlow::Break` if you must stop right now.
-    /// The boolean tell you if an index has been created by the batched task.
-    /// To ease the writing of the code. `true` can be returned when you don't need to create an index
-    /// but false can't be returned if you needs to create an index.
-    // TODO use an AutoBatchKind as input
-    pub fn new(
-        task_id: TaskId,
-        kind_with_content: KindWithContent,
-        primary_key: Option<&str>,
-    ) -> (ControlFlow<(BatchKind, BatchStopReason), BatchKind>, bool) {
-        use AutobatchKind as K;
-
-        let kind = kind_with_content.as_kind();
-
-        match AutobatchKind::from(kind_with_content) {
-            K::IndexCreation => (
-                Break((
-                    BatchKind::IndexCreation { id: task_id },
-                    BatchStopReason::TaskCannotBeBatched { kind, id: task_id },
-                )),
-                true,
-            ),
-            K::IndexDeletion => (
-                Break((
-                    BatchKind::IndexDeletion { ids: vec![task_id] },
-                    BatchStopReason::IndexDeletion { id: task_id },
-                )),
-                false,
-            ),
-            K::IndexUpdate => (
-                Break((
-                    BatchKind::IndexUpdate { id: task_id },
-                    BatchStopReason::TaskCannotBeBatched { kind, id: task_id },
-                )),
-                false,
-            ),
-            K::IndexSwap => (
-                Break((
-                    BatchKind::IndexSwap { id: task_id },
-                    BatchStopReason::TaskCannotBeBatched { kind, id: task_id },
-                )),
-                false,
-            ),
-            K::DocumentClear => (Continue(BatchKind::DocumentClear { ids: vec![task_id] }), false),
-            K::DocumentImport { allow_index_creation, primary_key: pk }
-                if primary_key.is_none() || pk.is_none() || primary_key == pk.as_deref() =>
-            {
-                (
-                    Continue(BatchKind::DocumentOperation {
-                        allow_index_creation,
-                        primary_key: pk,
-                        operation_ids: vec![task_id],
-                    }),
-                    allow_index_creation,
-                )
-            }
-            // if the primary key set in the task was different than ours we should stop and make this batch fail asap.
-            K::DocumentImport { allow_index_creation, primary_key: pk } => (
-                Break((
-                    BatchKind::DocumentOperation {
-                        allow_index_creation,
-                        primary_key: pk.clone(),
-                        operation_ids: vec![task_id],
-                    },
-                    BatchStopReason::PrimaryKeyIndexMismatch {
-                        id: task_id,
-                        in_index: primary_key.unwrap().to_owned(),
-                        in_task: pk.unwrap(),
-                    },
-                )),
-                allow_index_creation,
-            ),
-            K::DocumentEdition => (
-                Break((
-                    BatchKind::DocumentEdition { id: task_id },
-                    BatchStopReason::TaskCannotBeBatched { kind, id: task_id },
-                )),
-                false,
-            ),
-            K::DocumentDeletion { by_filter: includes_by_filter } => (
-                Continue(BatchKind::DocumentDeletion {
-                    deletion_ids: vec![task_id],
-                    includes_by_filter,
-                }),
-                false,
-            ),
-            K::Settings { allow_index_creation } => (
-                Continue(BatchKind::Settings { allow_index_creation, settings_ids: vec![task_id] }),
-                allow_index_creation,
-            ),
-        }
-    }
-
-    /// Returns a `ControlFlow::Break` if you must stop right now.
-    /// The boolean tell you if an index has been created by the batched task.
-    /// To ease the writing of the code. `true` can be returned when you don't need to create an index
-    /// but false can't be returned if you needs to create an index.
-    #[rustfmt::skip]
-    fn accumulate(self, id: TaskId, kind_with_content: KindWithContent, index_already_exists: bool, primary_key: Option<&str>) -> ControlFlow<(BatchKind, BatchStopReason), BatchKind> {
-        use AutobatchKind as K;
-
-        let kind = kind_with_content.as_kind();
-        let autobatch_kind = AutobatchKind::from(kind_with_content);
-
-        let pk: Option<String> = match (self.primary_key(), autobatch_kind.primary_key(), primary_key) {
-            // 1. If incoming task don't interact with primary key -> we can continue
-            (batch_pk, None | Some(None), _) => {
-                batch_pk.flatten().map(ToOwned::to_owned)
-            },
-            // 2.1 If we already have a primary-key ->
-            // 2.1.1 If the task we're trying to accumulate have a pk it must be equal to our primary key
-            (_batch_pk, Some(Some(task_pk)), Some(index_pk)) => if task_pk == index_pk {
-                Some(task_pk.to_owned())
-            } else {
-                return Break((self, BatchStopReason::PrimaryKeyMismatch {
-                    id,
-                    reason: PrimaryKeyMismatchReason::TaskPrimaryKeyDifferFromIndexPrimaryKey {
-                        task_pk: task_pk.to_owned(),
-                        index_pk: index_pk.to_owned(),
-                    },
-                }))
-            },
-            // 2.2 If we don't have a primary-key ->
-            // 2.2.2 If the batch is set to Some(None), the task should be too
-            (Some(None), Some(Some(task_pk)), None) => return Break((self, BatchStopReason::PrimaryKeyMismatch {
-                id,
-                reason: PrimaryKeyMismatchReason::CannotInterfereWithPrimaryKeyGuessing {
-                    task_pk: task_pk.to_owned(),
-                },
-            })),
-            (Some(Some(batch_pk)), Some(Some(task_pk)), None) => if task_pk == batch_pk {
-                Some(task_pk.to_owned())
-            } else {
-                let batch_pk = batch_pk.to_owned();
-                let task_pk = task_pk.to_owned();
-                return Break((self, BatchStopReason::PrimaryKeyMismatch {
-                    id,
-                    reason: PrimaryKeyMismatchReason::TaskPrimaryKeyDifferFromCurrentBatchPrimaryKey {
-                        batch_pk,
-                        task_pk
-                    },
-                }))
-            },
-            (None, Some(Some(task_pk)), None) => Some(task_pk.to_owned())
-        };
-
-        match (self, autobatch_kind) {
-            // We don't batch any of these operations
-            (this, K::IndexCreation | K::IndexUpdate | K::IndexSwap | K::DocumentEdition) => {
-                Break((this, BatchStopReason::TaskCannotBeBatched { kind, id }))
-            },
-            // We must not batch tasks that don't have the same index creation rights if the index doesn't already exists.
-            (this, kind) if !index_already_exists && this.allow_index_creation() == Some(false) && kind.allow_index_creation() == Some(true) => {
-                Break((this, BatchStopReason::IndexCreationMismatch { id }))
-            },
-            // The index deletion can batch with everything but must stop after
-            (
-                BatchKind::DocumentClear { mut ids }
-                | BatchKind::DocumentDeletion { deletion_ids: mut ids, includes_by_filter: _ }
-                | BatchKind::DocumentOperation { allow_index_creation: _, primary_key: _, operation_ids: mut ids }
-                | BatchKind::Settings { allow_index_creation: _, settings_ids: mut ids },
-                K::IndexDeletion,
-            ) => {
-                ids.push(id);
-                Break((BatchKind::IndexDeletion { ids }, BatchStopReason::IndexDeletion { id }))
-            }
-            (
-                BatchKind::ClearAndSettings { settings_ids: mut ids, allow_index_creation: _, mut other },
-                K::IndexDeletion,
-            ) => {
-                ids.push(id);
-                ids.append(&mut other);
-                Break((BatchKind::IndexDeletion { ids }, BatchStopReason::IndexDeletion { id }))
-            }
-
-            (
-                BatchKind::DocumentClear { mut ids },
-                K::DocumentClear | K::DocumentDeletion { by_filter: _ },
-            ) => {
-                ids.push(id);
-                Continue(BatchKind::DocumentClear { ids })
-            }
-            (
-                this @ BatchKind::DocumentClear { .. },
-                K::DocumentImport { .. } | K::Settings { .. },
-            ) => Break((this, BatchStopReason::DocumentOperationWithSettings { id })),
-            (
-                BatchKind::DocumentOperation { allow_index_creation: _, primary_key: _, mut operation_ids },
-                K::DocumentClear,
-            ) => {
-                operation_ids.push(id);
-                Continue(BatchKind::DocumentClear { ids: operation_ids })
-            }
-
-            // we can autobatch different kind of document operations and mix replacements with updates
-            (
-                BatchKind::DocumentOperation { allow_index_creation, primary_key: _, mut operation_ids },
-                K::DocumentImport { primary_key: _, .. },
-            ) => {
-                operation_ids.push(id);
-                Continue(BatchKind::DocumentOperation {
-                    allow_index_creation,
-                    operation_ids,
-                    primary_key: pk,
-                })
-            }
-            (
-                BatchKind::DocumentOperation { allow_index_creation, primary_key: _, mut operation_ids },
-                K::DocumentDeletion { by_filter: false },
-            ) => {
-                operation_ids.push(id);
-
-                Continue(BatchKind::DocumentOperation {
-                    allow_index_creation,
-                    operation_ids,
-                    primary_key: pk,
-                })
-            }
-            // We can't batch a document operation with a delete by filter
-            (
-                this @ BatchKind::DocumentOperation { .. },
-                K::DocumentDeletion { by_filter: true },
-            ) => {
-                Break((this, BatchStopReason::DocumentOperationWithDeletionByFilter { id }))
-            }
-            (
-                this @ BatchKind::DocumentOperation { .. },
-                K::Settings { .. },
-            ) => Break((this, BatchStopReason::DocumentOperationWithSettings { id })),
-
-            (BatchKind::DocumentDeletion { mut deletion_ids, includes_by_filter: _ }, K::DocumentClear) => {
-                deletion_ids.push(id);
-                Continue(BatchKind::DocumentClear { ids: deletion_ids })
-            }
-            // we can't autobatch the deletion and import if the document deletion contained a filter
-            (
-                this @ BatchKind::DocumentDeletion { deletion_ids: _, includes_by_filter: true },
-                K::DocumentImport { .. }
-            ) => Break((this, BatchStopReason::DeletionByFilterWithDocumentOperation { id })),
-            // we can autobatch the deletion and import if the index already exists
-            (
-                BatchKind::DocumentDeletion { mut deletion_ids, includes_by_filter: false },
-                K::DocumentImport { allow_index_creation, primary_key }
-            ) if index_already_exists => {
-                deletion_ids.push(id);
-
-                Continue(BatchKind::DocumentOperation {
-                    allow_index_creation,
-                    primary_key,
-                    operation_ids: deletion_ids,
-                })
-            }
-            // we can autobatch the deletion and import if both can't create an index
-            (
-                BatchKind::DocumentDeletion { mut deletion_ids, includes_by_filter: false },
-                K::DocumentImport { allow_index_creation, primary_key }
-            ) if !allow_index_creation => {
-                deletion_ids.push(id);
-
-                Continue(BatchKind::DocumentOperation {
-                    allow_index_creation,
-                    primary_key,
-                    operation_ids: deletion_ids,
-                })
-            }
-            // we can't autobatch a deletion and an import if the index does not exist but would be created by an addition
-            (
-                this @ BatchKind::DocumentDeletion { .. },
-                K::DocumentImport { .. }
-            ) => {
-                Break((this, BatchStopReason::IndexCreationMismatch { id }))
-            }
-            (BatchKind::DocumentDeletion { mut deletion_ids, includes_by_filter }, K::DocumentDeletion { by_filter }) => {
-                deletion_ids.push(id);
-                Continue(BatchKind::DocumentDeletion { deletion_ids, includes_by_filter: includes_by_filter | by_filter })
-            }
-            (this @ BatchKind::DocumentDeletion { .. }, K::Settings { .. }) => Break((this, BatchStopReason::DocumentOperationWithSettings { id })),
-
-            (
-                BatchKind::Settings { settings_ids, allow_index_creation },
-                K::DocumentClear,
-            ) => Continue(BatchKind::ClearAndSettings {
-                settings_ids,
-                allow_index_creation,
-                other: vec![id],
-            }),
-            (
-                this @ BatchKind::Settings { .. },
-                K::DocumentImport { .. } | K::DocumentDeletion { .. },
-            ) => Break((this, BatchStopReason::SettingsWithDocumentOperation { id })),
-            (
-                BatchKind::Settings { mut settings_ids, allow_index_creation },
-                K::Settings { .. },
-            ) => {
-                settings_ids.push(id);
-                Continue(BatchKind::Settings {
-                    allow_index_creation,
-                    settings_ids,
-                })
-            }
-
-            (
-                BatchKind::ClearAndSettings { mut other, settings_ids, allow_index_creation },
-                K::DocumentClear,
-            ) => {
-                other.push(id);
-                Continue(BatchKind::ClearAndSettings {
-                    other,
-                    settings_ids,
-                    allow_index_creation,
-                })
-            }
-            (this @ BatchKind::ClearAndSettings { .. }, K::DocumentImport { .. }) => Break((this, BatchStopReason::SettingsWithDocumentOperation { id })),
-            (
-                BatchKind::ClearAndSettings {
-                    mut other,
-                    settings_ids,
-                    allow_index_creation,
-                },
-                K::DocumentDeletion { .. },
-            ) => {
-                other.push(id);
-                Continue(BatchKind::ClearAndSettings {
-                    other,
-                    settings_ids,
-                    allow_index_creation,
-                })
-            }
-            (
-                BatchKind::ClearAndSettings { mut settings_ids, other, allow_index_creation },
-                K::Settings { .. },
-            ) => {
-                settings_ids.push(id);
-                Continue(BatchKind::ClearAndSettings {
-                    other,
-                    settings_ids,
-                    allow_index_creation,
-                })
-            }
-
-            (
-                BatchKind::IndexCreation { .. }
-                | BatchKind::IndexDeletion { .. }
-                | BatchKind::IndexUpdate { .. }
-                | BatchKind::IndexSwap { .. }
-                | BatchKind::DocumentEdition { .. },
-                _,
-            ) => {
-                unreachable!()
-            }
-        }
-    }
-}
-
-/// Create a batch from an ordered list of tasks.
-///
-/// ## Preconditions
-/// 1. The tasks must be enqueued and given in the order in which they were enqueued
-/// 2. The tasks must not be prioritised tasks (e.g. task cancellation, dump, snapshot, task deletion)
-/// 3. The tasks must all be related to the same index
-///
-/// ## Return
-/// `None` if the list of tasks is empty. Otherwise, an [`AutoBatch`] that represents
-/// a subset of the given tasks.
-pub fn autobatch(
-    enqueued: Vec<(TaskId, KindWithContent)>,
-    index_already_exists: bool,
-    primary_key: Option<&str>,
-) -> Option<(BatchKind, bool, Option<BatchStopReason>)> {
-    let mut enqueued = enqueued.into_iter();
-    let (id, kind) = enqueued.next()?;
-
-    // index_exist will keep track of if the index should exist at this point after the tasks we batched.
-    let mut index_exist = index_already_exists;
-
-    let (mut acc, must_create_index) = match BatchKind::new(id, kind, primary_key) {
-        (Continue(acc), create) => (acc, create),
-        (Break((acc, batch_stop_reason)), create) => {
-            return Some((acc, create, Some(batch_stop_reason)))
-        }
-    };
-
-    // if an index has been created in the previous step we can consider it as existing.
-    index_exist |= must_create_index;
-
-    for (id, kind_with_content) in enqueued {
-        acc = match acc.accumulate(id, kind_with_content, index_exist, primary_key) {
-            Continue(acc) => acc,
-            Break((acc, batch_stop_reason)) => {
-                return Some((acc, must_create_index, Some(batch_stop_reason)))
-            }
-        };
-    }
-
-    Some((acc, must_create_index, None))
-}

crates/index-scheduler/src/scheduler/autobatcher_test.rs

@@ -1,402 +0,0 @@
-use meilisearch_types::milli::update::IndexDocumentsMethod::{
-    self, ReplaceDocuments, UpdateDocuments,
-};
-use meilisearch_types::tasks::{BatchStopReason, IndexSwap, KindWithContent};
-use uuid::Uuid;
-
-use self::autobatcher::{autobatch, BatchKind};
-use super::*;
-use crate::TaskId;
-
-#[macro_export]
-macro_rules! debug_snapshot {
-        ($value:expr, @$snapshot:literal) => {{
-            let value = format!("{:?}", $value);
-            meili_snap::snapshot!(value, @$snapshot);
-        }};
-    }
-
-fn autobatch_from(
-    index_already_exists: bool,
-    primary_key: Option<&str>,
-    input: impl IntoIterator<Item = KindWithContent>,
-) -> Option<(BatchKind, bool, Option<BatchStopReason>)> {
-    autobatch(
-        input.into_iter().enumerate().map(|(id, kind)| (id as TaskId, kind)).collect(),
-        index_already_exists,
-        primary_key,
-    )
-}
-
-fn doc_imp(
-    method: IndexDocumentsMethod,
-    allow_index_creation: bool,
-    primary_key: Option<&str>,
-) -> KindWithContent {
-    KindWithContent::DocumentAdditionOrUpdate {
-        index_uid: String::from("doggo"),
-        primary_key: primary_key.map(|pk| pk.to_string()),
-        method,
-        content_file: Uuid::new_v4(),
-        documents_count: 0,
-        allow_index_creation,
-    }
-}
-
-fn doc_del() -> KindWithContent {
-    KindWithContent::DocumentDeletion {
-        index_uid: String::from("doggo"),
-        documents_ids: Vec::new(),
-    }
-}
-
-fn doc_del_fil() -> KindWithContent {
-    KindWithContent::DocumentDeletionByFilter {
-        index_uid: String::from("doggo"),
-        filter_expr: serde_json::json!("cuteness > 100"),
-    }
-}
-
-fn doc_clr() -> KindWithContent {
-    KindWithContent::DocumentClear { index_uid: String::from("doggo") }
-}
-
-fn settings(allow_index_creation: bool) -> KindWithContent {
-    KindWithContent::SettingsUpdate {
-        index_uid: String::from("doggo"),
-        new_settings: Default::default(),
-        is_deletion: false,
-        allow_index_creation,
-    }
-}
-
-fn idx_create() -> KindWithContent {
-    KindWithContent::IndexCreation { index_uid: String::from("doggo"), primary_key: None }
-}
-
-fn idx_update() -> KindWithContent {
-    KindWithContent::IndexUpdate {
-        index_uid: String::from("doggo"),
-        primary_key: None,
-        new_index_uid: None,
-    }
-}
-
-fn idx_del() -> KindWithContent {
-    KindWithContent::IndexDeletion { index_uid: String::from("doggo") }
-}
-
-fn idx_swap() -> KindWithContent {
-    KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap {
-            indexes: (String::from("doggo"), String::from("catto")),
-            rename: false,
-        }],
-    }
-}
-
-#[test]
-fn autobatch_simple_operation_together() {
-    // we can autobatch one or multiple `ReplaceDocuments` together.
-    // if the index exists.
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp( ReplaceDocuments, true , None), doc_imp(ReplaceDocuments, true , None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), doc_imp( ReplaceDocuments, false , None), doc_imp(ReplaceDocuments, false , None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1, 2] }, false, None))");
-
-    // if it doesn't exists.
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, true, None), doc_imp( ReplaceDocuments, true , None), doc_imp(ReplaceDocuments, true , None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, false, None), doc_imp( ReplaceDocuments, true , None), doc_imp(ReplaceDocuments, true , None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(IndexCreationMismatch { id: 1 })))");
-
-    // we can autobatch one or multiple `UpdateDocuments` together.
-    // if the index exists.
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, None), doc_imp(UpdateDocuments, false, None), doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1, 2] }, false, None))");
-
-    // if it doesn't exists.
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, false, None), doc_imp(UpdateDocuments, false, None), doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1, 2] }, false, None))");
-
-    // we can autobatch one or multiple DocumentDeletion together
-    debug_snapshot!(autobatch_from(true, None, [doc_del()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_del(), doc_del()]), @"Some((DocumentDeletion { deletion_ids: [0, 1, 2], includes_by_filter: false }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_del()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_del(), doc_del(), doc_del()]), @"Some((DocumentDeletion { deletion_ids: [0, 1, 2], includes_by_filter: false }, false, None))");
-
-    // we can autobatch one or multiple DocumentDeletionByFilter together
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_del_fil(), doc_del_fil()]), @"Some((DocumentDeletion { deletion_ids: [0, 1, 2], includes_by_filter: true }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_del_fil()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_del_fil(), doc_del_fil(), doc_del_fil()]), @"Some((DocumentDeletion { deletion_ids: [0, 1, 2], includes_by_filter: true }, false, None))");
-
-    // we can autobatch one or multiple Settings together
-    debug_snapshot!(autobatch_from(true, None, [settings(true)]), @"Some((Settings { allow_index_creation: true, settings_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [settings(true), settings(true), settings(true)]), @"Some((Settings { allow_index_creation: true, settings_ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [settings(false)]), @"Some((Settings { allow_index_creation: false, settings_ids: [0] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [settings(false), settings(false), settings(false)]), @"Some((Settings { allow_index_creation: false, settings_ids: [0, 1, 2] }, false, None))");
-
-    debug_snapshot!(autobatch_from(false,None,  [settings(true)]), @"Some((Settings { allow_index_creation: true, settings_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [settings(true), settings(true), settings(true)]), @"Some((Settings { allow_index_creation: true, settings_ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [settings(false)]), @"Some((Settings { allow_index_creation: false, settings_ids: [0] }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [settings(false), settings(false), settings(false)]), @"Some((Settings { allow_index_creation: false, settings_ids: [0, 1, 2] }, false, None))");
-
-    // We can autobatch document addition with document deletion
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, true, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, true, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, false, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, false, None), doc_del()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, true, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, true, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, false, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, false, Some("catto")), doc_del()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    // And the other way around
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(ReplaceDocuments, true, Some("catto"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(UpdateDocuments, true, Some("catto"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(ReplaceDocuments, false, Some("catto"))]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), doc_imp(UpdateDocuments, false, Some("catto"))]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(ReplaceDocuments, false, Some("catto"))]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(UpdateDocuments, false, Some("catto"))]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0, 1] }, false, None))"###);
-
-    // But we can't autobatch document addition with document deletion by filter
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, true, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, true, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, false, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, false, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, true, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, true, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("catto"), operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(ReplaceDocuments, false, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    debug_snapshot!(autobatch_from(false, None, [doc_imp(UpdateDocuments, false, Some("catto")), doc_del_fil()]), @r###"Some((DocumentOperation { allow_index_creation: false, primary_key: Some("catto"), operation_ids: [0] }, false, Some(DocumentOperationWithDeletionByFilter { id: 1 })))"###);
-    // And the other way around
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(ReplaceDocuments, true, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(UpdateDocuments, true, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(ReplaceDocuments, false, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(UpdateDocuments, false, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del_fil(), doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del_fil(), doc_imp(UpdateDocuments, false, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del_fil(), doc_imp(ReplaceDocuments, false, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del_fil(), doc_imp(UpdateDocuments, false, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-}
-
-#[test]
-fn simple_different_document_operations_autobatch_together() {
-    // addition and updates with deletion by filter can't batch together
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_del_fil()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithDeletionByFilter { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(DeletionByFilterWithDocumentOperation { id: 1 })))");
-
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), idx_create()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(TaskCannotBeBatched { kind: IndexCreation, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), idx_create()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(TaskCannotBeBatched { kind: IndexCreation, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), idx_create()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(TaskCannotBeBatched { kind: IndexCreation, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), idx_create()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(TaskCannotBeBatched { kind: IndexCreation, id: 1 })))");
-
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), idx_update()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(TaskCannotBeBatched { kind: IndexUpdate, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), idx_update()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(TaskCannotBeBatched { kind: IndexUpdate, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), idx_update()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(TaskCannotBeBatched { kind: IndexUpdate, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), idx_update()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(TaskCannotBeBatched { kind: IndexUpdate, id: 1 })))");
-
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), idx_swap()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(TaskCannotBeBatched { kind: IndexSwap, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), idx_swap()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(TaskCannotBeBatched { kind: IndexSwap, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), idx_swap()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(TaskCannotBeBatched { kind: IndexSwap, id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), idx_swap()]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: true }, false, Some(TaskCannotBeBatched { kind: IndexSwap, id: 1 })))");
-}
-
-#[test]
-fn document_addition_doesnt_batch_with_settings() {
-    // simple case
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-
-    // multiple settings and doc addition
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None), settings(true), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, Some(DocumentOperationWithSettings { id: 2 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None), settings(true), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, Some(DocumentOperationWithSettings { id: 2 })))");
-
-    // addition and setting unordered
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), doc_imp(ReplaceDocuments, true, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), doc_imp(UpdateDocuments, true, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-
-    // Doesn't batch with other forbidden operations
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), doc_del()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), doc_del()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), idx_create()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), idx_create()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), idx_update()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), idx_update()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), idx_swap()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), idx_swap()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-}
-
-#[test]
-fn clear_and_additions() {
-    // these two doesn't need to batch
-    debug_snapshot!(autobatch_from(true, None, [doc_clr(), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentClear { ids: [0] }, false, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_clr(), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentClear { ids: [0] }, false, Some(DocumentOperationWithSettings { id: 1 })))");
-
-    // Basic use case
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None), doc_clr()]), @"Some((DocumentClear { ids: [0, 1, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None), doc_clr()]), @"Some((DocumentClear { ids: [0, 1, 2] }, true, None))");
-
-    // This batch kind doesn't mix with other document addition
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None), doc_clr(), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentClear { ids: [0, 1, 2] }, true, Some(DocumentOperationWithSettings { id: 3 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None), doc_clr(), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentClear { ids: [0, 1, 2] }, true, Some(DocumentOperationWithSettings { id: 3 })))");
-
-    // But you can batch multiple clear together
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None), doc_clr(), doc_clr(), doc_clr()]), @"Some((DocumentClear { ids: [0, 1, 2, 3, 4] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), doc_imp(UpdateDocuments, true, None), doc_clr(), doc_clr(), doc_clr()]), @"Some((DocumentClear { ids: [0, 1, 2, 3, 4] }, true, None))");
-}
-
-#[test]
-fn clear_and_additions_and_settings() {
-    // A clear don't need to autobatch the settings that happens AFTER there is no documents
-    debug_snapshot!(autobatch_from(true, None, [doc_clr(), settings(true)]), @"Some((DocumentClear { ids: [0] }, false, Some(DocumentOperationWithSettings { id: 1 })))");
-
-    debug_snapshot!(autobatch_from(true, None, [settings(true), doc_clr(), settings(true)]), @"Some((ClearAndSettings { other: [1], allow_index_creation: true, settings_ids: [0, 2] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true), doc_clr()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), settings(true), doc_clr()]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-}
-
-#[test]
-fn anything_and_index_deletion() {
-    // The `IndexDeletion` doesn't batch with anything that happens AFTER.
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_imp(ReplaceDocuments, true, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_imp(UpdateDocuments, true, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_imp(ReplaceDocuments, false, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_imp(UpdateDocuments, false, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_del()]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_del_fil()]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), doc_clr()]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), settings(true)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(true, None, [idx_del(), settings(false)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_imp(ReplaceDocuments, true, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_imp(UpdateDocuments, true, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_imp(ReplaceDocuments, false, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_imp(UpdateDocuments, false, None)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_del()]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_del_fil()]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), doc_clr()]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), settings(true)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-    debug_snapshot!(autobatch_from(false,None,  [idx_del(), settings(false)]), @"Some((IndexDeletion { ids: [0] }, false, Some(IndexDeletion { id: 0 })))");
-
-    // The index deletion can accept almost any type of `BatchKind` and transform it to an `IndexDeletion`.
-    // First, the basic cases
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, true, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, true, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, true, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(UpdateDocuments, false, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del(), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_del_fil(), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_clr(), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [settings(true), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, true, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [settings(false), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, true, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, true, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, true, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, true, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, false, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(UpdateDocuments, false, None), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_del(), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_del_fil(), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_clr(), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [settings(true), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, true, Some(IndexDeletion { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [settings(false), idx_del()]), @"Some((IndexDeletion { ids: [0, 1] }, false, Some(IndexDeletion { id: 1 })))");
-}
-
-#[test]
-fn allowed_and_disallowed_index_creation() {
-    // `DocumentImport` can't be mixed with those disallowed to do so except if the index already exists.
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, false, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(DocumentOperationWithSettings { id: 1 })))");
-
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, false, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(IndexCreationMismatch { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, false, None), doc_imp(ReplaceDocuments, false, None)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0, 1] }, false, None))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, true, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(DocumentOperationWithSettings { id: 1 })))");
-    debug_snapshot!(autobatch_from(false,None,  [doc_imp(ReplaceDocuments, false, None), settings(true)]), @"Some((DocumentOperation { allow_index_creation: false, primary_key: None, operation_ids: [0] }, false, Some(IndexCreationMismatch { id: 1 })))");
-
-    // batch deletion and addition
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(ReplaceDocuments, true, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(IndexCreationMismatch { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(UpdateDocuments, true, Some("catto"))]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(IndexCreationMismatch { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(IndexCreationMismatch { id: 1 })))");
-    debug_snapshot!(autobatch_from(false, None, [doc_del(), doc_imp(UpdateDocuments, true, None)]), @"Some((DocumentDeletion { deletion_ids: [0], includes_by_filter: false }, false, Some(IndexCreationMismatch { id: 1 })))");
-}
-
-#[test]
-fn autobatch_primary_key() {
-    // ==> If I have a pk
-    // With a single update
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-
-    // With a multiple updates
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0, 1, 2] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, Some("other"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: TaskPrimaryKeyDifferFromIndexPrimaryKey { task_pk: "other", index_pk: "id" } })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: TaskPrimaryKeyDifferFromIndexPrimaryKey { task_pk: "other", index_pk: "id" } })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: TaskPrimaryKeyDifferFromIndexPrimaryKey { task_pk: "other", index_pk: "id" } })))"###);
-
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0, 1] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0, 1, 2] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, Some("other"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: TaskPrimaryKeyDifferFromIndexPrimaryKey { task_pk: "other", index_pk: "id" } })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: TaskPrimaryKeyDifferFromIndexPrimaryKey { task_pk: "other", index_pk: "id" } })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: TaskPrimaryKeyDifferFromIndexPrimaryKey { task_pk: "other", index_pk: "id" } })))"###);
-
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("other"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-    debug_snapshot!(autobatch_from(true, Some("id"), [doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("other")), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, Some(PrimaryKeyIndexMismatch { id: 0, in_index: "id", in_task: "other" })))"###);
-
-    // ==> If I don't have a pk
-    // With a single update
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0] }, true, None))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, Some("other"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("other"), operation_ids: [0] }, true, None))"###);
-
-    // With a multiple updates
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, None)]), @"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0, 1] }, true, None))");
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, None), doc_imp(ReplaceDocuments, true, Some("id"))]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: None, operation_ids: [0] }, true, Some(PrimaryKeyMismatch { id: 1, reason: CannotInterfereWithPrimaryKeyGuessing { task_pk: "id" } })))"###);
-    debug_snapshot!(autobatch_from(true, None, [doc_imp(ReplaceDocuments, true, Some("id")), doc_imp(ReplaceDocuments, true, None)]), @r###"Some((DocumentOperation { allow_index_creation: true, primary_key: Some("id"), operation_ids: [0, 1] }, true, None))"###);
-}

crates/index-scheduler/src/scheduler/create_batch.rs

@@ -1,669 +0,0 @@
-use std::fmt;
-use std::io::ErrorKind;
-
-use meilisearch_types::heed::RoTxn;
-use meilisearch_types::milli::update::IndexDocumentsMethod;
-use meilisearch_types::settings::{Settings, Unchecked};
-use meilisearch_types::tasks::{BatchStopReason, Kind, KindWithContent, Status, Task};
-use roaring::RoaringBitmap;
-use uuid::Uuid;
-
-use super::autobatcher::{self, BatchKind};
-use crate::utils::ProcessingBatch;
-use crate::{Error, IndexScheduler, Result};
-
-/// Represents a combination of tasks that can all be processed at the same time.
-///
-/// A batch contains the set of tasks that it represents (accessible through
-/// [`self.ids()`](Batch::ids)), as well as additional information on how to
-/// be processed.
-#[derive(Debug)]
-pub(crate) enum Batch {
-    TaskCancelation {
-        /// The task cancelation itself.
-        task: Task,
-    },
-    TaskDeletions(Vec<Task>),
-    SnapshotCreation(Vec<Task>),
-    Dump(Task),
-    IndexOperation {
-        op: IndexOperation,
-        must_create_index: bool,
-    },
-    IndexCreation {
-        index_uid: String,
-        primary_key: Option<String>,
-        task: Task,
-    },
-    IndexUpdate {
-        index_uid: String,
-        primary_key: Option<String>,
-        new_index_uid: Option<String>,
-        task: Task,
-    },
-    IndexDeletion {
-        index_uid: String,
-        tasks: Vec<Task>,
-        index_has_been_created: bool,
-    },
-    IndexSwap {
-        task: Task,
-    },
-    Export {
-        task: Task,
-    },
-    UpgradeDatabase {
-        tasks: Vec<Task>,
-    },
-    IndexCompaction {
-        index_uid: String,
-        task: Task,
-    },
-}
-
-#[derive(Debug)]
-pub(crate) enum DocumentOperation {
-    Replace(Uuid),
-    Update(Uuid),
-    Delete(Vec<String>),
-}
-
-/// A [batch](Batch) that combines multiple tasks operating on an index.
-#[derive(Debug)]
-#[allow(clippy::large_enum_variant)]
-pub(crate) enum IndexOperation {
-    DocumentOperation {
-        index_uid: String,
-        primary_key: Option<String>,
-        operations: Vec<DocumentOperation>,
-        tasks: Vec<Task>,
-    },
-    DocumentEdition {
-        index_uid: String,
-        task: Task,
-    },
-    DocumentDeletion {
-        index_uid: String,
-        tasks: Vec<Task>,
-    },
-    DocumentClear {
-        index_uid: String,
-        tasks: Vec<Task>,
-    },
-    Settings {
-        index_uid: String,
-        // The boolean indicates if it's a settings deletion or creation.
-        settings: Vec<(bool, Settings<Unchecked>)>,
-        tasks: Vec<Task>,
-    },
-    DocumentClearAndSetting {
-        index_uid: String,
-        cleared_tasks: Vec<Task>,
-
-        // The boolean indicates if it's a settings deletion or creation.
-        settings: Vec<(bool, Settings<Unchecked>)>,
-        settings_tasks: Vec<Task>,
-    },
-}
-
-impl Batch {
-    /// Return the task ids associated with this batch.
-    pub fn ids(&self) -> RoaringBitmap {
-        match self {
-            Batch::TaskCancelation { task, .. }
-            | Batch::Dump(task)
-            | Batch::IndexCreation { task, .. }
-            | Batch::Export { task }
-            | Batch::IndexUpdate { task, .. }
-            | Batch::IndexCompaction { task, .. } => {
-                RoaringBitmap::from_sorted_iter(std::iter::once(task.uid)).unwrap()
-            }
-            Batch::SnapshotCreation(tasks)
-            | Batch::TaskDeletions(tasks)
-            | Batch::UpgradeDatabase { tasks }
-            | Batch::IndexDeletion { tasks, .. } => {
-                RoaringBitmap::from_iter(tasks.iter().map(|task| task.uid))
-            }
-            Batch::IndexOperation { op, .. } => match op {
-                IndexOperation::DocumentOperation { tasks, .. }
-                | IndexOperation::Settings { tasks, .. }
-                | IndexOperation::DocumentDeletion { tasks, .. }
-                | IndexOperation::DocumentClear { tasks, .. } => {
-                    RoaringBitmap::from_iter(tasks.iter().map(|task| task.uid))
-                }
-                IndexOperation::DocumentEdition { task, .. } => {
-                    RoaringBitmap::from_sorted_iter(std::iter::once(task.uid)).unwrap()
-                }
-                IndexOperation::DocumentClearAndSetting {
-                    cleared_tasks: tasks,
-                    settings_tasks: other,
-                    ..
-                } => RoaringBitmap::from_iter(tasks.iter().chain(other).map(|task| task.uid)),
-            },
-            Batch::IndexSwap { task } => {
-                RoaringBitmap::from_sorted_iter(std::iter::once(task.uid)).unwrap()
-            }
-        }
-    }
-
-    /// Return the index UID associated with this batch
-    pub fn index_uid(&self) -> Option<&str> {
-        use Batch::*;
-        match self {
-            TaskCancelation { .. }
-            | TaskDeletions(_)
-            | SnapshotCreation(_)
-            | Dump(_)
-            | Export { .. }
-            | UpgradeDatabase { .. }
-            | IndexSwap { .. } => None,
-            IndexOperation { op, .. } => Some(op.index_uid()),
-            IndexCreation { index_uid, .. }
-            | IndexUpdate { index_uid, .. }
-            | IndexDeletion { index_uid, .. }
-            | IndexCompaction { index_uid, .. } => Some(index_uid),
-        }
-    }
-}
-
-impl fmt::Display for Batch {
-    /// A text used when we debug the profiling reports.
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let index_uid = self.index_uid();
-        let tasks = self.ids();
-        match self {
-            Batch::TaskCancelation { .. } => f.write_str("TaskCancelation")?,
-            Batch::TaskDeletions(_) => f.write_str("TaskDeletion")?,
-            Batch::SnapshotCreation(_) => f.write_str("SnapshotCreation")?,
-            Batch::Dump(_) => f.write_str("Dump")?,
-            Batch::IndexOperation { op, .. } => write!(f, "{op}")?,
-            Batch::IndexCreation { .. } => f.write_str("IndexCreation")?,
-            Batch::IndexUpdate { .. } => f.write_str("IndexUpdate")?,
-            Batch::IndexDeletion { .. } => f.write_str("IndexDeletion")?,
-            Batch::IndexSwap { .. } => f.write_str("IndexSwap")?,
-            Batch::IndexCompaction { .. } => f.write_str("IndexCompaction")?,
-            Batch::Export { .. } => f.write_str("Export")?,
-            Batch::UpgradeDatabase { .. } => f.write_str("UpgradeDatabase")?,
-        };
-        match index_uid {
-            Some(name) => f.write_fmt(format_args!(" on {name:?} from tasks: {tasks:?}")),
-            None => f.write_fmt(format_args!(" from tasks: {tasks:?}")),
-        }
-    }
-}
-
-impl IndexOperation {
-    pub fn index_uid(&self) -> &str {
-        match self {
-            IndexOperation::DocumentOperation { index_uid, .. }
-            | IndexOperation::DocumentEdition { index_uid, .. }
-            | IndexOperation::DocumentDeletion { index_uid, .. }
-            | IndexOperation::DocumentClear { index_uid, .. }
-            | IndexOperation::Settings { index_uid, .. }
-            | IndexOperation::DocumentClearAndSetting { index_uid, .. } => index_uid,
-        }
-    }
-}
-
-impl fmt::Display for IndexOperation {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        match self {
-            IndexOperation::DocumentOperation { .. } => {
-                f.write_str("IndexOperation::DocumentOperation")
-            }
-            IndexOperation::DocumentEdition { .. } => {
-                f.write_str("IndexOperation::DocumentEdition")
-            }
-            IndexOperation::DocumentDeletion { .. } => {
-                f.write_str("IndexOperation::DocumentDeletion")
-            }
-            IndexOperation::DocumentClear { .. } => f.write_str("IndexOperation::DocumentClear"),
-            IndexOperation::Settings { .. } => f.write_str("IndexOperation::Settings"),
-            IndexOperation::DocumentClearAndSetting { .. } => {
-                f.write_str("IndexOperation::DocumentClearAndSetting")
-            }
-        }
-    }
-}
-
-impl IndexScheduler {
-    /// Convert an [`BatchKind`](crate::autobatcher::BatchKind) into a [`Batch`].
-    ///
-    /// ## Arguments
-    /// - `rtxn`: read transaction
-    /// - `index_uid`: name of the index affected by the operations of the autobatch
-    /// - `batch`: the result of the autobatcher
-    pub(crate) fn create_next_batch_index(
-        &self,
-        rtxn: &RoTxn,
-        index_uid: String,
-        batch: BatchKind,
-        current_batch: &mut ProcessingBatch,
-        must_create_index: bool,
-    ) -> Result<Option<Batch>> {
-        match batch {
-            BatchKind::DocumentClear { ids } => Ok(Some(Batch::IndexOperation {
-                op: IndexOperation::DocumentClear {
-                    tasks: self.queue.get_existing_tasks_for_processing_batch(
-                        rtxn,
-                        current_batch,
-                        ids,
-                    )?,
-                    index_uid,
-                },
-                must_create_index,
-            })),
-            BatchKind::DocumentEdition { id } => {
-                let mut task =
-                    self.queue.tasks.get_task(rtxn, id)?.ok_or(Error::CorruptedTaskQueue)?;
-                current_batch.processing(Some(&mut task));
-                match &task.kind {
-                    KindWithContent::DocumentEdition { index_uid, .. } => {
-                        Ok(Some(Batch::IndexOperation {
-                            op: IndexOperation::DocumentEdition {
-                                index_uid: index_uid.clone(),
-                                task,
-                            },
-                            must_create_index: false,
-                        }))
-                    }
-                    _ => unreachable!(),
-                }
-            }
-            BatchKind::DocumentOperation { operation_ids, .. } => {
-                let tasks = self.queue.get_existing_tasks_for_processing_batch(
-                    rtxn,
-                    current_batch,
-                    operation_ids,
-                )?;
-                let primary_key = tasks
-                    .iter()
-                    .find_map(|task| match task.kind {
-                        KindWithContent::DocumentAdditionOrUpdate { ref primary_key, .. } => {
-                            // we want to stop on the first document addition
-                            Some(primary_key.clone())
-                        }
-                        KindWithContent::DocumentDeletion { .. } => None,
-                        _ => unreachable!(),
-                    })
-                    .flatten();
-
-                let mut operations = Vec::new();
-
-                for task in tasks.iter() {
-                    match task.kind {
-                        KindWithContent::DocumentAdditionOrUpdate {
-                            content_file, method, ..
-                        } => match method {
-                            IndexDocumentsMethod::ReplaceDocuments => {
-                                operations.push(DocumentOperation::Replace(content_file))
-                            }
-                            IndexDocumentsMethod::UpdateDocuments => {
-                                operations.push(DocumentOperation::Update(content_file))
-                            }
-                            _ => unreachable!("Unknown document merging method"),
-                        },
-                        KindWithContent::DocumentDeletion { ref documents_ids, .. } => {
-                            operations.push(DocumentOperation::Delete(documents_ids.clone()));
-                        }
-                        _ => unreachable!(),
-                    }
-                }
-
-                Ok(Some(Batch::IndexOperation {
-                    op: IndexOperation::DocumentOperation {
-                        index_uid,
-                        primary_key,
-                        operations,
-                        tasks,
-                    },
-                    must_create_index,
-                }))
-            }
-            BatchKind::DocumentDeletion { deletion_ids, includes_by_filter: _ } => {
-                let tasks = self.queue.get_existing_tasks_for_processing_batch(
-                    rtxn,
-                    current_batch,
-                    deletion_ids,
-                )?;
-
-                Ok(Some(Batch::IndexOperation {
-                    op: IndexOperation::DocumentDeletion { index_uid, tasks },
-                    must_create_index,
-                }))
-            }
-            BatchKind::Settings { settings_ids, .. } => {
-                let tasks = self.queue.get_existing_tasks_for_processing_batch(
-                    rtxn,
-                    current_batch,
-                    settings_ids,
-                )?;
-
-                let mut settings = Vec::new();
-                for task in &tasks {
-                    match task.kind {
-                        KindWithContent::SettingsUpdate {
-                            ref new_settings, is_deletion, ..
-                        } => settings.push((is_deletion, *new_settings.clone())),
-                        _ => unreachable!(),
-                    }
-                }
-
-                Ok(Some(Batch::IndexOperation {
-                    op: IndexOperation::Settings { index_uid, settings, tasks },
-                    must_create_index,
-                }))
-            }
-            BatchKind::ClearAndSettings { other, settings_ids, allow_index_creation } => {
-                let (index_uid, settings, settings_tasks) = match self
-                    .create_next_batch_index(
-                        rtxn,
-                        index_uid,
-                        BatchKind::Settings { settings_ids, allow_index_creation },
-                        current_batch,
-                        must_create_index,
-                    )?
-                    .unwrap()
-                {
-                    Batch::IndexOperation {
-                        op: IndexOperation::Settings { index_uid, settings, tasks, .. },
-                        ..
-                    } => (index_uid, settings, tasks),
-                    _ => unreachable!(),
-                };
-                let (index_uid, cleared_tasks) = match self
-                    .create_next_batch_index(
-                        rtxn,
-                        index_uid,
-                        BatchKind::DocumentClear { ids: other },
-                        current_batch,
-                        must_create_index,
-                    )?
-                    .unwrap()
-                {
-                    Batch::IndexOperation {
-                        op: IndexOperation::DocumentClear { index_uid, tasks },
-                        ..
-                    } => (index_uid, tasks),
-                    _ => unreachable!(),
-                };
-
-                Ok(Some(Batch::IndexOperation {
-                    op: IndexOperation::DocumentClearAndSetting {
-                        index_uid,
-                        cleared_tasks,
-                        settings,
-                        settings_tasks,
-                    },
-                    must_create_index,
-                }))
-            }
-            BatchKind::IndexCreation { id } => {
-                let mut task =
-                    self.queue.tasks.get_task(rtxn, id)?.ok_or(Error::CorruptedTaskQueue)?;
-                current_batch.processing(Some(&mut task));
-                let (index_uid, primary_key) = match &task.kind {
-                    KindWithContent::IndexCreation { index_uid, primary_key } => {
-                        (index_uid.clone(), primary_key.clone())
-                    }
-                    _ => unreachable!(),
-                };
-                Ok(Some(Batch::IndexCreation { index_uid, primary_key, task }))
-            }
-            BatchKind::IndexUpdate { id } => {
-                let mut task =
-                    self.queue.tasks.get_task(rtxn, id)?.ok_or(Error::CorruptedTaskQueue)?;
-                current_batch.processing(Some(&mut task));
-                let (primary_key, new_index_uid) = match &task.kind {
-                    KindWithContent::IndexUpdate { primary_key, new_index_uid, .. } => {
-                        (primary_key.clone(), new_index_uid.clone())
-                    }
-                    _ => unreachable!(),
-                };
-                Ok(Some(Batch::IndexUpdate { index_uid, primary_key, new_index_uid, task }))
-            }
-            BatchKind::IndexDeletion { ids } => Ok(Some(Batch::IndexDeletion {
-                index_uid,
-                index_has_been_created: must_create_index,
-                tasks: self.queue.get_existing_tasks_for_processing_batch(
-                    rtxn,
-                    current_batch,
-                    ids,
-                )?,
-            })),
-            BatchKind::IndexSwap { id } => {
-                let mut task =
-                    self.queue.tasks.get_task(rtxn, id)?.ok_or(Error::CorruptedTaskQueue)?;
-                current_batch.processing(Some(&mut task));
-                Ok(Some(Batch::IndexSwap { task }))
-            }
-        }
-    }
-
-    /// Create the next batch to be processed;
-    /// 0. We get the *last* task to cancel.
-    /// 1. We get the tasks to upgrade.
-    /// 2. We get the *next* task to delete.
-    /// 3. We get the *next* export to process.
-    /// 4. We get the *next* snapshot to process.
-    /// 5. We get the *next* dump to process.
-    /// 6. We get the *next* tasks to process for a specific index.
-    #[tracing::instrument(level = "trace", skip(self, rtxn), target = "indexing::scheduler")]
-    pub(crate) fn create_next_batch(
-        &self,
-        rtxn: &RoTxn,
-    ) -> Result<Option<(Batch, ProcessingBatch)>> {
-        #[cfg(test)]
-        self.maybe_fail(crate::test_utils::FailureLocation::InsideCreateBatch)?;
-
-        let batch_id = self.queue.batches.next_batch_id(rtxn)?;
-        let mut current_batch = ProcessingBatch::new(batch_id);
-
-        let enqueued = &self.queue.tasks.get_status(rtxn, Status::Enqueued)?;
-        let count_total_enqueued = enqueued.len();
-        let failed = &self.queue.tasks.get_status(rtxn, Status::Failed)?;
-
-        // 0. we get the last task to cancel.
-        let to_cancel = self.queue.tasks.get_kind(rtxn, Kind::TaskCancelation)? & enqueued;
-        if let Some(task_id) = to_cancel.max() {
-            let mut task =
-                self.queue.tasks.get_task(rtxn, task_id)?.ok_or(Error::CorruptedTaskQueue)?;
-            current_batch.processing(Some(&mut task));
-            current_batch.reason(BatchStopReason::TaskCannotBeBatched {
-                kind: Kind::TaskCancelation,
-                id: task_id,
-            });
-            return Ok(Some((Batch::TaskCancelation { task }, current_batch)));
-        }
-
-        // 1. We upgrade the instance
-        // There shouldn't be multiple upgrade tasks but just in case we're going to batch all of them at the same time
-        let upgrade = self.queue.tasks.get_kind(rtxn, Kind::UpgradeDatabase)? & (enqueued | failed);
-        if !upgrade.is_empty() {
-            let mut tasks = self.queue.tasks.get_existing_tasks(rtxn, upgrade)?;
-            // In the case of an upgrade database batch, we want to find back the original batch that tried processing it
-            // and re-use its id
-            if let Some(batch_uid) = tasks.last().unwrap().batch_uid {
-                current_batch.uid = batch_uid;
-            }
-            current_batch.processing(&mut tasks);
-            current_batch
-                .reason(BatchStopReason::TaskKindCannotBeBatched { kind: Kind::UpgradeDatabase });
-            return Ok(Some((Batch::UpgradeDatabase { tasks }, current_batch)));
-        }
-
-        // check the version of the scheduler here.
-        // if the version is not the current, refuse to batch any additional task.
-        let version = self.version.get_version(rtxn)?;
-        let package_version = (
-            meilisearch_types::versioning::VERSION_MAJOR,
-            meilisearch_types::versioning::VERSION_MINOR,
-            meilisearch_types::versioning::VERSION_PATCH,
-        );
-        if version != Some(package_version) {
-            return Err(Error::UnrecoverableError(Box::new(
-                Error::IndexSchedulerVersionMismatch {
-                    index_scheduler_version: version.unwrap_or((1, 12, 0)),
-                    package_version,
-                },
-            )));
-        }
-
-        // 2. we get the next task to delete
-        let to_delete = self.queue.tasks.get_kind(rtxn, Kind::TaskDeletion)? & enqueued;
-        if !to_delete.is_empty() {
-            let mut tasks = self.queue.tasks.get_existing_tasks(rtxn, to_delete)?;
-            current_batch.processing(&mut tasks);
-            current_batch
-                .reason(BatchStopReason::TaskKindCannotBeBatched { kind: Kind::TaskDeletion });
-            return Ok(Some((Batch::TaskDeletions(tasks), current_batch)));
-        }
-
-        // 3. we get the next task to compact
-        let to_compact = self.queue.tasks.get_kind(rtxn, Kind::IndexCompaction)? & enqueued;
-        if let Some(task_id) = to_compact.min() {
-            let mut task =
-                self.queue.tasks.get_task(rtxn, task_id)?.ok_or(Error::CorruptedTaskQueue)?;
-            current_batch.processing(Some(&mut task));
-            current_batch.reason(BatchStopReason::TaskCannotBeBatched {
-                kind: Kind::IndexCompaction,
-                id: task_id,
-            });
-            let index_uid =
-                task.index_uid().expect("Compaction task must have an index uid").to_owned();
-            return Ok(Some((Batch::IndexCompaction { index_uid, task }, current_batch)));
-        }
-
-        // 4. we batch the export.
-        let to_export = self.queue.tasks.get_kind(rtxn, Kind::Export)? & enqueued;
-        if !to_export.is_empty() {
-            let task_id = to_export.iter().next().expect("There must be at least one export task");
-            let mut task = self.queue.tasks.get_task(rtxn, task_id)?.unwrap();
-            current_batch.processing([&mut task]);
-            current_batch
-                .reason(BatchStopReason::TaskCannotBeBatched { kind: Kind::Export, id: task_id });
-            return Ok(Some((Batch::Export { task }, current_batch)));
-        }
-
-        // 5. we batch the snapshot.
-        let to_snapshot = self.queue.tasks.get_kind(rtxn, Kind::SnapshotCreation)? & enqueued;
-        if !to_snapshot.is_empty() {
-            let mut tasks = self.queue.tasks.get_existing_tasks(rtxn, to_snapshot)?;
-            current_batch.processing(&mut tasks);
-            current_batch
-                .reason(BatchStopReason::TaskKindCannotBeBatched { kind: Kind::SnapshotCreation });
-            return Ok(Some((Batch::SnapshotCreation(tasks), current_batch)));
-        }
-
-        // 6. we batch the dumps.
-        let to_dump = self.queue.tasks.get_kind(rtxn, Kind::DumpCreation)? & enqueued;
-        if let Some(to_dump) = to_dump.min() {
-            let mut task =
-                self.queue.tasks.get_task(rtxn, to_dump)?.ok_or(Error::CorruptedTaskQueue)?;
-            current_batch.processing(Some(&mut task));
-            current_batch.reason(BatchStopReason::TaskCannotBeBatched {
-                kind: Kind::DumpCreation,
-                id: task.uid,
-            });
-            return Ok(Some((Batch::Dump(task), current_batch)));
-        }
-
-        // 7. We make a batch from the unprioritised tasks. Start by taking the next enqueued task.
-        let task_id = if let Some(task_id) = enqueued.min() { task_id } else { return Ok(None) };
-        let mut task =
-            self.queue.tasks.get_task(rtxn, task_id)?.ok_or(Error::CorruptedTaskQueue)?;
-
-        // If the task is not associated with any index, verify that it is an index swap and
-        // create the batch directly. Otherwise, get the index name associated with the task
-        // and use the autobatcher to batch the enqueued tasks associated with it
-
-        let index_name = if let Some(&index_name) = task.indexes().first() {
-            index_name
-        } else {
-            assert!(matches!(&task.kind, KindWithContent::IndexSwap { swaps } if swaps.is_empty()));
-            current_batch.processing(Some(&mut task));
-            current_batch.reason(BatchStopReason::TaskCannotBeBatched {
-                kind: Kind::IndexSwap,
-                id: task.uid,
-            });
-            return Ok(Some((Batch::IndexSwap { task }, current_batch)));
-        };
-
-        let index_already_exists = self.index_mapper.exists(rtxn, index_name)?;
-        let mut primary_key = None;
-        if index_already_exists {
-            let index = self.index_mapper.index(rtxn, index_name)?;
-            let rtxn = index.read_txn()?;
-            primary_key = index.primary_key(&rtxn)?.map(|pk| pk.to_string());
-        }
-
-        let index_tasks = self.queue.tasks.index_tasks(rtxn, index_name)? & enqueued;
-
-        // If autobatching is disabled we only take one task at a time.
-        // Otherwise, we take only a maximum of tasks to create batches.
-        let tasks_limit = if self.scheduler.autobatching_enabled {
-            self.scheduler.max_number_of_batched_tasks
-        } else {
-            1
-        };
-
-        let mut stop_reason = BatchStopReason::default();
-        let mut enqueued = Vec::new();
-        let mut total_size: u64 = 0;
-        for task_id in index_tasks.into_iter() {
-            if enqueued.len() >= tasks_limit {
-                stop_reason = BatchStopReason::ReachedTaskLimit { task_limit: tasks_limit };
-                break;
-            }
-            let task = self
-                .queue
-                .tasks
-                .get_task(rtxn, task_id)
-                .and_then(|task| task.ok_or(Error::CorruptedTaskQueue))?;
-
-            if let Some(uuid) = task.content_uuid() {
-                let content_size = match self.queue.file_store.compute_size(uuid) {
-                    Ok(content_size) => content_size,
-                    Err(file_store::Error::IoError(err)) if err.kind() == ErrorKind::NotFound => 0,
-                    Err(otherwise) => return Err(otherwise.into()),
-                };
-                total_size = total_size.saturating_add(content_size);
-            }
-
-            let size_limit = self.scheduler.batched_tasks_size_limit;
-            if total_size > size_limit && !enqueued.is_empty() {
-                stop_reason = BatchStopReason::ReachedSizeLimit { size_limit, size: total_size };
-                break;
-            }
-
-            enqueued.push((task.uid, task.kind));
-        }
-
-        stop_reason.replace_unspecified({
-            if enqueued.len() == count_total_enqueued as usize {
-                BatchStopReason::ExhaustedEnqueuedTasks
-            } else {
-                BatchStopReason::ExhaustedEnqueuedTasksForIndex { index: index_name.to_owned() }
-            }
-        });
-
-        if let Some((batchkind, create_index, autobatch_stop_reason)) =
-            autobatcher::autobatch(enqueued, index_already_exists, primary_key.as_deref())
-        {
-            current_batch.reason(autobatch_stop_reason.unwrap_or(stop_reason));
-            return Ok(self
-                .create_next_batch_index(
-                    rtxn,
-                    index_name.to_string(),
-                    batchkind,
-                    &mut current_batch,
-                    create_index,
-                )?
-                .map(|batch| (batch, current_batch)));
-        }
-
-        // If we found no tasks then we were notified for something that got autobatched
-        // somehow and there is nothing to do.
-        Ok(None)
-    }
-}

crates/index-scheduler/src/scheduler/mod.rs

@@ -1,493 +0,0 @@
-mod autobatcher;
-#[cfg(test)]
-mod autobatcher_test;
-mod create_batch;
-mod process_batch;
-mod process_dump_creation;
-mod process_export;
-mod process_index_operation;
-mod process_snapshot_creation;
-mod process_upgrade;
-#[cfg(test)]
-mod test;
-#[cfg(test)]
-mod test_document_addition;
-#[cfg(test)]
-mod test_embedders;
-#[cfg(test)]
-mod test_failure;
-
-use std::path::PathBuf;
-use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
-use std::sync::Arc;
-
-use convert_case::{Case, Casing as _};
-use meilisearch_types::error::ResponseError;
-use meilisearch_types::heed::{Env, WithoutTls};
-use meilisearch_types::milli;
-use meilisearch_types::milli::update::S3SnapshotOptions;
-use meilisearch_types::tasks::Status;
-use process_batch::ProcessBatchInfo;
-use rayon::current_num_threads;
-use rayon::iter::{IntoParallelIterator, ParallelIterator};
-use roaring::RoaringBitmap;
-use synchronoise::SignalEvent;
-
-use crate::processing::{AtomicTaskStep, BatchProgress};
-use crate::{Error, IndexScheduler, IndexSchedulerOptions, Result, TickOutcome};
-
-#[derive(Default, Clone, Debug)]
-pub struct MustStopProcessing(Arc<AtomicBool>);
-
-impl MustStopProcessing {
-    pub fn get(&self) -> bool {
-        self.0.load(Ordering::Relaxed)
-    }
-
-    pub fn must_stop(&self) {
-        self.0.store(true, Ordering::Relaxed);
-    }
-
-    pub fn reset(&self) {
-        self.0.store(false, Ordering::Relaxed);
-    }
-}
-
-pub struct Scheduler {
-    /// A boolean that can be set to true to stop the currently processing tasks.
-    pub must_stop_processing: MustStopProcessing,
-
-    /// Get a signal when a batch needs to be processed.
-    pub(crate) wake_up: Arc<SignalEvent>,
-
-    /// Whether auto-batching is enabled or not.
-    pub(crate) autobatching_enabled: bool,
-
-    /// The maximum number of tasks that will be batched together.
-    pub(crate) max_number_of_batched_tasks: usize,
-
-    /// The maximum size, in bytes, of tasks in a batch.
-    pub(crate) batched_tasks_size_limit: u64,
-
-    /// The path used to create the dumps.
-    pub(crate) dumps_path: PathBuf,
-
-    /// The path used to create the snapshots.
-    pub(crate) snapshots_path: PathBuf,
-
-    /// The path to the folder containing the auth LMDB env.
-    pub(crate) auth_env: Env<WithoutTls>,
-
-    /// The path to the version file of Meilisearch.
-    pub(crate) version_file_path: PathBuf,
-
-    /// The maximal number of entries in the search query cache of an embedder.
-    ///
-    /// 0 disables the cache.
-    pub(crate) embedding_cache_cap: usize,
-
-    /// Snapshot compaction status.
-    pub(crate) experimental_no_snapshot_compaction: bool,
-
-    /// S3 Snapshot options.
-    pub(crate) s3_snapshot_options: Option<S3SnapshotOptions>,
-}
-
-impl Scheduler {
-    pub(crate) fn private_clone(&self) -> Self {
-        Self {
-            must_stop_processing: self.must_stop_processing.clone(),
-            wake_up: self.wake_up.clone(),
-            autobatching_enabled: self.autobatching_enabled,
-            max_number_of_batched_tasks: self.max_number_of_batched_tasks,
-            batched_tasks_size_limit: self.batched_tasks_size_limit,
-            dumps_path: self.dumps_path.clone(),
-            snapshots_path: self.snapshots_path.clone(),
-            auth_env: self.auth_env.clone(),
-            version_file_path: self.version_file_path.clone(),
-            embedding_cache_cap: self.embedding_cache_cap,
-            experimental_no_snapshot_compaction: self.experimental_no_snapshot_compaction,
-            s3_snapshot_options: self.s3_snapshot_options.clone(),
-        }
-    }
-
-    pub fn new(options: &IndexSchedulerOptions, auth_env: Env<WithoutTls>) -> Scheduler {
-        let IndexSchedulerOptions {
-            version_file_path,
-            auth_path: _,
-            tasks_path: _,
-            update_file_path: _,
-            indexes_path: _,
-            snapshots_path,
-            dumps_path,
-            cli_webhook_url: _,
-            cli_webhook_authorization: _,
-            task_db_size: _,
-            index_base_map_size: _,
-            enable_mdb_writemap: _,
-            index_growth_amount: _,
-            index_count: _,
-            indexer_config,
-            autobatching_enabled,
-            cleanup_enabled: _,
-            max_number_of_tasks: _,
-            max_number_of_batched_tasks,
-            batched_tasks_size_limit,
-            instance_features: _,
-            auto_upgrade: _,
-            embedding_cache_cap,
-            experimental_no_snapshot_compaction,
-        } = options;
-
-        Scheduler {
-            must_stop_processing: MustStopProcessing::default(),
-            // we want to start the loop right away in case meilisearch was ctrl+Ced while processing things
-            wake_up: Arc::new(SignalEvent::auto(true)),
-            autobatching_enabled: *autobatching_enabled,
-            max_number_of_batched_tasks: *max_number_of_batched_tasks,
-            batched_tasks_size_limit: *batched_tasks_size_limit,
-            dumps_path: dumps_path.clone(),
-            snapshots_path: snapshots_path.clone(),
-            auth_env,
-            version_file_path: version_file_path.clone(),
-            embedding_cache_cap: *embedding_cache_cap,
-            experimental_no_snapshot_compaction: *experimental_no_snapshot_compaction,
-            s3_snapshot_options: indexer_config.s3_snapshot_options.clone(),
-        }
-    }
-}
-
-impl IndexScheduler {
-    /// Perform one iteration of the run loop.
-    ///
-    /// 1. See if we need to cleanup the task queue
-    /// 2. Find the next batch of tasks to be processed.
-    /// 3. Update the information of these tasks following the start of their processing.
-    /// 4. Update the in-memory list of processed tasks accordingly.
-    /// 5. Process the batch:
-    ///    - perform the actions of each batched task
-    ///    - update the information of each batched task following the end
-    ///      of their processing.
-    /// 6. Reset the in-memory list of processed tasks.
-    ///
-    /// Returns the number of processed tasks.
-    pub(crate) fn tick(&self) -> Result<TickOutcome> {
-        #[cfg(test)]
-        {
-            *self.run_loop_iteration.write().unwrap() += 1;
-            self.breakpoint(crate::test_utils::Breakpoint::Start);
-        }
-
-        if self.cleanup_enabled {
-            let mut wtxn = self.env.write_txn()?;
-            self.queue.cleanup_task_queue(&mut wtxn)?;
-            wtxn.commit()?;
-        }
-
-        let rtxn = self.env.read_txn().map_err(Error::HeedTransaction)?;
-        let (batch, mut processing_batch) =
-            match self.create_next_batch(&rtxn).map_err(|e| Error::CreateBatch(Box::new(e)))? {
-                Some(batch) => batch,
-                None => return Ok(TickOutcome::WaitForSignal),
-            };
-        let index_uid = batch.index_uid().map(ToOwned::to_owned);
-        drop(rtxn);
-
-        // 1. store the starting date with the bitmap of processing tasks.
-        let mut ids = batch.ids();
-        let processed_tasks = ids.len();
-
-        // We reset the must_stop flag to be sure that we don't stop processing tasks
-        self.scheduler.must_stop_processing.reset();
-        let progress = self
-            .processing_tasks
-            .write()
-            .unwrap()
-            // We can clone the processing batch here because we don't want its modification to affect the view of the processing batches
-            .start_processing(processing_batch.clone(), ids.clone());
-
-        #[cfg(test)]
-        self.breakpoint(crate::test_utils::Breakpoint::BatchCreated);
-
-        // 2. Process the tasks
-        let res = {
-            let cloned_index_scheduler = self.private_clone();
-            let processing_batch = &mut processing_batch;
-            let progress = progress.clone();
-            std::thread::scope(|s| {
-                let p = progress.clone();
-                let handle = std::thread::Builder::new()
-                    .name(String::from("batch-operation"))
-                    .spawn_scoped(s, move || {
-                        cloned_index_scheduler.process_batch(batch, processing_batch, p)
-                    })
-                    .unwrap();
-
-                match handle.join() {
-                    Ok(ret) => {
-                        if ret.is_err() {
-                            if let Ok(progress_view) =
-                                serde_json::to_string(&progress.as_progress_view())
-                            {
-                                tracing::warn!("Batch failed while doing: {progress_view}")
-                            }
-                        }
-                        ret
-                    }
-                    Err(panic) => {
-                        if let Ok(progress_view) =
-                            serde_json::to_string(&progress.as_progress_view())
-                        {
-                            tracing::warn!("Batch failed while doing: {progress_view}")
-                        }
-                        let msg = match panic.downcast_ref::<&'static str>() {
-                            Some(s) => *s,
-                            None => match panic.downcast_ref::<String>() {
-                                Some(s) => &s[..],
-                                None => "Box<dyn Any>",
-                            },
-                        };
-                        Err(Error::ProcessBatchPanicked(msg.to_string()))
-                    }
-                }
-            })
-        };
-
-        // Reset the currently updating index to relinquish the index handle
-        self.index_mapper.set_currently_updating_index(None);
-
-        #[cfg(test)]
-        self.maybe_fail(crate::test_utils::FailureLocation::AcquiringWtxn)?;
-
-        progress.update_progress(BatchProgress::WritingTasksToDisk);
-        processing_batch.finished();
-        let mut stop_scheduler_forever = false;
-        let mut wtxn = self.env.write_txn().map_err(Error::HeedTransaction)?;
-        let mut canceled = RoaringBitmap::new();
-        let mut process_batch_info = ProcessBatchInfo::default();
-
-        match res {
-            Ok((tasks, info)) => {
-                #[cfg(test)]
-                self.breakpoint(crate::test_utils::Breakpoint::ProcessBatchSucceeded);
-
-                let (task_progress, task_progress_obj) = AtomicTaskStep::new(tasks.len() as u32);
-                progress.update_progress(task_progress_obj);
-                process_batch_info = info;
-                let mut success = 0;
-                let mut failure = 0;
-                let mut canceled_by = None;
-
-                #[allow(unused_variables)]
-                for (i, mut task) in tasks.into_iter().enumerate() {
-                    task_progress.fetch_add(1, Ordering::Relaxed);
-                    processing_batch.update(&mut task);
-                    if task.status == Status::Canceled {
-                        canceled.insert(task.uid);
-                        canceled_by = task.canceled_by;
-                    }
-
-                    #[cfg(test)]
-                    self.maybe_fail(
-                        crate::test_utils::FailureLocation::UpdatingTaskAfterProcessBatchSuccess {
-                            task_uid: i as u32,
-                        },
-                    )?;
-
-                    match task.error {
-                        Some(_) => failure += 1,
-                        None => success += 1,
-                    }
-
-                    self.queue
-                        .tasks
-                        .update_task(&mut wtxn, &mut task)
-                        .map_err(|e| Error::UnrecoverableError(Box::new(e)))?;
-                }
-                if let Some(canceled_by) = canceled_by {
-                    self.queue.tasks.canceled_by.put(&mut wtxn, &canceled_by, &canceled)?;
-                }
-                tracing::info!("A batch of tasks was successfully completed with {success} successful tasks and {failure} failed tasks.");
-            }
-            // If we have an abortion error we must stop the tick here and re-schedule tasks.
-            Err(Error::Milli {
-                error: milli::Error::InternalError(milli::InternalError::AbortedIndexation),
-                ..
-            })
-            | Err(Error::AbortedTask) => {
-                #[cfg(test)]
-                self.breakpoint(crate::test_utils::Breakpoint::AbortedIndexation);
-                wtxn.abort();
-
-                tracing::info!("A batch of tasks was aborted.");
-                // We make sure that we don't call `stop_processing` on the `processing_tasks`,
-                // this is because we want to let the next tick call `create_next_batch` and keep
-                // the `started_at` date times and `processings` of the current processing tasks.
-                // This date time is used by the task cancelation to store the right `started_at`
-                // date in the task on disk.
-                return Ok(TickOutcome::TickAgain(0));
-            }
-            // If an index said it was full, we need to:
-            // 1. identify which index is full
-            // 2. close the associated environment
-            // 3. resize it
-            // 4. re-schedule tasks
-            Err(Error::Milli {
-                error: milli::Error::UserError(milli::UserError::MaxDatabaseSizeReached),
-                ..
-            }) if index_uid.is_some() => {
-                // fixme: add index_uid to match to avoid the unwrap
-                let index_uid = index_uid.unwrap();
-                // fixme: handle error more gracefully? not sure when this could happen
-                self.index_mapper.resize_index(&wtxn, &index_uid)?;
-                wtxn.abort();
-
-                tracing::info!("The max database size was reached. Resizing the index.");
-
-                return Ok(TickOutcome::TickAgain(0));
-            }
-            // In case of a failure we must get back and patch all the tasks with the error.
-            Err(err) => {
-                #[cfg(test)]
-                self.breakpoint(crate::test_utils::Breakpoint::ProcessBatchFailed);
-                let (task_progress, task_progress_obj) = AtomicTaskStep::new(ids.len() as u32);
-                progress.update_progress(task_progress_obj);
-
-                if matches!(err, Error::DatabaseUpgrade(_)) {
-                    tracing::error!(
-                        "Upgrade task failed, tasks won't be processed until the following issue is fixed: {err}"
-                    );
-                    stop_scheduler_forever = true;
-                }
-                let error: ResponseError = err.into();
-                for id in ids.iter() {
-                    task_progress.fetch_add(1, Ordering::Relaxed);
-                    let mut task = self
-                        .queue
-                        .tasks
-                        .get_task(&wtxn, id)
-                        .map_err(|e| Error::UnrecoverableError(Box::new(e)))?
-                        .ok_or(Error::CorruptedTaskQueue)?;
-                    task.status = Status::Failed;
-                    task.error = Some(error.clone());
-                    task.details = task.details.map(|d| d.to_failed());
-                    processing_batch.update(&mut task);
-
-                    #[cfg(test)]
-                    self.maybe_fail(
-                        crate::test_utils::FailureLocation::UpdatingTaskAfterProcessBatchFailure,
-                    )?;
-
-                    tracing::error!("Batch failed {}", error);
-
-                    self.queue
-                        .tasks
-                        .update_task(&mut wtxn, &mut task)
-                        .map_err(|e| Error::UnrecoverableError(Box::new(e)))?;
-                }
-            }
-        }
-
-        // We must re-add the canceled task so they're part of the same batch.
-        ids |= canceled;
-
-        let ProcessBatchInfo { congestion, pre_commit_dabases_sizes, post_commit_dabases_sizes } =
-            process_batch_info;
-
-        processing_batch.stats.progress_trace =
-            progress.accumulated_durations().into_iter().map(|(k, v)| (k, v.into())).collect();
-        processing_batch.stats.write_channel_congestion = congestion.map(|congestion| {
-            let mut congestion_info = serde_json::Map::new();
-            congestion_info.insert("attempts".into(), congestion.attempts.into());
-            congestion_info.insert("blocking_attempts".into(), congestion.blocking_attempts.into());
-            congestion_info.insert("blocking_ratio".into(), congestion.congestion_ratio().into());
-            congestion_info
-        });
-        processing_batch.stats.internal_database_sizes = pre_commit_dabases_sizes
-            .iter()
-            .flat_map(|(dbname, pre_size)| {
-                post_commit_dabases_sizes
-                    .get(dbname)
-                    .map(|post_size| {
-                        use std::cmp::Ordering::{Equal, Greater, Less};
-
-                        use byte_unit::Byte;
-                        use byte_unit::UnitType::Binary;
-
-                        let post = Byte::from_u64(*post_size as u64).get_appropriate_unit(Binary);
-                        let diff_size = post_size.abs_diff(*pre_size) as u64;
-                        let diff = Byte::from_u64(diff_size).get_appropriate_unit(Binary);
-                        let sign = match post_size.cmp(pre_size) {
-                            Equal => return None,
-                            Greater => "+",
-                            Less => "-",
-                        };
-
-                        Some((
-                            dbname.to_case(Case::Camel),
-                            format!("{post:#.2} ({sign}{diff:#.2})").into(),
-                        ))
-                    })
-                    .into_iter()
-                    .flatten()
-            })
-            .collect();
-
-        if let Some(congestion) = congestion {
-            tracing::debug!(
-                "Channel congestion metrics - Attempts: {}, Blocked attempts: {}  ({:.1}% congestion)",
-                congestion.attempts,
-                congestion.blocking_attempts,
-                congestion.congestion_ratio(),
-            );
-        }
-
-        tracing::debug!("call trace: {:?}", progress.accumulated_durations());
-
-        self.queue.write_batch(&mut wtxn, processing_batch, &ids)?;
-
-        #[cfg(test)]
-        self.maybe_fail(crate::test_utils::FailureLocation::CommittingWtxn)?;
-
-        wtxn.commit().map_err(Error::HeedTransaction)?;
-
-        // We should stop processing AFTER everything is processed and written to disk otherwise, a batch (which only lives in RAM) may appear in the processing task
-        // and then become « not found » for some time until the commit everything is written and the final commit is made.
-        self.processing_tasks.write().unwrap().stop_processing();
-
-        // Once the tasks are committed, we should delete all the update files associated ASAP to avoid leaking files in case of a restart
-        tracing::debug!("Deleting the update files");
-
-        //We take one read transaction **per thread**. Then, every thread is going to pull out new IDs from the roaring bitmap with the help of an atomic shared index into the bitmap
-        let idx = AtomicU32::new(0);
-        (0..current_num_threads()).into_par_iter().try_for_each(|_| -> Result<()> {
-            let rtxn = self.read_txn()?;
-            while let Some(id) = ids.select(idx.fetch_add(1, Ordering::Relaxed)) {
-                let task = self
-                    .queue
-                    .tasks
-                    .get_task(&rtxn, id)
-                    .map_err(|e| Error::UnrecoverableError(Box::new(e)))?
-                    .ok_or(Error::CorruptedTaskQueue)?;
-                if let Err(e) = self.queue.delete_persisted_task_data(&task) {
-                    tracing::error!(
-                        "Failure to delete the content files associated with task {}. Error: {e}",
-                        task.uid
-                    );
-                }
-            }
-            Ok(())
-        })?;
-
-        self.notify_webhooks(ids);
-
-        #[cfg(test)]
-        self.breakpoint(crate::test_utils::Breakpoint::AfterProcessing);
-
-        if stop_scheduler_forever {
-            Ok(TickOutcome::StopProcessingForever)
-        } else {
-            Ok(TickOutcome::TickAgain(processed_tasks))
-        }
-    }
-}