Merge pull request #5844 from meilisearch/prepare-v1.18

Prepare v1.18.0

.github/ISSUE_TEMPLATE/new_feature_issue.md

@@ -1,28 +1,26 @@
 ---
-name: New sprint issue
-about: ⚠️ Should only be used by the engine team ⚠️
+name: New feature issue
+about: ⚠️ Should only be used by the internal Meili team ⚠️
 title: ''
-labels: 'missing usage in PRD, impacts docs'
+labels: 'impacts docs, impacts integrations'
 assignees: ''
 
 ---
 
 Related product team resources: [PRD]() (_internal only_)
-Related product discussion:
-
-## Motivation
-
-<!---Copy/paste the information in PRD or briefly detail the product motivation. Ask product team if any hesitation.-->
 
 ## Usage
 
 <!---Link to the public part of the PRD, or to the related product discussion for experimental features-->
 
+TBD
+
 ## TODO
 
 <!---If necessary, create a list with technical/product steps-->
 
 ### Are you modifying a database?
+
 - [ ] If not, add the `no db change` label to your PR, and you're good to merge.
 - [ ] If yes, add the `db change` label to your PR. You'll receive a message explaining you what to do.
 
@@ -54,5 +52,5 @@ Related product discussion:
 
 ## Impacted teams
 
-<!---Ping the related teams. Ask for the engine manager if any hesitation-->
-<!---@meilisearch/docs-team when there is any API change, e.g. settings addition-->
+<!---Ping the related teams. Ask on Slack if any hesitation-->
+<!---@meilisearch/docs-team and @meilisearch/integration-team when there is any API change, e.g. settings addition-->

.github/pull_request_template.md

@@ -0,0 +1,16 @@
+## Related issue
+
+Fixes #...
+
+## Requirements
+
+⚠️ Ensure the following requirements before merging ⚠️
+- [ ] Automated tests have been added.
+- [ ] If some tests cannot be automated, manual rigorous tests should be applied.
+- [ ] ⚠️ If there is any change in the DB: 
+    - [ ] Test that any impacted DB still works as expected after using `--experimental-dumpless-upgrade` on a DB created with the last released Meilisearch
+    - [ ] Test that during the upgrade, **search is still available** (artificially make the upgrade longer if needed)
+    - [ ] Set the `db change` label.
+- [ ] If necessary, the feature have been tested in the Cloud production environment (with [prototypes](./documentation/prototypes.md)) and the Cloud UI is ready.
+- [ ] If necessary, the [documentation](https://github.com/meilisearch/documentation) related to the implemented feature in the PR is ready.
+- [ ] If necessary, the [integrations](https://github.com/meilisearch/integration-guides) related to the implemented feature in the PR are ready.

.github/release-draft-template.yml

@@ -0,0 +1,33 @@
+name-template: 'v$RESOLVED_VERSION'
+tag-template: 'v$RESOLVED_VERSION'
+exclude-labels:
+  - 'skip changelog'
+version-resolver:
+  minor:
+    labels:
+      - 'enhancement'
+  default: patch
+categories:
+  - title: '⚠️ Breaking changes'
+    label: 'breaking-change'
+  - title: '🚀 Enhancements'
+    label: 'enhancement'
+  - title: '🐛 Bug Fixes'
+    label: 'bug'
+  - title: '🔒 Security'
+    label: 'security'
+  - title: '⚙️ Maintenance/misc'
+    label:
+      - 'maintenance'
+      - 'documentation'
+template: |
+  $CHANGES
+
+  ❤️  Huge thanks to our contributors: $CONTRIBUTORS.
+no-changes-template: 'Changes are coming soon 😎'
+sort-direction: 'ascending'
+replacers:
+  - search: '/(?:and )?@dependabot-preview(?:\[bot\])?,?/g'
+    replace: ''
+  - search: '/(?:and )?@dependabot(?:\[bot\])?,?/g'
+    replace: ''

.github/templates/dependency-issue.md

@@ -0,0 +1,22 @@
+This issue is about updating Meilisearch dependencies:
+  - [ ] Update Meilisearch dependencies with the help of `cargo +nightly udeps --all-targets` (remove unused dependencies) and `cargo upgrade` (upgrade dependencies versions) - ⚠️ Some repositories may contain subdirectories (like heed, charabia, or deserr). Take care of updating these in the main crate as well. This won't be done automatically by `cargo upgrade`.
+    - [ ] [deserr](https://github.com/meilisearch/deserr)
+    - [ ] [charabia](https://github.com/meilisearch/charabia/)
+    - [ ] [heed](https://github.com/meilisearch/heed/)
+    - [ ] [roaring-rs](https://github.com/RoaringBitmap/roaring-rs/)
+    - [ ] [obkv](https://github.com/meilisearch/obkv)
+    - [ ] [grenad](https://github.com/meilisearch/grenad/)
+    - [ ] [arroy](https://github.com/meilisearch/arroy/)
+    - [ ] [segment](https://github.com/meilisearch/segment)
+    - [ ] [bumparaw-collections](https://github.com/meilisearch/bumparaw-collections)
+    - [ ] [bbqueue](https://github.com/meilisearch/bbqueue)
+    - [ ] Finally, [Meilisearch](https://github.com/meilisearch/MeiliSearch)
+  - [ ] If new Rust versions have been released, update the minimal Rust version in use at Meilisearch:
+    - [ ] in this [GitHub Action file](https://github.com/meilisearch/meilisearch/blob/main/.github/workflows/test-suite.yml), by changing the `toolchain` field of the `rustfmt` job to the latest available nightly (of the day before or the current day).
+    - [ ] in every [GitHub Action files](https://github.com/meilisearch/meilisearch/blob/main/.github/workflows), by changing all the `dtolnay/rust-toolchain@` references to use the latest stable version.
+    - [ ] in this [`rust-toolchain.toml`](https://github.com/meilisearch/meilisearch/blob/main/rust-toolchain.toml), by changing the `channel` field to the latest stable version.
+    - [ ] in the [Dockerfile](https://github.com/meilisearch/meilisearch/blob/main/Dockerfile), by changing the base image to `rust:<target_rust_version>-alpine<alpine_version>`. Check that the image exists on [Dockerhub](https://hub.docker.com/_/rust/tags?page=1&name=alpine). Also, build and run the image to check everything still works!
+
+⚠️ This issue should be prioritized to avoid any deprecation and vulnerability issues.
+
+The GitHub action dependencies are managed by [Dependabot](https://github.com/meilisearch/meilisearch/blob/main/.github/dependabot.yml), so no need to update them when solving this issue.

.github/workflows/check-valid-milestone.yml

@@ -1,100 +0,0 @@
-name: PR Milestone Check
-
-on:
-  pull_request:
-    types: [opened, reopened, edited, synchronize, milestoned, demilestoned]
-    branches:
-      - "main"
-      - "release-v*.*.*"
-
-jobs:
-  check-milestone:
-    name: Check PR Milestone
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v3
-
-      - name: Validate PR milestone
-        uses: actions/github-script@v7
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            // Get PR number directly from the event payload
-            const prNumber = context.payload.pull_request.number;
-
-            // Get PR details
-            const { data: prData } = await github.rest.pulls.get({
-              owner: 'meilisearch',
-              repo: 'meilisearch',
-              pull_number: prNumber
-            });
-
-            // Get base branch name
-            const baseBranch = prData.base.ref;
-            console.log(`Base branch: ${baseBranch}`);
-
-            // Get PR milestone
-            const prMilestone = prData.milestone;
-            if (!prMilestone) {
-              core.setFailed('PR must have a milestone assigned');
-              return;
-            }
-            console.log(`PR milestone: ${prMilestone.title}`);
-
-            // Validate milestone format: vx.y.z
-            const milestoneRegex = /^v\d+\.\d+\.\d+$/;
-            if (!milestoneRegex.test(prMilestone.title)) {
-              core.setFailed(`Milestone "${prMilestone.title}" does not follow the required format vx.y.z`);
-              return;
-            }
-
-            // For main branch PRs, check if the milestone is the highest one
-            if (baseBranch === 'main') {
-              // Get all milestones
-              const { data: milestones } = await github.rest.issues.listMilestones({
-                owner: 'meilisearch',
-                repo: 'meilisearch',
-                state: 'open',
-                sort: 'due_on',
-                direction: 'desc'
-              });
-
-              // Sort milestones by version number (vx.y.z)
-              const sortedMilestones = milestones
-                .filter(m => milestoneRegex.test(m.title))
-                .sort((a, b) => {
-                  const versionA = a.title.substring(1).split('.').map(Number);
-                  const versionB = b.title.substring(1).split('.').map(Number);
-
-                  // Compare major version
-                  if (versionA[0] !== versionB[0]) return versionB[0] - versionA[0];
-                  // Compare minor version
-                  if (versionA[1] !== versionB[1]) return versionB[1] - versionA[1];
-                  // Compare patch version
-                  return versionB[2] - versionA[2];
-                });
-
-              if (sortedMilestones.length === 0) {
-                core.setFailed('No valid milestones found in the repository. Please create at least one milestone with the format vx.y.z');
-                return;
-              }
-
-              const highestMilestone = sortedMilestones[0];
-              console.log(`Highest milestone: ${highestMilestone.title}`);
-
-              if (prMilestone.title !== highestMilestone.title) {
-                core.setFailed(`PRs targeting the main branch must use the highest milestone (${highestMilestone.title}), but this PR uses ${prMilestone.title}`);
-                return;
-              }
-            } else {
-              // For release branches, the milestone should match the branch version
-              const branchVersion = baseBranch.substring(8); // remove 'release-'
-              if (prMilestone.title !== branchVersion) {
-                core.setFailed(`PRs targeting release branch "${baseBranch}" must use the matching milestone "${branchVersion}", but this PR uses "${prMilestone.title}"`);
-                return;
-              }
-            }
-
-            console.log('PR milestone validation passed!');

.github/workflows/dependency-issue.yml

@@ -15,7 +15,7 @@ jobs:
     steps:
     - uses: actions/checkout@v3
     - name: Download the issue template
-      run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/dependency-issue.md > $ISSUE_TEMPLATE
+      run: curl -s https://raw.githubusercontent.com/meilisearch/meilisearch/main/.github/templates/dependency-issue.md > $ISSUE_TEMPLATE
     - name: Create issue
       run: |
         gh issue create \

.github/workflows/flaky-tests.yml

@@ -3,7 +3,7 @@ name: Look for flaky tests
 on:
   workflow_dispatch:
   schedule:
-    - cron: "0 12 * * FRI" # Every Friday at 12:00PM
+    - cron: '0 4 * * *' # Every day at 4:00AM
 
 jobs:
   flaky:

.github/workflows/milestone-workflow.yml

@@ -1,224 +0,0 @@
-name: Milestone's workflow
-
-# /!\ No git flow are handled here
-
-# For each Milestone created (not opened!), and if the release is NOT a patch release (only the patch changed)
-# - the roadmap issue is created, see https://github.com/meilisearch/engine-team/blob/main/issue-templates/roadmap-issue.md
-# - the changelog issue is created, see https://github.com/meilisearch/engine-team/blob/main/issue-templates/changelog-issue.md
-# - update the ruleset to add the current release version to the list of allowed versions and be able to use the merge queue.
-
-# For each Milestone closed
-# - the `release_version` label is created
-# - this label is applied to all issues/PRs in the Milestone
-
-on:
-  milestone:
-    types: [created, closed]
-
-env:
-  MILESTONE_VERSION: ${{ github.event.milestone.title }}
-  MILESTONE_URL: ${{ github.event.milestone.html_url }}
-  MILESTONE_DUE_ON: ${{ github.event.milestone.due_on }}
-  GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-
-jobs:
-  # -----------------
-  # MILESTONE CREATED
-  # -----------------
-
-  get-release-version:
-    if: github.event.action == 'created'
-    runs-on: ubuntu-latest
-    outputs:
-      is-patch: ${{ steps.check-patch.outputs.is-patch }}
-    steps:
-      - uses: actions/checkout@v3
-      - name: Check if this release is a patch release only
-        id: check-patch
-        run: |
-          echo version: $MILESTONE_VERSION
-          if [[ $MILESTONE_VERSION =~ ^v[0-9]+\.[0-9]+\.0$ ]]; then
-            echo 'This is NOT a patch release'
-            echo "is-patch=false" >> $GITHUB_OUTPUT
-          elif [[ $MILESTONE_VERSION =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
-            echo 'This is a patch release'
-            echo "is-patch=true" >> $GITHUB_OUTPUT
-          else
-            echo "Not a valid format of release, check the Milestone's title."
-            echo 'Should be vX.Y.Z'
-            exit 1
-          fi
-
-  create-roadmap-issue:
-    needs: get-release-version
-    # Create the roadmap issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/roadmap-issue.md > $ISSUE_TEMPLATE
-      - name: Replace all empty occurrences in the templates
-        run: |
-          # Replace all <<version>> occurrences
-          sed -i "s/<<version>>/$MILESTONE_VERSION/g" $ISSUE_TEMPLATE
-
-          # Replace all <<milestone_id>> occurrences
-          milestone_id=$(echo $MILESTONE_URL | cut -d '/' -f 7)
-          sed -i "s/<<milestone_id>>/$milestone_id/g" $ISSUE_TEMPLATE
-
-          # Replace release date if exists
-          if [[ ! -z $MILESTONE_DUE_ON ]]; then
-            date=$(echo $MILESTONE_DUE_ON | cut -d 'T' -f 1)
-            sed -i "s/Release date\: 20XX-XX-XX/Release date\: $date/g" $ISSUE_TEMPLATE
-          fi
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "$MILESTONE_VERSION ROADMAP" \
-            --label 'epic,impacts docs,impacts integrations,impacts cloud' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  create-changelog-issue:
-    needs: get-release-version
-    # Create the changelog issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/changelog-issue.md > $ISSUE_TEMPLATE
-      - name: Replace all empty occurrences in the templates
-        run: |
-          # Replace all <<version>> occurrences
-          sed -i "s/<<version>>/$MILESTONE_VERSION/g" $ISSUE_TEMPLATE
-
-          # Replace all <<milestone_id>> occurrences
-          milestone_id=$(echo $MILESTONE_URL | cut -d '/' -f 7)
-          sed -i "s/<<milestone_id>>/$milestone_id/g" $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Create release changelogs for $MILESTONE_VERSION" \
-            --label 'impacts docs,documentation' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION \
-            --assignee curquiza
-
-  create-update-version-issue:
-    needs: get-release-version
-    # Create the update-version issue even if the release is a patch release
-    if: github.event.action == 'created'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-version-issue.md > $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Update version in Cargo.toml for $MILESTONE_VERSION" \
-            --label 'maintenance' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  create-update-openapi-issue:
-    needs: get-release-version
-    # Create the openAPI issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-openapi-issue.md > $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Update Open API file for $MILESTONE_VERSION" \
-            --label 'maintenance' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  update-ruleset:
-    runs-on: ubuntu-latest
-    if: github.event.action == 'created'
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install jq
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y jq
-      - name: Update ruleset
-        env:
-          # gh api repos/meilisearch/meilisearch/rulesets --jq '.[] | {name: .name, id: .id}'
-          RULESET_ID: 4253297
-          BRANCH_NAME: ${{ github.event.inputs.branch_name }}
-        run: |
-          echo "RULESET_ID: ${{ env.RULESET_ID }}"
-          echo "BRANCH_NAME: ${{ env.BRANCH_NAME }}"
-
-          # Get current ruleset conditions
-          CONDITIONS=$(gh api repos/meilisearch/meilisearch/rulesets/${{ env.RULESET_ID }} --jq '{ conditions: .conditions }')
-
-          # Update the conditions by appending the milestone version
-          UPDATED_CONDITIONS=$(echo $CONDITIONS | jq '.conditions.ref_name.include += ["refs/heads/release-'${{ env.MILESTONE_VERSION }}'"]')
-
-          # Update the ruleset from stdin (-)
-          echo $UPDATED_CONDITIONS |
-            gh api repos/meilisearch/meilisearch/rulesets/${{ env.RULESET_ID }} \
-              --method PUT \
-              -H "Accept: application/vnd.github+json" \
-              -H "X-GitHub-Api-Version: 2022-11-28" \
-              --input -
-
-  # ----------------
-  # MILESTONE CLOSED
-  # ----------------
-
-  create-release-label:
-    if: github.event.action == 'closed'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Create the ${{ env.MILESTONE_VERSION }} label
-        run: |
-          label_description="PRs/issues solved in $MILESTONE_VERSION"
-          if [[ ! -z $MILESTONE_DUE_ON ]]; then
-            date=$(echo $MILESTONE_DUE_ON | cut -d 'T' -f 1)
-            label_description="$label_description released on $date"
-          fi
-
-          gh api repos/meilisearch/meilisearch/labels \
-            --method POST \
-            -H "Accept: application/vnd.github+json" \
-            -f name="$MILESTONE_VERSION" \
-            -f description="$label_description" \
-            -f color='ff5ba3'
-
-  labelize-all-milestone-content:
-    if: github.event.action == 'closed'
-    needs: create-release-label
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Add label ${{ env.MILESTONE_VERSION }} to all PRs in the Milestone
-        run: |
-          prs=$(gh pr list --search milestone:"$MILESTONE_VERSION" --limit 1000 --state all --json number --template '{{range .}}{{tablerow (printf "%v" .number)}}{{end}}')
-          for pr in $prs; do
-              gh pr edit $pr --add-label $MILESTONE_VERSION
-          done
-      - name: Add label ${{ env.MILESTONE_VERSION }} to all issues in the Milestone
-        run: |
-          issues=$(gh issue list --search milestone:"$MILESTONE_VERSION" --limit 1000 --state all --json number --template '{{range .}}{{tablerow (printf "%v" .number)}}{{end}}')
-          for issue in $issues; do
-              gh issue edit $issue --add-label $MILESTONE_VERSION
-          done

.github/workflows/publish-apt-brew-pkg.yml

@@ -32,7 +32,7 @@ jobs:
       - name: Build deb package
         run: cargo deb -p meilisearch -o target/debian/meilisearch.deb
       - name: Upload debian pkg to release
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/debian/meilisearch.deb

.github/workflows/publish-docker-images.yml

@@ -16,6 +16,8 @@ on:
 jobs:
   docker:
     runs-on: docker
+    permissions:
+      id-token: write # This is needed to use Cosign in keyless mode
     steps:
       - uses: actions/checkout@v3
 
@@ -62,6 +64,9 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
+      - name: Install cosign
+        uses: sigstore/cosign-installer@d58896d6a1865668819e1d91763c7751a165e159 # tag=v3.9.2
+
       - name: Login to Docker Hub
         uses: docker/login-action@v3
         with:
@@ -85,6 +90,7 @@ jobs:
 
       - name: Build and push
         uses: docker/build-push-action@v6
+        id: build-and-push
         with:
           push: true
           platforms: linux/amd64,linux/arm64
@@ -94,6 +100,17 @@ jobs:
             COMMIT_DATE=${{ steps.build-metadata.outputs.date }}
             GIT_TAG=${{ github.ref_name }}
 
+      - name: Sign the images with GitHub OIDC Token
+        env:
+          DIGEST: ${{ steps.build-and-push.outputs.digest }}
+          TAGS: ${{ steps.meta.outputs.tags }}
+        run: |
+          images=""
+          for tag in ${TAGS}; do
+            images+="${tag}@${DIGEST} "
+          done
+          cosign sign --yes ${images}
+
       # /!\ Don't touch this without checking with Cloud team
       - name: Send CI information to Cloud team
         # Do not send if nightly build (i.e. 'schedule' or 'workflow_dispatch' event)

.github/workflows/publish-release-assets.yml

@@ -1,4 +1,4 @@
-name: Publish binaries to GitHub release
+name: Publish assets to GitHub release
 
 on:
   workflow_dispatch:
@@ -51,7 +51,7 @@ jobs:
       # No need to upload binaries for dry run (cron)
       - name: Upload binaries to release
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/release/meilisearch
@@ -81,7 +81,7 @@ jobs:
       # No need to upload binaries for dry run (cron)
       - name: Upload binaries to release
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/release/${{ matrix.artifact_name }}
@@ -113,7 +113,7 @@ jobs:
       - name: Upload the binary to release
         # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch
@@ -178,9 +178,34 @@ jobs:
       - name: Upload the binary to release
         # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch
           asset_name: ${{ matrix.asset_name }}
           tag: ${{ github.ref }}
+
+  publish-openapi-file:
+    name: Publish OpenAPI file
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      - name: Setup Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
+      - name: Generate OpenAPI file
+        run: |
+          cd crates/openapi-generator
+          cargo run --release -- --pretty --output ../../meilisearch.json
+      - name: Upload OpenAPI to Release
+        # No need to upload for dry run (cron)
+        if: github.event_name == 'release'
+        uses: svenstaro/upload-release-action@2.11.2
+        with:
+          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+          file: ./meilisearch.json
+          asset_name: meilisearch-openapi.json
+          tag: ${{ github.ref }}

.github/workflows/release-drafter.yml

@@ -0,0 +1,20 @@
+name: Release Drafter
+
+permissions:
+  contents: read
+  pull-requests: write
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  update_release_draft:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: release-drafter/release-drafter@v6
+        with:
+          config-name: release-draft-template.yml
+        env:
+          GITHUB_TOKEN: ${{ secrets.RELEASE_DRAFTER_TOKEN }}

.github/workflows/sdks-tests.yml

@@ -9,7 +9,7 @@ on:
         required: false
         default: nightly
   schedule:
-    - cron: "0 6 * * MON" # Every Monday at 6:00AM
+    - cron: '0 6 * * *' # Every day at 6:00am
 
 env:
   MEILI_MASTER_KEY: 'masterKey'
@@ -114,7 +114,7 @@ jobs:
             dep ensure
           fi
       - name: Run integration tests
-        run: go test -v ./...
+        run: go test --race -v ./integration
 
   meilisearch-java-tests:
     needs: define-docker-image
@@ -344,15 +344,23 @@ jobs:
           MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
         ports:
           - '7700:7700'
+    env:
+      RAILS_VERSION: '7.0'
     steps:
       - uses: actions/checkout@v3
         with:
           repository: meilisearch/meilisearch-rails
-      - name: Set up Ruby 3
+      - name: Install SQLite dependencies
+        run: sudo apt-get update && sudo apt-get install -y libsqlite3-dev
+      - name: Set up Ruby
         uses: ruby/setup-ruby@v1
         with:
           ruby-version: 3
           bundler-cache: true
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.12.0
+        with:
+          mongodb-version: 8.0
       - name: Run tests
         run: bundle exec rspec
 

.github/workflows/test-suite.yml

@@ -3,7 +3,7 @@ name: Test suite
 on:
   workflow_dispatch:
   schedule:
-    # Everyday at 5:00am
+    # Every day at 5:00am
     - cron: "0 5 * * *"
   pull_request:
   merge_group:

.github/workflows/update-cargo-toml-version.yml

@@ -41,5 +41,4 @@ jobs:
             --title "Update version for the next release ($NEW_VERSION) in Cargo.toml" \
             --body '⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.' \
             --label 'skip changelog' \
-            --milestone $NEW_VERSION \
             --base $GITHUB_REF_NAME

.gitignore

@@ -5,18 +5,24 @@
 **/*.json_lines
 **/*.rs.bk
 /*.mdb
-/data.ms
+/*.ms
 /snapshots
 /dumps
 /bench
 /_xtask_benchmark.ms
 /benchmarks
+.DS_Store
 
 # Snapshots
 ## ... large
 *.full.snap
-##  ... unreviewed
+## ... unreviewed
 *.snap.new
+## ... pending
+*.pending-snap
+
+# Tmp files
+.tmp*
 
 # Database snapshot
 crates/meilisearch/db.snapshot

CONTRIBUTING.md

@@ -106,7 +106,19 @@ Run `cargo xtask --help` from the root of the repository to find out what is ava
 #### Update the openAPI file if the API changed
 
 To update the openAPI file in the code, see [sprint_issue.md](https://github.com/meilisearch/meilisearch/blob/main/.github/ISSUE_TEMPLATE/sprint_issue.md#reminders-when-modifying-the-api).
-If you want to update the openAPI file on the [open-api repository](https://github.com/meilisearch/open-api), see [update-openapi-issue.md](https://github.com/meilisearch/engine-team/blob/main/issue-templates/update-openapi-issue.md).
+
+If you want to generate OpenAPI file manually:
+
+With swagger:
+- Starts Meilisearch with the `swagger` feature flag: `cargo run --features swagger`
+- On a browser, open the following URL: http://localhost:7700/scalar
+- Click the « Download openAPI file »
+
+With the internal crate:
+```bash
+cd crates/openapi-generator
+cargo run --release -- --pretty --output meilisearch.json
+```
 
 ### Logging
 
@@ -160,25 +172,37 @@ Some notes on GitHub PRs:
   The draft PRs are recommended when you want to show that you are working on something and make your work visible.
 - The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [GitHub Merge Queues](https://github.blog/news-insights/product-news/github-merge-queue-is-generally-available/) to automatically enforce this requirement without the PR author having to rebase manually.
 
-## Release Process (for internal team only)
-
-Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org/).
-
-### Automation to rebase and Merge the PRs
+## Merging PRs
 
 This project uses GitHub Merge Queues that helps us manage pull requests merging.
 
-### How to Publish a new Release
+Before merging a PR, the maintainer should ensure the following requirements are met
+- Automated tests have been added.
+- If some tests cannot be automated, manual rigorous tests should be applied.
+- ⚠️ If there is an change in the DB: it's mandatory to manually test the `--experimental-dumpless-upgrade` on a DB of the previous Meilisearch minor version (e.g. v1.13 for the v1.14 release).
+- If necessary, the feature have been tested in the Cloud production environment (with [prototypes](./documentation/prototypes.md)) and the Cloud UI is ready.
+- If necessary, the [documentation](https://github.com/meilisearch/documentation) related to the implemented feature in the PR is ready.
+- If necessary, the [integrations](https://github.com/meilisearch/integration-guides) related to the implemented feature in the PR are ready.
 
-The full Meilisearch release process is described in [this guide](https://github.com/meilisearch/engine-team/blob/main/resources/meilisearch-release.md). Please follow it carefully before doing any release.
+## Publish Process (for internal team only)
+
+Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org/).
+
+### How to publish a new release
+
+The full Meilisearch release process is described in [this guide](./documentation/release.md).
 
 ### How to publish a prototype
 
 Depending on the developed feature, you might need to provide a prototyped version of Meilisearch to make it easier to test by the users.
 
 This happens in two steps:
-- [Release the prototype](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#how-to-publish-a-prototype)
-- [Communicate about it](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#communication)
+- [Release the prototype](./documentation/prototypes.md#how-to-publish-a-prototype)
+- [Communicate about it](./documentation/prototypes.md#communication)
+
+### How to implement and publish an experimental feature
+
+Here is our [guidelines and process](./documentation/experimental-features.md) to implement and publish an experimental feature.
 
 ### Release assets
 

Cargo.lock

@@ -444,12 +444,12 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
 
 [[package]]
 name = "arroy"
-version = "0.7.0"
-source = "git+https://github.com/meilisearch/arroy.git?rev=1a21e5c0cbe3e106b05d834076a5b727cf46bca1#1a21e5c0cbe3e106b05d834076a5b727cf46bca1"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08e6111f351d004bd13e95ab540721272136fd3218b39d3ec95a2ea1c4e6a0a6"
 dependencies = [
  "bytemuck",
  "byteorder",
- "crossbeam",
  "enum-iterator",
  "heed",
  "memmap2",
@@ -461,7 +461,6 @@ dependencies = [
  "roaring",
  "tempfile",
  "thiserror 2.0.12",
- "thread_local",
  "tracing",
 ]
 
@@ -581,7 +580,7 @@ source = "git+https://github.com/meilisearch/bbqueue#cbb87cc707b5af415ef203bdaf2
 
 [[package]]
 name = "benchmarks"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "anyhow",
  "bumpalo",
@@ -771,7 +770,7 @@ dependencies = [
 
 [[package]]
 name = "build-info"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "anyhow",
  "time",
@@ -1368,19 +1367,6 @@ dependencies = [
  "itertools 0.10.5",
 ]
 
-[[package]]
-name = "crossbeam"
-version = "0.8.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1137cd7e7fc0fb5d3c5a8678be38ec56e819125d8d7907411fe24ccb943faca8"
-dependencies = [
- "crossbeam-channel",
- "crossbeam-deque",
- "crossbeam-epoch",
- "crossbeam-queue",
- "crossbeam-utils",
-]
-
 [[package]]
 name = "crossbeam-channel"
 version = "0.5.15"
@@ -1788,7 +1774,7 @@ dependencies = [
 
 [[package]]
 name = "dump"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "anyhow",
  "big_s",
@@ -2020,7 +2006,7 @@ checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
 
 [[package]]
 name = "file-store"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "tempfile",
  "thiserror 2.0.12",
@@ -2042,9 +2028,10 @@ dependencies = [
 
 [[package]]
 name = "filter-parser"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "insta",
+ "levenshtein_automata",
  "nom",
  "nom_locate",
  "unescaper",
@@ -2063,7 +2050,7 @@ dependencies = [
 
 [[package]]
 name = "flatten-serde-json"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "criterion",
  "serde_json",
@@ -2208,7 +2195,7 @@ dependencies = [
 
 [[package]]
 name = "fuzzers"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "arbitrary",
  "bumpalo",
@@ -3008,7 +2995,7 @@ dependencies = [
 
 [[package]]
 name = "index-scheduler"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "anyhow",
  "backoff",
@@ -3244,7 +3231,7 @@ dependencies = [
 
 [[package]]
 name = "json-depth-checker"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "criterion",
  "serde_json",
@@ -3738,7 +3725,7 @@ checksum = "490cc448043f947bae3cbee9c203358d62dbee0db12107a74be5c30ccfd09771"
 
 [[package]]
 name = "meili-snap"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "insta",
  "md5",
@@ -3749,7 +3736,7 @@ dependencies = [
 
 [[package]]
 name = "meilisearch"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "actix-cors",
  "actix-http",
@@ -3789,6 +3776,7 @@ dependencies = [
  "meili-snap",
  "meilisearch-auth",
  "meilisearch-types",
+ "memmap2",
  "mimalloc",
  "mime",
  "mopa-maintained",
@@ -3844,7 +3832,7 @@ dependencies = [
 
 [[package]]
 name = "meilisearch-auth"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "base64 0.22.1",
  "enum-iterator",
@@ -3863,7 +3851,7 @@ dependencies = [
 
 [[package]]
 name = "meilisearch-types"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "actix-web",
  "anyhow",
@@ -3898,7 +3886,7 @@ dependencies = [
 
 [[package]]
 name = "meilitool"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "anyhow",
  "clap",
@@ -3922,9 +3910,9 @@ checksum = "32a282da65faaf38286cf3be983213fcf1d2e2a58700e808f83f4ea9a4804bc0"
 
 [[package]]
 name = "memmap2"
-version = "0.9.5"
+version = "0.9.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd3f7eed9d3848f8b98834af67102b720745c4ec028fcd0aa0239277e7de374f"
+checksum = "483758ad303d734cec05e5c12b41d7e93e6a6390c5e9dae6bdeb7c1259012d28"
 dependencies = [
  "libc",
  "stable_deref_trait",
@@ -3932,7 +3920,7 @@ dependencies = [
 
 [[package]]
 name = "milli"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "allocator-api2 0.3.0",
  "arroy",
@@ -4351,6 +4339,17 @@ version = "11.1.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
 
+[[package]]
+name = "openapi-generator"
+version = "0.1.0"
+dependencies = [
+ "anyhow",
+ "clap",
+ "meilisearch",
+ "serde_json",
+ "utoipa",
+]
+
 [[package]]
 name = "openssl-probe"
 version = "0.1.6"
@@ -4484,7 +4483,7 @@ checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e"
 
 [[package]]
 name = "permissive-json-pointer"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "big_s",
  "serde_json",
@@ -7272,7 +7271,7 @@ dependencies = [
 
 [[package]]
 name = "xtask"
-version = "1.16.0"
+version = "1.18.0"
 dependencies = [
  "anyhow",
  "build-info",

Cargo.toml

@@ -19,10 +19,11 @@ members = [
     "crates/tracing-trace",
     "crates/xtask",
     "crates/build-info",
+    "crates/openapi-generator",
 ]
 
 [workspace.package]
-version = "1.16.0"
+version = "1.18.0"
 authors = [
     "Quentin de Quelen <quentin@dequelen.me>",
     "Clément Renault <clement@meilisearch.com>",

README.md

@@ -119,6 +119,6 @@ Meilisearch is, and will always be, open-source! If you want to contribute to th
 
 Meilisearch releases and their associated binaries are available on the project's [releases page](https://github.com/meilisearch/meilisearch/releases).
 
-The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](https://github.com/meilisearch/engine-team/blob/main/resources/versioning-policy.md).
+The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](./documentation/versioning-policy.md).
 
 Differently from the binaries, crates in this repository are not currently available on [crates.io](https://crates.io/) and do not follow [SemVer conventions](https://semver.org).

crates/benchmarks/Cargo.toml

@@ -14,7 +14,7 @@ license.workspace = true
 anyhow = "1.0.98"
 bumpalo = "3.18.1"
 csv = "1.3.1"
-memmap2 = "0.9.5"
+memmap2 = "0.9.7"
 milli = { path = "../milli" }
 mimalloc = { version = "0.1.47", default-features = false }
 serde_json = { version = "1.0.140", features = ["preserve_order"] }
@@ -51,3 +51,11 @@ harness = false
 [[bench]]
 name = "indexing"
 harness = false
+
+[[bench]]
+name = "sort"
+harness = false
+
+[[bench]]
+name = "filter_starts_with"
+harness = false

crates/benchmarks/benches/filter_starts_with.rs

@@ -0,0 +1,66 @@
+mod datasets_paths;
+mod utils;
+
+use criterion::{criterion_group, criterion_main};
+use milli::update::Settings;
+use milli::FilterableAttributesRule;
+use utils::Conf;
+
+#[cfg(not(windows))]
+#[global_allocator]
+static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
+
+fn base_conf(builder: &mut Settings) {
+    let displayed_fields = ["geonameid", "name"].iter().map(|s| s.to_string()).collect();
+    builder.set_displayed_fields(displayed_fields);
+
+    let filterable_fields =
+        ["name"].iter().map(|s| FilterableAttributesRule::Field(s.to_string())).collect();
+    builder.set_filterable_fields(filterable_fields);
+}
+
+#[rustfmt::skip]
+const BASE_CONF: Conf = Conf {
+    dataset: datasets_paths::SMOL_ALL_COUNTRIES,
+    dataset_format: "jsonl",
+    queries: &[
+        "",
+    ],
+    configure: base_conf,
+    primary_key: Some("geonameid"),
+    ..Conf::BASE
+};
+
+fn filter_starts_with(c: &mut criterion::Criterion) {
+    #[rustfmt::skip]
+    let confs = &[
+        utils::Conf {
+            group_name: "1 letter",
+            filter: Some("name STARTS WITH e"),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "2 letters",
+            filter: Some("name STARTS WITH es"),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "3 letters",
+            filter: Some("name STARTS WITH est"),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "6 letters",
+            filter: Some("name STARTS WITH estoni"),
+            ..BASE_CONF
+        }
+    ];
+
+    utils::run_benches(c, confs);
+}
+
+criterion_group!(benches, filter_starts_with);
+criterion_main!(benches);

crates/benchmarks/benches/sort.rs

@@ -0,0 +1,114 @@
+//! This benchmark module is used to compare the performance of sorting documents in /search VS /documents
+//!
+//! The tests/benchmarks were designed in the context of a query returning only 20 documents.
+
+mod datasets_paths;
+mod utils;
+
+use criterion::{criterion_group, criterion_main};
+use milli::update::Settings;
+use utils::Conf;
+
+#[cfg(not(windows))]
+#[global_allocator]
+static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
+
+fn base_conf(builder: &mut Settings) {
+    let displayed_fields =
+        ["geonameid", "name", "asciiname", "alternatenames", "_geo", "population"]
+            .iter()
+            .map(|s| s.to_string())
+            .collect();
+    builder.set_displayed_fields(displayed_fields);
+
+    let sortable_fields =
+        ["_geo", "name", "population", "elevation", "timezone", "modification-date"]
+            .iter()
+            .map(|s| s.to_string())
+            .collect();
+    builder.set_sortable_fields(sortable_fields);
+}
+
+#[rustfmt::skip]
+const BASE_CONF: Conf = Conf {
+    dataset: datasets_paths::SMOL_ALL_COUNTRIES,
+    dataset_format: "jsonl",
+    configure: base_conf,
+    primary_key: Some("geonameid"),
+    queries: &[""],
+    offsets: &[
+        Some((0, 20)), // The most common query in the real world
+        Some((0, 500)), // A query that ranges over many documents
+        Some((980, 20)), // The worst query that could happen in the real world
+        Some((800_000, 20)) // The worst query
+    ],
+    get_documents: true,
+    ..Conf::BASE
+};
+
+fn bench_sort(c: &mut criterion::Criterion) {
+    #[rustfmt::skip]
+    let confs = &[
+        utils::Conf {
+            group_name: "without sort",
+            sort: None,
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many different values",
+            sort: Some(vec!["name:asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many similar values",
+            sort: Some(vec!["timezone:desc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many similar then different values",
+            sort: Some(vec!["timezone:desc", "name:asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many different then similar values",
+            sort: Some(vec!["timezone:desc", "name:asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "geo sort",
+            sample_size: Some(10),
+            sort: Some(vec!["_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many similar values then geo sort",
+            sample_size: Some(50),
+            sort: Some(vec!["timezone:desc", "_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many different values then geo sort",
+            sample_size: Some(50),
+            sort: Some(vec!["name:desc", "_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many fields",
+            sort: Some(vec!["population:asc", "name:asc", "elevation:asc", "timezone:asc"]),
+            ..BASE_CONF
+        },
+    ];
+
+    utils::run_benches(c, confs);
+}
+
+criterion_group!(benches, bench_sort);
+criterion_main!(benches);

crates/benchmarks/benches/utils.rs

@@ -9,6 +9,7 @@ use anyhow::Context;
 use bumpalo::Bump;
 use criterion::BenchmarkId;
 use memmap2::Mmap;
+use milli::documents::sort::recursive_sort;
 use milli::heed::EnvOpenOptions;
 use milli::progress::Progress;
 use milli::update::new::indexer;
@@ -35,6 +36,12 @@ pub struct Conf<'a> {
     pub configure: fn(&mut Settings),
     pub filter: Option<&'a str>,
     pub sort: Option<Vec<&'a str>>,
+    /// set to skip documents (offset, limit)
+    pub offsets: &'a [Option<(usize, usize)>],
+    /// enable if you want to bench getting documents without querying
+    pub get_documents: bool,
+    /// configure the benchmark sample size
+    pub sample_size: Option<usize>,
     /// enable or disable the optional words on the query
     pub optional_words: bool,
     /// primary key, if there is None we'll auto-generate docids for every documents
@@ -52,6 +59,9 @@ impl Conf<'_> {
         configure: |_| (),
         filter: None,
         sort: None,
+        offsets: &[None],
+        get_documents: false,
+        sample_size: None,
         optional_words: true,
         primary_key: None,
     };
@@ -145,25 +155,79 @@ pub fn run_benches(c: &mut criterion::Criterion, confs: &[Conf]) {
         let file_name = Path::new(conf.dataset).file_name().and_then(|f| f.to_str()).unwrap();
         let name = format!("{}: {}", file_name, conf.group_name);
         let mut group = c.benchmark_group(&name);
+        if let Some(sample_size) = conf.sample_size {
+            group.sample_size(sample_size);
+        }
 
         for &query in conf.queries {
-            group.bench_with_input(BenchmarkId::from_parameter(query), &query, |b, &query| {
-                b.iter(|| {
-                    let rtxn = index.read_txn().unwrap();
-                    let mut search = index.search(&rtxn);
-                    search.query(query).terms_matching_strategy(TermsMatchingStrategy::default());
-                    if let Some(filter) = conf.filter {
-                        let filter = Filter::from_str(filter).unwrap().unwrap();
-                        search.filter(filter);
-                    }
-                    if let Some(sort) = &conf.sort {
-                        let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
-                        search.sort_criteria(sort);
-                    }
-                    let _ids = search.execute().unwrap();
-                });
-            });
+            for offset in conf.offsets {
+                let parameter = match offset {
+                    None => query.to_string(),
+                    Some((offset, limit)) => format!("{query}[{offset}:{limit}]"),
+                };
+                group.bench_with_input(
+                    BenchmarkId::from_parameter(parameter),
+                    &query,
+                    |b, &query| {
+                        b.iter(|| {
+                            let rtxn = index.read_txn().unwrap();
+                            let mut search = index.search(&rtxn);
+                            search
+                                .query(query)
+                                .terms_matching_strategy(TermsMatchingStrategy::default());
+                            if let Some(filter) = conf.filter {
+                                let filter = Filter::from_str(filter).unwrap().unwrap();
+                                search.filter(filter);
+                            }
+                            if let Some(sort) = &conf.sort {
+                                let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
+                                search.sort_criteria(sort);
+                            }
+                            if let Some((offset, limit)) = offset {
+                                search.offset(*offset).limit(*limit);
+                            }
+
+                            let _ids = search.execute().unwrap();
+                        });
+                    },
+                );
+            }
         }
+
+        if conf.get_documents {
+            for offset in conf.offsets {
+                let parameter = match offset {
+                    None => String::from("get_documents"),
+                    Some((offset, limit)) => format!("get_documents[{offset}:{limit}]"),
+                };
+                group.bench_with_input(BenchmarkId::from_parameter(parameter), &(), |b, &()| {
+                    b.iter(|| {
+                        let rtxn = index.read_txn().unwrap();
+                        if let Some(sort) = &conf.sort {
+                            let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
+                            let all_docs = index.documents_ids(&rtxn).unwrap();
+                            let facet_sort =
+                                recursive_sort(&index, &rtxn, sort, &all_docs).unwrap();
+                            let iter = facet_sort.iter().unwrap();
+                            if let Some((offset, limit)) = offset {
+                                let _results = iter.skip(*offset).take(*limit).collect::<Vec<_>>();
+                            } else {
+                                let _results = iter.collect::<Vec<_>>();
+                            }
+                        } else {
+                            let all_docs = index.documents_ids(&rtxn).unwrap();
+                            if let Some((offset, limit)) = offset {
+                                let _results =
+                                    all_docs.iter().skip(*offset).take(*limit).collect::<Vec<_>>();
+                            } else {
+                                let _results = all_docs.iter().collect::<Vec<_>>();
+                            }
+                        }
+                    });
+                });
+            }
+        }
+
         group.finish();
 
         index.prepare_for_closing().wait();

crates/dump/src/lib.rs

@@ -129,6 +129,7 @@ pub enum KindDump {
     },
     IndexUpdate {
         primary_key: Option<String>,
+        uid: Option<String>,
     },
     IndexSwap {
         swaps: Vec<IndexSwap>,
@@ -210,8 +211,8 @@ impl From<KindWithContent> for KindDump {
             KindWithContent::IndexCreation { primary_key, .. } => {
                 KindDump::IndexCreation { primary_key }
             }
-            KindWithContent::IndexUpdate { primary_key, .. } => {
-                KindDump::IndexUpdate { primary_key }
+            KindWithContent::IndexUpdate { primary_key, new_index_uid: uid, .. } => {
+                KindDump::IndexUpdate { primary_key, uid }
             }
             KindWithContent::IndexSwap { swaps } => KindDump::IndexSwap { swaps },
             KindWithContent::TaskCancelation { query, tasks } => {

crates/dump/src/reader/compat/v1_to_v2.rs

@@ -1,3 +1,4 @@
+use std::fs::File;
 use std::str::FromStr;
 
 use super::v2_to_v3::CompatV2ToV3;
@@ -94,6 +95,10 @@ impl CompatIndexV1ToV2 {
         self.from.documents().map(|it| Box::new(it) as Box<dyn Iterator<Item = _>>)
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.from.documents_file()
+    }
+
     pub fn settings(&mut self) -> Result<v2::settings::Settings<v2::settings::Checked>> {
         Ok(v2::settings::Settings::<v2::settings::Unchecked>::from(self.from.settings()?).check())
     }

crates/dump/src/reader/compat/v2_to_v3.rs

@@ -1,3 +1,4 @@
+use std::fs::File;
 use std::str::FromStr;
 
 use time::OffsetDateTime;
@@ -122,6 +123,13 @@ impl CompatIndexV2ToV3 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV2ToV3::V2(v2) => v2.documents_file(),
+            CompatIndexV2ToV3::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v3::Settings<v3::Checked>> {
         let settings = match self {
             CompatIndexV2ToV3::V2(from) => from.settings()?,

crates/dump/src/reader/compat/v3_to_v4.rs

@@ -1,3 +1,5 @@
+use std::fs::File;
+
 use super::v2_to_v3::{CompatIndexV2ToV3, CompatV2ToV3};
 use super::v4_to_v5::CompatV4ToV5;
 use crate::reader::{v3, v4, UpdateFile};
@@ -252,6 +254,13 @@ impl CompatIndexV3ToV4 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV3ToV4::V3(v3) => v3.documents_file(),
+            CompatIndexV3ToV4::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v4::Settings<v4::Checked>> {
         Ok(match self {
             CompatIndexV3ToV4::V3(v3) => {

crates/dump/src/reader/compat/v4_to_v5.rs

@@ -1,3 +1,5 @@
+use std::fs::File;
+
 use super::v3_to_v4::{CompatIndexV3ToV4, CompatV3ToV4};
 use super::v5_to_v6::CompatV5ToV6;
 use crate::reader::{v4, v5, Document};
@@ -241,6 +243,13 @@ impl CompatIndexV4ToV5 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV4ToV5::V4(v4) => v4.documents_file(),
+            CompatIndexV4ToV5::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v5::Settings<v5::Checked>> {
         match self {
             CompatIndexV4ToV5::V4(v4) => Ok(v5::Settings::from(v4.settings()?).check()),

crates/dump/src/reader/compat/v5_to_v6.rs

@@ -1,3 +1,4 @@
+use std::fs::File;
 use std::num::NonZeroUsize;
 use std::str::FromStr;
 
@@ -84,7 +85,7 @@ impl CompatV5ToV6 {
                             v6::Kind::IndexCreation { primary_key }
                         }
                         v5::tasks::TaskContent::IndexUpdate { primary_key, .. } => {
-                            v6::Kind::IndexUpdate { primary_key }
+                            v6::Kind::IndexUpdate { primary_key, uid: None }
                         }
                         v5::tasks::TaskContent::IndexDeletion { .. } => v6::Kind::IndexDeletion,
                         v5::tasks::TaskContent::DocumentAddition {
@@ -139,9 +140,11 @@ impl CompatV5ToV6 {
                         v5::Details::Settings { settings } => {
                             v6::Details::SettingsUpdate { settings: Box::new(settings.into()) }
                         }
-                        v5::Details::IndexInfo { primary_key } => {
-                            v6::Details::IndexInfo { primary_key }
-                        }
+                        v5::Details::IndexInfo { primary_key } => v6::Details::IndexInfo {
+                            primary_key,
+                            new_index_uid: None,
+                            old_index_uid: None,
+                        },
                         v5::Details::DocumentDeletion {
                             received_document_ids,
                             deleted_documents,
@@ -201,6 +204,10 @@ impl CompatV5ToV6 {
     pub fn network(&self) -> Result<Option<&v6::Network>> {
         Ok(None)
     }
+
+    pub fn webhooks(&self) -> Option<&v6::Webhooks> {
+        None
+    }
 }
 
 pub enum CompatIndexV5ToV6 {
@@ -243,6 +250,13 @@ impl CompatIndexV5ToV6 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV5ToV6::V5(v5) => v5.documents_file(),
+            CompatIndexV5ToV6::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v6::Settings<v6::Checked>> {
         match self {
             CompatIndexV5ToV6::V5(v5) => Ok(v6::Settings::from(v5.settings()?).check()),

crates/dump/src/reader/mod.rs

@@ -138,6 +138,13 @@ impl DumpReader {
             DumpReader::Compat(compat) => compat.network(),
         }
     }
+
+    pub fn webhooks(&self) -> Option<&v6::Webhooks> {
+        match self {
+            DumpReader::Current(current) => current.webhooks(),
+            DumpReader::Compat(compat) => compat.webhooks(),
+        }
+    }
 }
 
 impl From<V6Reader> for DumpReader {
@@ -192,6 +199,14 @@ impl DumpIndexReader {
         }
     }
 
+    /// A reference to a file in the NDJSON format containing all the documents of the index
+    pub fn documents_file(&self) -> &File {
+        match self {
+            DumpIndexReader::Current(v6) => v6.documents_file(),
+            DumpIndexReader::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v6::Settings<v6::Checked>> {
         match self {
             DumpIndexReader::Current(v6) => v6.settings(),
@@ -357,6 +372,7 @@ pub(crate) mod test {
 
         assert_eq!(dump.features().unwrap().unwrap(), RuntimeTogglableFeatures::default());
         assert_eq!(dump.network().unwrap(), None);
+        assert_eq!(dump.webhooks(), None);
     }
 
     #[test]
@@ -427,6 +443,43 @@ pub(crate) mod test {
         insta::assert_snapshot!(network.remotes.get("ms-2").as_ref().unwrap().search_api_key.as_ref().unwrap(), @"foo");
     }
 
+    #[test]
+    fn import_dump_v6_webhooks() {
+        let dump = File::open("tests/assets/v6-with-webhooks.dump").unwrap();
+        let dump = DumpReader::open(dump).unwrap();
+
+        // top level infos
+        insta::assert_snapshot!(dump.date().unwrap(), @"2025-07-31 9:21:30.479544 +00:00:00");
+        insta::assert_debug_snapshot!(dump.instance_uid().unwrap(), @r"
+        Some(
+            cb887dcc-34b3-48d1-addd-9815ae721a81,
+        )
+        ");
+
+        // webhooks
+        let webhooks = dump.webhooks().unwrap();
+        insta::assert_json_snapshot!(webhooks, @r#"
+        {
+          "webhooks": {
+            "627ea538-733d-4545-8d2d-03526eb381ce": {
+              "url": "https://example.com/authorization-less",
+              "headers": {}
+            },
+            "771b0a28-ef28-4082-b984-536f82958c65": {
+              "url": "https://example.com/hook",
+              "headers": {
+                "authorization": "TOKEN"
+              }
+            },
+            "f3583083-f8a7-4cbf-a5e7-fb3f1e28a7e9": {
+              "url": "https://third.com",
+              "headers": {}
+            }
+          }
+        }
+        "#);
+    }
+
     #[test]
     fn import_dump_v5() {
         let dump = File::open("tests/assets/v5.dump").unwrap();

crates/dump/src/reader/v1/mod.rs

@@ -72,6 +72,10 @@ impl V1IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<self::settings::Settings> {
         Ok(serde_json::from_reader(&mut self.settings)?)
     }

crates/dump/src/reader/v2/mod.rs

@@ -203,6 +203,10 @@ impl V2IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v3/mod.rs

@@ -215,6 +215,10 @@ impl V3IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v4/mod.rs

@@ -210,6 +210,10 @@ impl V4IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v5/mod.rs

@@ -247,6 +247,10 @@ impl V5IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v6/mod.rs

@@ -25,6 +25,7 @@ pub type Key = meilisearch_types::keys::Key;
 pub type ChatCompletionSettings = meilisearch_types::features::ChatCompletionSettings;
 pub type RuntimeTogglableFeatures = meilisearch_types::features::RuntimeTogglableFeatures;
 pub type Network = meilisearch_types::features::Network;
+pub type Webhooks = meilisearch_types::webhooks::WebhooksDumpView;
 
 // ===== Other types to clarify the code of the compat module
 // everything related to the tasks
@@ -59,6 +60,7 @@ pub struct V6Reader {
     keys: BufReader<File>,
     features: Option<RuntimeTogglableFeatures>,
     network: Option<Network>,
+    webhooks: Option<Webhooks>,
 }
 
 impl V6Reader {
@@ -93,8 +95,8 @@ impl V6Reader {
             Err(e) => return Err(e.into()),
         };
 
-        let network_file = match fs::read(dump.path().join("network.json")) {
-            Ok(network_file) => Some(network_file),
+        let network = match fs::read(dump.path().join("network.json")) {
+            Ok(network_file) => Some(serde_json::from_reader(&*network_file)?),
             Err(error) => match error.kind() {
                 // Allows the file to be missing, this will only result in all experimental features disabled.
                 ErrorKind::NotFound => {
@@ -104,10 +106,16 @@ impl V6Reader {
                 _ => return Err(error.into()),
             },
         };
-        let network = if let Some(network_file) = network_file {
-            Some(serde_json::from_reader(&*network_file)?)
-        } else {
-            None
+
+        let webhooks = match fs::read(dump.path().join("webhooks.json")) {
+            Ok(webhooks_file) => Some(serde_json::from_reader(&*webhooks_file)?),
+            Err(error) => match error.kind() {
+                ErrorKind::NotFound => {
+                    debug!("`webhooks.json` not found in dump");
+                    None
+                }
+                _ => return Err(error.into()),
+            },
         };
 
         Ok(V6Reader {
@@ -119,6 +127,7 @@ impl V6Reader {
             features,
             network,
             dump,
+            webhooks,
         })
     }
 
@@ -229,6 +238,10 @@ impl V6Reader {
     pub fn network(&self) -> Option<&Network> {
         self.network.as_ref()
     }
+
+    pub fn webhooks(&self) -> Option<&Webhooks> {
+        self.webhooks.as_ref()
+    }
 }
 
 pub struct UpdateFile {
@@ -284,6 +297,10 @@ impl V6IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         let mut settings: Settings<Unchecked> = serde_json::from_reader(&mut self.settings)?;
         patch_embedders(&mut settings);

crates/dump/src/writer.rs

@@ -8,6 +8,7 @@ use meilisearch_types::batches::Batch;
 use meilisearch_types::features::{ChatCompletionSettings, Network, RuntimeTogglableFeatures};
 use meilisearch_types::keys::Key;
 use meilisearch_types::settings::{Checked, Settings};
+use meilisearch_types::webhooks::WebhooksDumpView;
 use serde_json::{Map, Value};
 use tempfile::TempDir;
 use time::OffsetDateTime;
@@ -74,6 +75,13 @@ impl DumpWriter {
         Ok(std::fs::write(self.dir.path().join("network.json"), serde_json::to_string(&network)?)?)
     }
 
+    pub fn create_webhooks(&self, webhooks: WebhooksDumpView) -> Result<()> {
+        Ok(std::fs::write(
+            self.dir.path().join("webhooks.json"),
+            serde_json::to_string(&webhooks)?,
+        )?)
+    }
+
     pub fn persist_to(self, mut writer: impl Write) -> Result<()> {
         let gz_encoder = GzEncoder::new(&mut writer, Compression::default());
         let mut tar_encoder = tar::Builder::new(gz_encoder);

crates/filter-parser/Cargo.toml

@@ -15,6 +15,7 @@ license.workspace = true
 nom = "7.1.3"
 nom_locate = "4.2.0"
 unescaper = "0.1.6"
+levenshtein_automata = { version = "0.2.1", features = ["fst_automaton"] }
 
 [dev-dependencies]
 # fixed version due to format breakages in v1.40

crates/filter-parser/src/condition.rs

@@ -7,11 +7,22 @@
 
 use nom::branch::alt;
 use nom::bytes::complete::tag;
+use nom::character::complete::char;
+use nom::character::complete::multispace0;
 use nom::character::complete::multispace1;
 use nom::combinator::cut;
+use nom::combinator::map;
+use nom::combinator::value;
+use nom::sequence::preceded;
 use nom::sequence::{terminated, tuple};
 use Condition::*;
 
+use crate::error::IResultExt;
+use crate::value::parse_vector_value;
+use crate::value::parse_vector_value_cut;
+use crate::Error;
+use crate::ErrorKind;
+use crate::VectorFilter;
 use crate::{parse_value, FilterCondition, IResult, Span, Token};
 
 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -113,6 +124,83 @@ pub fn parse_not_exists(input: Span) -> IResult<FilterCondition> {
     Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Exists }))))
 }
 
+fn parse_vectors(input: Span) -> IResult<(Token, Option<Token>, VectorFilter<'_>)> {
+    let (input, _) = multispace0(input)?;
+    let (input, fid) = tag("_vectors")(input)?;
+
+    if let Ok((input, _)) = multispace1::<_, crate::Error>(input) {
+        return Ok((input, (Token::from(fid), None, VectorFilter::None)));
+    }
+
+    let (input, _) = char('.')(input)?;
+
+    // From this point, we are certain this is a vector filter, so our errors must be final.
+    // We could use nom's `cut` but it's better to be explicit about the errors
+
+    if let Ok((_, space)) = tag::<_, _, ()>(" ")(input) {
+        return Err(crate::Error::failure_from_kind(space, ErrorKind::VectorFilterMissingEmbedder));
+    }
+
+    let (input, embedder_name) =
+        parse_vector_value_cut(input, ErrorKind::VectorFilterInvalidEmbedder)?;
+
+    let (input, filter) = alt((
+        map(
+            preceded(tag(".fragments"), |input| {
+                let (input, _) = tag(".")(input).map_cut(ErrorKind::VectorFilterMissingFragment)?;
+                parse_vector_value_cut(input, ErrorKind::VectorFilterInvalidFragment)
+            }),
+            VectorFilter::Fragment,
+        ),
+        value(VectorFilter::UserProvided, tag(".userProvided")),
+        value(VectorFilter::DocumentTemplate, tag(".documentTemplate")),
+        value(VectorFilter::Regenerate, tag(".regenerate")),
+        value(VectorFilter::None, nom::combinator::success("")),
+    ))(input)?;
+
+    if let Ok((input, point)) = tag::<_, _, ()>(".")(input) {
+        let opt_value = parse_vector_value(input).ok().map(|(_, v)| v);
+        let value =
+            opt_value.as_ref().map(|v| v.value().to_owned()).unwrap_or_else(|| point.to_string());
+        let context = opt_value.map(|v| v.original_span()).unwrap_or(point);
+        let previous_kind = match filter {
+            VectorFilter::Fragment(_) => Some("fragments"),
+            VectorFilter::DocumentTemplate => Some("documentTemplate"),
+            VectorFilter::UserProvided => Some("userProvided"),
+            VectorFilter::Regenerate => Some("regenerate"),
+            VectorFilter::None => None,
+        };
+        return Err(Error::failure_from_kind(
+            context,
+            ErrorKind::VectorFilterUnknownSuffix(previous_kind, value),
+        ));
+    }
+
+    let (input, _) = multispace1(input).map_cut(ErrorKind::VectorFilterLeftover)?;
+
+    Ok((input, (Token::from(fid), Some(embedder_name), filter)))
+}
+
+/// vectors_exists          = vectors ("EXISTS" | ("NOT" WS+ "EXISTS"))
+pub fn parse_vectors_exists(input: Span) -> IResult<FilterCondition> {
+    let (input, (fid, embedder, filter)) = parse_vectors(input)?;
+
+    // Try parsing "EXISTS" first
+    if let Ok((input, _)) = tag::<_, _, ()>("EXISTS")(input) {
+        return Ok((input, FilterCondition::VectorExists { fid, embedder, filter }));
+    }
+
+    // Try parsing "NOT EXISTS"
+    if let Ok((input, _)) = tuple::<_, _, (), _>((tag("NOT"), multispace1, tag("EXISTS")))(input) {
+        return Ok((
+            input,
+            FilterCondition::Not(Box::new(FilterCondition::VectorExists { fid, embedder, filter })),
+        ));
+    }
+
+    Err(crate::Error::failure_from_kind(input, ErrorKind::VectorFilterOperation))
+}
+
 /// contains        = value "CONTAINS" value
 pub fn parse_contains(input: Span) -> IResult<FilterCondition> {
     let (input, (fid, contains, value)) =

crates/filter-parser/src/error.rs

@@ -42,6 +42,23 @@ pub fn cut_with_err<'a, O>(
     }
 }
 
+pub trait IResultExt<'a> {
+    fn map_cut(self, kind: ErrorKind<'a>) -> Self;
+}
+
+impl<'a, T> IResultExt<'a> for IResult<'a, T> {
+    fn map_cut(self, kind: ErrorKind<'a>) -> Self {
+        self.map_err(move |e: nom::Err<Error<'a>>| {
+            let input = match e {
+                nom::Err::Incomplete(_) => return e,
+                nom::Err::Error(e) => *e.context(),
+                nom::Err::Failure(e) => *e.context(),
+            };
+            Error::failure_from_kind(input, kind)
+        })
+    }
+}
+
 #[derive(Debug)]
 pub struct Error<'a> {
     context: Span<'a>,
@@ -61,6 +78,14 @@ pub enum ErrorKind<'a> {
     GeoBoundingBox,
     MisusedGeoRadius,
     MisusedGeoBoundingBox,
+    VectorFilterLeftover,
+    VectorFilterInvalidQuotes,
+    VectorFilterMissingEmbedder,
+    VectorFilterInvalidEmbedder,
+    VectorFilterMissingFragment,
+    VectorFilterInvalidFragment,
+    VectorFilterUnknownSuffix(Option<&'static str>, String),
+    VectorFilterOperation,
     InvalidPrimary,
     InvalidEscapedNumber,
     ExpectedEof,
@@ -91,6 +116,10 @@ impl<'a> Error<'a> {
         Self { context, kind }
     }
 
+    pub fn failure_from_kind(context: Span<'a>, kind: ErrorKind<'a>) -> nom::Err<Self> {
+        nom::Err::Failure(Self::new_from_kind(context, kind))
+    }
+
     pub fn new_from_external(context: Span<'a>, error: impl std::error::Error) -> Self {
         Self::new_from_kind(context, ErrorKind::External(error.to_string()))
     }
@@ -128,6 +157,20 @@ impl Display for Error<'_> {
         // first line being the diagnostic and the second line being the incriminated filter.
         let escaped_input = input.escape_debug();
 
+        fn key_suggestion<'a>(key: &str, keys: &[&'a str]) -> Option<&'a str> {
+            let typos =
+                levenshtein_automata::LevenshteinAutomatonBuilder::new(2, true).build_dfa(key);
+            for key in keys.iter() {
+                match typos.eval(key) {
+                    levenshtein_automata::Distance::Exact(_) => {
+                        return Some(key);
+                    }
+                    levenshtein_automata::Distance::AtLeast(_) => continue,
+                }
+            }
+            None
+        }
+
         match &self.kind {
             ErrorKind::ExpectedValue(_) if input.trim().is_empty() => {
                 writeln!(f, "Was expecting a value but instead got nothing.")?
@@ -169,6 +212,44 @@ impl Display for Error<'_> {
             ErrorKind::MisusedGeoBoundingBox => {
                 writeln!(f, "The `_geoBoundingBox` filter is an operation and can't be used as a value.")?
             }
+            ErrorKind::VectorFilterLeftover => {
+                writeln!(f, "The vector filter has leftover tokens.")?
+            }
+            ErrorKind::VectorFilterUnknownSuffix(_, value) if value.as_str() == "." => {
+                writeln!(f, "Was expecting one of `.fragments`, `.userProvided`, `.documentTemplate`, `.regenerate` or nothing, but instead found a point without a valid value.")?;
+            }
+            ErrorKind::VectorFilterUnknownSuffix(None, value) if ["fragments", "userProvided", "documentTemplate", "regenerate"].contains(&value.as_str()) => {
+                // This will happen with "_vectors.rest.\"userProvided\"" for instance
+                writeln!(f, "Was expecting this part to be unquoted.")?
+            }
+            ErrorKind::VectorFilterUnknownSuffix(None, value) => {
+                if let Some(suggestion) = key_suggestion(value, &["fragments", "userProvided", "documentTemplate", "regenerate"]) {
+                    writeln!(f, "Was expecting one of `fragments`, `userProvided`, `documentTemplate`, `regenerate` or nothing, but instead found `{value}`. Did you mean `{suggestion}`?")?;
+                } else {
+                    writeln!(f, "Was expecting one of `fragments`, `userProvided`, `documentTemplate`, `regenerate` or nothing, but instead found `{value}`.")?;
+                }
+            }
+            ErrorKind::VectorFilterUnknownSuffix(Some(previous_filter_kind), value) => {
+                writeln!(f, "Vector filter can only accept one of `fragments`, `userProvided`, `documentTemplate` or `regenerate`, but found both `{previous_filter_kind}` and `{value}`.")?
+            },
+            ErrorKind::VectorFilterInvalidFragment => {
+                writeln!(f, "The vector filter's fragment name is invalid.")?
+            }
+            ErrorKind::VectorFilterMissingFragment => {
+                writeln!(f, "The vector filter is missing a fragment name.")?
+            }
+            ErrorKind::VectorFilterMissingEmbedder => {
+                writeln!(f, "Was expecting embedder name but found nothing.")?
+            }
+            ErrorKind::VectorFilterInvalidEmbedder => {
+                writeln!(f, "The vector filter's embedder name is invalid.")?
+            }
+            ErrorKind::VectorFilterOperation => {
+                writeln!(f, "Was expecting an operation like `EXISTS` or `NOT EXISTS` after the vector filter.")?
+            }
+            ErrorKind::VectorFilterInvalidQuotes => {
+                writeln!(f, "The quotes in one of the values are inconsistent.")?
+            }
             ErrorKind::ReservedKeyword(word) => {
                 writeln!(f, "`{word}` is a reserved keyword and thus cannot be used as a field name unless it is put inside quotes. Use \"{word}\" or \'{word}\' instead.")?
             }

crates/filter-parser/src/lib.rs

@@ -65,6 +65,9 @@ use nom_locate::LocatedSpan;
 pub(crate) use value::parse_value;
 use value::word_exact;
 
+use crate::condition::parse_vectors_exists;
+use crate::error::IResultExt;
+
 pub type Span<'a> = LocatedSpan<&'a str, &'a str>;
 
 type IResult<'a, Ret> = nom::IResult<Span<'a>, Ret, Error<'a>>;
@@ -136,6 +139,15 @@ impl<'a> From<&'a str> for Token<'a> {
     }
 }
 
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum VectorFilter<'a> {
+    Fragment(Token<'a>),
+    DocumentTemplate,
+    UserProvided,
+    Regenerate,
+    None,
+}
+
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub enum FilterCondition<'a> {
     Not(Box<Self>),
@@ -143,6 +155,7 @@ pub enum FilterCondition<'a> {
     In { fid: Token<'a>, els: Vec<Token<'a>> },
     Or(Vec<Self>),
     And(Vec<Self>),
+    VectorExists { fid: Token<'a>, embedder: Option<Token<'a>>, filter: VectorFilter<'a> },
     GeoLowerThan { point: [Token<'a>; 2], radius: Token<'a> },
     GeoBoundingBox { top_right_point: [Token<'a>; 2], bottom_left_point: [Token<'a>; 2] },
 }
@@ -165,17 +178,32 @@ impl<'a> FilterCondition<'a> {
                 | Condition::Exists
                 | Condition::LowerThan(_)
                 | Condition::LowerThanOrEqual(_)
-                | Condition::Between { .. } => None,
-                Condition::Contains { keyword, word: _ }
-                | Condition::StartsWith { keyword, word: _ } => Some(keyword),
+                | Condition::Between { .. }
+                | Condition::StartsWith { .. } => None,
+                Condition::Contains { keyword, word: _ } => Some(keyword),
             },
             FilterCondition::Not(this) => this.use_contains_operator(),
             FilterCondition::Or(seq) | FilterCondition::And(seq) => {
                 seq.iter().find_map(|filter| filter.use_contains_operator())
             }
+            FilterCondition::VectorExists { .. }
+            | FilterCondition::GeoLowerThan { .. }
+            | FilterCondition::GeoBoundingBox { .. }
+            | FilterCondition::In { .. } => None,
+        }
+    }
+
+    pub fn use_vector_filter(&self) -> Option<&Token> {
+        match self {
+            FilterCondition::Condition { .. } => None,
+            FilterCondition::Not(this) => this.use_vector_filter(),
+            FilterCondition::Or(seq) | FilterCondition::And(seq) => {
+                seq.iter().find_map(|filter| filter.use_vector_filter())
+            }
             FilterCondition::GeoLowerThan { .. }
             | FilterCondition::GeoBoundingBox { .. }
             | FilterCondition::In { .. } => None,
+            FilterCondition::VectorExists { fid, .. } => Some(fid),
         }
     }
 
@@ -263,10 +291,7 @@ fn parse_in_body(input: Span) -> IResult<Vec<Token>> {
     let (input, _) = ws(word_exact("IN"))(input)?;
 
     // everything after `IN` can be a failure
-    let (input, _) =
-        cut_with_err(tag("["), |_| Error::new_from_kind(input, ErrorKind::InOpeningBracket))(
-            input,
-        )?;
+    let (input, _) = tag("[")(input).map_cut(ErrorKind::InOpeningBracket)?;
 
     let (input, content) = cut(parse_value_list)(input)?;
 
@@ -412,7 +437,7 @@ fn parse_geo_bounding_box(input: Span) -> IResult<FilterCondition> {
     let (input, args) = parsed?;
 
     if args.len() != 2 || args[0].len() != 2 || args[1].len() != 2 {
-        return Err(nom::Err::Failure(Error::new_from_kind(input, ErrorKind::GeoBoundingBox)));
+        return Err(Error::failure_from_kind(input, ErrorKind::GeoBoundingBox));
     }
 
     let res = FilterCondition::GeoBoundingBox {
@@ -433,7 +458,7 @@ fn parse_geo_point(input: Span) -> IResult<FilterCondition> {
     ))(input)
     .map_err(|e| e.map(|_| Error::new_from_kind(input, ErrorKind::ReservedGeo("_geoPoint"))))?;
     // if we succeeded we still return a `Failure` because geoPoints are not allowed
-    Err(nom::Err::Failure(Error::new_from_kind(input, ErrorKind::ReservedGeo("_geoPoint"))))
+    Err(Error::failure_from_kind(input, ErrorKind::ReservedGeo("_geoPoint")))
 }
 
 /// geoPoint      = WS* "_geoDistance(float WS* "," WS* float WS* "," WS* float)
@@ -447,7 +472,7 @@ fn parse_geo_distance(input: Span) -> IResult<FilterCondition> {
     ))(input)
     .map_err(|e| e.map(|_| Error::new_from_kind(input, ErrorKind::ReservedGeo("_geoDistance"))))?;
     // if we succeeded we still return a `Failure` because `geoDistance` filters are not allowed
-    Err(nom::Err::Failure(Error::new_from_kind(input, ErrorKind::ReservedGeo("_geoDistance"))))
+    Err(Error::failure_from_kind(input, ErrorKind::ReservedGeo("_geoDistance")))
 }
 
 /// geo      = WS* "_geo(float WS* "," WS* float WS* "," WS* float)
@@ -461,7 +486,7 @@ fn parse_geo(input: Span) -> IResult<FilterCondition> {
     ))(input)
     .map_err(|e| e.map(|_| Error::new_from_kind(input, ErrorKind::ReservedGeo("_geo"))))?;
     // if we succeeded we still return a `Failure` because `_geo` filter is not allowed
-    Err(nom::Err::Failure(Error::new_from_kind(input, ErrorKind::ReservedGeo("_geo"))))
+    Err(Error::failure_from_kind(input, ErrorKind::ReservedGeo("_geo")))
 }
 
 fn parse_error_reserved_keyword(input: Span) -> IResult<FilterCondition> {
@@ -500,8 +525,7 @@ fn parse_primary(input: Span, depth: usize) -> IResult<FilterCondition> {
         parse_is_not_null,
         parse_is_empty,
         parse_is_not_empty,
-        parse_exists,
-        parse_not_exists,
+        alt((parse_vectors_exists, parse_exists, parse_not_exists)),
         parse_to,
         parse_contains,
         parse_not_contains,
@@ -557,6 +581,22 @@ impl std::fmt::Display for FilterCondition<'_> {
                 }
                 write!(f, "]")
             }
+            FilterCondition::VectorExists { fid: _, embedder, filter: inner } => {
+                write!(f, "_vectors")?;
+                if let Some(embedder) = embedder {
+                    write!(f, ".{:?}", embedder.value())?;
+                }
+                match inner {
+                    VectorFilter::Fragment(fragment) => {
+                        write!(f, ".fragments.{:?}", fragment.value())?
+                    }
+                    VectorFilter::DocumentTemplate => write!(f, ".documentTemplate")?,
+                    VectorFilter::UserProvided => write!(f, ".userProvided")?,
+                    VectorFilter::Regenerate => write!(f, ".regenerate")?,
+                    VectorFilter::None => (),
+                }
+                write!(f, " EXISTS")
+            }
             FilterCondition::GeoLowerThan { point, radius } => {
                 write!(f, "_geoRadius({}, {}, {})", point[0], point[1], radius)
             }
@@ -630,6 +670,9 @@ pub mod tests {
         insta::assert_snapshot!(p(r"title = 'foo\\\\\\\\'"), @r#"{title} = {foo\\\\}"#);
         // but it also works with other sequences
         insta::assert_snapshot!(p(r#"title = 'foo\x20\n\t\"\'"'"#), @"{title} = {foo \n\t\"\'\"}");
+
+        insta::assert_snapshot!(p(r#"_vectors." valid.name  ".fragments."also.. valid! " EXISTS"#), @r#"_vectors." valid.name  ".fragments."also.. valid! " EXISTS"#);
+        insta::assert_snapshot!(p("_vectors.\"\n\t\r\\\"\" EXISTS"), @r#"_vectors."\n\t\r\"" EXISTS"#);
     }
 
     #[test]
@@ -692,6 +735,18 @@ pub mod tests {
         insta::assert_snapshot!(p("NOT subscribers IS NOT EMPTY"), @"{subscribers} IS EMPTY");
         insta::assert_snapshot!(p("subscribers  IS   NOT   EMPTY"), @"NOT ({subscribers} IS EMPTY)");
 
+        // Test _vectors EXISTS + _vectors NOT EXITS
+        insta::assert_snapshot!(p("_vectors EXISTS"), @"_vectors EXISTS");
+        insta::assert_snapshot!(p("_vectors.embedderName EXISTS"), @r#"_vectors."embedderName" EXISTS"#);
+        insta::assert_snapshot!(p("_vectors.embedderName.documentTemplate EXISTS"), @r#"_vectors."embedderName".documentTemplate EXISTS"#);
+        insta::assert_snapshot!(p("_vectors.embedderName.regenerate EXISTS"), @r#"_vectors."embedderName".regenerate EXISTS"#);
+        insta::assert_snapshot!(p("_vectors.embedderName.regenerate EXISTS"), @r#"_vectors."embedderName".regenerate EXISTS"#);
+        insta::assert_snapshot!(p("_vectors.embedderName.fragments.fragmentName EXISTS"), @r#"_vectors."embedderName".fragments."fragmentName" EXISTS"#);
+        insta::assert_snapshot!(p("  _vectors.embedderName.fragments.fragmentName   EXISTS"), @r#"_vectors."embedderName".fragments."fragmentName" EXISTS"#);
+        insta::assert_snapshot!(p("NOT _vectors EXISTS"), @"NOT (_vectors EXISTS)");
+        insta::assert_snapshot!(p(" NOT  _vectors   EXISTS"), @"NOT (_vectors EXISTS)");
+        insta::assert_snapshot!(p("  _vectors  NOT  EXISTS"), @"NOT (_vectors EXISTS)");
+
         // Test EXISTS + NOT EXITS
         insta::assert_snapshot!(p("subscribers EXISTS"), @"{subscribers} EXISTS");
         insta::assert_snapshot!(p("NOT subscribers EXISTS"), @"NOT ({subscribers} EXISTS)");
@@ -946,6 +1001,71 @@ pub mod tests {
         "###
         );
 
+        insta::assert_snapshot!(p(r#"_vectors _vectors EXISTS"#), @r"
+        Was expecting an operation like `EXISTS` or `NOT EXISTS` after the vector filter.
+        10:25 _vectors _vectors EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors. embedderName EXISTS"#), @r"
+        Was expecting embedder name but found nothing.
+        10:11 _vectors. embedderName EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors .embedderName EXISTS"#), @r"
+        Was expecting an operation like `EXISTS` or `NOT EXISTS` after the vector filter.
+        10:30 _vectors .embedderName EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName. EXISTS"#), @r"
+        Was expecting one of `.fragments`, `.userProvided`, `.documentTemplate`, `.regenerate` or nothing, but instead found a point without a valid value.
+        22:23 _vectors.embedderName. EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors."embedderName EXISTS"#), @r#"
+        The quotes in one of the values are inconsistent.
+        10:30 _vectors."embedderName EXISTS
+        "#);
+        insta::assert_snapshot!(p(r#"_vectors."embedderNam"e EXISTS"#), @r#"
+        The vector filter has leftover tokens.
+        23:31 _vectors."embedderNam"e EXISTS
+        "#);
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.documentTemplate. EXISTS"#), @r"
+        Was expecting one of `.fragments`, `.userProvided`, `.documentTemplate`, `.regenerate` or nothing, but instead found a point without a valid value.
+        39:40 _vectors.embedderName.documentTemplate. EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.fragments EXISTS"#), @r"
+        The vector filter is missing a fragment name.
+        32:39 _vectors.embedderName.fragments EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.fragments. EXISTS"#), @r"
+        The vector filter's fragment name is invalid.
+        33:40 _vectors.embedderName.fragments. EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.fragments.test test EXISTS"#), @r"
+        Was expecting an operation like `EXISTS` or `NOT EXISTS` after the vector filter.
+        38:49 _vectors.embedderName.fragments.test test EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.fragments. test EXISTS"#), @r"
+        The vector filter's fragment name is invalid.
+        33:45 _vectors.embedderName.fragments. test EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName .fragments. test EXISTS"#), @r"
+        Was expecting an operation like `EXISTS` or `NOT EXISTS` after the vector filter.
+        23:46 _vectors.embedderName .fragments. test EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName .fragments.test EXISTS"#), @r"
+        Was expecting an operation like `EXISTS` or `NOT EXISTS` after the vector filter.
+        23:45 _vectors.embedderName .fragments.test EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.fargments.test EXISTS"#), @r"
+        Was expecting one of `fragments`, `userProvided`, `documentTemplate`, `regenerate` or nothing, but instead found `fargments`. Did you mean `fragments`?
+        23:32 _vectors.embedderName.fargments.test EXISTS
+        ");
+        insta::assert_snapshot!(p(r#"_vectors.embedderName."userProvided" EXISTS"#), @r#"
+        Was expecting this part to be unquoted.
+        24:36 _vectors.embedderName."userProvided" EXISTS
+        "#);
+        insta::assert_snapshot!(p(r#"_vectors.embedderName.userProvided.fragments.test EXISTS"#), @r"
+        Vector filter can only accept one of `fragments`, `userProvided`, `documentTemplate` or `regenerate`, but found both `userProvided` and `fragments`.
+        36:45 _vectors.embedderName.userProvided.fragments.test EXISTS
+        ");
+
         insta::assert_snapshot!(p(r#"NOT OR EXISTS AND EXISTS NOT EXISTS"#), @r###"
         Was expecting a value but instead got `OR`, which is a reserved keyword. To use `OR` as a field name or a value, surround it by quotes.
         5:7 NOT OR EXISTS AND EXISTS NOT EXISTS

crates/filter-parser/src/value.rs

@@ -80,6 +80,51 @@ pub fn word_exact<'a, 'b: 'a>(tag: &'b str) -> impl Fn(Span<'a>) -> IResult<'a,
     }
 }
 
+/// vector_value          = ( non_dot_word | singleQuoted | doubleQuoted)
+pub fn parse_vector_value(input: Span) -> IResult<Token> {
+    pub fn non_dot_word(input: Span) -> IResult<Token> {
+        let (input, word) = take_while1(|c| is_value_component(c) && c != '.')(input)?;
+        Ok((input, word.into()))
+    }
+
+    let (input, value) = alt((
+        delimited(char('\''), cut(|input| quoted_by('\'', input)), cut(char('\''))),
+        delimited(char('"'), cut(|input| quoted_by('"', input)), cut(char('"'))),
+        non_dot_word,
+    ))(input)?;
+
+    match unescaper::unescape(value.value()) {
+        Ok(content) => {
+            if content.len() != value.value().len() {
+                Ok((input, Token::new(value.original_span(), Some(content))))
+            } else {
+                Ok((input, value))
+            }
+        }
+        Err(unescaper::Error::IncompleteStr(_)) => Err(nom::Err::Incomplete(nom::Needed::Unknown)),
+        Err(unescaper::Error::ParseIntError { .. }) => Err(nom::Err::Error(Error::new_from_kind(
+            value.original_span(),
+            ErrorKind::InvalidEscapedNumber,
+        ))),
+        Err(unescaper::Error::InvalidChar { .. }) => Err(nom::Err::Error(Error::new_from_kind(
+            value.original_span(),
+            ErrorKind::MalformedValue,
+        ))),
+    }
+}
+
+pub fn parse_vector_value_cut<'a>(input: Span<'a>, kind: ErrorKind<'a>) -> IResult<'a, Token<'a>> {
+    parse_vector_value(input).map_err(|e| match e {
+        nom::Err::Failure(e) => match e.kind() {
+            ErrorKind::Char(c) if *c == '"' || *c == '\'' => {
+                crate::Error::failure_from_kind(input, ErrorKind::VectorFilterInvalidQuotes)
+            }
+            _ => crate::Error::failure_from_kind(input, kind),
+        },
+        _ => crate::Error::failure_from_kind(input, kind),
+    })
+}
+
 /// value          = WS* ( word | singleQuoted | doubleQuoted) WS+
 pub fn parse_value(input: Span) -> IResult<Token> {
     // to get better diagnostic message we are going to strip the left whitespaces from the input right now
@@ -99,31 +144,21 @@ pub fn parse_value(input: Span) -> IResult<Token> {
     }
 
     match parse_geo_radius(input) {
-        Ok(_) => {
-            return Err(nom::Err::Failure(Error::new_from_kind(input, ErrorKind::MisusedGeoRadius)))
-        }
+        Ok(_) => return Err(Error::failure_from_kind(input, ErrorKind::MisusedGeoRadius)),
         // if we encountered a failure it means the user badly wrote a _geoRadius filter.
         // But instead of showing them how to fix his syntax we are going to tell them they should not use this filter as a value.
         Err(e) if e.is_failure() => {
-            return Err(nom::Err::Failure(Error::new_from_kind(input, ErrorKind::MisusedGeoRadius)))
+            return Err(Error::failure_from_kind(input, ErrorKind::MisusedGeoRadius))
         }
         _ => (),
     }
 
     match parse_geo_bounding_box(input) {
-        Ok(_) => {
-            return Err(nom::Err::Failure(Error::new_from_kind(
-                input,
-                ErrorKind::MisusedGeoBoundingBox,
-            )))
-        }
+        Ok(_) => return Err(Error::failure_from_kind(input, ErrorKind::MisusedGeoBoundingBox)),
         // if we encountered a failure it means the user badly wrote a _geoBoundingBox filter.
         // But instead of showing them how to fix his syntax we are going to tell them they should not use this filter as a value.
         Err(e) if e.is_failure() => {
-            return Err(nom::Err::Failure(Error::new_from_kind(
-                input,
-                ErrorKind::MisusedGeoBoundingBox,
-            )))
+            return Err(Error::failure_from_kind(input, ErrorKind::MisusedGeoBoundingBox))
         }
         _ => (),
     }

crates/index-scheduler/Cargo.toml

@@ -26,7 +26,7 @@ flate2 = "1.1.2"
 indexmap = "2.9.0"
 meilisearch-auth = { path = "../meilisearch-auth" }
 meilisearch-types = { path = "../meilisearch-types" }
-memmap2 = "0.9.5"
+memmap2 = "0.9.7"
 page_size = "0.6.0"
 rayon = "1.10.0"
 roaring = { version = "0.10.12", features = ["serde"] }

crates/index-scheduler/src/dump.rs

@@ -197,9 +197,10 @@ impl<'a> Dump<'a> {
                     index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
                     primary_key,
                 },
-                KindDump::IndexUpdate { primary_key } => KindWithContent::IndexUpdate {
+                KindDump::IndexUpdate { primary_key, uid } => KindWithContent::IndexUpdate {
                     index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
                     primary_key,
+                    new_index_uid: uid,
                 },
                 KindDump::IndexSwap { swaps } => KindWithContent::IndexSwap { swaps },
                 KindDump::TaskCancelation { query, tasks } => {

crates/index-scheduler/src/error.rs

@@ -67,6 +67,8 @@ pub enum Error {
     SwapDuplicateIndexesFound(Vec<String>),
     #[error("Index `{0}` not found.")]
     SwapIndexNotFound(String),
+    #[error("Cannot rename `{0}` to `{1}` as the index already exists. Hint: You can remove `{1}` first and then do your remove.")]
+    SwapIndexFoundDuringRename(String, String),
     #[error("Meilisearch cannot receive write operations because the limit of the task database has been reached. Please delete tasks to continue performing write operations.")]
     NoSpaceLeftInTaskQueue,
     #[error(
@@ -74,6 +76,10 @@ pub enum Error {
         .0.iter().map(|s| format!("`{}`", s)).collect::<Vec<_>>().join(", ")
     )]
     SwapIndexesNotFound(Vec<String>),
+    #[error("The following indexes are being renamed but cannot because their new name conflicts with an already existing index: {}. Renaming doesn't overwrite the other index name.",
+        .0.iter().map(|s| format!("`{}`", s)).collect::<Vec<_>>().join(", ")
+    )]
+    SwapIndexesFoundDuringRename(Vec<String>),
     #[error("Corrupted dump.")]
     CorruptedDump,
     #[error(
@@ -203,6 +209,8 @@ impl Error {
             | Error::SwapIndexNotFound(_)
             | Error::NoSpaceLeftInTaskQueue
             | Error::SwapIndexesNotFound(_)
+            | Error::SwapIndexFoundDuringRename(_, _)
+            | Error::SwapIndexesFoundDuringRename(_)
             | Error::CorruptedDump
             | Error::InvalidTaskDate { .. }
             | Error::InvalidTaskUid { .. }
@@ -271,6 +279,8 @@ impl ErrorCode for Error {
             Error::SwapDuplicateIndexFound(_) => Code::InvalidSwapDuplicateIndexFound,
             Error::SwapIndexNotFound(_) => Code::IndexNotFound,
             Error::SwapIndexesNotFound(_) => Code::IndexNotFound,
+            Error::SwapIndexFoundDuringRename(_, _) => Code::IndexAlreadyExists,
+            Error::SwapIndexesFoundDuringRename(_) => Code::IndexAlreadyExists,
             Error::InvalidTaskDate { field, .. } => (*field).into(),
             Error::InvalidTaskUid { .. } => Code::InvalidTaskUids,
             Error::InvalidBatchUid { .. } => Code::InvalidBatchUids,

crates/index-scheduler/src/features.rs

@@ -85,7 +85,7 @@ impl RoFeatures {
             Ok(())
         } else {
             Err(FeatureNotEnabledError {
-                disabled_action: "Using `CONTAINS` or `STARTS WITH` in a filter",
+                disabled_action: "Using `CONTAINS` in a filter",
                 feature: "contains filter",
                 issue_link: "https://github.com/orgs/meilisearch/discussions/763",
             }
@@ -182,6 +182,7 @@ impl FeatureData {
             ..persisted_features
         }));
 
+        // Once this is stabilized, network should be stored along with webhooks in index-scheduler's persisted database
         let network_db = runtime_features_db.remap_data_type::<SerdeJson<Network>>();
         let network: Network = network_db.get(wtxn, db_keys::NETWORK)?.unwrap_or_default();
 

crates/index-scheduler/src/index_mapper/mod.rs

@@ -71,7 +71,7 @@ pub struct IndexMapper {
     /// Path to the folder where the LMDB environments of each index are.
     base_path: PathBuf,
     /// The map size an index is opened with on the first time.
-    index_base_map_size: usize,
+    pub(crate) index_base_map_size: usize,
     /// The quantity by which the map size of an index is incremented upon reopening, in bytes.
     index_growth_amount: usize,
     /// Whether we open a meilisearch index with the MDB_WRITEMAP option or not.
@@ -526,6 +526,20 @@ impl IndexMapper {
         Ok(())
     }
 
+    /// Rename an index.
+    pub fn rename(&self, wtxn: &mut RwTxn, current: &str, new: &str) -> Result<()> {
+        let uuid = self
+            .index_mapping
+            .get(wtxn, current)?
+            .ok_or_else(|| Error::IndexNotFound(current.to_string()))?;
+        if self.index_mapping.get(wtxn, new)?.is_some() {
+            return Err(Error::IndexAlreadyExists(new.to_string()));
+        }
+        self.index_mapping.delete(wtxn, current)?;
+        self.index_mapping.put(wtxn, new, &uuid)?;
+        Ok(())
+    }
+
     /// The stats of an index.
     ///
     /// If available in the cache, they are directly returned.

crates/index-scheduler/src/insta_snapshot.rs

@@ -20,16 +20,17 @@ pub fn snapshot_index_scheduler(scheduler: &IndexScheduler) -> String {
 
     let IndexScheduler {
         cleanup_enabled: _,
+        experimental_no_edition_2024_for_dumps: _,
         processing_tasks,
         env,
         version,
         queue,
         scheduler,
+        persisted,
 
         index_mapper,
         features: _,
-        webhook_url: _,
-        webhook_authorization_header: _,
+        webhooks: _,
         test_breakpoint_sdr: _,
         planned_failures: _,
         run_loop_iteration: _,
@@ -61,6 +62,13 @@ pub fn snapshot_index_scheduler(scheduler: &IndexScheduler) -> String {
     }
     snap.push_str("\n----------------------------------------------------------------------\n");
 
+    let persisted_db_snapshot = snapshot_persisted_db(&rtxn, persisted);
+    if !persisted_db_snapshot.is_empty() {
+        snap.push_str("### Persisted:\n");
+        snap.push_str(&persisted_db_snapshot);
+        snap.push_str("----------------------------------------------------------------------\n");
+    }
+
     snap.push_str("### All Tasks:\n");
     snap.push_str(&snapshot_all_tasks(&rtxn, queue.tasks.all_tasks));
     snap.push_str("----------------------------------------------------------------------\n");
@@ -199,6 +207,16 @@ pub fn snapshot_date_db(rtxn: &RoTxn, db: Database<BEI128, CboRoaringBitmapCodec
     snap
 }
 
+pub fn snapshot_persisted_db(rtxn: &RoTxn, db: &Database<Str, Str>) -> String {
+    let mut snap = String::new();
+    let iter = db.iter(rtxn).unwrap();
+    for next in iter {
+        let (key, value) = next.unwrap();
+        snap.push_str(&format!("{key}: {value}\n"));
+    }
+    snap
+}
+
 pub fn snapshot_task(task: &Task) -> String {
     let mut snap = String::new();
     let Task {
@@ -256,8 +274,8 @@ fn snapshot_details(d: &Details) -> String {
         Details::SettingsUpdate { settings } => {
             format!("{{ settings: {settings:?} }}")
         }
-        Details::IndexInfo { primary_key } => {
-            format!("{{ primary_key: {primary_key:?} }}")
+        Details::IndexInfo { primary_key, new_index_uid, old_index_uid } => {
+            format!("{{ primary_key: {primary_key:?}, old_new_uid: {old_index_uid:?}, new_index_uid: {new_index_uid:?} }}")
         }
         Details::DocumentDeletion {
             provided_ids: received_document_ids,
@@ -310,6 +328,7 @@ pub fn snapshot_status(
     }
     snap
 }
+
 pub fn snapshot_kind(rtxn: &RoTxn, db: Database<SerdeBincode<Kind>, RoaringBitmapCodec>) -> String {
     let mut snap = String::new();
     let iter = db.iter(rtxn).unwrap();
@@ -330,6 +349,7 @@ pub fn snapshot_index_tasks(rtxn: &RoTxn, db: Database<Str, RoaringBitmapCodec>)
     }
     snap
 }
+
 pub fn snapshot_canceled_by(rtxn: &RoTxn, db: Database<BEU32, RoaringBitmapCodec>) -> String {
     let mut snap = String::new();
     let iter = db.iter(rtxn).unwrap();

crates/index-scheduler/src/lib.rs

@@ -65,13 +65,16 @@ use meilisearch_types::milli::vector::{
 use meilisearch_types::milli::{self, Index};
 use meilisearch_types::task_view::TaskView;
 use meilisearch_types::tasks::{KindWithContent, Task};
+use meilisearch_types::webhooks::{Webhook, WebhooksDumpView, WebhooksView};
 use milli::vector::db::IndexEmbeddingConfig;
 use processing::ProcessingTasks;
 pub use queue::Query;
 use queue::Queue;
 use roaring::RoaringBitmap;
 use scheduler::Scheduler;
+use serde::{Deserialize, Serialize};
 use time::OffsetDateTime;
+use uuid::Uuid;
 use versioning::Versioning;
 
 use crate::index_mapper::IndexMapper;
@@ -80,7 +83,15 @@ use crate::utils::clamp_to_page_size;
 pub(crate) type BEI128 = I128<BE>;
 
 const TASK_SCHEDULER_SIZE_THRESHOLD_PERCENT_INT: u64 = 40;
-const CHAT_SETTINGS_DB_NAME: &str = "chat-settings";
+
+mod db_name {
+    pub const CHAT_SETTINGS: &str = "chat-settings";
+    pub const PERSISTED: &str = "persisted";
+}
+
+mod db_keys {
+    pub const WEBHOOKS: &str = "webhooks";
+}
 
 #[derive(Debug)]
 pub struct IndexSchedulerOptions {
@@ -98,10 +109,10 @@ pub struct IndexSchedulerOptions {
     pub snapshots_path: PathBuf,
     /// The path to the folder containing the dumps.
     pub dumps_path: PathBuf,
-    /// The URL on which we must send the tasks statuses
-    pub webhook_url: Option<String>,
-    /// The value we will send into the Authorization HTTP header on the webhook URL
-    pub webhook_authorization_header: Option<String>,
+    /// The webhook url that was set by the CLI.
+    pub cli_webhook_url: Option<String>,
+    /// The Authorization header to send to the webhook URL that was set by the CLI.
+    pub cli_webhook_authorization: Option<String>,
     /// The maximum size, in bytes, of the task index.
     pub task_db_size: usize,
     /// The size, in bytes, with which a meilisearch index is opened the first time of each meilisearch index.
@@ -168,10 +179,14 @@ pub struct IndexScheduler {
     /// Whether we should automatically cleanup the task queue or not.
     pub(crate) cleanup_enabled: bool,
 
-    /// The webhook url we should send tasks to after processing every batches.
-    pub(crate) webhook_url: Option<String>,
-    /// The Authorization header to send to the webhook URL.
-    pub(crate) webhook_authorization_header: Option<String>,
+    /// Whether we should use the old document indexer or the new one.
+    pub(crate) experimental_no_edition_2024_for_dumps: bool,
+
+    /// A database to store single-keyed data that is persisted across restarts.
+    persisted: Database<Str, Str>,
+
+    /// Webhook, loaded and stored in the `persisted` database
+    webhooks: Arc<Webhooks>,
 
     /// A map to retrieve the runtime representation of an embedder depending on its configuration.
     ///
@@ -210,8 +225,10 @@ impl IndexScheduler {
 
             index_mapper: self.index_mapper.clone(),
             cleanup_enabled: self.cleanup_enabled,
-            webhook_url: self.webhook_url.clone(),
-            webhook_authorization_header: self.webhook_authorization_header.clone(),
+            experimental_no_edition_2024_for_dumps: self.experimental_no_edition_2024_for_dumps,
+            persisted: self.persisted,
+
+            webhooks: self.webhooks.clone(),
             embedders: self.embedders.clone(),
             #[cfg(test)]
             test_breakpoint_sdr: self.test_breakpoint_sdr.clone(),
@@ -230,6 +247,7 @@ impl IndexScheduler {
             + IndexMapper::nb_db()
             + features::FeatureData::nb_db()
             + 1 // chat-prompts
+            + 1 // persisted
     }
 
     /// Create an index scheduler and start its run loop.
@@ -280,10 +298,18 @@ impl IndexScheduler {
         let version = versioning::Versioning::new(&env, from_db_version)?;
 
         let mut wtxn = env.write_txn()?;
+
         let features = features::FeatureData::new(&env, &mut wtxn, options.instance_features)?;
         let queue = Queue::new(&env, &mut wtxn, &options)?;
         let index_mapper = IndexMapper::new(&env, &mut wtxn, &options, budget)?;
-        let chat_settings = env.create_database(&mut wtxn, Some(CHAT_SETTINGS_DB_NAME))?;
+        let chat_settings = env.create_database(&mut wtxn, Some(db_name::CHAT_SETTINGS))?;
+
+        let persisted = env.create_database(&mut wtxn, Some(db_name::PERSISTED))?;
+        let webhooks_db = persisted.remap_data_type::<SerdeJson<Webhooks>>();
+        let mut webhooks = webhooks_db.get(&wtxn, db_keys::WEBHOOKS)?.unwrap_or_default();
+        webhooks
+            .with_cli(options.cli_webhook_url.clone(), options.cli_webhook_authorization.clone());
+
         wtxn.commit()?;
 
         // allow unreachable_code to get rids of the warning in the case of a test build.
@@ -296,8 +322,11 @@ impl IndexScheduler {
             index_mapper,
             env,
             cleanup_enabled: options.cleanup_enabled,
-            webhook_url: options.webhook_url,
-            webhook_authorization_header: options.webhook_authorization_header,
+            experimental_no_edition_2024_for_dumps: options
+                .indexer_config
+                .experimental_no_edition_2024_for_dumps,
+            persisted,
+            webhooks: Arc::new(webhooks),
             embedders: Default::default(),
 
             #[cfg(test)]
@@ -594,6 +623,11 @@ impl IndexScheduler {
         Ok(nbr_index_processing_tasks > 0)
     }
 
+    /// Whether the index should use the old document indexer.
+    pub fn no_edition_2024_for_dumps(&self) -> bool {
+        self.experimental_no_edition_2024_for_dumps
+    }
+
     /// Return the tasks matching the query from the user's point of view along
     /// with the total number of tasks matching the query, ignoring from and limit.
     ///
@@ -740,86 +774,92 @@ impl IndexScheduler {
         Ok(())
     }
 
-    /// Once the tasks changes have been committed we must send all the tasks that were updated to our webhook if there is one.
-    fn notify_webhook(&self, updated: &RoaringBitmap) -> Result<()> {
-        if let Some(ref url) = self.webhook_url {
-            struct TaskReader<'a, 'b> {
-                rtxn: &'a RoTxn<'a>,
-                index_scheduler: &'a IndexScheduler,
-                tasks: &'b mut roaring::bitmap::Iter<'b>,
-                buffer: Vec<u8>,
-                written: usize,
-            }
+    /// Once the tasks changes have been committed we must send all the tasks that were updated to our webhooks
+    fn notify_webhooks(&self, updated: RoaringBitmap) {
+        struct TaskReader<'a, 'b> {
+            rtxn: &'a RoTxn<'a>,
+            index_scheduler: &'a IndexScheduler,
+            tasks: &'b mut roaring::bitmap::Iter<'b>,
+            buffer: Vec<u8>,
+            written: usize,
+        }
 
-            impl Read for TaskReader<'_, '_> {
-                fn read(&mut self, mut buf: &mut [u8]) -> std::io::Result<usize> {
-                    if self.buffer.is_empty() {
-                        match self.tasks.next() {
-                            None => return Ok(0),
-                            Some(task_id) => {
-                                let task = self
-                                    .index_scheduler
-                                    .queue
-                                    .tasks
-                                    .get_task(self.rtxn, task_id)
-                                    .map_err(|err| io::Error::new(io::ErrorKind::Other, err))?
-                                    .ok_or_else(|| {
-                                        io::Error::new(
-                                            io::ErrorKind::Other,
-                                            Error::CorruptedTaskQueue,
-                                        )
-                                    })?;
+        impl Read for TaskReader<'_, '_> {
+            fn read(&mut self, mut buf: &mut [u8]) -> std::io::Result<usize> {
+                if self.buffer.is_empty() {
+                    match self.tasks.next() {
+                        None => return Ok(0),
+                        Some(task_id) => {
+                            let task = self
+                                .index_scheduler
+                                .queue
+                                .tasks
+                                .get_task(self.rtxn, task_id)
+                                .map_err(|err| io::Error::new(io::ErrorKind::Other, err))?
+                                .ok_or_else(|| {
+                                    io::Error::new(io::ErrorKind::Other, Error::CorruptedTaskQueue)
+                                })?;
 
-                                serde_json::to_writer(
-                                    &mut self.buffer,
-                                    &TaskView::from_task(&task),
-                                )?;
-                                self.buffer.push(b'\n');
-                            }
+                            serde_json::to_writer(&mut self.buffer, &TaskView::from_task(&task))?;
+                            self.buffer.push(b'\n');
                         }
                     }
-
-                    let mut to_write = &self.buffer[self.written..];
-                    let wrote = io::copy(&mut to_write, &mut buf)?;
-                    self.written += wrote as usize;
-
-                    // we wrote everything and must refresh our buffer on the next call
-                    if self.written == self.buffer.len() {
-                        self.written = 0;
-                        self.buffer.clear();
-                    }
-
-                    Ok(wrote as usize)
                 }
-            }
 
-            let rtxn = self.env.read_txn()?;
+                let mut to_write = &self.buffer[self.written..];
+                let wrote = io::copy(&mut to_write, &mut buf)?;
+                self.written += wrote as usize;
 
-            let task_reader = TaskReader {
-                rtxn: &rtxn,
-                index_scheduler: self,
-                tasks: &mut updated.into_iter(),
-                buffer: Vec::with_capacity(50), // on average a task is around ~100 bytes
-                written: 0,
-            };
+                // we wrote everything and must refresh our buffer on the next call
+                if self.written == self.buffer.len() {
+                    self.written = 0;
+                    self.buffer.clear();
+                }
 
-            // let reader = GzEncoder::new(BufReader::new(task_reader), Compression::default());
-            let reader = GzEncoder::new(BufReader::new(task_reader), Compression::default());
-            let request = ureq::post(url)
-                .timeout(Duration::from_secs(30))
-                .set("Content-Encoding", "gzip")
-                .set("Content-Type", "application/x-ndjson");
-            let request = match &self.webhook_authorization_header {
-                Some(header) => request.set("Authorization", header),
-                None => request,
-            };
-
-            if let Err(e) = request.send(reader) {
-                tracing::error!("While sending data to the webhook: {e}");
+                Ok(wrote as usize)
             }
         }
 
-        Ok(())
+        let webhooks = self.webhooks.get_all();
+        if webhooks.is_empty() {
+            return;
+        }
+        let this = self.private_clone();
+        // We must take the RoTxn before entering the thread::spawn otherwise another batch may be
+        // processed before we had the time to take our txn.
+        let rtxn = match self.env.clone().static_read_txn() {
+            Ok(rtxn) => rtxn,
+            Err(e) => {
+                tracing::error!("Couldn't get an rtxn to notify the webhook: {e}");
+                return;
+            }
+        };
+
+        std::thread::spawn(move || {
+            for (uuid, Webhook { url, headers }) in webhooks.iter() {
+                let task_reader = TaskReader {
+                    rtxn: &rtxn,
+                    index_scheduler: &this,
+                    tasks: &mut updated.iter(),
+                    buffer: Vec::with_capacity(page_size::get()),
+                    written: 0,
+                };
+
+                let reader = GzEncoder::new(BufReader::new(task_reader), Compression::default());
+
+                let mut request = ureq::post(url)
+                    .timeout(Duration::from_secs(30))
+                    .set("Content-Encoding", "gzip")
+                    .set("Content-Type", "application/x-ndjson");
+                for (header_name, header_value) in headers.iter() {
+                    request = request.set(header_name, header_value);
+                }
+
+                if let Err(e) = request.send(reader) {
+                    tracing::error!("While sending data to the webhook {uuid}: {e}");
+                }
+            }
+        });
     }
 
     pub fn index_stats(&self, index_uid: &str) -> Result<IndexStats> {
@@ -850,6 +890,29 @@ impl IndexScheduler {
         self.features.network()
     }
 
+    pub fn update_runtime_webhooks(&self, runtime: RuntimeWebhooks) -> Result<()> {
+        let webhooks = Webhooks::from_runtime(runtime);
+        let mut wtxn = self.env.write_txn()?;
+        let webhooks_db = self.persisted.remap_data_type::<SerdeJson<Webhooks>>();
+        webhooks_db.put(&mut wtxn, db_keys::WEBHOOKS, &webhooks)?;
+        wtxn.commit()?;
+        self.webhooks.update_runtime(webhooks.into_runtime());
+        Ok(())
+    }
+
+    pub fn webhooks_dump_view(&self) -> WebhooksDumpView {
+        // We must not dump the cli api key
+        WebhooksDumpView { webhooks: self.webhooks.get_runtime() }
+    }
+
+    pub fn webhooks_view(&self) -> WebhooksView {
+        WebhooksView { webhooks: self.webhooks.get_all() }
+    }
+
+    pub fn retrieve_runtime_webhooks(&self) -> RuntimeWebhooks {
+        self.webhooks.get_runtime()
+    }
+
     pub fn embedders(
         &self,
         index_uid: String,
@@ -978,3 +1041,72 @@ pub struct IndexStats {
     /// Internal stats computed from the index.
     pub inner_stats: index_mapper::IndexStats,
 }
+
+/// These structure are not meant to be exposed to the end user, if needed, use the meilisearch-types::webhooks structure instead.
+/// /!\ Everytime you deserialize this structure you should fill the cli_webhook later on with the `with_cli` method. /!\
+#[derive(Debug, Serialize, Deserialize, Default)]
+#[serde(rename_all = "camelCase")]
+struct Webhooks {
+    // The cli webhook should *never* be stored in a database.
+    // It represent a state that only exists for this execution of meilisearch
+    #[serde(skip)]
+    pub cli: Option<CliWebhook>,
+
+    #[serde(default)]
+    pub runtime: RwLock<RuntimeWebhooks>,
+}
+
+type RuntimeWebhooks = BTreeMap<Uuid, Webhook>;
+
+impl Webhooks {
+    pub fn with_cli(&mut self, url: Option<String>, auth: Option<String>) {
+        if let Some(url) = url {
+            let webhook = CliWebhook { url, auth };
+            self.cli = Some(webhook);
+        }
+    }
+
+    pub fn from_runtime(webhooks: RuntimeWebhooks) -> Self {
+        Self { cli: None, runtime: RwLock::new(webhooks) }
+    }
+
+    pub fn into_runtime(self) -> RuntimeWebhooks {
+        // safe because we own self and it cannot be cloned
+        self.runtime.into_inner().unwrap()
+    }
+
+    pub fn update_runtime(&self, webhooks: RuntimeWebhooks) {
+        *self.runtime.write().unwrap() = webhooks;
+    }
+
+    /// Returns all the webhooks in an unified view. The cli webhook is represented with an uuid set to 0
+    pub fn get_all(&self) -> BTreeMap<Uuid, Webhook> {
+        self.cli
+            .as_ref()
+            .map(|wh| (Uuid::nil(), Webhook::from(wh)))
+            .into_iter()
+            .chain(self.runtime.read().unwrap().iter().map(|(uuid, wh)| (*uuid, wh.clone())))
+            .collect()
+    }
+
+    /// Returns all the runtime webhooks.
+    pub fn get_runtime(&self) -> BTreeMap<Uuid, Webhook> {
+        self.runtime.read().unwrap().iter().map(|(uuid, wh)| (*uuid, wh.clone())).collect()
+    }
+}
+
+#[derive(Debug, Serialize, Deserialize, Default, Clone, PartialEq)]
+struct CliWebhook {
+    pub url: String,
+    pub auth: Option<String>,
+}
+
+impl From<&CliWebhook> for Webhook {
+    fn from(webhook: &CliWebhook) -> Self {
+        let mut headers = BTreeMap::new();
+        if let Some(ref auth) = webhook.auth {
+            headers.insert("Authorization".to_string(), auth.to_string());
+        }
+        Self { url: webhook.url.to_string(), headers }
+    }
+}

crates/index-scheduler/src/processing.rs

@@ -108,6 +108,7 @@ make_enum_progress! {
         DumpTheBatches,
         DumpTheIndexes,
         DumpTheExperimentalFeatures,
+        DumpTheWebhooks,
         CompressTheDump,
     }
 }

crates/index-scheduler/src/queue/batches_test.rs

@@ -334,11 +334,11 @@ fn query_batches_special_rules() {
     let kind = index_creation_task("doggo", "sheep");
     let _task = index_scheduler.register(kind, None, false).unwrap();
     let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()) }],
+        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
     };
     let _task = index_scheduler.register(kind, None, false).unwrap();
     let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()) }],
+        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()), rename: false }],
     };
     let _task = index_scheduler.register(kind, None, false).unwrap();
 
@@ -442,7 +442,7 @@ fn query_batches_canceled_by() {
     let kind = index_creation_task("doggo", "sheep");
     let _ = index_scheduler.register(kind, None, false).unwrap();
     let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()) }],
+        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
     };
     let _task = index_scheduler.register(kind, None, false).unwrap();
 

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_canceled_by/start.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/queue/batches_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
 3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
 ----------------------------------------------------------------------
 ### Status:
@@ -49,7 +49,7 @@ catto: { number_of_documents: 0, field_distribution: {} }
 ----------------------------------------------------------------------
 ### All Batches:
 0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"]}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 3 of type `taskCancelation` that cannot be batched with any other task.", }
+1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"],"rename":false}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 3 of type `taskCancelation` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### Batch to tasks mapping:
 0 [0,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/processed_all_tasks.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/queue/batches_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
+2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_second_task.snap

@@ -1,14 +1,13 @@
 ---
 source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_third_task.snap

@@ -1,15 +1,14 @@
 ---
 source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
+2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/after-advancing-a-bit.snap

@@ -7,9 +7,9 @@ source: crates/index-scheduler/src/queue/batches_test.rs
 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, status: enqueued, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [1,2,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/end.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/queue/batches_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/start.snap

@@ -1,15 +1,14 @@
 ---
 source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, status: enqueued, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_special_rules/after-processing-everything.snap

@@ -6,10 +6,10 @@ source: crates/index-scheduler/src/queue/batches_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, batch_uid: 3, status: failed, error: ResponseError { code: 200, message: "Index `whalo` not found.", error_code: "index_not_found", error_type: "invalid_request", error_link: "https://docs.meilisearch.com/errors#index_not_found" }, details: { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
+3 {uid: 3, batch_uid: 3, status: failed, error: ResponseError { code: 200, message: "Index `whalo` not found.", error_code: "index_not_found", error_type: "invalid_request", error_link: "https://docs.meilisearch.com/errors#index_not_found" }, details: { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []
@@ -54,8 +54,8 @@ doggo: { number_of_documents: 0, field_distribution: {} }
 ### All Batches:
 0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
 1 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 1 of type `indexCreation` that cannot be batched with any other task.", }
-2 {uid: 2, details: {"swaps":[{"indexes":["catto","doggo"]}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, stop reason: "created batch containing only task with id 2 of type `indexSwap` that cannot be batched with any other task.", }
-3 {uid: 3, details: {"swaps":[{"indexes":["catto","whalo"]}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, stop reason: "created batch containing only task with id 3 of type `indexSwap` that cannot be batched with any other task.", }
+2 {uid: 2, details: {"swaps":[{"indexes":["catto","doggo"],"rename":false}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, stop reason: "created batch containing only task with id 2 of type `indexSwap` that cannot be batched with any other task.", }
+3 {uid: 3, details: {"swaps":[{"indexes":["catto","whalo"],"rename":false}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, stop reason: "created batch containing only task with id 3 of type `indexSwap` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### Batch to tasks mapping:
 0 [0,]

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_special_rules/start.snap

@@ -1,16 +1,15 @@
 ---
 source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
+3 {uid: 3, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,3,]

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_canceled_by/start.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/queue/tasks_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
 3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
 ----------------------------------------------------------------------
 ### Status:
@@ -49,7 +49,7 @@ catto: { number_of_documents: 0, field_distribution: {} }
 ----------------------------------------------------------------------
 ### All Batches:
 0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
-1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"]}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 3 of type `taskCancelation` that cannot be batched with any other task.", }
+1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"],"rename":false}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, stop reason: "created batch containing only task with id 3 of type `taskCancelation` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### Batch to tasks mapping:
 0 [0,]

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_from_and_limit/processed_all_tasks.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/queue/tasks_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
+2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_from_and_limit/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/queue/tasks_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_from_and_limit/registered_the_second_task.snap

@@ -1,14 +1,13 @@
 ---
 source: crates/index-scheduler/src/queue/tasks_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,]

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_from_and_limit/registered_the_third_task.snap

@@ -1,15 +1,14 @@
 ---
 source: crates/index-scheduler/src/queue/tasks_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("bone"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
+2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,]

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_simple/end.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/queue/tasks_test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued []

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_simple/start.snap

@@ -1,15 +1,14 @@
 ---
 source: crates/index-scheduler/src/queue/tasks_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, status: enqueued, details: { primary_key: Some("fish"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,]

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_special_rules/start.snap

@@ -1,16 +1,15 @@
 ---
 source: crates/index-scheduler/src/queue/tasks_test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
+2 {uid: 2, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo"), rename: false }] }}
+3 {uid: 3, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo"), rename: false }] }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,2,3,]

crates/index-scheduler/src/queue/snapshots/test.rs/register/everything_is_successfully_registered.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
 1 {uid: 1, status: enqueued, details: { received_documents: 12, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 12, allow_index_creation: true }}
 2 {uid: 2, status: enqueued, details: { received_documents: 50, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "catto", primary_key: None, method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000001, documents_count: 50, allow_index_creation: true }}
 3 {uid: 3, status: enqueued, details: { received_documents: 5000, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggo", primary_key: Some("bone"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000002, documents_count: 5000, allow_index_creation: true }}

crates/index-scheduler/src/queue/snapshots/test.rs/test_auto_deletion_of_tasks/after_the_second_task_deletion.snap

@@ -1,6 +1,5 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 [
   {
@@ -13,7 +12,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",

crates/index-scheduler/src/queue/snapshots/test.rs/test_auto_deletion_of_tasks/task_deletion_have_been_enqueued.snap

@@ -1,6 +1,5 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 [
   {
@@ -13,7 +12,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "succeeded",
@@ -39,7 +40,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "failed",
@@ -60,7 +63,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",
@@ -81,7 +86,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",

crates/index-scheduler/src/queue/snapshots/test.rs/test_auto_deletion_of_tasks/task_deletion_have_been_processed.snap

@@ -1,6 +1,5 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 [
   {
@@ -13,7 +12,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",
@@ -34,7 +35,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",

crates/index-scheduler/src/queue/snapshots/test.rs/test_auto_deletion_of_tasks/task_queue_is_full.snap

@@ -1,6 +1,5 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 [
   {
@@ -13,7 +12,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "succeeded",
@@ -39,7 +40,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "failed",
@@ -60,7 +63,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",
@@ -81,7 +86,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",

crates/index-scheduler/src/queue/snapshots/test.rs/test_disable_auto_deletion_of_tasks/task_deletion_have_not_been_enqueued.snap

@@ -1,6 +1,5 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 [
   {
@@ -13,7 +12,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "succeeded",
@@ -39,7 +40,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "failed",
@@ -60,7 +63,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",
@@ -81,7 +86,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",

crates/index-scheduler/src/queue/snapshots/test.rs/test_disable_auto_deletion_of_tasks/task_queue_is_full.snap

@@ -1,6 +1,5 @@
 ---
 source: crates/index-scheduler/src/queue/test.rs
-snapshot_kind: text
 ---
 [
   {
@@ -13,7 +12,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "succeeded",
@@ -39,7 +40,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "failed",
@@ -60,7 +63,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",
@@ -81,7 +86,9 @@ snapshot_kind: text
     "canceledBy": null,
     "details": {
       "IndexInfo": {
-        "primary_key": null
+        "primary_key": null,
+        "new_index_uid": null,
+        "old_index_uid": null
       }
     },
     "status": "enqueued",

crates/index-scheduler/src/queue/tasks_test.rs

@@ -304,11 +304,11 @@ fn query_tasks_special_rules() {
     let kind = index_creation_task("doggo", "sheep");
     let _task = index_scheduler.register(kind, None, false).unwrap();
     let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()) }],
+        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
     };
     let _task = index_scheduler.register(kind, None, false).unwrap();
     let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()) }],
+        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()), rename: false }],
     };
     let _task = index_scheduler.register(kind, None, false).unwrap();
 
@@ -399,7 +399,7 @@ fn query_tasks_canceled_by() {
     let kind = index_creation_task("doggo", "sheep");
     let _ = index_scheduler.register(kind, None, false).unwrap();
     let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()) }],
+        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()), rename: false }],
     };
     let _task = index_scheduler.register(kind, None, false).unwrap();
 

crates/index-scheduler/src/scheduler/autobatcher.rs

@@ -287,7 +287,7 @@ impl BatchKind {
         };
 
         match (self, autobatch_kind) {
-            // We don't batch any of these operations
+            // We don't batch any of these operations  
             (this, K::IndexCreation | K::IndexUpdate | K::IndexSwap | K::DocumentEdition) => Break((this, BatchStopReason::TaskCannotBeBatched { kind, id })),
             // We must not batch tasks that don't have the same index creation rights if the index doesn't already exists.
             (this, kind) if !index_already_exists && this.allow_index_creation() == Some(false) && kind.allow_index_creation() == Some(true) => {

crates/index-scheduler/src/scheduler/autobatcher_test.rs

@@ -75,7 +75,11 @@ fn idx_create() -> KindWithContent {
 }
 
 fn idx_update() -> KindWithContent {
-    KindWithContent::IndexUpdate { index_uid: String::from("doggo"), primary_key: None }
+    KindWithContent::IndexUpdate {
+        index_uid: String::from("doggo"),
+        primary_key: None,
+        new_index_uid: None,
+    }
 }
 
 fn idx_del() -> KindWithContent {
@@ -84,7 +88,10 @@ fn idx_del() -> KindWithContent {
 
 fn idx_swap() -> KindWithContent {
     KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: (String::from("doggo"), String::from("catto")) }],
+        swaps: vec![IndexSwap {
+            indexes: (String::from("doggo"), String::from("catto")),
+            rename: false,
+        }],
     }
 }
 

crates/index-scheduler/src/scheduler/create_batch.rs

@@ -38,6 +38,7 @@ pub(crate) enum Batch {
     IndexUpdate {
         index_uid: String,
         primary_key: Option<String>,
+        new_index_uid: Option<String>,
         task: Task,
     },
     IndexDeletion {
@@ -405,11 +406,13 @@ impl IndexScheduler {
                 let mut task =
                     self.queue.tasks.get_task(rtxn, id)?.ok_or(Error::CorruptedTaskQueue)?;
                 current_batch.processing(Some(&mut task));
-                let primary_key = match &task.kind {
-                    KindWithContent::IndexUpdate { primary_key, .. } => primary_key.clone(),
+                let (primary_key, new_index_uid) = match &task.kind {
+                    KindWithContent::IndexUpdate { primary_key, new_index_uid, .. } => {
+                        (primary_key.clone(), new_index_uid.clone())
+                    }
                     _ => unreachable!(),
                 };
-                Ok(Some(Batch::IndexUpdate { index_uid, primary_key, task }))
+                Ok(Some(Batch::IndexUpdate { index_uid, primary_key, new_index_uid, task }))
             }
             BatchKind::IndexDeletion { ids } => Ok(Some(Batch::IndexDeletion {
                 index_uid,

crates/index-scheduler/src/scheduler/mod.rs

@@ -238,7 +238,7 @@ impl IndexScheduler {
                 #[cfg(test)]
                 self.breakpoint(crate::test_utils::Breakpoint::ProcessBatchSucceeded);
 
-                let (task_progress, task_progress_obj) = AtomicTaskStep::new(tasks.len() as u64);
+                let (task_progress, task_progress_obj) = AtomicTaskStep::new(tasks.len() as u32);
                 progress.update_progress(task_progress_obj);
                 process_batch_info = info;
                 let mut success = 0;
@@ -317,7 +317,7 @@ impl IndexScheduler {
             Err(err) => {
                 #[cfg(test)]
                 self.breakpoint(crate::test_utils::Breakpoint::ProcessBatchFailed);
-                let (task_progress, task_progress_obj) = AtomicTaskStep::new(ids.len() as u64);
+                let (task_progress, task_progress_obj) = AtomicTaskStep::new(ids.len() as u32);
                 progress.update_progress(task_progress_obj);
 
                 if matches!(err, Error::DatabaseUpgrade(_)) {
@@ -446,8 +446,7 @@ impl IndexScheduler {
             Ok(())
         })?;
 
-        // We shouldn't crash the tick function if we can't send data to the webhook.
-        let _ = self.notify_webhook(&ids);
+        self.notify_webhooks(ids);
 
         #[cfg(test)]
         self.breakpoint(crate::test_utils::Breakpoint::AfterProcessing);

crates/index-scheduler/src/scheduler/process_batch.rs

@@ -10,6 +10,7 @@ use meilisearch_types::tasks::{Details, IndexSwap, Kind, KindWithContent, Status
 use meilisearch_types::versioning::{VERSION_MAJOR, VERSION_MINOR, VERSION_PATCH};
 use milli::update::Settings as MilliSettings;
 use roaring::RoaringBitmap;
+use time::OffsetDateTime;
 
 use super::create_batch::Batch;
 use crate::processing::{
@@ -224,24 +225,46 @@ impl IndexScheduler {
                 self.index_mapper.create_index(wtxn, &index_uid, None)?;
 
                 self.process_batch(
-                    Batch::IndexUpdate { index_uid, primary_key, task },
+                    Batch::IndexUpdate { index_uid, primary_key, new_index_uid: None, task },
                     current_batch,
                     progress,
                 )
             }
-            Batch::IndexUpdate { index_uid, primary_key, mut task } => {
+            Batch::IndexUpdate { index_uid, primary_key, new_index_uid, mut task } => {
                 progress.update_progress(UpdateIndexProgress::UpdatingTheIndex);
+
+                // Get the index (renamed or not)
                 let rtxn = self.env.read_txn()?;
                 let index = self.index_mapper.index(&rtxn, &index_uid)?;
+                let mut index_wtxn = index.write_txn()?;
 
-                if let Some(primary_key) = primary_key.clone() {
-                    let mut index_wtxn = index.write_txn()?;
+                // Handle rename if new_index_uid is provided
+                let final_index_uid = if let Some(new_uid) = &new_index_uid {
+                    if new_uid != &index_uid {
+                        index.set_updated_at(&mut index_wtxn, &OffsetDateTime::now_utc())?;
+
+                        let mut wtxn = self.env.write_txn()?;
+                        self.apply_index_swap(
+                            &mut wtxn, &progress, task.uid, &index_uid, new_uid, true,
+                        )?;
+                        wtxn.commit()?;
+
+                        new_uid.clone()
+                    } else {
+                        new_uid.clone()
+                    }
+                } else {
+                    index_uid.clone()
+                };
+
+                // Handle primary key update if provided
+                if let Some(ref primary_key) = primary_key {
                     let mut builder = MilliSettings::new(
                         &mut index_wtxn,
                         &index,
                         self.index_mapper.indexer_config(),
                     );
-                    builder.set_primary_key(primary_key);
+                    builder.set_primary_key(primary_key.clone());
                     let must_stop_processing = self.scheduler.must_stop_processing.clone();
 
                     builder
@@ -250,15 +273,20 @@ impl IndexScheduler {
                             &progress,
                             current_batch.embedder_stats.clone(),
                         )
-                        .map_err(|e| Error::from_milli(e, Some(index_uid.to_string())))?;
-                    index_wtxn.commit()?;
+                        .map_err(|e| Error::from_milli(e, Some(final_index_uid.to_string())))?;
                 }
 
+                index_wtxn.commit()?;
                 // drop rtxn before starting a new wtxn on the same db
                 rtxn.commit()?;
 
                 task.status = Status::Succeeded;
-                task.details = Some(Details::IndexInfo { primary_key });
+                task.details = Some(Details::IndexInfo {
+                    primary_key: primary_key.clone(),
+                    new_index_uid: new_index_uid.clone(),
+                    // we only display the old index uid if a rename happened => there is a new_index_uid
+                    old_index_uid: new_index_uid.map(|_| index_uid.clone()),
+                });
 
                 // if the update processed successfully, we're going to store the new
                 // stats of the index. Since the tasks have already been processed and
@@ -268,8 +296,8 @@ impl IndexScheduler {
                     let mut wtxn = self.env.write_txn()?;
                     let index_rtxn = index.read_txn()?;
                     let stats = crate::index_mapper::IndexStats::new(&index, &index_rtxn)
-                        .map_err(|e| Error::from_milli(e, Some(index_uid.clone())))?;
-                    self.index_mapper.store_stats_of(&mut wtxn, &index_uid, &stats)?;
+                        .map_err(|e| Error::from_milli(e, Some(final_index_uid.clone())))?;
+                    self.index_mapper.store_stats_of(&mut wtxn, &final_index_uid, &stats)?;
                     wtxn.commit()?;
                     Ok(())
                 }();
@@ -330,13 +358,18 @@ impl IndexScheduler {
                     unreachable!()
                 };
                 let mut not_found_indexes = BTreeSet::new();
-                for IndexSwap { indexes: (lhs, rhs) } in swaps {
-                    for index in [lhs, rhs] {
-                        let index_exists = self.index_mapper.index_exists(&wtxn, index)?;
-                        if !index_exists {
-                            not_found_indexes.insert(index);
-                        }
+                let mut found_indexes_but_should_not = BTreeSet::new();
+                for IndexSwap { indexes: (lhs, rhs), rename } in swaps {
+                    let index_exists = self.index_mapper.index_exists(&wtxn, lhs)?;
+                    if !index_exists {
+                        not_found_indexes.insert(lhs);
                     }
+                    let index_exists = self.index_mapper.index_exists(&wtxn, rhs)?;
+                    match (index_exists, rename) {
+                        (true, true) => found_indexes_but_should_not.insert((lhs, rhs)),
+                        (false, false) => not_found_indexes.insert(rhs),
+                        (true, false) | (false, true) => true, // random value we don't read it anyway
+                    };
                 }
                 if !not_found_indexes.is_empty() {
                     if not_found_indexes.len() == 1 {
@@ -349,12 +382,29 @@ impl IndexScheduler {
                         ));
                     }
                 }
+                if !found_indexes_but_should_not.is_empty() {
+                    if found_indexes_but_should_not.len() == 1 {
+                        let (lhs, rhs) = found_indexes_but_should_not
+                            .into_iter()
+                            .next()
+                            .map(|(lhs, rhs)| (lhs.clone(), rhs.clone()))
+                            .unwrap();
+                        return Err(Error::SwapIndexFoundDuringRename(lhs, rhs));
+                    } else {
+                        return Err(Error::SwapIndexesFoundDuringRename(
+                            found_indexes_but_should_not
+                                .into_iter()
+                                .map(|(_, rhs)| rhs.to_string())
+                                .collect(),
+                        ));
+                    }
+                }
                 progress.update_progress(SwappingTheIndexes::SwappingTheIndexes);
                 for (step, swap) in swaps.iter().enumerate() {
                     progress.update_progress(VariableNameStep::<SwappingTheIndexes>::new(
                         format!("swapping index {} and {}", swap.indexes.0, swap.indexes.1),
-                        step as u64,
-                        swaps.len() as u64,
+                        step as u32,
+                        swaps.len() as u32,
                     ));
                     self.apply_index_swap(
                         &mut wtxn,
@@ -362,6 +412,7 @@ impl IndexScheduler {
                         task.uid,
                         &swap.indexes.0,
                         &swap.indexes.1,
+                        swap.rename,
                     )?;
                 }
                 wtxn.commit()?;
@@ -451,6 +502,7 @@ impl IndexScheduler {
         task_id: u32,
         lhs: &str,
         rhs: &str,
+        rename: bool,
     ) -> Result<()> {
         progress.update_progress(InnerSwappingTwoIndexes::RetrieveTheTasks);
         // 1. Verify that both lhs and rhs are existing indexes
@@ -458,21 +510,28 @@ impl IndexScheduler {
         if !index_lhs_exists {
             return Err(Error::IndexNotFound(lhs.to_owned()));
         }
-        let index_rhs_exists = self.index_mapper.index_exists(wtxn, rhs)?;
-        if !index_rhs_exists {
-            return Err(Error::IndexNotFound(rhs.to_owned()));
+        if !rename {
+            let index_rhs_exists = self.index_mapper.index_exists(wtxn, rhs)?;
+            if !index_rhs_exists {
+                return Err(Error::IndexNotFound(rhs.to_owned()));
+            }
         }
 
         // 2. Get the task set for index = name that appeared before the index swap task
         let mut index_lhs_task_ids = self.queue.tasks.index_tasks(wtxn, lhs)?;
         index_lhs_task_ids.remove_range(task_id..);
-        let mut index_rhs_task_ids = self.queue.tasks.index_tasks(wtxn, rhs)?;
-        index_rhs_task_ids.remove_range(task_id..);
+        let index_rhs_task_ids = if rename {
+            let mut index_rhs_task_ids = self.queue.tasks.index_tasks(wtxn, rhs)?;
+            index_rhs_task_ids.remove_range(task_id..);
+            index_rhs_task_ids
+        } else {
+            RoaringBitmap::new()
+        };
 
         // 3. before_name -> new_name in the task's KindWithContent
         progress.update_progress(InnerSwappingTwoIndexes::UpdateTheTasks);
         let tasks_to_update = &index_lhs_task_ids | &index_rhs_task_ids;
-        let (atomic, task_progress) = AtomicTaskStep::new(tasks_to_update.len() as u64);
+        let (atomic, task_progress) = AtomicTaskStep::new(tasks_to_update.len() as u32);
         progress.update_progress(task_progress);
 
         for task_id in tasks_to_update {
@@ -496,7 +555,11 @@ impl IndexScheduler {
         })?;
 
         // 6. Swap in the index mapper
-        self.index_mapper.swap(wtxn, lhs, rhs)?;
+        if rename {
+            self.index_mapper.rename(wtxn, lhs, rhs)?;
+        } else {
+            self.index_mapper.swap(wtxn, lhs, rhs)?;
+        }
 
         Ok(())
     }
@@ -529,7 +592,7 @@ impl IndexScheduler {
         // The tasks that have been removed *per batches*.
         let mut affected_batches: HashMap<BatchId, RoaringBitmap> = HashMap::new();
 
-        let (atomic_progress, task_progress) = AtomicTaskStep::new(to_delete_tasks.len() as u64);
+        let (atomic_progress, task_progress) = AtomicTaskStep::new(to_delete_tasks.len() as u32);
         progress.update_progress(task_progress);
         for task_id in to_delete_tasks.iter() {
             let task =
@@ -575,7 +638,7 @@ impl IndexScheduler {
 
         progress.update_progress(TaskDeletionProgress::DeletingTasksMetadata);
         let (atomic_progress, task_progress) = AtomicTaskStep::new(
-            (affected_indexes.len() + affected_statuses.len() + affected_kinds.len()) as u64,
+            (affected_indexes.len() + affected_statuses.len() + affected_kinds.len()) as u32,
         );
         progress.update_progress(task_progress);
         for index in affected_indexes.iter() {
@@ -594,7 +657,7 @@ impl IndexScheduler {
         }
 
         progress.update_progress(TaskDeletionProgress::DeletingTasks);
-        let (atomic_progress, task_progress) = AtomicTaskStep::new(to_delete_tasks.len() as u64);
+        let (atomic_progress, task_progress) = AtomicTaskStep::new(to_delete_tasks.len() as u32);
         progress.update_progress(task_progress);
         for task in to_delete_tasks.iter() {
             self.queue.tasks.all_tasks.delete(wtxn, &task)?;
@@ -611,7 +674,7 @@ impl IndexScheduler {
             }
         }
         progress.update_progress(TaskDeletionProgress::DeletingBatches);
-        let (atomic_progress, batch_progress) = AtomicBatchStep::new(affected_batches.len() as u64);
+        let (atomic_progress, batch_progress) = AtomicBatchStep::new(affected_batches.len() as u32);
         progress.update_progress(batch_progress);
         for (batch_id, to_delete_tasks) in affected_batches {
             if let Some(mut tasks) = self.queue.batch_to_tasks_mapping.get(wtxn, &batch_id)? {
@@ -786,7 +849,7 @@ impl IndexScheduler {
         }
 
         // 3. We now have a list of tasks to cancel, cancel them
-        let (task_progress, progress_obj) = AtomicTaskStep::new(tasks_to_cancel.len() as u64);
+        let (task_progress, progress_obj) = AtomicTaskStep::new(tasks_to_cancel.len() as u32);
         progress.update_progress(progress_obj);
 
         let mut tasks = self.queue.tasks.get_existing_tasks(
@@ -797,7 +860,7 @@ impl IndexScheduler {
         )?;
 
         progress.update_progress(TaskCancelationProgress::UpdatingTasks);
-        let (task_progress, progress_obj) = AtomicTaskStep::new(tasks_to_cancel.len() as u64);
+        let (task_progress, progress_obj) = AtomicTaskStep::new(tasks_to_cancel.len() as u32);
         progress.update_progress(progress_obj);
         for task in tasks.iter_mut() {
             task.status = Status::Canceled;

crates/index-scheduler/src/scheduler/process_dump_creation.rs

@@ -5,6 +5,7 @@ use std::sync::atomic::Ordering;
 
 use dump::IndexMetadata;
 use meilisearch_types::milli::constants::RESERVED_VECTORS_FIELD_NAME;
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::milli::vector::parsed_vectors::{ExplicitVectors, VectorOrArrayOfVectors};
 use meilisearch_types::milli::{self};
@@ -57,7 +58,7 @@ impl IndexScheduler {
         let mut dump_tasks = dump.create_tasks_queue()?;
 
         let (atomic, update_task_progress) =
-            AtomicTaskStep::new(self.queue.tasks.all_tasks.len(&rtxn)? as u64);
+            AtomicTaskStep::new(self.queue.tasks.all_tasks.len(&rtxn)? as u32);
         progress.update_progress(update_task_progress);
 
         for ret in self.queue.tasks.all_tasks.iter(&rtxn)? {
@@ -119,7 +120,7 @@ impl IndexScheduler {
         let mut dump_batches = dump.create_batches_queue()?;
 
         let (atomic_batch_progress, update_batch_progress) =
-            AtomicBatchStep::new(self.queue.batches.all_batches.len(&rtxn)? as u64);
+            AtomicBatchStep::new(self.queue.batches.all_batches.len(&rtxn)? as u32);
         progress.update_progress(update_batch_progress);
 
         for ret in self.queue.batches.all_batches.iter(&rtxn)? {
@@ -149,7 +150,7 @@ impl IndexScheduler {
 
         // 5. Dump the indexes
         progress.update_progress(DumpCreationProgress::DumpTheIndexes);
-        let nb_indexes = self.index_mapper.index_mapping.len(&rtxn)? as u64;
+        let nb_indexes = self.index_mapper.index_mapping.len(&rtxn)? as u32;
         let mut count = 0;
         let () = self.index_mapper.try_for_each_index(&rtxn, |uid, index| -> Result<()> {
             progress.update_progress(VariableNameStep::<DumpCreationProgress>::new(
@@ -178,7 +179,7 @@ impl IndexScheduler {
             let nb_documents = index
                 .number_of_documents(&rtxn)
                 .map_err(|e| Error::from_milli(e, Some(uid.to_string())))?
-                as u64;
+                as u32;
             let (atomic, update_document_progress) = AtomicDocumentStep::new(nb_documents);
             progress.update_progress(update_document_progress);
             let documents = index
@@ -227,12 +228,21 @@ impl IndexScheduler {
                         return Err(Error::from_milli(user_err, Some(uid.to_string())));
                     };
 
-                    for (embedder_name, (embeddings, regenerate)) in embeddings {
+                    for (
+                        embedder_name,
+                        EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
+                    ) in embeddings
+                    {
                         let embeddings = ExplicitVectors {
                             embeddings: Some(VectorOrArrayOfVectors::from_array_of_vectors(
                                 embeddings,
                             )),
-                            regenerate,
+                            regenerate: regenerate &&
+                            // Meilisearch does not handle well dumps with fragments, because as the fragments
+                            // are marked as user-provided,
+                            // all embeddings would be regenerated on any settings change or document update.
+                            // To prevent this, we mark embeddings has non regenerate in this case.
+                            !has_fragments,
                         };
                         vectors.insert(embedder_name, serde_json::to_value(embeddings).unwrap());
                     }
@@ -260,6 +270,11 @@ impl IndexScheduler {
         let network = self.network();
         dump.create_network(network)?;
 
+        // 7. Dump the webhooks
+        progress.update_progress(DumpCreationProgress::DumpTheWebhooks);
+        let webhooks = self.webhooks_dump_view();
+        dump.create_webhooks(webhooks)?;
+
         let dump_uid = started_at.format(format_description!(
                     "[year repr:full][month repr:numerical][day padding:zero]-[hour padding:zero][minute padding:zero][second padding:zero][subsecond digits:3]"
                 )).unwrap();

crates/index-scheduler/src/scheduler/process_export.rs

@@ -9,6 +9,7 @@ use flate2::write::GzEncoder;
 use flate2::Compression;
 use meilisearch_types::index_uid_pattern::IndexUidPattern;
 use meilisearch_types::milli::constants::RESERVED_VECTORS_FIELD_NAME;
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::milli::update::{request_threads, Setting};
 use meilisearch_types::milli::vector::parsed_vectors::{ExplicitVectors, VectorOrArrayOfVectors};
@@ -55,20 +56,21 @@ impl IndexScheduler {
 
             progress.update_progress(VariableNameStep::<ExportIndex>::new(
                 format!("Exporting index `{uid}`"),
-                i as u64,
-                indexes.len() as u64,
+                i as u32,
+                indexes.len() as u32,
             ));
 
             let ExportIndexSettings { filter, override_settings } = export_settings;
             let index = self.index(uid)?;
             let index_rtxn = index.read_txn()?;
+            let bearer = api_key.map(|api_key| format!("Bearer {api_key}"));
 
             // First, check if the index already exists
             let url = format!("{base_url}/indexes/{uid}");
             let response = retry(&must_stop_processing, || {
                 let mut request = agent.get(&url);
-                if let Some(api_key) = api_key {
-                    request = request.set("Authorization", &format!("Bearer {api_key}"));
+                if let Some(bearer) = &bearer {
+                    request = request.set("Authorization", bearer);
                 }
 
                 request.send_bytes(Default::default()).map_err(into_backoff_error)
@@ -90,8 +92,8 @@ impl IndexScheduler {
                 let url = format!("{base_url}/indexes");
                 retry(&must_stop_processing, || {
                     let mut request = agent.post(&url);
-                    if let Some(api_key) = api_key {
-                        request = request.set("Authorization", &format!("Bearer {api_key}"));
+                    if let Some(bearer) = &bearer {
+                        request = request.set("Authorization", bearer);
                     }
                     let index_param = json!({ "uid": uid, "primaryKey": primary_key });
                     request.send_json(&index_param).map_err(into_backoff_error)
@@ -103,8 +105,8 @@ impl IndexScheduler {
                 let url = format!("{base_url}/indexes/{uid}");
                 retry(&must_stop_processing, || {
                     let mut request = agent.patch(&url);
-                    if let Some(api_key) = api_key {
-                        request = request.set("Authorization", &format!("Bearer {api_key}"));
+                    if let Some(bearer) = &bearer {
+                        request = request.set("Authorization", bearer);
                     }
                     let index_param = json!({ "primaryKey": primary_key });
                     request.send_json(&index_param).map_err(into_backoff_error)
@@ -122,7 +124,6 @@ impl IndexScheduler {
                 }
                 // Retry logic for sending settings
                 let url = format!("{base_url}/indexes/{uid}/settings");
-                let bearer = api_key.map(|api_key| format!("Bearer {api_key}"));
                 retry(&must_stop_processing, || {
                     let mut request = agent.patch(&url);
                     if let Some(bearer) = bearer.as_ref() {
@@ -155,7 +156,7 @@ impl IndexScheduler {
             // spawn many threads to process the documents
             drop(index_rtxn);
 
-            let total_documents = universe.len();
+            let total_documents = universe.len() as u32;
             let (step, progress_step) = AtomicDocumentStep::new(total_documents);
             progress.update_progress(progress_step);
 
@@ -163,14 +164,14 @@ impl IndexScheduler {
                 IndexUidPattern::new_unchecked(uid.clone()),
                 DetailsExportIndexSettings {
                     settings: (*export_settings).clone(),
-                    matched_documents: Some(total_documents),
+                    matched_documents: Some(total_documents as u64),
                 },
             );
 
-            let limit = payload_size.map(|ps| ps.as_u64() as usize).unwrap_or(50 * 1024 * 1024); // defaults to 50 MiB
+            let limit = payload_size.map(|ps| ps.as_u64() as usize).unwrap_or(20 * 1024 * 1024); // defaults to 20 MiB
             let documents_url = format!("{base_url}/indexes/{uid}/documents");
 
-            request_threads()
+            let results = request_threads()
                 .broadcast(|ctx| {
                     let index_rtxn = index
                         .read_txn()
@@ -229,12 +230,21 @@ impl IndexScheduler {
                                 ));
                             };
 
-                            for (embedder_name, (embeddings, regenerate)) in embeddings {
+                            for (
+                                embedder_name,
+                                EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
+                            ) in embeddings
+                            {
                                 let embeddings = ExplicitVectors {
                                     embeddings: Some(
                                         VectorOrArrayOfVectors::from_array_of_vectors(embeddings),
                                     ),
-                                    regenerate,
+                                    regenerate: regenerate &&
+                                    // Meilisearch does not handle well dumps with fragments, because as the fragments
+                                    // are marked as user-provided,
+                                    // all embeddings would be regenerated on any settings change or document update.
+                                    // To prevent this, we mark embeddings has non regenerate in this case.
+                                    !has_fragments,
                                 };
                                 vectors.insert(
                                     embedder_name,
@@ -265,9 +275,8 @@ impl IndexScheduler {
                                 let mut request = agent.post(&documents_url);
                                 request = request.set("Content-Type", "application/x-ndjson");
                                 request = request.set("Content-Encoding", "gzip");
-                                if let Some(api_key) = api_key {
-                                    request = request
-                                        .set("Authorization", &(format!("Bearer {api_key}")));
+                                if let Some(bearer) = &bearer {
+                                    request = request.set("Authorization", bearer);
                                 }
                                 request.send_bytes(&compressed_buffer).map_err(into_backoff_error)
                             })?;
@@ -276,7 +285,7 @@ impl IndexScheduler {
                         }
                         buffer.extend_from_slice(&tmp_buffer);
 
-                        if i % 100 == 0 {
+                        if i > 0 && i % 100 == 0 {
                             step.fetch_add(100, atomic::Ordering::Relaxed);
                         }
                     }
@@ -284,8 +293,8 @@ impl IndexScheduler {
                     retry(&must_stop_processing, || {
                         let mut request = agent.post(&documents_url);
                         request = request.set("Content-Type", "application/x-ndjson");
-                        if let Some(api_key) = api_key {
-                            request = request.set("Authorization", &(format!("Bearer {api_key}")));
+                        if let Some(bearer) = &bearer {
+                            request = request.set("Authorization", bearer);
                         }
                         request.send_bytes(&buffer).map_err(into_backoff_error)
                     })?;
@@ -298,6 +307,9 @@ impl IndexScheduler {
                         Some(uid.to_string()),
                     )
                 })?;
+            for result in results {
+                result?;
+            }
 
             step.store(total_documents, atomic::Ordering::Relaxed);
         }

crates/index-scheduler/src/scheduler/process_snapshot_creation.rs

@@ -7,9 +7,73 @@ use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::tasks::{Status, Task};
 use meilisearch_types::{compression, VERSION_FILE_NAME};
 
+use crate::heed::EnvOpenOptions;
 use crate::processing::{AtomicUpdateFileStep, SnapshotCreationProgress};
+use crate::queue::TaskQueue;
 use crate::{Error, IndexScheduler, Result};
 
+/// # Safety
+///
+/// See [`EnvOpenOptions::open`].
+unsafe fn remove_tasks(
+    tasks: &[Task],
+    dst: &std::path::Path,
+    index_base_map_size: usize,
+) -> Result<()> {
+    let env_options = EnvOpenOptions::new();
+    let mut env_options = env_options.read_txn_without_tls();
+    let env = env_options.max_dbs(TaskQueue::nb_db()).map_size(index_base_map_size).open(dst)?;
+    let mut wtxn = env.write_txn()?;
+    let task_queue = TaskQueue::new(&env, &mut wtxn)?;
+
+    // Destructuring to ensure the code below gets updated if a database gets added in the future.
+    let TaskQueue {
+        all_tasks,
+        status,
+        kind,
+        index_tasks: _, // snapshot creation tasks are not index tasks
+        canceled_by,
+        enqueued_at,
+        started_at,
+        finished_at,
+    } = task_queue;
+
+    for task in tasks {
+        all_tasks.delete(&mut wtxn, &task.uid)?;
+
+        let mut tasks = status.get(&wtxn, &task.status)?.unwrap_or_default();
+        tasks.remove(task.uid);
+        status.put(&mut wtxn, &task.status, &tasks)?;
+
+        let mut tasks = kind.get(&wtxn, &task.kind.as_kind())?.unwrap_or_default();
+        tasks.remove(task.uid);
+        kind.put(&mut wtxn, &task.kind.as_kind(), &tasks)?;
+
+        canceled_by.delete(&mut wtxn, &task.uid)?;
+
+        let timestamp = task.enqueued_at.unix_timestamp_nanos();
+        let mut tasks = enqueued_at.get(&wtxn, &timestamp)?.unwrap_or_default();
+        tasks.remove(task.uid);
+        enqueued_at.put(&mut wtxn, &timestamp, &tasks)?;
+
+        if let Some(task_started_at) = task.started_at {
+            let timestamp = task_started_at.unix_timestamp_nanos();
+            let mut tasks = started_at.get(&wtxn, &timestamp)?.unwrap_or_default();
+            tasks.remove(task.uid);
+            started_at.put(&mut wtxn, &timestamp, &tasks)?;
+        }
+
+        if let Some(task_finished_at) = task.finished_at {
+            let timestamp = task_finished_at.unix_timestamp_nanos();
+            let mut tasks = finished_at.get(&wtxn, &timestamp)?.unwrap_or_default();
+            tasks.remove(task.uid);
+            finished_at.put(&mut wtxn, &timestamp, &tasks)?;
+        }
+    }
+    wtxn.commit()?;
+    Ok(())
+}
+
 impl IndexScheduler {
     pub(super) fn process_snapshot(
         &self,
@@ -48,17 +112,29 @@ impl IndexScheduler {
         };
         self.env.copy_to_path(dst.join("data.mdb"), compaction_option)?;
 
-        // 2.2 Create a read transaction on the index-scheduler
+        // 2.2 Remove the current snapshot tasks
+        //
+        // This is done to ensure that the tasks are not processed again when the snapshot is imported
+        //
+        // # Safety
+        //
+        // This is safe because we open the env file we just created in a temporary directory.
+        // We are sure it's not being used by any other process nor thread.
+        unsafe {
+            remove_tasks(&tasks, &dst, self.index_mapper.index_base_map_size)?;
+        }
+
+        // 2.3 Create a read transaction on the index-scheduler
         let rtxn = self.env.read_txn()?;
 
-        // 2.3 Create the update files directory
+        // 2.4 Create the update files directory
         let update_files_dir = temp_snapshot_dir.path().join("update_files");
         fs::create_dir_all(&update_files_dir)?;
 
-        // 2.4 Only copy the update files of the enqueued tasks
+        // 2.5 Only copy the update files of the enqueued tasks
         progress.update_progress(SnapshotCreationProgress::SnapshotTheUpdateFiles);
         let enqueued = self.queue.tasks.get_status(&rtxn, Status::Enqueued)?;
-        let (atomic, update_file_progress) = AtomicUpdateFileStep::new(enqueued.len() as u64);
+        let (atomic, update_file_progress) = AtomicUpdateFileStep::new(enqueued.len() as u32);
         progress.update_progress(update_file_progress);
         for task_id in enqueued {
             let task =
@@ -74,12 +150,12 @@ impl IndexScheduler {
         // 3. Snapshot every indexes
         progress.update_progress(SnapshotCreationProgress::SnapshotTheIndexes);
         let index_mapping = self.index_mapper.index_mapping;
-        let nb_indexes = index_mapping.len(&rtxn)? as u64;
+        let nb_indexes = index_mapping.len(&rtxn)? as u32;
 
         for (i, result) in index_mapping.iter(&rtxn)?.enumerate() {
             let (name, uuid) = result?;
             progress.update_progress(VariableNameStep::<SnapshotCreationProgress>::new(
-                name, i as u64, nb_indexes,
+                name, i as u32, nb_indexes,
             ));
             let index = self.index_mapper.index(&rtxn, name)?;
             let dst = temp_snapshot_dir.path().join("indexes").join(uuid.to_string());

crates/index-scheduler/src/scheduler/process_upgrade.rs

@@ -22,8 +22,8 @@ impl IndexScheduler {
             }
             progress.update_progress(VariableNameStep::<UpgradeIndex>::new(
                 format!("Upgrading index `{uid}`"),
-                i as u64,
-                indexes.len() as u64,
+                i as u32,
+                indexes.len() as u32,
             ));
             let index = self.index(uid)?;
             let mut index_wtxn = index.write_txn()?;
@@ -65,8 +65,8 @@ impl IndexScheduler {
         for (i, uid) in indexes.iter().enumerate() {
             progress.update_progress(VariableNameStep::<UpgradeIndex>::new(
                 format!("Rollbacking index `{uid}`"),
-                i as u64,
-                indexes.len() as u64,
+                i as u32,
+                indexes.len() as u32,
             ));
             let index_schd_rtxn = self.env.read_txn()?;
 

crates/index-scheduler/src/scheduler/snapshots/test.rs/do_not_batch_task_of_different_indexes/all_tasks_processed.snap

@@ -6,9 +6,9 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "cattos", primary_key: None }}
-2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "girafos", primary_key: None }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "cattos", primary_key: None }}
+2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "girafos", primary_key: None }}
 3 {uid: 3, batch_uid: 3, status: succeeded, details: { deleted_documents: Some(0) }, kind: DocumentClear { index_uid: "doggos" }}
 4 {uid: 4, batch_uid: 4, status: succeeded, details: { deleted_documents: Some(0) }, kind: DocumentClear { index_uid: "cattos" }}
 5 {uid: 5, batch_uid: 5, status: succeeded, details: { deleted_documents: Some(0) }, kind: DocumentClear { index_uid: "girafos" }}

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/before_index_creation.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
 1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/both_task_succeeded.snap

@@ -6,7 +6,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
 1 {uid: 1, batch_uid: 1, status: succeeded, details: { received_documents: 1, indexed_documents: Some(0) }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
 2 {uid: 2, batch_uid: 1, status: succeeded, details: { deleted_documents: Some(0) }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+0 {uid: 0, status: enqueued, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/registered_the_second_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+0 {uid: 0, status: enqueued, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
 1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/document_addition_and_index_deletion/registered_the_third_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+0 {uid: 0, status: enqueued, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
 1 {uid: 1, status: enqueued, details: { received_documents: 1, indexed_documents: None }, kind: DocumentAdditionOrUpdate { index_uid: "doggos", primary_key: Some("id"), method: ReplaceDocuments, content_file: 00000000-0000-0000-0000-000000000000, documents_count: 1, allow_index_creation: true }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------

crates/index-scheduler/src/scheduler/snapshots/test.rs/insert_task_while_another_task_is_processing/after_batch_creation.snap

@@ -7,7 +7,7 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 0, details: {"primaryKey":"id"}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"indexCreation":1},"indexUids":{"index_a":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("id") }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/insert_task_while_another_task_is_processing/registered_the_first_task.snap

@@ -1,13 +1,12 @@
 ---
 source: crates/index-scheduler/src/scheduler/test.rs
-snapshot_kind: text
 ---
 ### Autobatching Enabled = true
 ### Processing batch None:
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("id") }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/insert_task_while_another_task_is_processing/registered_the_second_task.snap

@@ -7,8 +7,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 0, details: {"primaryKey":"id"}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"indexCreation":1},"indexUids":{"index_a":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("id") }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("id") }, kind: IndexCreation { index_uid: "index_b", primary_key: Some("id") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "index_b", primary_key: Some("id") }}
 ----------------------------------------------------------------------
 ### Status:
 enqueued [0,1,]

crates/index-scheduler/src/scheduler/snapshots/test.rs/insert_task_while_another_task_is_processing/registered_the_third_task.snap

@@ -7,8 +7,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 {uid: 0, details: {"primaryKey":"id"}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"indexCreation":1},"indexUids":{"index_a":1}}, stop reason: "created batch containing only task with id 0 of type `indexCreation` that cannot be batched with any other task.", }
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("id") }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("id") }, kind: IndexCreation { index_uid: "index_b", primary_key: Some("id") }}
+0 {uid: 0, status: enqueued, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "index_a", primary_key: Some("id") }}
+1 {uid: 1, status: enqueued, details: { primary_key: Some("id"), old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "index_b", primary_key: Some("id") }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "index_a" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/process_tasks_inserted_without_new_signal/processed_the_first_task.snap

@@ -6,8 +6,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, status: enqueued, details: { primary_key: None }, kind: IndexCreation { index_uid: "cattos", primary_key: None }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+1 {uid: 1, status: enqueued, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "cattos", primary_key: None }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status:

crates/index-scheduler/src/scheduler/snapshots/test.rs/process_tasks_inserted_without_new_signal/processed_the_second_task.snap

@@ -6,8 +6,8 @@ source: crates/index-scheduler/src/scheduler/test.rs
 []
 ----------------------------------------------------------------------
 ### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: None }, kind: IndexCreation { index_uid: "cattos", primary_key: None }}
+0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "doggos", primary_key: None }}
+1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: None, old_new_uid: None, new_index_uid: None }, kind: IndexCreation { index_uid: "cattos", primary_key: None }}
 2 {uid: 2, status: enqueued, details: { deleted_documents: None }, kind: IndexDeletion { index_uid: "doggos" }}
 ----------------------------------------------------------------------
 ### Status: