Merge branch 'main' into starts-with-optim

.github/ISSUE_TEMPLATE/new_feature_issue.md

@@ -1,28 +1,26 @@
 ---
-name: New sprint issue
-about: ⚠️ Should only be used by the engine team ⚠️
+name: New feature issue
+about: ⚠️ Should only be used by the internal Meili team ⚠️
 title: ''
-labels: 'missing usage in PRD, impacts docs'
+labels: 'impacts docs, impacts integrations'
 assignees: ''
 
 ---
 
 Related product team resources: [PRD]() (_internal only_)
-Related product discussion:
-
-## Motivation
-
-<!---Copy/paste the information in PRD or briefly detail the product motivation. Ask product team if any hesitation.-->
 
 ## Usage
 
 <!---Link to the public part of the PRD, or to the related product discussion for experimental features-->
 
+TBD
+
 ## TODO
 
 <!---If necessary, create a list with technical/product steps-->
 
 ### Are you modifying a database?
+
 - [ ] If not, add the `no db change` label to your PR, and you're good to merge.
 - [ ] If yes, add the `db change` label to your PR. You'll receive a message explaining you what to do.
 
@@ -54,5 +52,5 @@ Related product discussion:
 
 ## Impacted teams
 
-<!---Ping the related teams. Ask for the engine manager if any hesitation-->
-<!---@meilisearch/docs-team when there is any API change, e.g. settings addition-->
+<!---Ping the related teams. Ask on Slack if any hesitation-->
+<!---@meilisearch/docs-team and @meilisearch/integration-team when there is any API change, e.g. settings addition-->

.github/pull_request_template.md

@@ -0,0 +1,16 @@
+## Related issue
+
+Fixes #...
+
+## Requirements
+
+⚠️ Ensure the following requirements before merging ⚠️
+- [] Automated tests have been added.
+- [] If some tests cannot be automated, manual rigorous tests should be applied.
+- [] ⚠️ If there is any change in the DB: 
+    - [ ] Test that any impacted DB still works as expected after using `--experimental-dumpless-upgrade` on a DB created with the last released Meilisearch
+    - [ ] Test that during the upgrade, **search is still available** (artificially make the upgrade longer if needed)
+    - [ ] Set the `db change` label.
+- [] If necessary, the feature have been tested in the Cloud production environment (with [prototypes](./documentation/prototypes.md)) and the Cloud UI is ready.
+- [] If necessary, the [documentation](https://github.com/meilisearch/documentation) related to the implemented feature in the PR is ready.
+- [] If necessary, the [integrations](https://github.com/meilisearch/integration-guides) related to the implemented feature in the PR are ready.

.github/release-draft-template.yml

@@ -0,0 +1,23 @@
+name-template: 'v$RESOLVED_VERSION'
+tag-template: 'v$RESOLVED_VERSION'
+exclude-labels:
+  - 'skip changelog'
+version-resolver:
+  major:
+    labels:
+      - 'breaking-change'
+  minor:
+    labels:
+      - 'enhancement'
+  default: patch
+template: |
+  $CHANGES
+
+  Thanks again to $CONTRIBUTORS! 🎉
+no-changes-template: 'Changes are coming soon 😎'
+sort-direction: 'ascending'
+replacers:
+  - search: '/(?:and )?@dependabot-preview(?:\[bot\])?,?/g'
+    replace: ''
+  - search: '/(?:and )?@dependabot(?:\[bot\])?,?/g'
+    replace: ''

.github/templates/dependency-issue.md

@@ -0,0 +1,22 @@
+This issue is about updating Meilisearch dependencies:
+  - [ ] Update Meilisearch dependencies with the help of `cargo +nightly udeps --all-targets` (remove unused dependencies) and `cargo upgrade` (upgrade dependencies versions) - ⚠️ Some repositories may contain subdirectories (like heed, charabia, or deserr). Take care of updating these in the main crate as well. This won't be done automatically by `cargo upgrade`.
+    - [ ] [deserr](https://github.com/meilisearch/deserr)
+    - [ ] [charabia](https://github.com/meilisearch/charabia/)
+    - [ ] [heed](https://github.com/meilisearch/heed/)
+    - [ ] [roaring-rs](https://github.com/RoaringBitmap/roaring-rs/)
+    - [ ] [obkv](https://github.com/meilisearch/obkv)
+    - [ ] [grenad](https://github.com/meilisearch/grenad/)
+    - [ ] [arroy](https://github.com/meilisearch/arroy/)
+    - [ ] [segment](https://github.com/meilisearch/segment)
+    - [ ] [bumparaw-collections](https://github.com/meilisearch/bumparaw-collections)
+    - [ ] [bbqueue](https://github.com/meilisearch/bbqueue)
+    - [ ] Finally, [Meilisearch](https://github.com/meilisearch/MeiliSearch)
+  - [ ] If new Rust versions have been released, update the minimal Rust version in use at Meilisearch:
+    - [ ] in this [GitHub Action file](https://github.com/meilisearch/meilisearch/blob/main/.github/workflows/test-suite.yml), by changing the `toolchain` field of the `rustfmt` job to the latest available nightly (of the day before or the current day).
+    - [ ] in every [GitHub Action files](https://github.com/meilisearch/meilisearch/blob/main/.github/workflows), by changing all the `dtolnay/rust-toolchain@` references to use the latest stable version.
+    - [ ] in this [`rust-toolchain.toml`](https://github.com/meilisearch/meilisearch/blob/main/rust-toolchain.toml), by changing the `channel` field to the latest stable version.
+    - [ ] in the [Dockerfile](https://github.com/meilisearch/meilisearch/blob/main/Dockerfile), by changing the base image to `rust:<target_rust_version>-alpine<alpine_version>`. Check that the image exists on [Dockerhub](https://hub.docker.com/_/rust/tags?page=1&name=alpine). Also, build and run the image to check everything still works!
+
+⚠️ This issue should be prioritized to avoid any deprecation and vulnerability issues.
+
+The GitHub action dependencies are managed by [Dependabot](https://github.com/meilisearch/meilisearch/blob/main/.github/dependabot.yml), so no need to update them when solving this issue.

.github/workflows/check-valid-milestone.yml

@@ -1,100 +0,0 @@
-name: PR Milestone Check
-
-on:
-  pull_request:
-    types: [opened, reopened, edited, synchronize, milestoned, demilestoned]
-    branches:
-      - "main"
-      - "release-v*.*.*"
-
-jobs:
-  check-milestone:
-    name: Check PR Milestone
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v3
-
-      - name: Validate PR milestone
-        uses: actions/github-script@v7
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            // Get PR number directly from the event payload
-            const prNumber = context.payload.pull_request.number;
-
-            // Get PR details
-            const { data: prData } = await github.rest.pulls.get({
-              owner: 'meilisearch',
-              repo: 'meilisearch',
-              pull_number: prNumber
-            });
-
-            // Get base branch name
-            const baseBranch = prData.base.ref;
-            console.log(`Base branch: ${baseBranch}`);
-
-            // Get PR milestone
-            const prMilestone = prData.milestone;
-            if (!prMilestone) {
-              core.setFailed('PR must have a milestone assigned');
-              return;
-            }
-            console.log(`PR milestone: ${prMilestone.title}`);
-
-            // Validate milestone format: vx.y.z
-            const milestoneRegex = /^v\d+\.\d+\.\d+$/;
-            if (!milestoneRegex.test(prMilestone.title)) {
-              core.setFailed(`Milestone "${prMilestone.title}" does not follow the required format vx.y.z`);
-              return;
-            }
-
-            // For main branch PRs, check if the milestone is the highest one
-            if (baseBranch === 'main') {
-              // Get all milestones
-              const { data: milestones } = await github.rest.issues.listMilestones({
-                owner: 'meilisearch',
-                repo: 'meilisearch',
-                state: 'open',
-                sort: 'due_on',
-                direction: 'desc'
-              });
-
-              // Sort milestones by version number (vx.y.z)
-              const sortedMilestones = milestones
-                .filter(m => milestoneRegex.test(m.title))
-                .sort((a, b) => {
-                  const versionA = a.title.substring(1).split('.').map(Number);
-                  const versionB = b.title.substring(1).split('.').map(Number);
-
-                  // Compare major version
-                  if (versionA[0] !== versionB[0]) return versionB[0] - versionA[0];
-                  // Compare minor version
-                  if (versionA[1] !== versionB[1]) return versionB[1] - versionA[1];
-                  // Compare patch version
-                  return versionB[2] - versionA[2];
-                });
-
-              if (sortedMilestones.length === 0) {
-                core.setFailed('No valid milestones found in the repository. Please create at least one milestone with the format vx.y.z');
-                return;
-              }
-
-              const highestMilestone = sortedMilestones[0];
-              console.log(`Highest milestone: ${highestMilestone.title}`);
-
-              if (prMilestone.title !== highestMilestone.title) {
-                core.setFailed(`PRs targeting the main branch must use the highest milestone (${highestMilestone.title}), but this PR uses ${prMilestone.title}`);
-                return;
-              }
-            } else {
-              // For release branches, the milestone should match the branch version
-              const branchVersion = baseBranch.substring(8); // remove 'release-'
-              if (prMilestone.title !== branchVersion) {
-                core.setFailed(`PRs targeting release branch "${baseBranch}" must use the matching milestone "${branchVersion}", but this PR uses "${prMilestone.title}"`);
-                return;
-              }
-            }
-
-            console.log('PR milestone validation passed!');

.github/workflows/dependency-issue.yml

@@ -15,7 +15,7 @@ jobs:
     steps:
     - uses: actions/checkout@v3
     - name: Download the issue template
-      run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/dependency-issue.md > $ISSUE_TEMPLATE
+      run: curl -s https://raw.githubusercontent.com/meilisearch/meilisearch/main/.github/templates/dependency-issue.md > $ISSUE_TEMPLATE
     - name: Create issue
       run: |
         gh issue create \

.github/workflows/flaky-tests.yml

@@ -3,7 +3,7 @@ name: Look for flaky tests
 on:
   workflow_dispatch:
   schedule:
-    - cron: "0 12 * * FRI" # Every Friday at 12:00PM
+    - cron: '0 4 * * *' # Every day at 4:00AM
 
 jobs:
   flaky:

.github/workflows/milestone-workflow.yml

@@ -1,224 +0,0 @@
-name: Milestone's workflow
-
-# /!\ No git flow are handled here
-
-# For each Milestone created (not opened!), and if the release is NOT a patch release (only the patch changed)
-# - the roadmap issue is created, see https://github.com/meilisearch/engine-team/blob/main/issue-templates/roadmap-issue.md
-# - the changelog issue is created, see https://github.com/meilisearch/engine-team/blob/main/issue-templates/changelog-issue.md
-# - update the ruleset to add the current release version to the list of allowed versions and be able to use the merge queue.
-
-# For each Milestone closed
-# - the `release_version` label is created
-# - this label is applied to all issues/PRs in the Milestone
-
-on:
-  milestone:
-    types: [created, closed]
-
-env:
-  MILESTONE_VERSION: ${{ github.event.milestone.title }}
-  MILESTONE_URL: ${{ github.event.milestone.html_url }}
-  MILESTONE_DUE_ON: ${{ github.event.milestone.due_on }}
-  GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-
-jobs:
-  # -----------------
-  # MILESTONE CREATED
-  # -----------------
-
-  get-release-version:
-    if: github.event.action == 'created'
-    runs-on: ubuntu-latest
-    outputs:
-      is-patch: ${{ steps.check-patch.outputs.is-patch }}
-    steps:
-      - uses: actions/checkout@v3
-      - name: Check if this release is a patch release only
-        id: check-patch
-        run: |
-          echo version: $MILESTONE_VERSION
-          if [[ $MILESTONE_VERSION =~ ^v[0-9]+\.[0-9]+\.0$ ]]; then
-            echo 'This is NOT a patch release'
-            echo "is-patch=false" >> $GITHUB_OUTPUT
-          elif [[ $MILESTONE_VERSION =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
-            echo 'This is a patch release'
-            echo "is-patch=true" >> $GITHUB_OUTPUT
-          else
-            echo "Not a valid format of release, check the Milestone's title."
-            echo 'Should be vX.Y.Z'
-            exit 1
-          fi
-
-  create-roadmap-issue:
-    needs: get-release-version
-    # Create the roadmap issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/roadmap-issue.md > $ISSUE_TEMPLATE
-      - name: Replace all empty occurrences in the templates
-        run: |
-          # Replace all <<version>> occurrences
-          sed -i "s/<<version>>/$MILESTONE_VERSION/g" $ISSUE_TEMPLATE
-
-          # Replace all <<milestone_id>> occurrences
-          milestone_id=$(echo $MILESTONE_URL | cut -d '/' -f 7)
-          sed -i "s/<<milestone_id>>/$milestone_id/g" $ISSUE_TEMPLATE
-
-          # Replace release date if exists
-          if [[ ! -z $MILESTONE_DUE_ON ]]; then
-            date=$(echo $MILESTONE_DUE_ON | cut -d 'T' -f 1)
-            sed -i "s/Release date\: 20XX-XX-XX/Release date\: $date/g" $ISSUE_TEMPLATE
-          fi
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "$MILESTONE_VERSION ROADMAP" \
-            --label 'epic,impacts docs,impacts integrations,impacts cloud' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  create-changelog-issue:
-    needs: get-release-version
-    # Create the changelog issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/changelog-issue.md > $ISSUE_TEMPLATE
-      - name: Replace all empty occurrences in the templates
-        run: |
-          # Replace all <<version>> occurrences
-          sed -i "s/<<version>>/$MILESTONE_VERSION/g" $ISSUE_TEMPLATE
-
-          # Replace all <<milestone_id>> occurrences
-          milestone_id=$(echo $MILESTONE_URL | cut -d '/' -f 7)
-          sed -i "s/<<milestone_id>>/$milestone_id/g" $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Create release changelogs for $MILESTONE_VERSION" \
-            --label 'impacts docs,documentation' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION \
-            --assignee curquiza
-
-  create-update-version-issue:
-    needs: get-release-version
-    # Create the update-version issue even if the release is a patch release
-    if: github.event.action == 'created'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-version-issue.md > $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Update version in Cargo.toml for $MILESTONE_VERSION" \
-            --label 'maintenance' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  create-update-openapi-issue:
-    needs: get-release-version
-    # Create the openAPI issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-openapi-issue.md > $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Update Open API file for $MILESTONE_VERSION" \
-            --label 'maintenance' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  update-ruleset:
-    runs-on: ubuntu-latest
-    if: github.event.action == 'created'
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install jq
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y jq
-      - name: Update ruleset
-        env:
-          # gh api repos/meilisearch/meilisearch/rulesets --jq '.[] | {name: .name, id: .id}'
-          RULESET_ID: 4253297
-          BRANCH_NAME: ${{ github.event.inputs.branch_name }}
-        run: |
-          echo "RULESET_ID: ${{ env.RULESET_ID }}"
-          echo "BRANCH_NAME: ${{ env.BRANCH_NAME }}"
-
-          # Get current ruleset conditions
-          CONDITIONS=$(gh api repos/meilisearch/meilisearch/rulesets/${{ env.RULESET_ID }} --jq '{ conditions: .conditions }')
-
-          # Update the conditions by appending the milestone version
-          UPDATED_CONDITIONS=$(echo $CONDITIONS | jq '.conditions.ref_name.include += ["refs/heads/release-'${{ env.MILESTONE_VERSION }}'"]')
-
-          # Update the ruleset from stdin (-)
-          echo $UPDATED_CONDITIONS |
-            gh api repos/meilisearch/meilisearch/rulesets/${{ env.RULESET_ID }} \
-              --method PUT \
-              -H "Accept: application/vnd.github+json" \
-              -H "X-GitHub-Api-Version: 2022-11-28" \
-              --input -
-
-  # ----------------
-  # MILESTONE CLOSED
-  # ----------------
-
-  create-release-label:
-    if: github.event.action == 'closed'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Create the ${{ env.MILESTONE_VERSION }} label
-        run: |
-          label_description="PRs/issues solved in $MILESTONE_VERSION"
-          if [[ ! -z $MILESTONE_DUE_ON ]]; then
-            date=$(echo $MILESTONE_DUE_ON | cut -d 'T' -f 1)
-            label_description="$label_description released on $date"
-          fi
-
-          gh api repos/meilisearch/meilisearch/labels \
-            --method POST \
-            -H "Accept: application/vnd.github+json" \
-            -f name="$MILESTONE_VERSION" \
-            -f description="$label_description" \
-            -f color='ff5ba3'
-
-  labelize-all-milestone-content:
-    if: github.event.action == 'closed'
-    needs: create-release-label
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Add label ${{ env.MILESTONE_VERSION }} to all PRs in the Milestone
-        run: |
-          prs=$(gh pr list --search milestone:"$MILESTONE_VERSION" --limit 1000 --state all --json number --template '{{range .}}{{tablerow (printf "%v" .number)}}{{end}}')
-          for pr in $prs; do
-              gh pr edit $pr --add-label $MILESTONE_VERSION
-          done
-      - name: Add label ${{ env.MILESTONE_VERSION }} to all issues in the Milestone
-        run: |
-          issues=$(gh issue list --search milestone:"$MILESTONE_VERSION" --limit 1000 --state all --json number --template '{{range .}}{{tablerow (printf "%v" .number)}}{{end}}')
-          for issue in $issues; do
-              gh issue edit $issue --add-label $MILESTONE_VERSION
-          done

.github/workflows/publish-apt-brew-pkg.yml

@@ -32,7 +32,7 @@ jobs:
       - name: Build deb package
         run: cargo deb -p meilisearch -o target/debian/meilisearch.deb
       - name: Upload debian pkg to release
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/debian/meilisearch.deb

.github/workflows/publish-binaries.yml

@@ -51,7 +51,7 @@ jobs:
       # No need to upload binaries for dry run (cron)
       - name: Upload binaries to release
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/release/meilisearch
@@ -81,7 +81,7 @@ jobs:
       # No need to upload binaries for dry run (cron)
       - name: Upload binaries to release
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/release/${{ matrix.artifact_name }}
@@ -113,7 +113,7 @@ jobs:
       - name: Upload the binary to release
         # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch
@@ -178,7 +178,7 @@ jobs:
       - name: Upload the binary to release
         # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.11.1
+        uses: svenstaro/upload-release-action@2.11.2
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch

.github/workflows/publish-docker-images.yml

@@ -65,7 +65,7 @@ jobs:
         uses: docker/setup-buildx-action@v3
 
       - name: Install cosign
-        uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # tag=v3.8.2
+        uses: sigstore/cosign-installer@d58896d6a1865668819e1d91763c7751a165e159 # tag=v3.9.2
 
       - name: Login to Docker Hub
         uses: docker/login-action@v3

.github/workflows/release-drafter.yml

@@ -0,0 +1,20 @@
+name: Release Drafter
+
+permissions:
+  contents: read
+  pull-requests: write
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  update_release_draft:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: release-drafter/release-drafter@v6
+        with:
+          config-name: release-draft-template.yml
+        env:
+          GITHUB_TOKEN: ${{ secrets.RELEASE_DRAFTER_TOKEN }}

.github/workflows/sdks-tests.yml

@@ -9,7 +9,7 @@ on:
         required: false
         default: nightly
   schedule:
-    - cron: "0 6 * * MON" # Every Monday at 6:00AM
+    - cron: '0 6 * * *' # Every day at 6:00am
 
 env:
   MEILI_MASTER_KEY: 'masterKey'
@@ -114,7 +114,7 @@ jobs:
             dep ensure
           fi
       - name: Run integration tests
-        run: go test -v ./...
+        run: go test --race -v ./integration
 
   meilisearch-java-tests:
     needs: define-docker-image

.github/workflows/test-suite.yml

@@ -3,7 +3,7 @@ name: Test suite
 on:
   workflow_dispatch:
   schedule:
-    # Everyday at 5:00am
+    # Every day at 5:00am
     - cron: "0 5 * * *"
   pull_request:
   merge_group:

.gitignore

@@ -5,18 +5,24 @@
 **/*.json_lines
 **/*.rs.bk
 /*.mdb
-/data.ms
+/*.ms
 /snapshots
 /dumps
 /bench
 /_xtask_benchmark.ms
 /benchmarks
+.DS_Store
 
 # Snapshots
 ## ... large
 *.full.snap
-##  ... unreviewed
+## ... unreviewed
 *.snap.new
+## ... pending
+*.pending-snap
+
+# Tmp files
+.tmp*
 
 # Database snapshot
 crates/meilisearch/db.snapshot

CONTRIBUTING.md

@@ -106,7 +106,13 @@ Run `cargo xtask --help` from the root of the repository to find out what is ava
 #### Update the openAPI file if the API changed
 
 To update the openAPI file in the code, see [sprint_issue.md](https://github.com/meilisearch/meilisearch/blob/main/.github/ISSUE_TEMPLATE/sprint_issue.md#reminders-when-modifying-the-api).
-If you want to update the openAPI file on the [open-api repository](https://github.com/meilisearch/open-api), see [update-openapi-issue.md](https://github.com/meilisearch/engine-team/blob/main/issue-templates/update-openapi-issue.md).
+
+If you want to update the openAPI file on the [open-api repository](https://github.com/meilisearch/open-api):
+- Pull the latest version of the latest rc of Meilisearch `git checkout release-vX.Y.Z; git pull`
+- Starts Meilisearch with the `swagger` feature flag: `cargo run --features swagger`
+- On a browser, open the following URL: http://localhost:7700/scalar
+- Click the « Download openAPI file »
+- Open a PR replacing [this file](https://github.com/meilisearch/open-api/blob/main/open-api.json) with the one downloaded
 
 ### Logging
 
@@ -160,25 +166,37 @@ Some notes on GitHub PRs:
   The draft PRs are recommended when you want to show that you are working on something and make your work visible.
 - The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [GitHub Merge Queues](https://github.blog/news-insights/product-news/github-merge-queue-is-generally-available/) to automatically enforce this requirement without the PR author having to rebase manually.
 
-## Release Process (for internal team only)
-
-Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org/).
-
-### Automation to rebase and Merge the PRs
+## Merging PRs
 
 This project uses GitHub Merge Queues that helps us manage pull requests merging.
 
-### How to Publish a new Release
+Before merging a PR, the maintainer should ensure the following requirements are met
+- Automated tests have been added.
+- If some tests cannot be automated, manual rigorous tests should be applied.
+- ⚠️ If there is an change in the DB: it's mandatory to manually test the `--experimental-dumpless-upgrade` on a DB of the previous Meilisearch minor version (e.g. v1.13 for the v1.14 release).
+- If necessary, the feature have been tested in the Cloud production environment (with [prototypes](./documentation/prototypes.md)) and the Cloud UI is ready.
+- If necessary, the [documentation](https://github.com/meilisearch/documentation) related to the implemented feature in the PR is ready.
+- If necessary, the [integrations](https://github.com/meilisearch/integration-guides) related to the implemented feature in the PR are ready.
 
-The full Meilisearch release process is described in [this guide](https://github.com/meilisearch/engine-team/blob/main/resources/meilisearch-release.md). Please follow it carefully before doing any release.
+## Publish Process (for internal team only)
+
+Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org/).
+
+### How to publish a new release
+
+The full Meilisearch release process is described in [this guide](./documentation/release.md).
 
 ### How to publish a prototype
 
 Depending on the developed feature, you might need to provide a prototyped version of Meilisearch to make it easier to test by the users.
 
 This happens in two steps:
-- [Release the prototype](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#how-to-publish-a-prototype)
-- [Communicate about it](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#communication)
+- [Release the prototype](./documentation/prototypes.md#how-to-publish-a-prototype)
+- [Communicate about it](./documentation/prototypes.md#communication)
+
+### How to implement and publish an experimental feature
+
+Here is our [guidelines and process](./documentation/experimental-features.md) to implement and publish an experimental feature.
 
 ### Release assets
 

Cargo.lock

@@ -3775,6 +3775,7 @@ dependencies = [
  "meili-snap",
  "meilisearch-auth",
  "meilisearch-types",
+ "memmap2",
  "mimalloc",
  "mime",
  "mopa-maintained",
@@ -3908,9 +3909,9 @@ checksum = "32a282da65faaf38286cf3be983213fcf1d2e2a58700e808f83f4ea9a4804bc0"
 
 [[package]]
 name = "memmap2"
-version = "0.9.5"
+version = "0.9.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd3f7eed9d3848f8b98834af67102b720745c4ec028fcd0aa0239277e7de374f"
+checksum = "483758ad303d734cec05e5c12b41d7e93e6a6390c5e9dae6bdeb7c1259012d28"
 dependencies = [
  "libc",
  "stable_deref_trait",

README.md

@@ -119,6 +119,6 @@ Meilisearch is, and will always be, open-source! If you want to contribute to th
 
 Meilisearch releases and their associated binaries are available on the project's [releases page](https://github.com/meilisearch/meilisearch/releases).
 
-The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](https://github.com/meilisearch/engine-team/blob/main/resources/versioning-policy.md).
+The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](./documentation/versioning-policy.md).
 
 Differently from the binaries, crates in this repository are not currently available on [crates.io](https://crates.io/) and do not follow [SemVer conventions](https://semver.org).

crates/benchmarks/Cargo.toml

@@ -14,7 +14,7 @@ license.workspace = true
 anyhow = "1.0.98"
 bumpalo = "3.18.1"
 csv = "1.3.1"
-memmap2 = "0.9.5"
+memmap2 = "0.9.7"
 milli = { path = "../milli" }
 mimalloc = { version = "0.1.47", default-features = false }
 serde_json = { version = "1.0.140", features = ["preserve_order"] }
@@ -52,6 +52,10 @@ harness = false
 name = "indexing"
 harness = false
 
+[[bench]]
+name = "sort"
+harness = false
+
 [[bench]]
 name = "filter_starts_with"
 harness = false

crates/benchmarks/benches/sort.rs

@@ -0,0 +1,114 @@
+//! This benchmark module is used to compare the performance of sorting documents in /search VS /documents
+//!
+//! The tests/benchmarks were designed in the context of a query returning only 20 documents.
+
+mod datasets_paths;
+mod utils;
+
+use criterion::{criterion_group, criterion_main};
+use milli::update::Settings;
+use utils::Conf;
+
+#[cfg(not(windows))]
+#[global_allocator]
+static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
+
+fn base_conf(builder: &mut Settings) {
+    let displayed_fields =
+        ["geonameid", "name", "asciiname", "alternatenames", "_geo", "population"]
+            .iter()
+            .map(|s| s.to_string())
+            .collect();
+    builder.set_displayed_fields(displayed_fields);
+
+    let sortable_fields =
+        ["_geo", "name", "population", "elevation", "timezone", "modification-date"]
+            .iter()
+            .map(|s| s.to_string())
+            .collect();
+    builder.set_sortable_fields(sortable_fields);
+}
+
+#[rustfmt::skip]
+const BASE_CONF: Conf = Conf {
+    dataset: datasets_paths::SMOL_ALL_COUNTRIES,
+    dataset_format: "jsonl",
+    configure: base_conf,
+    primary_key: Some("geonameid"),
+    queries: &[""],
+    offsets: &[
+        Some((0, 20)), // The most common query in the real world
+        Some((0, 500)), // A query that ranges over many documents
+        Some((980, 20)), // The worst query that could happen in the real world
+        Some((800_000, 20)) // The worst query
+    ],
+    get_documents: true,
+    ..Conf::BASE
+};
+
+fn bench_sort(c: &mut criterion::Criterion) {
+    #[rustfmt::skip]
+    let confs = &[
+        utils::Conf {
+            group_name: "without sort",
+            sort: None,
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many different values",
+            sort: Some(vec!["name:asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many similar values",
+            sort: Some(vec!["timezone:desc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many similar then different values",
+            sort: Some(vec!["timezone:desc", "name:asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many different then similar values",
+            sort: Some(vec!["timezone:desc", "name:asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "geo sort",
+            sample_size: Some(10),
+            sort: Some(vec!["_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many similar values then geo sort",
+            sample_size: Some(50),
+            sort: Some(vec!["timezone:desc", "_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many different values then geo sort",
+            sample_size: Some(50),
+            sort: Some(vec!["name:desc", "_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..BASE_CONF
+        },
+
+        utils::Conf {
+            group_name: "sort on many fields",
+            sort: Some(vec!["population:asc", "name:asc", "elevation:asc", "timezone:asc"]),
+            ..BASE_CONF
+        },
+    ];
+
+    utils::run_benches(c, confs);
+}
+
+criterion_group!(benches, bench_sort);
+criterion_main!(benches);

crates/benchmarks/benches/utils.rs

@@ -9,6 +9,7 @@ use anyhow::Context;
 use bumpalo::Bump;
 use criterion::BenchmarkId;
 use memmap2::Mmap;
+use milli::documents::sort::recursive_sort;
 use milli::heed::EnvOpenOptions;
 use milli::progress::Progress;
 use milli::update::new::indexer;
@@ -35,6 +36,12 @@ pub struct Conf<'a> {
     pub configure: fn(&mut Settings),
     pub filter: Option<&'a str>,
     pub sort: Option<Vec<&'a str>>,
+    /// set to skip documents (offset, limit)
+    pub offsets: &'a [Option<(usize, usize)>],
+    /// enable if you want to bench getting documents without querying
+    pub get_documents: bool,
+    /// configure the benchmark sample size
+    pub sample_size: Option<usize>,
     /// enable or disable the optional words on the query
     pub optional_words: bool,
     /// primary key, if there is None we'll auto-generate docids for every documents
@@ -52,6 +59,9 @@ impl Conf<'_> {
         configure: |_| (),
         filter: None,
         sort: None,
+        offsets: &[None],
+        get_documents: false,
+        sample_size: None,
         optional_words: true,
         primary_key: None,
     };
@@ -145,25 +155,79 @@ pub fn run_benches(c: &mut criterion::Criterion, confs: &[Conf]) {
         let file_name = Path::new(conf.dataset).file_name().and_then(|f| f.to_str()).unwrap();
         let name = format!("{}: {}", file_name, conf.group_name);
         let mut group = c.benchmark_group(&name);
+        if let Some(sample_size) = conf.sample_size {
+            group.sample_size(sample_size);
+        }
 
         for &query in conf.queries {
-            group.bench_with_input(BenchmarkId::from_parameter(query), &query, |b, &query| {
-                b.iter(|| {
-                    let rtxn = index.read_txn().unwrap();
-                    let mut search = index.search(&rtxn);
-                    search.query(query).terms_matching_strategy(TermsMatchingStrategy::default());
-                    if let Some(filter) = conf.filter {
-                        let filter = Filter::from_str(filter).unwrap().unwrap();
-                        search.filter(filter);
-                    }
-                    if let Some(sort) = &conf.sort {
-                        let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
-                        search.sort_criteria(sort);
-                    }
-                    let _ids = search.execute().unwrap();
-                });
-            });
+            for offset in conf.offsets {
+                let parameter = match offset {
+                    None => query.to_string(),
+                    Some((offset, limit)) => format!("{query}[{offset}:{limit}]"),
+                };
+                group.bench_with_input(
+                    BenchmarkId::from_parameter(parameter),
+                    &query,
+                    |b, &query| {
+                        b.iter(|| {
+                            let rtxn = index.read_txn().unwrap();
+                            let mut search = index.search(&rtxn);
+                            search
+                                .query(query)
+                                .terms_matching_strategy(TermsMatchingStrategy::default());
+                            if let Some(filter) = conf.filter {
+                                let filter = Filter::from_str(filter).unwrap().unwrap();
+                                search.filter(filter);
+                            }
+                            if let Some(sort) = &conf.sort {
+                                let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
+                                search.sort_criteria(sort);
+                            }
+                            if let Some((offset, limit)) = offset {
+                                search.offset(*offset).limit(*limit);
+                            }
+
+                            let _ids = search.execute().unwrap();
+                        });
+                    },
+                );
+            }
         }
+
+        if conf.get_documents {
+            for offset in conf.offsets {
+                let parameter = match offset {
+                    None => String::from("get_documents"),
+                    Some((offset, limit)) => format!("get_documents[{offset}:{limit}]"),
+                };
+                group.bench_with_input(BenchmarkId::from_parameter(parameter), &(), |b, &()| {
+                    b.iter(|| {
+                        let rtxn = index.read_txn().unwrap();
+                        if let Some(sort) = &conf.sort {
+                            let sort = sort.iter().map(|sort| sort.parse().unwrap()).collect();
+                            let all_docs = index.documents_ids(&rtxn).unwrap();
+                            let facet_sort =
+                                recursive_sort(&index, &rtxn, sort, &all_docs).unwrap();
+                            let iter = facet_sort.iter().unwrap();
+                            if let Some((offset, limit)) = offset {
+                                let _results = iter.skip(*offset).take(*limit).collect::<Vec<_>>();
+                            } else {
+                                let _results = iter.collect::<Vec<_>>();
+                            }
+                        } else {
+                            let all_docs = index.documents_ids(&rtxn).unwrap();
+                            if let Some((offset, limit)) = offset {
+                                let _results =
+                                    all_docs.iter().skip(*offset).take(*limit).collect::<Vec<_>>();
+                            } else {
+                                let _results = all_docs.iter().collect::<Vec<_>>();
+                            }
+                        }
+                    });
+                });
+            }
+        }
+
         group.finish();
 
         index.prepare_for_closing().wait();

crates/dump/src/reader/compat/v1_to_v2.rs

@@ -1,3 +1,4 @@
+use std::fs::File;
 use std::str::FromStr;
 
 use super::v2_to_v3::CompatV2ToV3;
@@ -94,6 +95,10 @@ impl CompatIndexV1ToV2 {
         self.from.documents().map(|it| Box::new(it) as Box<dyn Iterator<Item = _>>)
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.from.documents_file()
+    }
+
     pub fn settings(&mut self) -> Result<v2::settings::Settings<v2::settings::Checked>> {
         Ok(v2::settings::Settings::<v2::settings::Unchecked>::from(self.from.settings()?).check())
     }

crates/dump/src/reader/compat/v2_to_v3.rs

@@ -1,3 +1,4 @@
+use std::fs::File;
 use std::str::FromStr;
 
 use time::OffsetDateTime;
@@ -122,6 +123,13 @@ impl CompatIndexV2ToV3 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV2ToV3::V2(v2) => v2.documents_file(),
+            CompatIndexV2ToV3::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v3::Settings<v3::Checked>> {
         let settings = match self {
             CompatIndexV2ToV3::V2(from) => from.settings()?,

crates/dump/src/reader/compat/v3_to_v4.rs

@@ -1,3 +1,5 @@
+use std::fs::File;
+
 use super::v2_to_v3::{CompatIndexV2ToV3, CompatV2ToV3};
 use super::v4_to_v5::CompatV4ToV5;
 use crate::reader::{v3, v4, UpdateFile};
@@ -252,6 +254,13 @@ impl CompatIndexV3ToV4 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV3ToV4::V3(v3) => v3.documents_file(),
+            CompatIndexV3ToV4::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v4::Settings<v4::Checked>> {
         Ok(match self {
             CompatIndexV3ToV4::V3(v3) => {

crates/dump/src/reader/compat/v4_to_v5.rs

@@ -1,3 +1,5 @@
+use std::fs::File;
+
 use super::v3_to_v4::{CompatIndexV3ToV4, CompatV3ToV4};
 use super::v5_to_v6::CompatV5ToV6;
 use crate::reader::{v4, v5, Document};
@@ -241,6 +243,13 @@ impl CompatIndexV4ToV5 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV4ToV5::V4(v4) => v4.documents_file(),
+            CompatIndexV4ToV5::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v5::Settings<v5::Checked>> {
         match self {
             CompatIndexV4ToV5::V4(v4) => Ok(v5::Settings::from(v4.settings()?).check()),

crates/dump/src/reader/compat/v5_to_v6.rs

@@ -1,3 +1,4 @@
+use std::fs::File;
 use std::num::NonZeroUsize;
 use std::str::FromStr;
 
@@ -243,6 +244,13 @@ impl CompatIndexV5ToV6 {
         }
     }
 
+    pub fn documents_file(&self) -> &File {
+        match self {
+            CompatIndexV5ToV6::V5(v5) => v5.documents_file(),
+            CompatIndexV5ToV6::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v6::Settings<v6::Checked>> {
         match self {
             CompatIndexV5ToV6::V5(v5) => Ok(v6::Settings::from(v5.settings()?).check()),

crates/dump/src/reader/mod.rs

@@ -192,6 +192,14 @@ impl DumpIndexReader {
         }
     }
 
+    /// A reference to a file in the NDJSON format containing all the documents of the index
+    pub fn documents_file(&self) -> &File {
+        match self {
+            DumpIndexReader::Current(v6) => v6.documents_file(),
+            DumpIndexReader::Compat(compat) => compat.documents_file(),
+        }
+    }
+
     pub fn settings(&mut self) -> Result<v6::Settings<v6::Checked>> {
         match self {
             DumpIndexReader::Current(v6) => v6.settings(),

crates/dump/src/reader/v1/mod.rs

@@ -72,6 +72,10 @@ impl V1IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<self::settings::Settings> {
         Ok(serde_json::from_reader(&mut self.settings)?)
     }

crates/dump/src/reader/v2/mod.rs

@@ -203,6 +203,10 @@ impl V2IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v3/mod.rs

@@ -215,6 +215,10 @@ impl V3IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v4/mod.rs

@@ -210,6 +210,10 @@ impl V4IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v5/mod.rs

@@ -247,6 +247,10 @@ impl V5IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         Ok(self.settings.clone())
     }

crates/dump/src/reader/v6/mod.rs

@@ -284,6 +284,10 @@ impl V6IndexReader {
             .map(|line| -> Result<_> { Ok(serde_json::from_str(&line?)?) }))
     }
 
+    pub fn documents_file(&self) -> &File {
+        self.documents.get_ref()
+    }
+
     pub fn settings(&mut self) -> Result<Settings<Checked>> {
         let mut settings: Settings<Unchecked> = serde_json::from_reader(&mut self.settings)?;
         patch_embedders(&mut settings);

crates/index-scheduler/Cargo.toml

@@ -26,7 +26,7 @@ flate2 = "1.1.2"
 indexmap = "2.9.0"
 meilisearch-auth = { path = "../meilisearch-auth" }
 meilisearch-types = { path = "../meilisearch-types" }
-memmap2 = "0.9.5"
+memmap2 = "0.9.7"
 page_size = "0.6.0"
 rayon = "1.10.0"
 roaring = { version = "0.10.12", features = ["serde"] }

crates/index-scheduler/src/index_mapper/mod.rs

@@ -71,7 +71,7 @@ pub struct IndexMapper {
     /// Path to the folder where the LMDB environments of each index are.
     base_path: PathBuf,
     /// The map size an index is opened with on the first time.
-    index_base_map_size: usize,
+    pub(crate) index_base_map_size: usize,
     /// The quantity by which the map size of an index is incremented upon reopening, in bytes.
     index_growth_amount: usize,
     /// Whether we open a meilisearch index with the MDB_WRITEMAP option or not.

crates/index-scheduler/src/insta_snapshot.rs

@@ -20,6 +20,7 @@ pub fn snapshot_index_scheduler(scheduler: &IndexScheduler) -> String {
 
     let IndexScheduler {
         cleanup_enabled: _,
+        experimental_no_edition_2024_for_dumps: _,
         processing_tasks,
         env,
         version,

crates/index-scheduler/src/lib.rs

@@ -168,6 +168,9 @@ pub struct IndexScheduler {
     /// Whether we should automatically cleanup the task queue or not.
     pub(crate) cleanup_enabled: bool,
 
+    /// Whether we should use the old document indexer or the new one.
+    pub(crate) experimental_no_edition_2024_for_dumps: bool,
+
     /// The webhook url we should send tasks to after processing every batches.
     pub(crate) webhook_url: Option<String>,
     /// The Authorization header to send to the webhook URL.
@@ -210,6 +213,7 @@ impl IndexScheduler {
 
             index_mapper: self.index_mapper.clone(),
             cleanup_enabled: self.cleanup_enabled,
+            experimental_no_edition_2024_for_dumps: self.experimental_no_edition_2024_for_dumps,
             webhook_url: self.webhook_url.clone(),
             webhook_authorization_header: self.webhook_authorization_header.clone(),
             embedders: self.embedders.clone(),
@@ -296,6 +300,9 @@ impl IndexScheduler {
             index_mapper,
             env,
             cleanup_enabled: options.cleanup_enabled,
+            experimental_no_edition_2024_for_dumps: options
+                .indexer_config
+                .experimental_no_edition_2024_for_dumps,
             webhook_url: options.webhook_url,
             webhook_authorization_header: options.webhook_authorization_header,
             embedders: Default::default(),
@@ -594,6 +601,11 @@ impl IndexScheduler {
         Ok(nbr_index_processing_tasks > 0)
     }
 
+    /// Whether the index should use the old document indexer.
+    pub fn no_edition_2024_for_dumps(&self) -> bool {
+        self.experimental_no_edition_2024_for_dumps
+    }
+
     /// Return the tasks matching the query from the user's point of view along
     /// with the total number of tasks matching the query, ignoring from and limit.
     ///

crates/index-scheduler/src/scheduler/process_dump_creation.rs

@@ -5,6 +5,7 @@ use std::sync::atomic::Ordering;
 
 use dump::IndexMetadata;
 use meilisearch_types::milli::constants::RESERVED_VECTORS_FIELD_NAME;
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::milli::vector::parsed_vectors::{ExplicitVectors, VectorOrArrayOfVectors};
 use meilisearch_types::milli::{self};
@@ -227,12 +228,21 @@ impl IndexScheduler {
                         return Err(Error::from_milli(user_err, Some(uid.to_string())));
                     };
 
-                    for (embedder_name, (embeddings, regenerate)) in embeddings {
+                    for (
+                        embedder_name,
+                        EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
+                    ) in embeddings
+                    {
                         let embeddings = ExplicitVectors {
                             embeddings: Some(VectorOrArrayOfVectors::from_array_of_vectors(
                                 embeddings,
                             )),
-                            regenerate,
+                            regenerate: regenerate &&
+                            // Meilisearch does not handle well dumps with fragments, because as the fragments
+                            // are marked as user-provided,
+                            // all embeddings would be regenerated on any settings change or document update.
+                            // To prevent this, we mark embeddings has non regenerate in this case.
+                            !has_fragments,
                         };
                         vectors.insert(embedder_name, serde_json::to_value(embeddings).unwrap());
                     }

crates/index-scheduler/src/scheduler/process_export.rs

@@ -9,6 +9,7 @@ use flate2::write::GzEncoder;
 use flate2::Compression;
 use meilisearch_types::index_uid_pattern::IndexUidPattern;
 use meilisearch_types::milli::constants::RESERVED_VECTORS_FIELD_NAME;
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::milli::update::{request_threads, Setting};
 use meilisearch_types::milli::vector::parsed_vectors::{ExplicitVectors, VectorOrArrayOfVectors};
@@ -62,13 +63,14 @@ impl IndexScheduler {
             let ExportIndexSettings { filter, override_settings } = export_settings;
             let index = self.index(uid)?;
             let index_rtxn = index.read_txn()?;
+            let bearer = api_key.map(|api_key| format!("Bearer {api_key}"));
 
             // First, check if the index already exists
             let url = format!("{base_url}/indexes/{uid}");
             let response = retry(&must_stop_processing, || {
                 let mut request = agent.get(&url);
-                if let Some(api_key) = api_key {
-                    request = request.set("Authorization", &format!("Bearer {api_key}"));
+                if let Some(bearer) = &bearer {
+                    request = request.set("Authorization", bearer);
                 }
 
                 request.send_bytes(Default::default()).map_err(into_backoff_error)
@@ -90,8 +92,8 @@ impl IndexScheduler {
                 let url = format!("{base_url}/indexes");
                 retry(&must_stop_processing, || {
                     let mut request = agent.post(&url);
-                    if let Some(api_key) = api_key {
-                        request = request.set("Authorization", &format!("Bearer {api_key}"));
+                    if let Some(bearer) = &bearer {
+                        request = request.set("Authorization", bearer);
                     }
                     let index_param = json!({ "uid": uid, "primaryKey": primary_key });
                     request.send_json(&index_param).map_err(into_backoff_error)
@@ -103,8 +105,8 @@ impl IndexScheduler {
                 let url = format!("{base_url}/indexes/{uid}");
                 retry(&must_stop_processing, || {
                     let mut request = agent.patch(&url);
-                    if let Some(api_key) = api_key {
-                        request = request.set("Authorization", &format!("Bearer {api_key}"));
+                    if let Some(bearer) = &bearer {
+                        request = request.set("Authorization", bearer);
                     }
                     let index_param = json!({ "primaryKey": primary_key });
                     request.send_json(&index_param).map_err(into_backoff_error)
@@ -122,7 +124,6 @@ impl IndexScheduler {
                 }
                 // Retry logic for sending settings
                 let url = format!("{base_url}/indexes/{uid}/settings");
-                let bearer = api_key.map(|api_key| format!("Bearer {api_key}"));
                 retry(&must_stop_processing, || {
                     let mut request = agent.patch(&url);
                     if let Some(bearer) = bearer.as_ref() {
@@ -167,10 +168,10 @@ impl IndexScheduler {
                 },
             );
 
-            let limit = payload_size.map(|ps| ps.as_u64() as usize).unwrap_or(50 * 1024 * 1024); // defaults to 50 MiB
+            let limit = payload_size.map(|ps| ps.as_u64() as usize).unwrap_or(20 * 1024 * 1024); // defaults to 20 MiB
             let documents_url = format!("{base_url}/indexes/{uid}/documents");
 
-            request_threads()
+            let results = request_threads()
                 .broadcast(|ctx| {
                     let index_rtxn = index
                         .read_txn()
@@ -229,12 +230,21 @@ impl IndexScheduler {
                                 ));
                             };
 
-                            for (embedder_name, (embeddings, regenerate)) in embeddings {
+                            for (
+                                embedder_name,
+                                EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
+                            ) in embeddings
+                            {
                                 let embeddings = ExplicitVectors {
                                     embeddings: Some(
                                         VectorOrArrayOfVectors::from_array_of_vectors(embeddings),
                                     ),
-                                    regenerate,
+                                    regenerate: regenerate &&
+                                    // Meilisearch does not handle well dumps with fragments, because as the fragments
+                                    // are marked as user-provided,
+                                    // all embeddings would be regenerated on any settings change or document update.
+                                    // To prevent this, we mark embeddings has non regenerate in this case.
+                                    !has_fragments,
                                 };
                                 vectors.insert(
                                     embedder_name,
@@ -265,9 +275,8 @@ impl IndexScheduler {
                                 let mut request = agent.post(&documents_url);
                                 request = request.set("Content-Type", "application/x-ndjson");
                                 request = request.set("Content-Encoding", "gzip");
-                                if let Some(api_key) = api_key {
-                                    request = request
-                                        .set("Authorization", &(format!("Bearer {api_key}")));
+                                if let Some(bearer) = &bearer {
+                                    request = request.set("Authorization", bearer);
                                 }
                                 request.send_bytes(&compressed_buffer).map_err(into_backoff_error)
                             })?;
@@ -276,7 +285,7 @@ impl IndexScheduler {
                         }
                         buffer.extend_from_slice(&tmp_buffer);
 
-                        if i % 100 == 0 {
+                        if i > 0 && i % 100 == 0 {
                             step.fetch_add(100, atomic::Ordering::Relaxed);
                         }
                     }
@@ -284,8 +293,8 @@ impl IndexScheduler {
                     retry(&must_stop_processing, || {
                         let mut request = agent.post(&documents_url);
                         request = request.set("Content-Type", "application/x-ndjson");
-                        if let Some(api_key) = api_key {
-                            request = request.set("Authorization", &(format!("Bearer {api_key}")));
+                        if let Some(bearer) = &bearer {
+                            request = request.set("Authorization", bearer);
                         }
                         request.send_bytes(&buffer).map_err(into_backoff_error)
                     })?;
@@ -298,6 +307,9 @@ impl IndexScheduler {
                         Some(uid.to_string()),
                     )
                 })?;
+            for result in results {
+                result?;
+            }
 
             step.store(total_documents, atomic::Ordering::Relaxed);
         }

crates/index-scheduler/src/scheduler/process_snapshot_creation.rs

@@ -7,9 +7,73 @@ use meilisearch_types::milli::progress::{Progress, VariableNameStep};
 use meilisearch_types::tasks::{Status, Task};
 use meilisearch_types::{compression, VERSION_FILE_NAME};
 
+use crate::heed::EnvOpenOptions;
 use crate::processing::{AtomicUpdateFileStep, SnapshotCreationProgress};
+use crate::queue::TaskQueue;
 use crate::{Error, IndexScheduler, Result};
 
+/// # Safety
+///
+/// See [`EnvOpenOptions::open`].
+unsafe fn remove_tasks(
+    tasks: &[Task],
+    dst: &std::path::Path,
+    index_base_map_size: usize,
+) -> Result<()> {
+    let env_options = EnvOpenOptions::new();
+    let mut env_options = env_options.read_txn_without_tls();
+    let env = env_options.max_dbs(TaskQueue::nb_db()).map_size(index_base_map_size).open(dst)?;
+    let mut wtxn = env.write_txn()?;
+    let task_queue = TaskQueue::new(&env, &mut wtxn)?;
+
+    // Destructuring to ensure the code below gets updated if a database gets added in the future.
+    let TaskQueue {
+        all_tasks,
+        status,
+        kind,
+        index_tasks: _, // snapshot creation tasks are not index tasks
+        canceled_by,
+        enqueued_at,
+        started_at,
+        finished_at,
+    } = task_queue;
+
+    for task in tasks {
+        all_tasks.delete(&mut wtxn, &task.uid)?;
+
+        let mut tasks = status.get(&wtxn, &task.status)?.unwrap_or_default();
+        tasks.remove(task.uid);
+        status.put(&mut wtxn, &task.status, &tasks)?;
+
+        let mut tasks = kind.get(&wtxn, &task.kind.as_kind())?.unwrap_or_default();
+        tasks.remove(task.uid);
+        kind.put(&mut wtxn, &task.kind.as_kind(), &tasks)?;
+
+        canceled_by.delete(&mut wtxn, &task.uid)?;
+
+        let timestamp = task.enqueued_at.unix_timestamp_nanos();
+        let mut tasks = enqueued_at.get(&wtxn, &timestamp)?.unwrap_or_default();
+        tasks.remove(task.uid);
+        enqueued_at.put(&mut wtxn, &timestamp, &tasks)?;
+
+        if let Some(task_started_at) = task.started_at {
+            let timestamp = task_started_at.unix_timestamp_nanos();
+            let mut tasks = started_at.get(&wtxn, &timestamp)?.unwrap_or_default();
+            tasks.remove(task.uid);
+            started_at.put(&mut wtxn, &timestamp, &tasks)?;
+        }
+
+        if let Some(task_finished_at) = task.finished_at {
+            let timestamp = task_finished_at.unix_timestamp_nanos();
+            let mut tasks = finished_at.get(&wtxn, &timestamp)?.unwrap_or_default();
+            tasks.remove(task.uid);
+            finished_at.put(&mut wtxn, &timestamp, &tasks)?;
+        }
+    }
+    wtxn.commit()?;
+    Ok(())
+}
+
 impl IndexScheduler {
     pub(super) fn process_snapshot(
         &self,
@@ -48,14 +112,26 @@ impl IndexScheduler {
         };
         self.env.copy_to_path(dst.join("data.mdb"), compaction_option)?;
 
-        // 2.2 Create a read transaction on the index-scheduler
+        // 2.2 Remove the current snapshot tasks
+        //
+        // This is done to ensure that the tasks are not processed again when the snapshot is imported
+        //
+        // # Safety
+        //
+        // This is safe because we open the env file we just created in a temporary directory.
+        // We are sure it's not being used by any other process nor thread.
+        unsafe {
+            remove_tasks(&tasks, &dst, self.index_mapper.index_base_map_size)?;
+        }
+
+        // 2.3 Create a read transaction on the index-scheduler
         let rtxn = self.env.read_txn()?;
 
-        // 2.3 Create the update files directory
+        // 2.4 Create the update files directory
         let update_files_dir = temp_snapshot_dir.path().join("update_files");
         fs::create_dir_all(&update_files_dir)?;
 
-        // 2.4 Only copy the update files of the enqueued tasks
+        // 2.5 Only copy the update files of the enqueued tasks
         progress.update_progress(SnapshotCreationProgress::SnapshotTheUpdateFiles);
         let enqueued = self.queue.tasks.get_status(&rtxn, Status::Enqueued)?;
         let (atomic, update_file_progress) = AtomicUpdateFileStep::new(enqueued.len() as u32);

crates/index-scheduler/src/scheduler/test_embedders.rs

@@ -3,6 +3,7 @@ use std::collections::BTreeMap;
 use big_s::S;
 use insta::assert_json_snapshot;
 use meili_snap::{json_string, snapshot};
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::update::Setting;
 use meilisearch_types::milli::vector::settings::EmbeddingSettings;
 use meilisearch_types::milli::vector::SearchQuery;
@@ -220,8 +221,8 @@ fn import_vectors() {
 
         let embeddings = index.embeddings(&rtxn, 0).unwrap();
 
-        assert_json_snapshot!(embeddings[&simple_hf_name].0[0] == lab_embed, @"true");
-        assert_json_snapshot!(embeddings[&fakerest_name].0[0] == beagle_embed, @"true");
+        assert_json_snapshot!(embeddings[&simple_hf_name].embeddings[0] == lab_embed, @"true");
+        assert_json_snapshot!(embeddings[&fakerest_name].embeddings[0] == beagle_embed, @"true");
 
         let doc = index.documents(&rtxn, std::iter::once(0)).unwrap()[0].1;
         let fields_ids_map = index.fields_ids_map(&rtxn).unwrap();
@@ -311,9 +312,9 @@ fn import_vectors() {
             let embeddings = index.embeddings(&rtxn, 0).unwrap();
 
             // automatically changed to patou because set to regenerate
-            assert_json_snapshot!(embeddings[&simple_hf_name].0[0] == patou_embed, @"true");
+            assert_json_snapshot!(embeddings[&simple_hf_name].embeddings[0] == patou_embed, @"true");
             // remained beagle
-            assert_json_snapshot!(embeddings[&fakerest_name].0[0] == beagle_embed, @"true");
+            assert_json_snapshot!(embeddings[&fakerest_name].embeddings[0] == beagle_embed, @"true");
 
             let doc = index.documents(&rtxn, std::iter::once(0)).unwrap()[0].1;
             let fields_ids_map = index.fields_ids_map(&rtxn).unwrap();
@@ -497,13 +498,13 @@ fn import_vectors_first_and_embedder_later() {
 
     let docid = index.external_documents_ids.get(&rtxn, "0").unwrap().unwrap();
     let embeddings = index.embeddings(&rtxn, docid).unwrap();
-    let (embedding, _) = &embeddings["my_doggo_embedder"];
-    assert!(!embedding.is_empty(), "{embedding:?}");
+    let EmbeddingsWithMetadata { embeddings, .. } = &embeddings["my_doggo_embedder"];
+    assert!(!embeddings.is_empty(), "{embeddings:?}");
 
     // the document with the id 3 should keep its original embedding
     let docid = index.external_documents_ids.get(&rtxn, "3").unwrap().unwrap();
     let embeddings = index.embeddings(&rtxn, docid).unwrap();
-    let (embeddings, _) = &embeddings["my_doggo_embedder"];
+    let EmbeddingsWithMetadata { embeddings, .. } = &embeddings["my_doggo_embedder"];
 
     snapshot!(embeddings.len(), @"1");
     assert!(embeddings[0].iter().all(|i| *i == 3.0), "{:?}", embeddings[0]);
@@ -558,7 +559,7 @@ fn import_vectors_first_and_embedder_later() {
         "###);
 
     let embeddings = index.embeddings(&rtxn, docid).unwrap();
-    let (embedding, _) = &embeddings["my_doggo_embedder"];
+    let EmbeddingsWithMetadata { embeddings: embedding, .. } = &embeddings["my_doggo_embedder"];
 
     assert!(!embedding.is_empty());
     assert!(!embedding[0].iter().all(|i| *i == 3.0), "{:?}", embedding[0]);
@@ -566,7 +567,7 @@ fn import_vectors_first_and_embedder_later() {
     // the document with the id 4 should generate an embedding
     let docid = index.external_documents_ids.get(&rtxn, "4").unwrap().unwrap();
     let embeddings = index.embeddings(&rtxn, docid).unwrap();
-    let (embedding, _) = &embeddings["my_doggo_embedder"];
+    let EmbeddingsWithMetadata { embeddings: embedding, .. } = &embeddings["my_doggo_embedder"];
 
     assert!(!embedding.is_empty());
 }
@@ -696,7 +697,7 @@ fn delete_document_containing_vector() {
     "###);
     let docid = index.external_documents_ids.get(&rtxn, "0").unwrap().unwrap();
     let embeddings = index.embeddings(&rtxn, docid).unwrap();
-    let (embedding, _) = &embeddings["manual"];
+    let EmbeddingsWithMetadata { embeddings: embedding, .. } = &embeddings["manual"];
     assert!(!embedding.is_empty(), "{embedding:?}");
 
     index_scheduler

crates/meilisearch-auth/src/lib.rs

@@ -158,7 +158,7 @@ impl AuthController {
         self.store.delete_all_keys()
     }
 
-    /// Delete all the keys in the DB.
+    /// Insert a key directly into the store.
     pub fn raw_insert_key(&mut self, key: Key) -> Result<()> {
         self.store.put_api_key(key)?;
         Ok(())
@@ -351,6 +351,7 @@ pub struct IndexSearchRules {
 
 fn generate_default_keys(store: &HeedAuthStore) -> Result<()> {
     store.put_api_key(Key::default_chat())?;
+    store.put_api_key(Key::default_read_only_admin())?;
     store.put_api_key(Key::default_admin())?;
     store.put_api_key(Key::default_search())?;
 

crates/meilisearch-auth/src/store.rs

@@ -88,7 +88,13 @@ impl HeedAuthStore {
         let mut actions = HashSet::new();
         for action in &key.actions {
             match action {
-                Action::All => actions.extend(enum_iterator::all::<Action>()),
+                Action::All => {
+                    actions.extend(enum_iterator::all::<Action>());
+                    actions.remove(&Action::AllGet);
+                }
+                Action::AllGet => {
+                    actions.extend(enum_iterator::all::<Action>().filter(|a| a.is_read()))
+                }
                 Action::DocumentsAll => {
                     actions.extend(
                         [Action::DocumentsGet, Action::DocumentsDelete, Action::DocumentsAdd]

crates/meilisearch-types/Cargo.toml

@@ -24,7 +24,7 @@ enum-iterator = "2.1.0"
 file-store = { path = "../file-store" }
 flate2 = "1.1.2"
 fst = "0.4.7"
-memmap2 = "0.9.5"
+memmap2 = "0.9.7"
 milli = { path = "../milli" }
 roaring = { version = "0.10.12", features = ["serde"] }
 rustc-hash = "2.1.1"

crates/meilisearch-types/src/error.rs

@@ -237,6 +237,7 @@ InvalidDocumentRetrieveVectors                 , InvalidRequest       , BAD_REQU
 MissingDocumentFilter                          , InvalidRequest       , BAD_REQUEST ;
 MissingDocumentEditionFunction                 , InvalidRequest       , BAD_REQUEST ;
 InvalidDocumentFilter                          , InvalidRequest       , BAD_REQUEST ;
+InvalidDocumentSort                            , InvalidRequest       , BAD_REQUEST ;
 InvalidDocumentGeoField                        , InvalidRequest       , BAD_REQUEST ;
 InvalidVectorDimensions                        , InvalidRequest       , BAD_REQUEST ;
 InvalidVectorsType                             , InvalidRequest       , BAD_REQUEST ;
@@ -415,6 +416,7 @@ InvalidChatCompletionPrompts                   , InvalidRequest       , BAD_REQU
 InvalidChatCompletionSystemPrompt              , InvalidRequest       , BAD_REQUEST ;
 InvalidChatCompletionSearchDescriptionPrompt   , InvalidRequest       , BAD_REQUEST ;
 InvalidChatCompletionSearchQueryParamPrompt    , InvalidRequest       , BAD_REQUEST ;
+InvalidChatCompletionSearchFilterParamPrompt   , InvalidRequest       , BAD_REQUEST ;
 InvalidChatCompletionSearchIndexUidParamPrompt , InvalidRequest       , BAD_REQUEST ;
 InvalidChatCompletionPreQueryPrompt            , InvalidRequest       , BAD_REQUEST
 }
@@ -476,7 +478,8 @@ impl ErrorCode for milli::Error {
                     UserError::InvalidDistinctAttribute { .. } => Code::InvalidSearchDistinct,
                     UserError::SortRankingRuleMissing => Code::InvalidSearchSort,
                     UserError::InvalidFacetsDistribution { .. } => Code::InvalidSearchFacets,
-                    UserError::InvalidSortableAttribute { .. } => Code::InvalidSearchSort,
+                    UserError::InvalidSearchSortableAttribute { .. } => Code::InvalidSearchSort,
+                    UserError::InvalidDocumentSortableAttribute { .. } => Code::InvalidDocumentSort,
                     UserError::InvalidSearchableAttribute { .. } => {
                         Code::InvalidSearchAttributesToSearchOn
                     }
@@ -492,7 +495,8 @@ impl ErrorCode for milli::Error {
                     UserError::InvalidVectorsMapType { .. }
                     | UserError::InvalidVectorsEmbedderConf { .. } => Code::InvalidVectorsType,
                     UserError::TooManyVectors(_, _) => Code::TooManyVectors,
-                    UserError::SortError(_) => Code::InvalidSearchSort,
+                    UserError::SortError { search: true, .. } => Code::InvalidSearchSort,
+                    UserError::SortError { search: false, .. } => Code::InvalidDocumentSort,
                     UserError::InvalidMinTypoWordLenSetting(_, _) => {
                         Code::InvalidSettingsTypoTolerance
                     }

crates/meilisearch-types/src/features.rs

@@ -4,10 +4,11 @@ use serde::{Deserialize, Serialize};
 
 use crate::error::{Code, ResponseError};
 
-pub const DEFAULT_CHAT_SYSTEM_PROMPT: &str = "You are a highly capable research assistant with access to powerful search tools. IMPORTANT INSTRUCTIONS:1. When answering questions, you MUST make multiple tool calls (at least 2-3) to gather comprehensive information.2. Use different search queries for each tool call - vary keywords, rephrase questions, and explore different semantic angles to ensure broad coverage.3. Always explicitly announce BEFORE making each tool call by saying: \"I'll search for [specific information] now.\"4. Combine information from ALL tool calls to provide complete, nuanced answers rather than relying on a single source.5. For complex topics, break down your research into multiple targeted queries rather than using a single generic search.";
+pub const DEFAULT_CHAT_SYSTEM_PROMPT: &str = "You are a highly capable research assistant with access to powerful search tools. IMPORTANT INSTRUCTIONS:1. When answering questions, you MUST make multiple tool calls (at least 2-3) to gather comprehensive information.2. Use different search queries for each tool call - vary keywords, rephrase questions, and explore different semantic angles to ensure broad coverage.3. Always explicitly announce BEFORE making each tool call by saying: \"I'll search for [specific information] now.\"4. Combine information from ALL tool calls to provide complete, nuanced answers rather than relying on a single source.5. For complex topics, break down your research into multiple targeted queries rather than using a single generic search. Meilisearch doesn't use the colon (:) syntax to filter but rather the equal (=) one. Separate filters from query and keep the q parameter empty if needed. Same for the filter parameter: keep it empty if need be. If you need to find documents that CONTAINS keywords simply put the keywords in the q parameter do no use a filter for this purpose. Whenever you get an error, read the error message and fix your error. ";
 pub const DEFAULT_CHAT_SEARCH_DESCRIPTION_PROMPT: &str =
-    "Search the database for relevant JSON documents using an optional query.";
+    "Query: 'best story about Rust before 2018' with year: 2018, 2020, 2021\nlabel: analysis, golang, javascript\ntype: story, link\nvote: 300, 298, 278\n: {\"q\": \"\", \"filter\": \"category = Rust AND type = story AND year < 2018 AND vote > 100\"}\nQuery: 'A black or green car that can go fast with red brakes' with maxspeed_kmh: 200, 150, 130\ncolor: black, grey, red, green\nbrand: Toyota, Renault, Jeep, Ferrari\n: {\"q\": \"red brakes\", \"filter\": \"maxspeed_kmh > 150 AND color IN ['black', green]\"}\nQuery: 'Superman movie released in 2018 or after' with year: 2018, 2020, 2021\ngenres: Drama, Comedy, Adventure, Fiction\n: {\"q\":\"Superman\",\"filter\":\"genres IN [Adventure, Fiction] AND year >= 2018\"}";
 pub const DEFAULT_CHAT_SEARCH_Q_PARAM_PROMPT: &str = "The search query string used to find relevant documents in the index. This should contain keywords or phrases that best represent what the user is looking for. More specific queries will yield more precise results.";
+pub const DEFAULT_CHAT_SEARCH_FILTER_PARAM_PROMPT: &str = "The search filter string used to find relevant documents in the index. It supports parentheses, `=`, `!=`, `>=`, `>`, `<=`, `<`, `IN`, `NOT IN`, `TO`, `EXISTS`, `NOT EXISTS`, `IS NULL`, `IS NOT NULL`, `IS EMPTY`, `IS NOT EMPTY`, `_geoRadius`, or `_geoBoundingBox`. Here is an example: \"price > 100 AND category = 'electronics'\". The following is a list of fields that can be filtered on: ";
 pub const DEFAULT_CHAT_SEARCH_INDEX_UID_PARAM_PROMPT: &str = "The name of the index to search within. An index is a collection of documents organized for search. Selecting the right index ensures the most relevant results for the user query.";
 
 #[derive(Serialize, Deserialize, Debug, Clone, Copy, Default, PartialEq, Eq)]
@@ -161,18 +162,31 @@ impl ChatCompletionSource {
 #[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
 #[serde(rename_all = "camelCase")]
 pub struct ChatCompletionPrompts {
+    #[serde(default)]
     pub system: String,
+    #[serde(default)]
     pub search_description: String,
+    #[serde(default)]
     pub search_q_param: String,
+    #[serde(default = "default_search_filter_param")]
+    pub search_filter_param: String,
+    #[serde(default)]
     pub search_index_uid_param: String,
 }
 
+/// This function is used for when the search_filter_param is
+/// not provided and this can happen when the database is in v1.15.
+fn default_search_filter_param() -> String {
+    DEFAULT_CHAT_SEARCH_FILTER_PARAM_PROMPT.to_string()
+}
+
 impl Default for ChatCompletionPrompts {
     fn default() -> Self {
         Self {
             system: DEFAULT_CHAT_SYSTEM_PROMPT.to_string(),
             search_description: DEFAULT_CHAT_SEARCH_DESCRIPTION_PROMPT.to_string(),
             search_q_param: DEFAULT_CHAT_SEARCH_Q_PARAM_PROMPT.to_string(),
+            search_filter_param: DEFAULT_CHAT_SEARCH_FILTER_PARAM_PROMPT.to_string(),
             search_index_uid_param: DEFAULT_CHAT_SEARCH_INDEX_UID_PARAM_PROMPT.to_string(),
         }
     }

crates/meilisearch-types/src/keys.rs

@@ -144,6 +144,21 @@ impl Key {
         }
     }
 
+    pub fn default_read_only_admin() -> Self {
+        let now = OffsetDateTime::now_utc();
+        let uid = Uuid::new_v4();
+        Self {
+            name: Some("Default Read-Only Admin API Key".to_string()),
+            description: Some("Use it to read information across the whole database. Caution! Do not expose this key on a public frontend".to_string()),
+            uid,
+            actions: vec![Action::AllGet, Action::KeysGet],
+            indexes: vec![IndexUidPattern::all()],
+            expires_at: None,
+            created_at: now,
+            updated_at: now,
+        }
+    }
+
     pub fn default_search() -> Self {
         let now = OffsetDateTime::now_utc();
         let uid = Uuid::new_v4();
@@ -347,6 +362,9 @@ pub enum Action {
     #[serde(rename = "chatsSettings.update")]
     #[deserr(rename = "chatsSettings.update")]
     ChatsSettingsUpdate,
+    #[serde(rename = "*.get")]
+    #[deserr(rename = "*.get")]
+    AllGet,
 }
 
 impl Action {
@@ -385,6 +403,7 @@ impl Action {
             METRICS_GET => Some(Self::MetricsGet),
             DUMPS_ALL => Some(Self::DumpsAll),
             DUMPS_CREATE => Some(Self::DumpsCreate),
+            SNAPSHOTS_ALL => Some(Self::SnapshotsAll),
             SNAPSHOTS_CREATE => Some(Self::SnapshotsCreate),
             VERSION => Some(Self::Version),
             KEYS_CREATE => Some(Self::KeysAdd),
@@ -393,12 +412,60 @@ impl Action {
             KEYS_DELETE => Some(Self::KeysDelete),
             EXPERIMENTAL_FEATURES_GET => Some(Self::ExperimentalFeaturesGet),
             EXPERIMENTAL_FEATURES_UPDATE => Some(Self::ExperimentalFeaturesUpdate),
+            EXPORT => Some(Self::Export),
             NETWORK_GET => Some(Self::NetworkGet),
             NETWORK_UPDATE => Some(Self::NetworkUpdate),
+            ALL_GET => Some(Self::AllGet),
             _otherwise => None,
         }
     }
 
+    /// Whether the action should be included in [Action::AllRead].
+    pub fn is_read(&self) -> bool {
+        use Action::*;
+
+        // It's using an exhaustive match to force the addition of new actions.
+        match self {
+            // Any action that expands to others must return false, as it wouldn't be able to expand recursively.
+            All | AllGet | DocumentsAll | IndexesAll | ChatsAll | TasksAll | SettingsAll
+            | StatsAll | MetricsAll | DumpsAll | SnapshotsAll | ChatsSettingsAll => false,
+
+            Search => true,
+            DocumentsAdd => false,
+            DocumentsGet => true,
+            DocumentsDelete => false,
+            Export => true,
+            IndexesAdd => false,
+            IndexesGet => true,
+            IndexesUpdate => false,
+            IndexesDelete => false,
+            IndexesSwap => false,
+            TasksCancel => false,
+            TasksDelete => false,
+            TasksGet => true,
+            SettingsGet => true,
+            SettingsUpdate => false,
+            StatsGet => true,
+            MetricsGet => true,
+            DumpsCreate => false,
+            SnapshotsCreate => false,
+            Version => true,
+            KeysAdd => false,
+            KeysGet => false, // Disabled in order to prevent privilege escalation
+            KeysUpdate => false,
+            KeysDelete => false,
+            ExperimentalFeaturesGet => true,
+            ExperimentalFeaturesUpdate => false,
+            NetworkGet => true,
+            NetworkUpdate => false,
+            ChatCompletions => false, // Disabled because it might trigger generation of new chats
+            ChatsGet => true,
+            ChatsDelete => false,
+            ChatsSettingsGet => true,
+            ChatsSettingsUpdate => false,
+        }
+    }
+
     pub const fn repr(&self) -> u8 {
         *self as u8
     }
@@ -408,6 +475,7 @@ pub mod actions {
     use super::Action::*;
 
     pub(crate) const ALL: u8 = All.repr();
+    pub const ALL_GET: u8 = AllGet.repr();
     pub const SEARCH: u8 = Search.repr();
     pub const DOCUMENTS_ALL: u8 = DocumentsAll.repr();
     pub const DOCUMENTS_ADD: u8 = DocumentsAdd.repr();
@@ -432,6 +500,7 @@ pub mod actions {
     pub const METRICS_GET: u8 = MetricsGet.repr();
     pub const DUMPS_ALL: u8 = DumpsAll.repr();
     pub const DUMPS_CREATE: u8 = DumpsCreate.repr();
+    pub const SNAPSHOTS_ALL: u8 = SnapshotsAll.repr();
     pub const SNAPSHOTS_CREATE: u8 = SnapshotsCreate.repr();
     pub const VERSION: u8 = Version.repr();
     pub const KEYS_CREATE: u8 = KeysAdd.repr();
@@ -454,3 +523,68 @@ pub mod actions {
     pub const CHATS_SETTINGS_GET: u8 = ChatsSettingsGet.repr();
     pub const CHATS_SETTINGS_UPDATE: u8 = ChatsSettingsUpdate.repr();
 }
+
+#[cfg(test)]
+pub(crate) mod test {
+    use super::actions::*;
+    use super::Action::*;
+    use super::*;
+
+    #[test]
+    fn test_action_repr_and_constants() {
+        assert!(All.repr() == 0 && ALL == 0);
+        assert!(Search.repr() == 1 && SEARCH == 1);
+        assert!(DocumentsAll.repr() == 2 && DOCUMENTS_ALL == 2);
+        assert!(DocumentsAdd.repr() == 3 && DOCUMENTS_ADD == 3);
+        assert!(DocumentsGet.repr() == 4 && DOCUMENTS_GET == 4);
+        assert!(DocumentsDelete.repr() == 5 && DOCUMENTS_DELETE == 5);
+        assert!(IndexesAll.repr() == 6 && INDEXES_ALL == 6);
+        assert!(IndexesAdd.repr() == 7 && INDEXES_CREATE == 7);
+        assert!(IndexesGet.repr() == 8 && INDEXES_GET == 8);
+        assert!(IndexesUpdate.repr() == 9 && INDEXES_UPDATE == 9);
+        assert!(IndexesDelete.repr() == 10 && INDEXES_DELETE == 10);
+        assert!(IndexesSwap.repr() == 11 && INDEXES_SWAP == 11);
+        assert!(TasksAll.repr() == 12 && TASKS_ALL == 12);
+        assert!(TasksCancel.repr() == 13 && TASKS_CANCEL == 13);
+        assert!(TasksDelete.repr() == 14 && TASKS_DELETE == 14);
+        assert!(TasksGet.repr() == 15 && TASKS_GET == 15);
+        assert!(SettingsAll.repr() == 16 && SETTINGS_ALL == 16);
+        assert!(SettingsGet.repr() == 17 && SETTINGS_GET == 17);
+        assert!(SettingsUpdate.repr() == 18 && SETTINGS_UPDATE == 18);
+        assert!(StatsAll.repr() == 19 && STATS_ALL == 19);
+        assert!(StatsGet.repr() == 20 && STATS_GET == 20);
+        assert!(MetricsAll.repr() == 21 && METRICS_ALL == 21);
+        assert!(MetricsGet.repr() == 22 && METRICS_GET == 22);
+        assert!(DumpsAll.repr() == 23 && DUMPS_ALL == 23);
+        assert!(DumpsCreate.repr() == 24 && DUMPS_CREATE == 24);
+        assert!(SnapshotsAll.repr() == 25 && SNAPSHOTS_ALL == 25);
+        assert!(SnapshotsCreate.repr() == 26 && SNAPSHOTS_CREATE == 26);
+        assert!(Version.repr() == 27 && VERSION == 27);
+        assert!(KeysAdd.repr() == 28 && KEYS_CREATE == 28);
+        assert!(KeysGet.repr() == 29 && KEYS_GET == 29);
+        assert!(KeysUpdate.repr() == 30 && KEYS_UPDATE == 30);
+        assert!(KeysDelete.repr() == 31 && KEYS_DELETE == 31);
+        assert!(ExperimentalFeaturesGet.repr() == 32 && EXPERIMENTAL_FEATURES_GET == 32);
+        assert!(ExperimentalFeaturesUpdate.repr() == 33 && EXPERIMENTAL_FEATURES_UPDATE == 33);
+        assert!(Export.repr() == 34 && EXPORT == 34);
+        assert!(NetworkGet.repr() == 35 && NETWORK_GET == 35);
+        assert!(NetworkUpdate.repr() == 36 && NETWORK_UPDATE == 36);
+        assert!(ChatCompletions.repr() == 37 && CHAT_COMPLETIONS == 37);
+        assert!(ChatsAll.repr() == 38 && CHATS_ALL == 38);
+        assert!(ChatsGet.repr() == 39 && CHATS_GET == 39);
+        assert!(ChatsDelete.repr() == 40 && CHATS_DELETE == 40);
+        assert!(ChatsSettingsAll.repr() == 41 && CHATS_SETTINGS_ALL == 41);
+        assert!(ChatsSettingsGet.repr() == 42 && CHATS_SETTINGS_GET == 42);
+        assert!(ChatsSettingsUpdate.repr() == 43 && CHATS_SETTINGS_UPDATE == 43);
+        assert!(AllGet.repr() == 44 && ALL_GET == 44);
+    }
+
+    #[test]
+    fn test_from_repr() {
+        for action in enum_iterator::all::<Action>() {
+            let repr = action.repr();
+            let action_from_repr = Action::from_repr(repr);
+            assert_eq!(Some(action), action_from_repr, "Failed for action: {:?}", action);
+        }
+    }
+}

crates/meilisearch/Cargo.toml

@@ -50,6 +50,7 @@ jsonwebtoken = "9.3.1"
 lazy_static = "1.5.0"
 meilisearch-auth = { path = "../meilisearch-auth" }
 meilisearch-types = { path = "../meilisearch-types" }
+memmap2 = "0.9.7"
 mimalloc = { version = "0.1.47", default-features = false }
 mime = "0.3.17"
 num_cpus = "1.17.0"
@@ -169,5 +170,5 @@ german = ["meilisearch-types/german"]
 turkish = ["meilisearch-types/turkish"]
 
 [package.metadata.mini-dashboard]
-assets-url = "https://github.com/meilisearch/mini-dashboard/releases/download/v0.2.20/build.zip"
-sha1 = "82a7ddd7bf14bb5323c3d235d2b62892a98b6a59"
+assets-url = "https://github.com/meilisearch/mini-dashboard/releases/download/v0.2.22/build.zip"
+sha1 = "b70b2036b5f167da9ea0b637da8b320c7ea88254"

crates/meilisearch/src/analytics/mock_analytics.rs

@@ -104,6 +104,4 @@ impl Analytics for MockAnalytics {
         _request: &HttpRequest,
     ) {
     }
-    fn get_fetch_documents(&self, _documents_query: &DocumentFetchKind, _request: &HttpRequest) {}
-    fn post_fetch_documents(&self, _documents_query: &DocumentFetchKind, _request: &HttpRequest) {}
 }

crates/meilisearch/src/analytics/mod.rs

@@ -73,12 +73,6 @@ pub enum DocumentDeletionKind {
     PerFilter,
 }
 
-#[derive(Copy, Clone, Debug, PartialEq, Eq)]
-pub enum DocumentFetchKind {
-    PerDocumentId { retrieve_vectors: bool },
-    Normal { with_filter: bool, limit: usize, offset: usize, retrieve_vectors: bool },
-}
-
 /// To send an event to segment, your event must be able to aggregate itself with another event of the same type.
 pub trait Aggregate: 'static + mopa::Any + Send {
     /// The name of the event that will be sent to segment.

crates/meilisearch/src/analytics/segment_analytics.rs

@@ -203,6 +203,7 @@ struct Infos {
     experimental_composite_embedders: bool,
     experimental_embedding_cache_entries: usize,
     experimental_no_snapshot_compaction: bool,
+    experimental_no_edition_2024_for_dumps: bool,
     experimental_no_edition_2024_for_settings: bool,
     gpu_enabled: bool,
     db_path: bool,
@@ -293,6 +294,7 @@ impl Infos {
             max_indexing_threads,
             skip_index_budget: _,
             experimental_no_edition_2024_for_settings,
+            experimental_no_edition_2024_for_dumps,
         } = indexer_options;
 
         let RuntimeTogglableFeatures {
@@ -329,6 +331,7 @@ impl Infos {
             experimental_composite_embedders: composite_embedders,
             experimental_embedding_cache_entries,
             experimental_no_snapshot_compaction,
+            experimental_no_edition_2024_for_dumps,
             gpu_enabled: meilisearch_types::milli::vector::is_cuda_enabled(),
             db_path: db_path != PathBuf::from("./data.ms"),
             import_dump: import_dump.is_some(),

crates/meilisearch/src/error.rs

@@ -49,7 +49,7 @@ pub enum MeilisearchHttpError {
     TooManySearchRequests(usize),
     #[error("Internal error: Search limiter is down.")]
     SearchLimiterIsDown,
-    #[error("The provided payload reached the size limit. The maximum accepted payload size is {}.",  Byte::from_u64(*.0 as u64).get_appropriate_unit(UnitType::Binary))]
+    #[error("The provided payload reached the size limit. The maximum accepted payload size is {}.", Byte::from_u64(*.0 as u64).get_appropriate_unit(if *.0 % 1024 == 0 { UnitType::Binary } else { UnitType::Decimal }))]
     PayloadTooLarge(usize),
     #[error("Two indexes must be given for each swap. The list `[{}]` contains {} indexes.",
         .0.iter().map(|uid| format!("\"{uid}\"")).collect::<Vec<_>>().join(", "), .0.len()

crates/meilisearch/src/lib.rs

@@ -30,6 +30,7 @@ use actix_web::web::Data;
 use actix_web::{web, HttpRequest};
 use analytics::Analytics;
 use anyhow::bail;
+use bumpalo::Bump;
 use error::PayloadError;
 use extractors::payload::PayloadConfig;
 use index_scheduler::versioning::Versioning;
@@ -38,6 +39,7 @@ use meilisearch_auth::{open_auth_store_env, AuthController};
 use meilisearch_types::milli::constants::VERSION_MAJOR;
 use meilisearch_types::milli::documents::{DocumentsBatchBuilder, DocumentsBatchReader};
 use meilisearch_types::milli::progress::{EmbedderStats, Progress};
+use meilisearch_types::milli::update::new::indexer;
 use meilisearch_types::milli::update::{
     default_thread_pool_and_threads, IndexDocumentsConfig, IndexDocumentsMethod, IndexerConfig,
 };
@@ -533,7 +535,7 @@ fn import_dump(
         let mut index_reader = index_reader?;
         let metadata = index_reader.metadata();
         let uid = metadata.uid.clone();
-        tracing::info!("Importing index `{}`.", metadata.uid);
+        tracing::info!("Importing index `{uid}`.");
 
         let date = Some((metadata.created_at, metadata.updated_at));
         let index = index_scheduler.create_raw_index(&metadata.uid, date)?;
@@ -552,48 +554,100 @@ fn import_dump(
         apply_settings_to_builder(&settings, &mut builder);
         let embedder_stats: Arc<EmbedderStats> = Default::default();
         builder.execute(&|| false, &progress, embedder_stats.clone())?;
+        wtxn.commit()?;
 
-        // 5.3 Import the documents.
-        // 5.3.1 We need to recreate the grenad+obkv format accepted by the index.
-        tracing::info!("Importing the documents.");
-        let file = tempfile::tempfile()?;
-        let mut builder = DocumentsBatchBuilder::new(BufWriter::new(file));
-        for document in index_reader.documents()? {
-            builder.append_json_object(&document?)?;
+        let mut wtxn = index.write_txn()?;
+        let rtxn = index.read_txn()?;
+
+        if index_scheduler.no_edition_2024_for_dumps() {
+            // 5.3 Import the documents.
+            // 5.3.1 We need to recreate the grenad+obkv format accepted by the index.
+            tracing::info!("Importing the documents.");
+            let file = tempfile::tempfile()?;
+            let mut builder = DocumentsBatchBuilder::new(BufWriter::new(file));
+            for document in index_reader.documents()? {
+                builder.append_json_object(&document?)?;
+            }
+
+            // This flush the content of the batch builder.
+            let file = builder.into_inner()?.into_inner()?;
+
+            // 5.3.2 We feed it to the milli index.
+            let reader = BufReader::new(file);
+            let reader = DocumentsBatchReader::from_reader(reader)?;
+
+            let embedder_configs = index.embedding_configs().embedding_configs(&wtxn)?;
+            let embedders = index_scheduler.embedders(uid.to_string(), embedder_configs)?;
+
+            let builder = milli::update::IndexDocuments::new(
+                &mut wtxn,
+                &index,
+                indexer_config,
+                IndexDocumentsConfig {
+                    update_method: IndexDocumentsMethod::ReplaceDocuments,
+                    ..Default::default()
+                },
+                |indexing_step| tracing::trace!("update: {:?}", indexing_step),
+                || false,
+                &embedder_stats,
+            )?;
+
+            let builder = builder.with_embedders(embedders);
+
+            let (builder, user_result) = builder.add_documents(reader)?;
+            let user_result = user_result?;
+            tracing::info!(documents_found = user_result, "{} documents found.", user_result);
+            builder.execute()?;
+        } else {
+            let db_fields_ids_map = index.fields_ids_map(&rtxn)?;
+            let primary_key = index.primary_key(&rtxn)?;
+            let mut new_fields_ids_map = db_fields_ids_map.clone();
+
+            let mut indexer = indexer::DocumentOperation::new();
+            let embedders = index.embedding_configs().embedding_configs(&rtxn)?;
+            let embedders = index_scheduler.embedders(uid.clone(), embedders)?;
+
+            let mmap = unsafe { memmap2::Mmap::map(index_reader.documents_file())? };
+
+            indexer.replace_documents(&mmap)?;
+
+            let indexer_config = index_scheduler.indexer_config();
+            let pool = &indexer_config.thread_pool;
+
+            let indexer_alloc = Bump::new();
+            let (document_changes, mut operation_stats, primary_key) = indexer.into_changes(
+                &indexer_alloc,
+                &index,
+                &rtxn,
+                primary_key,
+                &mut new_fields_ids_map,
+                &|| false, // never stop processing a dump
+                progress.clone(),
+            )?;
+
+            let operation_stats = operation_stats.pop().unwrap();
+            if let Some(error) = operation_stats.error {
+                return Err(error.into());
+            }
+
+            let _congestion = indexer::index(
+                &mut wtxn,
+                &index,
+                pool,
+                indexer_config.grenad_parameters(),
+                &db_fields_ids_map,
+                new_fields_ids_map,
+                primary_key,
+                &document_changes,
+                embedders,
+                &|| false, // never stop processing a dump
+                &progress,
+                &embedder_stats,
+            )?;
         }
 
-        // This flush the content of the batch builder.
-        let file = builder.into_inner()?.into_inner()?;
-
-        // 5.3.2 We feed it to the milli index.
-        let reader = BufReader::new(file);
-        let reader = DocumentsBatchReader::from_reader(reader)?;
-
-        let embedder_configs = index.embedding_configs().embedding_configs(&wtxn)?;
-        let embedders = index_scheduler.embedders(uid.to_string(), embedder_configs)?;
-
-        let builder = milli::update::IndexDocuments::new(
-            &mut wtxn,
-            &index,
-            indexer_config,
-            IndexDocumentsConfig {
-                update_method: IndexDocumentsMethod::ReplaceDocuments,
-                ..Default::default()
-            },
-            |indexing_step| tracing::trace!("update: {:?}", indexing_step),
-            || false,
-            &embedder_stats,
-        )?;
-
-        let builder = builder.with_embedders(embedders);
-
-        let (builder, user_result) = builder.add_documents(reader)?;
-        let user_result = user_result?;
-        tracing::info!(documents_found = user_result, "{} documents found.", user_result);
-        builder.execute()?;
         wtxn.commit()?;
         tracing::info!("All documents successfully imported.");
-
         index_scheduler.refresh_index_stats(&uid)?;
     }
 

crates/meilisearch/src/metrics.rs

@@ -15,30 +15,33 @@ lazy_static! {
         "Meilisearch number of degraded search requests"
     ))
     .expect("Can't create a metric");
-    pub static ref MEILISEARCH_CHAT_SEARCH_REQUESTS: IntCounterVec = register_int_counter_vec!(
+    pub static ref MEILISEARCH_CHAT_SEARCHES_TOTAL: IntCounterVec = register_int_counter_vec!(
         opts!(
-            "meilisearch_chat_search_requests",
-            "Meilisearch number of search requests performed by the chat route itself"
+            "meilisearch_chat_searches_total",
+            "Total number of searches performed by the chat route"
         ),
         &["type"]
     )
     .expect("Can't create a metric");
-    pub static ref MEILISEARCH_CHAT_PROMPT_TOKENS_USAGE: IntCounterVec = register_int_counter_vec!(
-        opts!("meilisearch_chat_prompt_tokens_usage", "Meilisearch Chat Prompt Tokens Usage"),
+    pub static ref MEILISEARCH_CHAT_PROMPT_TOKENS_TOTAL: IntCounterVec = register_int_counter_vec!(
+        opts!("meilisearch_chat_prompt_tokens_total", "Total number of prompt tokens consumed"),
         &["workspace", "model"]
     )
     .expect("Can't create a metric");
-    pub static ref MEILISEARCH_CHAT_COMPLETION_TOKENS_USAGE: IntCounterVec =
+    pub static ref MEILISEARCH_CHAT_COMPLETION_TOKENS_TOTAL: IntCounterVec =
         register_int_counter_vec!(
             opts!(
-                "meilisearch_chat_completion_tokens_usage",
-                "Meilisearch Chat Completion Tokens Usage"
+                "meilisearch_chat_completion_tokens_total",
+                "Total number of completion tokens consumed"
             ),
             &["workspace", "model"]
         )
         .expect("Can't create a metric");
-    pub static ref MEILISEARCH_CHAT_TOTAL_TOKENS_USAGE: IntCounterVec = register_int_counter_vec!(
-        opts!("meilisearch_chat_total_tokens_usage", "Meilisearch Chat Total Tokens Usage"),
+    pub static ref MEILISEARCH_CHAT_TOKENS_TOTAL: IntCounterVec = register_int_counter_vec!(
+        opts!(
+            "meilisearch_chat_tokens_total",
+            "Total number of tokens consumed (prompt + completion)"
+        ),
         &["workspace", "model"]
     )
     .expect("Can't create a metric");

crates/meilisearch/src/option.rs

@@ -68,6 +68,8 @@ const MEILI_EXPERIMENTAL_LIMIT_BATCHED_TASKS_TOTAL_SIZE: &str =
 const MEILI_EXPERIMENTAL_EMBEDDING_CACHE_ENTRIES: &str =
     "MEILI_EXPERIMENTAL_EMBEDDING_CACHE_ENTRIES";
 const MEILI_EXPERIMENTAL_NO_SNAPSHOT_COMPACTION: &str = "MEILI_EXPERIMENTAL_NO_SNAPSHOT_COMPACTION";
+const MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_DUMPS: &str =
+    "MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_DUMPS";
 const DEFAULT_CONFIG_FILE_PATH: &str = "./config.toml";
 const DEFAULT_DB_PATH: &str = "./data.ms";
 const DEFAULT_HTTP_ADDR: &str = "localhost:7700";
@@ -759,6 +761,15 @@ pub struct IndexerOpts {
     #[clap(long, env = MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_SETTINGS)]
     #[serde(default)]
     pub experimental_no_edition_2024_for_settings: bool,
+
+    /// Experimental make dump imports use the old document indexer.
+    ///
+    /// When enabled, Meilisearch will use the old document indexer when importing dumps.
+    ///
+    /// For more information, see <https://github.com/orgs/meilisearch/discussions/851>.
+    #[clap(long, env = MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_DUMPS)]
+    #[serde(default)]
+    pub experimental_no_edition_2024_for_dumps: bool,
 }
 
 impl IndexerOpts {
@@ -769,6 +780,7 @@ impl IndexerOpts {
             max_indexing_threads,
             skip_index_budget: _,
             experimental_no_edition_2024_for_settings,
+            experimental_no_edition_2024_for_dumps,
         } = self;
         if let Some(max_indexing_memory) = max_indexing_memory.0 {
             export_to_env_if_not_present(
@@ -788,6 +800,12 @@ impl IndexerOpts {
                 experimental_no_edition_2024_for_settings.to_string(),
             );
         }
+        if experimental_no_edition_2024_for_dumps {
+            export_to_env_if_not_present(
+                MEILI_EXPERIMENTAL_NO_EDITION_2024_FOR_DUMPS,
+                experimental_no_edition_2024_for_dumps.to_string(),
+            );
+        }
     }
 }
 
@@ -808,6 +826,7 @@ impl TryFrom<&IndexerOpts> for IndexerConfig {
             skip_index_budget: other.skip_index_budget,
             experimental_no_edition_2024_for_settings: other
                 .experimental_no_edition_2024_for_settings,
+            experimental_no_edition_2024_for_dumps: other.experimental_no_edition_2024_for_dumps,
             chunk_compression_type: Default::default(),
             chunk_compression_level: Default::default(),
             documents_chunk_size: Default::default(),

crates/meilisearch/src/routes/chats/chat_completions.rs

@@ -27,9 +27,10 @@ use meilisearch_types::features::{
     ChatCompletionPrompts as DbChatCompletionPrompts,
     ChatCompletionSource as DbChatCompletionSource, SystemRole,
 };
+use meilisearch_types::heed::RoTxn;
 use meilisearch_types::keys::actions;
 use meilisearch_types::milli::index::ChatConfig;
-use meilisearch_types::milli::{all_obkv_to_json, obkv_to_json, TimeBudget};
+use meilisearch_types::milli::{all_obkv_to_json, obkv_to_json, OrderBy, PatternMatch, TimeBudget};
 use meilisearch_types::{Document, Index};
 use serde::Deserialize;
 use serde_json::json;
@@ -49,8 +50,8 @@ use crate::error::MeilisearchHttpError;
 use crate::extractors::authentication::policies::ActionPolicy;
 use crate::extractors::authentication::{extract_token_from_request, GuardedData, Policy as _};
 use crate::metrics::{
-    MEILISEARCH_CHAT_COMPLETION_TOKENS_USAGE, MEILISEARCH_CHAT_PROMPT_TOKENS_USAGE,
-    MEILISEARCH_CHAT_SEARCH_REQUESTS, MEILISEARCH_CHAT_TOTAL_TOKENS_USAGE,
+    MEILISEARCH_CHAT_COMPLETION_TOKENS_TOTAL, MEILISEARCH_CHAT_PROMPT_TOKENS_TOTAL,
+    MEILISEARCH_CHAT_SEARCHES_TOTAL, MEILISEARCH_CHAT_TOKENS_TOTAL,
     MEILISEARCH_DEGRADED_SEARCH_REQUESTS,
 };
 use crate::routes::chats::utils::SseEventSender;
@@ -169,6 +170,7 @@ fn setup_search_tool(
 
     let mut index_uids = Vec::new();
     let mut function_description = prompts.search_description.clone();
+    let mut filter_description = prompts.search_filter_param.clone();
     index_scheduler.try_for_each_index::<_, ()>(|name, index| {
         // Make sure to skip unauthorized indexes
         if !filters.is_index_authorized(name) {
@@ -180,16 +182,22 @@ fn setup_search_tool(
         let index_description = chat_config.description;
         let _ = writeln!(&mut function_description, "\n\n - {name}: {index_description}\n");
         index_uids.push(name.to_string());
+        let facet_distributions = format_facet_distributions(index, &rtxn, 10).unwrap(); // TODO do not unwrap
+        let _ = writeln!(&mut filter_description, "\n## Facet distributions of the {name} index");
+        let _ = writeln!(&mut filter_description, "{facet_distributions}");
 
         Ok(())
     })?;
 
+    tracing::debug!("LLM function description: {function_description}");
+    tracing::debug!("LLM filter description: {filter_description}");
+
     let tool = ChatCompletionToolArgs::default()
         .r#type(ChatCompletionToolType::Function)
         .function(
             FunctionObjectArgs::default()
                 .name(MEILI_SEARCH_IN_INDEX_FUNCTION_NAME)
-                .description(&function_description)
+                .description(function_description)
                 .parameters(json!({
                     "type": "object",
                     "properties": {
@@ -203,9 +211,13 @@ fn setup_search_tool(
                             // "type": ["string", "null"],
                             "type": "string",
                             "description": prompts.search_q_param,
+                        },
+                        "filter": {
+                            "type": "string",
+                            "description": filter_description,
                         }
                     },
-                    "required": ["index_uid", "q"],
+                    "required": ["index_uid", "q", "filter"],
                     "additionalProperties": false,
                 }))
                 .strict(true)
@@ -247,11 +259,19 @@ async fn process_search_request(
     auth_token: &str,
     index_uid: String,
     q: Option<String>,
+    filter: Option<String>,
 ) -> Result<(Index, Vec<Document>, String), ResponseError> {
     let index = index_scheduler.index(&index_uid)?;
     let rtxn = index.static_read_txn()?;
     let ChatConfig { description: _, prompt: _, search_parameters } = index.chat_config(&rtxn)?;
-    let mut query = SearchQuery { q, ..SearchQuery::from(search_parameters) };
+    let mut query = SearchQuery {
+        q,
+        filter: filter.map(serde_json::Value::from),
+        ..SearchQuery::from(search_parameters)
+    };
+
+    tracing::debug!("LLM query: {:?}", query);
+
     let auth_filter = ActionPolicy::<{ actions::SEARCH }>::authenticate(
         auth_ctrl,
         auth_token,
@@ -280,17 +300,26 @@ async fn process_search_request(
         let (search, _is_finite_pagination, _max_total_hits, _offset) =
             prepare_search(&index_cloned, &rtxn, &query, &search_kind, time_budget, features)?;
 
-        search_from_kind(index_uid, search_kind, search)
-            .map(|(search_results, _)| (rtxn, search_results))
-            .map_err(ResponseError::from)
+        match search_from_kind(index_uid, search_kind, search) {
+            Ok((search_results, _)) => Ok((rtxn, Ok(search_results))),
+            Err(MeilisearchHttpError::Milli {
+                error: meilisearch_types::milli::Error::UserError(user_error),
+                index_name: _,
+            }) => Ok((rtxn, Err(user_error))),
+            Err(err) => Err(ResponseError::from(err)),
+        }
     })
     .await;
     permit.drop().await;
 
-    let output = output?;
+    let output = match output? {
+        Ok((rtxn, Ok(search_results))) => Ok((rtxn, search_results)),
+        Ok((_rtxn, Err(error))) => return Ok((index, Vec::new(), error.to_string())),
+        Err(err) => Err(err),
+    };
     let mut documents = Vec::new();
     if let Ok((ref rtxn, ref search_result)) = output {
-        MEILISEARCH_CHAT_SEARCH_REQUESTS.with_label_values(&["internal"]).inc();
+        MEILISEARCH_CHAT_SEARCHES_TOTAL.with_label_values(&["internal"]).inc();
         if search_result.degraded {
             MEILISEARCH_DEGRADED_SEARCH_REQUESTS.inc();
         }
@@ -395,16 +424,19 @@ async fn non_streamed_chat(
 
                 for call in meili_calls {
                     let result = match serde_json::from_str(&call.function.arguments) {
-                        Ok(SearchInIndexParameters { index_uid, q }) => process_search_request(
-                            &index_scheduler,
-                            auth_ctrl.clone(),
-                            &search_queue,
-                            auth_token,
-                            index_uid,
-                            q,
-                        )
-                        .await
-                        .map_err(|e| e.to_string()),
+                        Ok(SearchInIndexParameters { index_uid, q, filter }) => {
+                            process_search_request(
+                                &index_scheduler,
+                                auth_ctrl.clone(),
+                                &search_queue,
+                                auth_token,
+                                index_uid,
+                                q,
+                                filter,
+                            )
+                            .await
+                            .map_err(|e| e.to_string())
+                        }
                         Err(err) => Err(err.to_string()),
                     };
 
@@ -564,13 +596,13 @@ async fn run_conversation<C: async_openai::config::Config>(
         match result {
             Ok(resp) => {
                 if let Some(usage) = resp.usage.as_ref() {
-                    MEILISEARCH_CHAT_PROMPT_TOKENS_USAGE
+                    MEILISEARCH_CHAT_PROMPT_TOKENS_TOTAL
                         .with_label_values(&[workspace_uid, &chat_completion.model])
                         .inc_by(usage.prompt_tokens as u64);
-                    MEILISEARCH_CHAT_COMPLETION_TOKENS_USAGE
+                    MEILISEARCH_CHAT_COMPLETION_TOKENS_TOTAL
                         .with_label_values(&[workspace_uid, &chat_completion.model])
                         .inc_by(usage.completion_tokens as u64);
-                    MEILISEARCH_CHAT_TOTAL_TOKENS_USAGE
+                    MEILISEARCH_CHAT_TOKENS_TOTAL
                         .with_label_values(&[workspace_uid, &chat_completion.model])
                         .inc_by(usage.total_tokens as u64);
                 }
@@ -719,13 +751,14 @@ async fn handle_meili_tools(
         let mut error = None;
 
         let result = match serde_json::from_str(&call.function.arguments) {
-            Ok(SearchInIndexParameters { index_uid, q }) => match process_search_request(
+            Ok(SearchInIndexParameters { index_uid, q, filter }) => match process_search_request(
                 index_scheduler,
                 auth_ctrl.clone(),
                 search_queue,
                 auth_token,
                 index_uid,
                 q,
+                filter,
             )
             .await
             {
@@ -801,4 +834,42 @@ struct SearchInIndexParameters {
     index_uid: String,
     /// The query parameter to use.
     q: Option<String>,
+    /// The filter parameter to use.
+    filter: Option<String>,
+}
+
+fn format_facet_distributions(
+    index: &Index,
+    rtxn: &RoTxn,
+    max_values_per_facet: usize,
+) -> meilisearch_types::milli::Result<String> {
+    let universe = index.documents_ids(rtxn)?;
+    let rules = index.filterable_attributes_rules(rtxn)?;
+    let fields_ids_map = index.fields_ids_map(rtxn)?;
+    let filterable_attributes = fields_ids_map
+        .names()
+        .filter(|name| rules.iter().any(|rule| matches!(rule.match_str(name), PatternMatch::Match)))
+        .map(|name| (name, OrderBy::Count));
+    let facets_distribution = index
+        .facets_distribution(rtxn)
+        .max_values_per_facet(max_values_per_facet)
+        .candidates(universe)
+        .facets(filterable_attributes)
+        .execute()?;
+
+    let mut output = String::new();
+    for (facet_name, entries) in facets_distribution {
+        let _ = write!(&mut output, "{}: ", facet_name);
+        let total_entries = entries.len();
+        for (i, (value, _count)) in entries.into_iter().enumerate() {
+            let _ = if total_entries.saturating_sub(1) == i {
+                write!(&mut output, "{value}.")
+            } else {
+                write!(&mut output, "{value}, ")
+            };
+        }
+        let _ = writeln!(&mut output);
+    }
+
+    Ok(output)
 }

crates/meilisearch/src/routes/chats/settings.rs

@@ -8,8 +8,8 @@ use meilisearch_types::error::{Code, ResponseError};
 use meilisearch_types::features::{
     ChatCompletionPrompts as DbChatCompletionPrompts, ChatCompletionSettings,
     ChatCompletionSource as DbChatCompletionSource, DEFAULT_CHAT_SEARCH_DESCRIPTION_PROMPT,
-    DEFAULT_CHAT_SEARCH_INDEX_UID_PARAM_PROMPT, DEFAULT_CHAT_SEARCH_Q_PARAM_PROMPT,
-    DEFAULT_CHAT_SYSTEM_PROMPT,
+    DEFAULT_CHAT_SEARCH_FILTER_PARAM_PROMPT, DEFAULT_CHAT_SEARCH_INDEX_UID_PARAM_PROMPT,
+    DEFAULT_CHAT_SEARCH_Q_PARAM_PROMPT, DEFAULT_CHAT_SYSTEM_PROMPT,
 };
 use meilisearch_types::keys::actions;
 use meilisearch_types::milli::update::Setting;
@@ -84,6 +84,11 @@ async fn patch_settings(
                 Setting::Reset => DEFAULT_CHAT_SEARCH_Q_PARAM_PROMPT.to_string(),
                 Setting::NotSet => old_settings.prompts.search_q_param,
             },
+            search_filter_param: match new_prompts.search_filter_param {
+                Setting::Set(new_description) => new_description,
+                Setting::Reset => DEFAULT_CHAT_SEARCH_FILTER_PARAM_PROMPT.to_string(),
+                Setting::NotSet => old_settings.prompts.search_filter_param,
+            },
             search_index_uid_param: match new_prompts.search_index_uid_param {
                 Setting::Set(new_description) => new_description,
                 Setting::Reset => DEFAULT_CHAT_SEARCH_INDEX_UID_PARAM_PROMPT.to_string(),
@@ -252,6 +257,10 @@ pub struct ChatPrompts {
     #[schema(value_type = Option<String>, example = json!("This is query parameter..."))]
     pub search_q_param: Setting<String>,
     #[serde(default)]
+    #[deserr(default, error = DeserrJsonError<InvalidChatCompletionSearchFilterParamPrompt>)]
+    #[schema(value_type = Option<String>, example = json!("This is filter parameter..."))]
+    pub search_filter_param: Setting<String>,
+    #[serde(default)]
     #[deserr(default, error = DeserrJsonError<InvalidChatCompletionSearchIndexUidParamPrompt>)]
     #[schema(value_type = Option<String>, example = json!("This is index you want to search in..."))]
     pub search_index_uid_param: Setting<String>,

crates/meilisearch/src/routes/indexes/documents.rs

@@ -1,6 +1,7 @@
 use std::collections::HashSet;
 use std::io::{ErrorKind, Seek as _};
 use std::marker::PhantomData;
+use std::str::FromStr;
 
 use actix_web::http::header::CONTENT_TYPE;
 use actix_web::web::Data;
@@ -17,9 +18,11 @@ use meilisearch_types::error::deserr_codes::*;
 use meilisearch_types::error::{Code, ResponseError};
 use meilisearch_types::heed::RoTxn;
 use meilisearch_types::index_uid::IndexUid;
+use meilisearch_types::milli::documents::sort::recursive_sort;
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::update::IndexDocumentsMethod;
 use meilisearch_types::milli::vector::parsed_vectors::ExplicitVectors;
-use meilisearch_types::milli::DocumentId;
+use meilisearch_types::milli::{AscDesc, DocumentId};
 use meilisearch_types::serde_cs::vec::CS;
 use meilisearch_types::star_or::OptionStarOrList;
 use meilisearch_types::tasks::KindWithContent;
@@ -42,6 +45,7 @@ use crate::extractors::authentication::policies::*;
 use crate::extractors::authentication::GuardedData;
 use crate::extractors::payload::Payload;
 use crate::extractors::sequential_extractor::SeqHandler;
+use crate::routes::indexes::search::fix_sort_query_parameters;
 use crate::routes::{
     get_task_id, is_dry_run, PaginationView, SummarizedTaskView, PAGINATION_DEFAULT_LIMIT,
 };
@@ -135,6 +139,8 @@ pub struct DocumentsFetchAggregator<Method: AggregateMethod> {
     per_document_id: bool,
     // if a filter was used
     per_filter: bool,
+    // if documents were sorted
+    sort: bool,
 
     #[serde(rename = "vector.retrieve_vectors")]
     retrieve_vectors: bool,
@@ -151,39 +157,6 @@ pub struct DocumentsFetchAggregator<Method: AggregateMethod> {
     marker: std::marker::PhantomData<Method>,
 }
 
-#[derive(Copy, Clone, Debug, PartialEq, Eq)]
-pub enum DocumentFetchKind {
-    PerDocumentId { retrieve_vectors: bool },
-    Normal { with_filter: bool, limit: usize, offset: usize, retrieve_vectors: bool, ids: usize },
-}
-
-impl<Method: AggregateMethod> DocumentsFetchAggregator<Method> {
-    pub fn from_query(query: &DocumentFetchKind) -> Self {
-        let (limit, offset, retrieve_vectors) = match query {
-            DocumentFetchKind::PerDocumentId { retrieve_vectors } => (1, 0, *retrieve_vectors),
-            DocumentFetchKind::Normal { limit, offset, retrieve_vectors, .. } => {
-                (*limit, *offset, *retrieve_vectors)
-            }
-        };
-
-        let ids = match query {
-            DocumentFetchKind::Normal { ids, .. } => *ids,
-            DocumentFetchKind::PerDocumentId { .. } => 0,
-        };
-
-        Self {
-            per_document_id: matches!(query, DocumentFetchKind::PerDocumentId { .. }),
-            per_filter: matches!(query, DocumentFetchKind::Normal { with_filter, .. } if *with_filter),
-            max_limit: limit,
-            max_offset: offset,
-            retrieve_vectors,
-            max_document_ids: ids,
-
-            marker: PhantomData,
-        }
-    }
-}
-
 impl<Method: AggregateMethod> Aggregate for DocumentsFetchAggregator<Method> {
     fn event_name(&self) -> &'static str {
         Method::event_name()
@@ -193,6 +166,7 @@ impl<Method: AggregateMethod> Aggregate for DocumentsFetchAggregator<Method> {
         Box::new(Self {
             per_document_id: self.per_document_id | new.per_document_id,
             per_filter: self.per_filter | new.per_filter,
+            sort: self.sort | new.sort,
             retrieve_vectors: self.retrieve_vectors | new.retrieve_vectors,
             max_limit: self.max_limit.max(new.max_limit),
             max_offset: self.max_offset.max(new.max_offset),
@@ -276,6 +250,7 @@ pub async fn get_document(
             retrieve_vectors: param_retrieve_vectors.0,
             per_document_id: true,
             per_filter: false,
+            sort: false,
             max_limit: 0,
             max_offset: 0,
             max_document_ids: 0,
@@ -406,6 +381,8 @@ pub struct BrowseQueryGet {
     #[param(default, value_type = Option<String>, example = "popularity > 1000")]
     #[deserr(default, error = DeserrQueryParamError<InvalidDocumentFilter>)]
     filter: Option<String>,
+    #[deserr(default, error = DeserrQueryParamError<InvalidDocumentSort>)]
+    sort: Option<String>,
 }
 
 #[derive(Debug, Deserr, ToSchema)]
@@ -430,6 +407,9 @@ pub struct BrowseQuery {
     #[schema(default, value_type = Option<Value>, example = "popularity > 1000")]
     #[deserr(default, error = DeserrJsonError<InvalidDocumentFilter>)]
     filter: Option<Value>,
+    #[schema(default, value_type = Option<Vec<String>>, example = json!(["title:asc", "rating:desc"]))]
+    #[deserr(default, error = DeserrJsonError<InvalidDocumentSort>)]
+    sort: Option<Vec<String>>,
 }
 
 /// Get documents with POST
@@ -495,6 +475,7 @@ pub async fn documents_by_query_post(
     analytics.publish(
         DocumentsFetchAggregator::<DocumentsPOST> {
             per_filter: body.filter.is_some(),
+            sort: body.sort.is_some(),
             retrieve_vectors: body.retrieve_vectors,
             max_limit: body.limit,
             max_offset: body.offset,
@@ -571,7 +552,7 @@ pub async fn get_documents(
 ) -> Result<HttpResponse, ResponseError> {
     debug!(parameters = ?params, "Get documents GET");
 
-    let BrowseQueryGet { limit, offset, fields, retrieve_vectors, filter, ids } =
+    let BrowseQueryGet { limit, offset, fields, retrieve_vectors, filter, ids, sort } =
         params.into_inner();
 
     let filter = match filter {
@@ -582,20 +563,20 @@ pub async fn get_documents(
         None => None,
     };
 
-    let ids = ids.map(|ids| ids.into_iter().map(Into::into).collect());
-
     let query = BrowseQuery {
         offset: offset.0,
         limit: limit.0,
         fields: fields.merge_star_and_none(),
         retrieve_vectors: retrieve_vectors.0,
         filter,
-        ids,
+        ids: ids.map(|ids| ids.into_iter().map(Into::into).collect()),
+        sort: sort.map(|attr| fix_sort_query_parameters(&attr)),
     };
 
     analytics.publish(
         DocumentsFetchAggregator::<DocumentsGET> {
             per_filter: query.filter.is_some(),
+            sort: query.sort.is_some(),
             retrieve_vectors: query.retrieve_vectors,
             max_limit: query.limit,
             max_offset: query.offset,
@@ -615,7 +596,7 @@ fn documents_by_query(
     query: BrowseQuery,
 ) -> Result<HttpResponse, ResponseError> {
     let index_uid = IndexUid::try_from(index_uid.into_inner())?;
-    let BrowseQuery { offset, limit, fields, retrieve_vectors, filter, ids } = query;
+    let BrowseQuery { offset, limit, fields, retrieve_vectors, filter, ids, sort } = query;
 
     let retrieve_vectors = RetrieveVectors::new(retrieve_vectors);
 
@@ -633,6 +614,18 @@ fn documents_by_query(
         None
     };
 
+    let sort_criteria = if let Some(sort) = &sort {
+        let sorts: Vec<_> = match sort.iter().map(|s| milli::AscDesc::from_str(s)).collect() {
+            Ok(sorts) => sorts,
+            Err(asc_desc_error) => {
+                return Err(milli::SortError::from(asc_desc_error).into_document_error().into())
+            }
+        };
+        Some(sorts)
+    } else {
+        None
+    };
+
     let index = index_scheduler.index(&index_uid)?;
     let (total, documents) = retrieve_documents(
         &index,
@@ -643,6 +636,7 @@ fn documents_by_query(
         fields,
         retrieve_vectors,
         index_scheduler.features(),
+        sort_criteria,
     )?;
 
     let ret = PaginationView::new(offset, limit, total as usize, documents);
@@ -1467,9 +1461,13 @@ fn some_documents<'a, 't: 'a>(
                         Some(Value::Object(map)) => map,
                         _ => Default::default(),
                     };
-                    for (name, (vector, regenerate)) in index.embeddings(rtxn, key)? {
+                    for (
+                        name,
+                        EmbeddingsWithMetadata { embeddings, regenerate, has_fragments: _ },
+                    ) in index.embeddings(rtxn, key)?
+                    {
                         let embeddings =
-                            ExplicitVectors { embeddings: Some(vector.into()), regenerate };
+                            ExplicitVectors { embeddings: Some(embeddings.into()), regenerate };
                         vectors.insert(
                             name,
                             serde_json::to_value(embeddings).map_err(MeilisearchHttpError::from)?,
@@ -1494,6 +1492,7 @@ fn retrieve_documents<S: AsRef<str>>(
     attributes_to_retrieve: Option<Vec<S>>,
     retrieve_vectors: RetrieveVectors,
     features: RoFeatures,
+    sort_criteria: Option<Vec<AscDesc>>,
 ) -> Result<(u64, Vec<Document>), ResponseError> {
     let rtxn = index.read_txn()?;
     let filter = &filter;
@@ -1526,15 +1525,32 @@ fn retrieve_documents<S: AsRef<str>>(
         })?
     }
 
-    let (it, number_of_documents) = {
+    let (it, number_of_documents) = if let Some(sort) = sort_criteria {
+        let number_of_documents = candidates.len();
+        let facet_sort = recursive_sort(index, &rtxn, sort, &candidates)?;
+        let iter = facet_sort.iter()?;
+        let mut documents = Vec::with_capacity(limit);
+        for result in iter.skip(offset).take(limit) {
+            documents.push(result?);
+        }
+        (
+            itertools::Either::Left(some_documents(
+                index,
+                &rtxn,
+                documents.into_iter(),
+                retrieve_vectors,
+            )?),
+            number_of_documents,
+        )
+    } else {
         let number_of_documents = candidates.len();
         (
-            some_documents(
+            itertools::Either::Right(some_documents(
                 index,
                 &rtxn,
                 candidates.into_iter().skip(offset).take(limit),
                 retrieve_vectors,
-            )?,
+            )?),
             number_of_documents,
         )
     };

crates/meilisearch/src/search/federated/perform.rs

@@ -745,10 +745,9 @@ impl SearchByIndex {
                         match sort.iter().map(|s| milli::AscDesc::from_str(s)).collect() {
                             Ok(sorts) => sorts,
                             Err(asc_desc_error) => {
-                                return Err(milli::Error::from(milli::SortError::from(
-                                    asc_desc_error,
-                                ))
-                                .into())
+                                return Err(milli::SortError::from(asc_desc_error)
+                                    .into_search_error()
+                                    .into())
                             }
                         };
                     Some(sorts)

crates/meilisearch/src/search/mod.rs

@@ -16,7 +16,7 @@ use meilisearch_types::error::{Code, ResponseError};
 use meilisearch_types::heed::RoTxn;
 use meilisearch_types::index_uid::IndexUid;
 use meilisearch_types::locales::Locale;
-use meilisearch_types::milli::index::{self, SearchParameters};
+use meilisearch_types::milli::index::{self, EmbeddingsWithMetadata, SearchParameters};
 use meilisearch_types::milli::score_details::{ScoreDetails, ScoringStrategy};
 use meilisearch_types::milli::vector::parsed_vectors::ExplicitVectors;
 use meilisearch_types::milli::vector::Embedder;
@@ -1051,6 +1051,7 @@ pub fn prepare_search<'t>(
         .unwrap_or(DEFAULT_PAGINATION_MAX_TOTAL_HITS);
 
     search.exhaustive_number_hits(is_finite_pagination);
+    search.max_total_hits(Some(max_total_hits));
     search.scoring_strategy(
         if query.show_ranking_score
             || query.show_ranking_score_details
@@ -1091,7 +1092,7 @@ pub fn prepare_search<'t>(
         let sort = match sort.iter().map(|s| AscDesc::from_str(s)).collect() {
             Ok(sorts) => sorts,
             Err(asc_desc_error) => {
-                return Err(milli::Error::from(SortError::from(asc_desc_error)).into())
+                return Err(SortError::from(asc_desc_error).into_search_error().into())
             }
         };
 
@@ -1527,8 +1528,11 @@ impl<'a> HitMaker<'a> {
                 Some(Value::Object(map)) => map,
                 _ => Default::default(),
             };
-            for (name, (vector, regenerate)) in self.index.embeddings(self.rtxn, id)? {
-                let embeddings = ExplicitVectors { embeddings: Some(vector.into()), regenerate };
+            for (name, EmbeddingsWithMetadata { embeddings, regenerate, has_fragments: _ }) in
+                self.index.embeddings(self.rtxn, id)?
+            {
+                let embeddings =
+                    ExplicitVectors { embeddings: Some(embeddings.into()), regenerate };
                 vectors.insert(
                     name,
                     serde_json::to_value(embeddings).map_err(InternalError::SerdeJson)?,

crates/meilisearch/tests/auth/api_keys.rs

@@ -419,14 +419,14 @@ async fn error_add_api_key_invalid_parameters_actions() {
     let (response, code) = server.add_api_key(content).await;
 
     meili_snap::snapshot!(code, @"400 Bad Request");
-    meili_snap::snapshot!(meili_snap::json_string!(response, { ".createdAt" => "[ignored]", ".updatedAt" => "[ignored]" }), @r###"
+    meili_snap::snapshot!(meili_snap::json_string!(response, { ".createdAt" => "[ignored]", ".updatedAt" => "[ignored]" }), @r#"
     {
-      "message": "Unknown value `doc.add` at `.actions[0]`: expected one of `*`, `search`, `documents.*`, `documents.add`, `documents.get`, `documents.delete`, `indexes.*`, `indexes.create`, `indexes.get`, `indexes.update`, `indexes.delete`, `indexes.swap`, `tasks.*`, `tasks.cancel`, `tasks.delete`, `tasks.get`, `settings.*`, `settings.get`, `settings.update`, `stats.*`, `stats.get`, `metrics.*`, `metrics.get`, `dumps.*`, `dumps.create`, `snapshots.*`, `snapshots.create`, `version`, `keys.create`, `keys.get`, `keys.update`, `keys.delete`, `experimental.get`, `experimental.update`, `export`, `network.get`, `network.update`, `chatCompletions`, `chats.*`, `chats.get`, `chats.delete`, `chatsSettings.*`, `chatsSettings.get`, `chatsSettings.update`",
+      "message": "Unknown value `doc.add` at `.actions[0]`: expected one of `*`, `search`, `documents.*`, `documents.add`, `documents.get`, `documents.delete`, `indexes.*`, `indexes.create`, `indexes.get`, `indexes.update`, `indexes.delete`, `indexes.swap`, `tasks.*`, `tasks.cancel`, `tasks.delete`, `tasks.get`, `settings.*`, `settings.get`, `settings.update`, `stats.*`, `stats.get`, `metrics.*`, `metrics.get`, `dumps.*`, `dumps.create`, `snapshots.*`, `snapshots.create`, `version`, `keys.create`, `keys.get`, `keys.update`, `keys.delete`, `experimental.get`, `experimental.update`, `export`, `network.get`, `network.update`, `chatCompletions`, `chats.*`, `chats.get`, `chats.delete`, `chatsSettings.*`, `chatsSettings.get`, `chatsSettings.update`, `*.get`",
       "code": "invalid_api_key_actions",
       "type": "invalid_request",
       "link": "https://docs.meilisearch.com/errors#invalid_api_key_actions"
     }
-    "###);
+    "#);
 }
 
 #[actix_rt::test]
@@ -790,7 +790,7 @@ async fn list_api_keys() {
     meili_snap::snapshot!(code, @"201 Created");
 
     let (response, code) = server.list_api_keys("").await;
-    meili_snap::snapshot!(meili_snap::json_string!(response, { ".results[].createdAt" => "[ignored]", ".results[].updatedAt" => "[ignored]", ".results[].uid" => "[ignored]", ".results[].key" => "[ignored]" }), @r###"
+    meili_snap::snapshot!(meili_snap::json_string!(response, { ".results[].createdAt" => "[ignored]", ".results[].updatedAt" => "[ignored]", ".results[].uid" => "[ignored]", ".results[].key" => "[ignored]" }), @r#"
     {
       "results": [
         {
@@ -850,6 +850,22 @@ async fn list_api_keys() {
           "createdAt": "[ignored]",
           "updatedAt": "[ignored]"
         },
+        {
+          "name": "Default Read-Only Admin API Key",
+          "description": "Use it to read information across the whole database. Caution! Do not expose this key on a public frontend",
+          "key": "[ignored]",
+          "uid": "[ignored]",
+          "actions": [
+            "*.get",
+            "keys.get"
+          ],
+          "indexes": [
+            "*"
+          ],
+          "expiresAt": null,
+          "createdAt": "[ignored]",
+          "updatedAt": "[ignored]"
+        },
         {
           "name": "Default Chat API Key",
           "description": "Use it to chat and search from the frontend",
@@ -869,9 +885,9 @@ async fn list_api_keys() {
       ],
       "offset": 0,
       "limit": 20,
-      "total": 4
+      "total": 5
     }
-    "###);
+    "#);
     meili_snap::snapshot!(code, @"200 OK");
 }
 

crates/meilisearch/tests/auth/errors.rs

@@ -91,14 +91,14 @@ async fn create_api_key_bad_actions() {
     // can't parse
     let (response, code) = server.add_api_key(json!({ "actions": ["doggo"] })).await;
     snapshot!(code, @"400 Bad Request");
-    snapshot!(json_string!(response), @r###"
+    snapshot!(json_string!(response), @r#"
     {
-      "message": "Unknown value `doggo` at `.actions[0]`: expected one of `*`, `search`, `documents.*`, `documents.add`, `documents.get`, `documents.delete`, `indexes.*`, `indexes.create`, `indexes.get`, `indexes.update`, `indexes.delete`, `indexes.swap`, `tasks.*`, `tasks.cancel`, `tasks.delete`, `tasks.get`, `settings.*`, `settings.get`, `settings.update`, `stats.*`, `stats.get`, `metrics.*`, `metrics.get`, `dumps.*`, `dumps.create`, `snapshots.*`, `snapshots.create`, `version`, `keys.create`, `keys.get`, `keys.update`, `keys.delete`, `experimental.get`, `experimental.update`, `export`, `network.get`, `network.update`, `chatCompletions`, `chats.*`, `chats.get`, `chats.delete`, `chatsSettings.*`, `chatsSettings.get`, `chatsSettings.update`",
+      "message": "Unknown value `doggo` at `.actions[0]`: expected one of `*`, `search`, `documents.*`, `documents.add`, `documents.get`, `documents.delete`, `indexes.*`, `indexes.create`, `indexes.get`, `indexes.update`, `indexes.delete`, `indexes.swap`, `tasks.*`, `tasks.cancel`, `tasks.delete`, `tasks.get`, `settings.*`, `settings.get`, `settings.update`, `stats.*`, `stats.get`, `metrics.*`, `metrics.get`, `dumps.*`, `dumps.create`, `snapshots.*`, `snapshots.create`, `version`, `keys.create`, `keys.get`, `keys.update`, `keys.delete`, `experimental.get`, `experimental.update`, `export`, `network.get`, `network.update`, `chatCompletions`, `chats.*`, `chats.get`, `chats.delete`, `chatsSettings.*`, `chatsSettings.get`, `chatsSettings.update`, `*.get`",
       "code": "invalid_api_key_actions",
       "type": "invalid_request",
       "link": "https://docs.meilisearch.com/errors#invalid_api_key_actions"
     }
-    "###);
+    "#);
 }
 
 #[actix_rt::test]

crates/meilisearch/tests/common/index.rs

@@ -551,5 +551,7 @@ pub struct GetAllDocumentsOptions {
     pub offset: Option<usize>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub fields: Option<Vec<&'static str>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub sort: Option<Vec<&'static str>>,
     pub retrieve_vectors: bool,
 }

crates/meilisearch/tests/common/mod.rs

@@ -3,8 +3,10 @@ pub mod index;
 pub mod server;
 pub mod service;
 
+use std::collections::BTreeMap;
 use std::fmt::{self, Display};
 
+use actix_http::StatusCode;
 #[allow(unused)]
 pub use index::GetAllDocumentsOptions;
 use meili_snap::json_string;
@@ -13,6 +15,8 @@ use serde::{Deserialize, Serialize};
 #[allow(unused)]
 pub use server::{default_settings, Server};
 use tokio::sync::OnceCell;
+use wiremock::matchers::{method, path};
+use wiremock::{Mock, MockServer, Request, ResponseTemplate};
 
 use crate::common::index::Index;
 
@@ -517,3 +521,166 @@ pub async fn shared_index_with_geo_documents() -> &'static Index<'static, Shared
         })
         .await
 }
+
+pub async fn shared_index_for_fragments() -> Index<'static, Shared> {
+    static INDEX: OnceCell<(Server<Shared>, String)> = OnceCell::const_new();
+    let (server, uid) = INDEX
+        .get_or_init(|| async {
+            let (server, uid, _) = init_fragments_index().await;
+            (server.into_shared(), uid)
+        })
+        .await;
+    server._index(uid).to_shared()
+}
+
+async fn fragment_mock_server() -> String {
+    let text_to_embedding: BTreeMap<_, _> = vec![
+        ("kefir", [0.5, -0.5, 0.0]),
+        ("intel", [1.0, 1.0, 0.0]),
+        ("dustin", [-0.5, 0.5, 0.0]),
+        ("bulldog", [0.0, 0.0, 1.0]),
+        ("labrador", [0.0, 0.0, -1.0]),
+        ("{{ doc.", [-9999.0, -9999.0, -9999.0]), // If a template didn't render
+    ]
+    .into_iter()
+    .collect();
+
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
+
+    Mock::given(method("POST"))
+        .and(path("/"))
+        .respond_with(move |req: &Request| {
+            let text = String::from_utf8_lossy(&req.body).to_string();
+
+            let mut data = [0.0, 0.0, 0.0];
+            for (inner_text, inner_data) in &text_to_embedding {
+                if text.contains(inner_text) {
+                    for (i, &value) in inner_data.iter().enumerate() {
+                        data[i] += value;
+                    }
+                }
+            }
+            ResponseTemplate::new(200).set_body_json(json!({ "data": data }))
+        })
+        .mount(mock_server)
+        .await;
+
+    mock_server.uri()
+}
+
+pub async fn init_fragments_index() -> (Server<Owned>, String, crate::common::Value) {
+    let url = fragment_mock_server().await;
+    let server = Server::new().await;
+    let index = server.unique_index();
+
+    let (_response, code) = server.set_features(json!({"multimodal": true})).await;
+    assert_eq!(code, StatusCode::OK);
+
+    // Configure the index to use our mock embedder
+    let settings = json!({
+        "embedders": {
+            "rest": {
+                "source": "rest",
+                "url": url,
+                "dimensions": 3,
+                "request": "{{fragment}}",
+                "response": {
+                    "data": "{{embedding}}"
+                },
+                "indexingFragments": {
+                    "withBreed": {"value": "{{ doc.name }} is a {{ doc.breed }}"},
+                    "basic": {"value": "{{ doc.name }} is a dog"},
+                },
+                "searchFragments": {
+                    "justBreed": {"value": "It's a {{ media.breed }}"},
+                    "justName": {"value": "{{ media.name }} is a dog"},
+                    "query": {"value": "Some pre-prompt for query {{ q }}"},
+                }
+            },
+        },
+    });
+    let (response, code) = index.update_settings(settings.clone()).await;
+    assert_eq!(code, StatusCode::ACCEPTED);
+
+    server.wait_task(response.uid()).await.succeeded();
+
+    // Send documents
+    let documents = json!([
+        {"id": 0, "name": "kefir"},
+        {"id": 1, "name": "echo", "_vectors": { "rest": [1, 1, 1] }},
+        {"id": 2, "name": "intel", "breed": "labrador"},
+        {"id": 3, "name": "dustin", "breed": "bulldog"},
+    ]);
+    let (value, code) = index.add_documents(documents, None).await;
+    assert_eq!(code, StatusCode::ACCEPTED);
+
+    let _task = server.wait_task(value.uid()).await.succeeded();
+
+    let uid = index.uid.clone();
+    (server, uid, settings)
+}
+
+pub async fn init_fragments_index_composite() -> (Server<Owned>, String, crate::common::Value) {
+    let url = fragment_mock_server().await;
+    let server = Server::new().await;
+    let index = server.unique_index();
+
+    let (_response, code) = server.set_features(json!({"multimodal": true})).await;
+    assert_eq!(code, StatusCode::OK);
+
+    let (_response, code) = server.set_features(json!({"compositeEmbedders": true})).await;
+    assert_eq!(code, StatusCode::OK);
+
+    // Configure the index to use our mock embedder
+    let settings = json!({
+        "embedders": {
+            "rest": {
+                "source": "composite",
+                "searchEmbedder": {
+                    "source": "rest",
+                    "url": url,
+                    "dimensions": 3,
+                    "request": "{{fragment}}",
+                    "response": {
+                        "data": "{{embedding}}"
+                    },
+                    "searchFragments": {
+                        "query": {"value": "Some pre-prompt for query {{ q }}"},
+                    }
+                },
+                "indexingEmbedder": {
+                    "source": "rest",
+                    "url": url,
+                    "dimensions": 3,
+                    "request": "{{fragment}}",
+                    "response": {
+                        "data": "{{embedding}}"
+                    },
+                    "indexingFragments": {
+                        "withBreed": {"value": "{{ doc.name }} is a {{ doc.breed }}"},
+                        "basic": {"value": "{{ doc.name }} is a dog"},
+                    }
+                },
+            },
+        },
+    });
+    let (response, code) = index.update_settings(settings.clone()).await;
+    assert_eq!(code, StatusCode::ACCEPTED);
+
+    server.wait_task(response.uid()).await.succeeded();
+
+    // Send documents
+    let documents = json!([
+        {"id": 0, "name": "kefir"},
+        {"id": 1, "name": "echo", "_vectors": { "rest": [1, 1, 1] }},
+        {"id": 2, "name": "intel", "breed": "labrador"},
+        {"id": 3, "name": "dustin", "breed": "bulldog"},
+    ]);
+    let (value, code) = index.add_documents(documents, None).await;
+    assert_eq!(code, StatusCode::ACCEPTED);
+
+    server.wait_task(value.uid()).await.succeeded();
+
+    let uid = index.uid.clone();
+    (server, uid, settings)
+}

crates/meilisearch/tests/common/server.rs

@@ -35,7 +35,7 @@ pub struct Server<State = Owned> {
 pub static TEST_TEMP_DIR: Lazy<TempDir> = Lazy::new(|| TempDir::new().unwrap());
 
 impl Server<Owned> {
-    fn into_shared(self) -> Server<Shared> {
+    pub(super) fn into_shared(self) -> Server<Shared> {
         Server { service: self.service, _dir: self._dir, _marker: PhantomData }
     }
 
@@ -97,6 +97,7 @@ impl Server<Owned> {
         self.use_api_key(master_key);
         let (response, code) = self.list_api_keys("").await;
         assert_eq!(200, code, "{:?}", response);
+        // TODO: relying on the order of keys is not ideal, we should use the name instead
         let admin_key = &response["results"][1]["key"];
         self.use_api_key(admin_key.as_str().unwrap());
     }
@@ -465,6 +466,7 @@ pub fn default_settings(dir: impl AsRef<Path>) -> Opt {
             // Having 2 threads makes the tests way faster
             max_indexing_threads: MaxThreads::from_str("2").unwrap(),
             experimental_no_edition_2024_for_settings: false,
+            experimental_no_edition_2024_for_dumps: false,
         },
         experimental_enable_metrics: false,
         ..Parser::parse_from(None as Option<&str>)

crates/meilisearch/tests/documents/get_documents.rs

@@ -5,8 +5,8 @@ use urlencoding::encode as urlencode;
 
 use crate::common::encoder::Encoder;
 use crate::common::{
-    shared_does_not_exists_index, shared_empty_index, shared_index_with_test_set,
-    GetAllDocumentsOptions, Server, Value,
+    shared_does_not_exists_index, shared_empty_index, shared_index_with_geo_documents,
+    shared_index_with_test_set, GetAllDocumentsOptions, Server, Value,
 };
 use crate::json;
 
@@ -83,6 +83,311 @@ async fn get_document() {
     );
 }
 
+#[actix_rt::test]
+async fn get_document_sorted() {
+    let server = Server::new_shared();
+    let index = server.unique_index();
+    index.load_test_set(server).await;
+
+    let (task, _status_code) =
+        index.update_settings_sortable_attributes(json!(["age", "email", "gender", "name"])).await;
+    server.wait_task(task.uid()).await.succeeded();
+
+    let (response, _code) = index
+        .get_all_documents(GetAllDocumentsOptions {
+            fields: Some(vec!["id", "age", "email"]),
+            sort: Some(vec!["age:asc", "email:desc"]),
+            ..Default::default()
+        })
+        .await;
+    let results = response["results"].as_array().unwrap();
+    snapshot!(json_string!(results), @r#"
+    [
+      {
+        "id": 5,
+        "age": 20,
+        "email": "warrenwatson@chorizon.com"
+      },
+      {
+        "id": 6,
+        "age": 20,
+        "email": "sheliaberry@chorizon.com"
+      },
+      {
+        "id": 57,
+        "age": 20,
+        "email": "kaitlinconner@chorizon.com"
+      },
+      {
+        "id": 45,
+        "age": 20,
+        "email": "irenebennett@chorizon.com"
+      },
+      {
+        "id": 40,
+        "age": 21,
+        "email": "staffordemerson@chorizon.com"
+      },
+      {
+        "id": 41,
+        "age": 21,
+        "email": "salinasgamble@chorizon.com"
+      },
+      {
+        "id": 63,
+        "age": 21,
+        "email": "knowleshebert@chorizon.com"
+      },
+      {
+        "id": 50,
+        "age": 21,
+        "email": "guerramcintyre@chorizon.com"
+      },
+      {
+        "id": 44,
+        "age": 22,
+        "email": "jonispears@chorizon.com"
+      },
+      {
+        "id": 56,
+        "age": 23,
+        "email": "tuckerbarry@chorizon.com"
+      },
+      {
+        "id": 51,
+        "age": 23,
+        "email": "keycervantes@chorizon.com"
+      },
+      {
+        "id": 60,
+        "age": 23,
+        "email": "jodyherrera@chorizon.com"
+      },
+      {
+        "id": 70,
+        "age": 23,
+        "email": "glassperkins@chorizon.com"
+      },
+      {
+        "id": 75,
+        "age": 24,
+        "email": "emmajacobs@chorizon.com"
+      },
+      {
+        "id": 68,
+        "age": 24,
+        "email": "angelinadyer@chorizon.com"
+      },
+      {
+        "id": 17,
+        "age": 25,
+        "email": "ortegabrennan@chorizon.com"
+      },
+      {
+        "id": 76,
+        "age": 25,
+        "email": "claricegardner@chorizon.com"
+      },
+      {
+        "id": 43,
+        "age": 25,
+        "email": "arnoldbender@chorizon.com"
+      },
+      {
+        "id": 12,
+        "age": 25,
+        "email": "aidakirby@chorizon.com"
+      },
+      {
+        "id": 9,
+        "age": 26,
+        "email": "kellimendez@chorizon.com"
+      }
+    ]
+    "#);
+
+    let (response, _code) = index
+        .get_all_documents(GetAllDocumentsOptions {
+            fields: Some(vec!["id", "gender", "name"]),
+            sort: Some(vec!["gender:asc", "name:asc"]),
+            ..Default::default()
+        })
+        .await;
+    let results = response["results"].as_array().unwrap();
+    snapshot!(json_string!(results), @r#"
+    [
+      {
+        "id": 3,
+        "name": "Adeline Flynn",
+        "gender": "female"
+      },
+      {
+        "id": 12,
+        "name": "Aida Kirby",
+        "gender": "female"
+      },
+      {
+        "id": 68,
+        "name": "Angelina Dyer",
+        "gender": "female"
+      },
+      {
+        "id": 15,
+        "name": "Aurelia Contreras",
+        "gender": "female"
+      },
+      {
+        "id": 36,
+        "name": "Barbra Valenzuela",
+        "gender": "female"
+      },
+      {
+        "id": 23,
+        "name": "Blanca Mcclain",
+        "gender": "female"
+      },
+      {
+        "id": 53,
+        "name": "Caitlin Burnett",
+        "gender": "female"
+      },
+      {
+        "id": 71,
+        "name": "Candace Sawyer",
+        "gender": "female"
+      },
+      {
+        "id": 65,
+        "name": "Carole Rowland",
+        "gender": "female"
+      },
+      {
+        "id": 33,
+        "name": "Cecilia Greer",
+        "gender": "female"
+      },
+      {
+        "id": 1,
+        "name": "Cherry Orr",
+        "gender": "female"
+      },
+      {
+        "id": 38,
+        "name": "Christina Short",
+        "gender": "female"
+      },
+      {
+        "id": 7,
+        "name": "Chrystal Boyd",
+        "gender": "female"
+      },
+      {
+        "id": 76,
+        "name": "Clarice Gardner",
+        "gender": "female"
+      },
+      {
+        "id": 73,
+        "name": "Eleanor Shepherd",
+        "gender": "female"
+      },
+      {
+        "id": 75,
+        "name": "Emma Jacobs",
+        "gender": "female"
+      },
+      {
+        "id": 16,
+        "name": "Estella Bass",
+        "gender": "female"
+      },
+      {
+        "id": 62,
+        "name": "Estelle Ramirez",
+        "gender": "female"
+      },
+      {
+        "id": 20,
+        "name": "Florence Long",
+        "gender": "female"
+      },
+      {
+        "id": 42,
+        "name": "Graciela Russell",
+        "gender": "female"
+      }
+    ]
+    "#);
+}
+
+#[actix_rt::test]
+async fn get_document_geosorted() {
+    let index = shared_index_with_geo_documents().await;
+
+    let (response, _code) = index
+        .get_all_documents(GetAllDocumentsOptions {
+            sort: Some(vec!["_geoPoint(45.4777599, 9.1967508):asc"]),
+            ..Default::default()
+        })
+        .await;
+    let results = response["results"].as_array().unwrap();
+    snapshot!(json_string!(results), @r#"
+    [
+      {
+        "id": 2,
+        "name": "La Bella Italia",
+        "address": "456 Elm Street, Townsville",
+        "type": "Italian",
+        "rating": 9,
+        "_geo": {
+          "lat": "45.4777599",
+          "lng": "9.1967508"
+        }
+      },
+      {
+        "id": 1,
+        "name": "Taco Truck",
+        "address": "444 Salsa Street, Burritoville",
+        "type": "Mexican",
+        "rating": 9,
+        "_geo": {
+          "lat": 34.0522,
+          "lng": -118.2437
+        }
+      },
+      {
+        "id": 3,
+        "name": "Crêpe Truck",
+        "address": "2 Billig Avenue, Rouenville",
+        "type": "French",
+        "rating": 10
+      }
+    ]
+    "#);
+}
+
+#[actix_rt::test]
+async fn get_document_sort_the_unsortable() {
+    let index = shared_index_with_test_set().await;
+
+    let (response, _code) = index
+        .get_all_documents(GetAllDocumentsOptions {
+            fields: Some(vec!["id", "name"]),
+            sort: Some(vec!["name:asc"]),
+            ..Default::default()
+        })
+        .await;
+
+    snapshot!(json_string!(response), @r#"
+    {
+      "message": "Attribute `name` is not sortable. This index does not have configured sortable attributes.",
+      "code": "invalid_document_sort",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#invalid_document_sort"
+    }
+    "#);
+}
+
 #[actix_rt::test]
 async fn error_get_unexisting_index_all_documents() {
     let index = shared_does_not_exists_index().await;

crates/meilisearch/tests/index/stats.rs

@@ -1,5 +1,4 @@
 use crate::common::{shared_does_not_exists_index, Server};
-
 use crate::json;
 
 #[actix_rt::test]

crates/meilisearch/tests/search/multi/proxy.rs

@@ -2500,7 +2500,7 @@ pub struct LocalMeiliParams {
 
 /// A server that exploits [`MockServer`] to provide an URL for testing network and the network.
 pub struct LocalMeili {
-    mock_server: MockServer,
+    mock_server: &'static MockServer,
 }
 
 impl LocalMeili {
@@ -2509,7 +2509,7 @@ impl LocalMeili {
     }
 
     pub async fn with_params(server: Arc<Server>, params: LocalMeiliParams) -> Self {
-        let mock_server = MockServer::start().await;
+        let mock_server = Box::leak(Box::new(MockServer::start().await));
 
         // tokio won't let us execute asynchronous code from a sync function inside of an async test,
         // so instead we spawn another thread that will call the service on a brand new tokio runtime
@@ -2573,7 +2573,7 @@ impl LocalMeili {
                     response.set_body_json(value)
                 }
             })
-            .mount(&mock_server)
+            .mount(mock_server)
             .await;
         Self { mock_server }
     }

crates/meilisearch/tests/search/pagination.rs

@@ -1,6 +1,7 @@
 use super::shared_index_with_documents;
 use crate::common::Server;
 use crate::json;
+use meili_snap::{json_string, snapshot};
 
 #[actix_rt::test]
 async fn default_search_should_return_estimated_total_hit() {
@@ -133,3 +134,61 @@ async fn ensure_placeholder_search_hit_count_valid() {
             .await;
     }
 }
+
+#[actix_rt::test]
+async fn test_issue_5274() {
+    let server = Server::new_shared();
+    let index = server.unique_index();
+
+    let documents = json!([
+        {
+            "id": 1,
+            "title": "Document 1",
+            "content": "This is the first."
+        },
+        {
+            "id": 2,
+            "title": "Document 2",
+            "content": "This is the second doc."
+        }
+    ]);
+    let (task, _code) = index.add_documents(documents, None).await;
+    server.wait_task(task.uid()).await.succeeded();
+
+    // Find out the lowest ranking score among the documents
+    let (rep, _status) = index
+        .search_post(json!({"q": "doc", "page": 1, "hitsPerPage": 2, "showRankingScore": true}))
+        .await;
+    let hits = rep["hits"].as_array().expect("Missing hits array");
+    let second_hit = hits.get(1).expect("Missing second hit");
+    let ranking_score = second_hit
+        .get("_rankingScore")
+        .expect("Missing _rankingScore field")
+        .as_f64()
+        .expect("Expected _rankingScore to be a f64");
+
+    // Search with a ranking score threshold just above and expect to be a single hit
+    let (rep, _status) = index
+        .search_post(json!({"q": "doc", "page": 1, "hitsPerPage": 1, "rankingScoreThreshold": ranking_score + 0.0001}))
+        .await;
+
+    snapshot!(json_string!(rep, {
+        ".processingTimeMs" => "[ignored]",
+    }), @r#"
+    {
+      "hits": [
+        {
+          "id": 2,
+          "title": "Document 2",
+          "content": "This is the second doc."
+        }
+      ],
+      "query": "doc",
+      "processingTimeMs": "[ignored]",
+      "hitsPerPage": 1,
+      "page": 1,
+      "totalPages": 1,
+      "totalHits": 1
+    }
+    "#);
+}

crates/meilisearch/tests/settings/get_settings.rs

@@ -692,3 +692,68 @@ async fn granular_filterable_attributes() {
     ]
     "###);
 }
+
+#[actix_rt::test]
+async fn test_searchable_attributes_order() {
+    let server = Server::new_shared();
+    let index = server.unique_index();
+
+    // 1) Create an index with settings "searchableAttributes": ["title", "overview"]
+    let (response, code) = index.create(None).await;
+    assert_eq!(code, 202, "{response}");
+    server.wait_task(response.uid()).await.succeeded();
+
+    let (task, code) = index
+        .update_settings(json!({
+            "searchableAttributes": ["title", "overview"]
+        }))
+        .await;
+    assert_eq!(code, 202, "{task}");
+    server.wait_task(task.uid()).await.succeeded();
+
+    // 2) Add documents in the index
+    let documents = json!([
+        {
+            "id": 1,
+            "title": "The Matrix",
+            "overview": "A computer hacker learns from mysterious rebels about the true nature of his reality."
+        },
+        {
+            "id": 2,
+            "title": "Inception",
+            "overview": "A thief who steals corporate secrets through dream-sharing technology."
+        }
+    ]);
+
+    let (response, code) = index.add_documents(documents, None).await;
+    assert_eq!(code, 202, "{response}");
+    server.wait_task(response.uid()).await.succeeded();
+
+    // 3) Modify the settings "searchableAttributes": ["overview", "title"] (overview is put first)
+    let (task, code) = index
+        .update_settings(json!({
+            "searchableAttributes": ["overview", "title"]
+        }))
+        .await;
+    assert_eq!(code, 202, "{task}");
+    server.wait_task(task.uid()).await.succeeded();
+
+    // 4) Check if it has been applied
+    let (response, code) = index.settings().await;
+    assert_eq!(code, 200, "{response}");
+    assert_eq!(response["searchableAttributes"], json!(["overview", "title"]));
+
+    // 5) Re-modify the settings "searchableAttributes": ["title", "overview"] (title is put first)
+    let (task, code) = index
+        .update_settings(json!({
+            "searchableAttributes": ["title", "overview"]
+        }))
+        .await;
+    assert_eq!(code, 202, "{task}");
+    server.wait_task(task.uid()).await.succeeded();
+
+    // 6) Check if it has been applied
+    let (response, code) = index.settings().await;
+    assert_eq!(code, 200, "{response}");
+    assert_eq!(response["searchableAttributes"], json!(["title", "overview"]));
+}

crates/meilisearch/tests/snapshot/mod.rs

@@ -122,11 +122,7 @@ async fn perform_on_demand_snapshot() {
     let server = Server::new_with_options(options).await.unwrap();
 
     let index = server.index("catto");
-    index
-        .update_settings(json! ({
-        "searchableAttributes": [],
-        }))
-        .await;
+    index.update_settings(json! ({ "searchableAttributes": [] })).await;
 
     index.load_test_set(&server).await;
 
@@ -203,3 +199,70 @@ async fn perform_on_demand_snapshot() {
         server.index("doggo").settings(),
     );
 }
+
+#[actix_rt::test]
+#[cfg_attr(target_os = "windows", ignore)]
+async fn snapshotception_issue_4653() {
+    let temp = tempfile::tempdir().unwrap();
+    let snapshot_dir = tempfile::tempdir().unwrap();
+    let options =
+        Opt { snapshot_dir: snapshot_dir.path().to_owned(), ..default_settings(temp.path()) };
+
+    let server = Server::new_with_options(options).await.unwrap();
+
+    let (task, code) = server.create_snapshot().await;
+    snapshot!(code, @"202 Accepted");
+    snapshot!(json_string!(task, { ".enqueuedAt" => "[date]" }), @r###"
+    {
+      "taskUid": 0,
+      "indexUid": null,
+      "status": "enqueued",
+      "type": "snapshotCreation",
+      "enqueuedAt": "[date]"
+    }
+    "###);
+    server.wait_task(task.uid()).await.succeeded();
+
+    let temp = tempfile::tempdir().unwrap();
+    let snapshot_path = snapshot_dir.path().to_owned().join("db.snapshot");
+
+    let options = Opt { import_snapshot: Some(snapshot_path), ..default_settings(temp.path()) };
+    let snapshot_server = Server::new_with_options(options).await.unwrap();
+
+    // The snapshot should have been taken without the snapshot creation task
+    let (tasks, code) = snapshot_server.tasks().await;
+    snapshot!(code, @"200 OK");
+    snapshot!(tasks, @r#"
+    {
+      "results": [],
+      "total": 0,
+      "limit": 20,
+      "from": null,
+      "next": null
+    }
+    "#);
+
+    // Ensure the task is not present in the snapshot
+    let (task, code) = snapshot_server.get_task(0).await;
+    snapshot!(code, @"404 Not Found");
+    snapshot!(task, @r#"
+    {
+      "message": "Task `0` not found.",
+      "code": "task_not_found",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#task_not_found"
+    }
+    "#);
+
+    // Ensure the batch is also not present
+    let (batch, code) = snapshot_server.get_batch(0).await;
+    snapshot!(code, @"404 Not Found");
+    snapshot!(batch, @r#"
+    {
+      "message": "Batch `0` not found.",
+      "code": "batch_not_found",
+      "type": "invalid_request",
+      "link": "https://docs.meilisearch.com/errors#batch_not_found"
+    }
+    "#);
+}

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_features/kefir_settings.snap

@@ -61,7 +61,16 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
   "pagination": {
     "maxTotalHits": 15
   },
-  "embedders": {},
+  "embedders": {
+    "doggo_embedder": {
+      "source": "huggingFace",
+      "model": "sentence-transformers/all-MiniLM-L6-v2",
+      "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+      "pooling": "forceMean",
+      "documentTemplate": "{{doc.description}}",
+      "documentTemplateMaxBytes": 400
+    }
+  },
   "searchCutoffMs": 8000,
   "localizedAttributes": [
     {

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_features/search_with_retrieve_vectors.snap

@@ -0,0 +1,40 @@
+---
+source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
+---
+[
+  {
+    "id": 1,
+    "name": "kefir",
+    "surname": [
+      "kef",
+      "kefkef",
+      "kefirounet",
+      "boubou"
+    ],
+    "age": 1.4,
+    "description": "kefir est un petit chien blanc très mignon",
+    "_vectors": {
+      "doggo_embedder": {
+        "embeddings": "[vector]",
+        "regenerate": true
+      }
+    }
+  },
+  {
+    "id": 2,
+    "name": "intel",
+    "surname": [
+      "untel",
+      "tétel",
+      "iouiou"
+    ],
+    "age": 11.5,
+    "description": "intel est un grand beagle très mignon",
+    "_vectors": {
+      "doggo_embedder": {
+        "embeddings": "[vector]",
+        "regenerate": false
+      }
+    }
+  }
+]

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/batches_filter_afterEnqueuedAt_equal_2025-01-16T16_47_41.snap

@@ -4,7 +4,7 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 24,
+      "uid": 30,
       "progress": null,
       "details": {
         "upgradeFrom": "v1.12.0",
@@ -26,6 +26,155 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "finishedAt": "[date]",
       "batchStrategy": "stopped after the last task of type `upgradeDatabase` because they cannot be batched with tasks of any other type."
     },
+    {
+      "uid": 29,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.067201S",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 28,
+      "progress": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "indexDeletion": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.012727S",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 27,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "failed": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.059920S",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 26,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.088879S",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 25,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.312911S",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 24,
+      "progress": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "settingsUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.247378S",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z",
+      "batchStrategy": "unspecified"
+    },
     {
       "uid": 23,
       "progress": null,
@@ -348,179 +497,10 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "2025-01-16T17:01:14.112756687Z",
       "finishedAt": "2025-01-16T17:01:14.120064527Z",
       "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 10,
-      "progress": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 99
-        },
-        "pagination": {
-          "maxTotalHits": 15
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007391353S",
-      "startedAt": "2025-01-16T17:00:29.201180268Z",
-      "finishedAt": "2025-01-16T17:00:29.208571621Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 9,
-      "progress": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 100
-        },
-        "pagination": {
-          "maxTotalHits": 1000
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007445825S",
-      "startedAt": "2025-01-16T17:00:15.77629445Z",
-      "finishedAt": "2025-01-16T17:00:15.783740275Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 8,
-      "progress": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          },
-          "disableOnWords": [
-            "kefir"
-          ],
-          "disableOnAttributes": [
-            "surname"
-          ]
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.012020083S",
-      "startedAt": "2025-01-16T16:59:42.744086671Z",
-      "finishedAt": "2025-01-16T16:59:42.756106754Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 7,
-      "progress": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          }
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007440092S",
-      "startedAt": "2025-01-16T16:58:41.2155771Z",
-      "finishedAt": "2025-01-16T16:58:41.223017192Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 6,
-      "progress": null,
-      "details": {
-        "synonyms": {
-          "boubou": [
-            "kefir"
-          ]
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007565161S",
-      "startedAt": "2025-01-16T16:54:51.940332781Z",
-      "finishedAt": "2025-01-16T16:54:51.947897942Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 5,
-      "progress": null,
-      "details": {
-        "stopWords": [
-          "le",
-          "un"
-        ]
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.016307263S",
-      "startedAt": "2025-01-16T16:53:19.913351957Z",
-      "finishedAt": "2025-01-16T16:53:19.92965922Z",
-      "batchStrategy": "unspecified"
     }
   ],
-  "total": 23,
+  "total": 29,
   "limit": 20,
-  "from": 24,
-  "next": 4
+  "from": 30,
+  "next": 10
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/batches_filter_afterFinishedAt_equal_2025-01-16T16_47_41.snap

@@ -4,7 +4,7 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 24,
+      "uid": 30,
       "progress": null,
       "details": {
         "upgradeFrom": "v1.12.0",
@@ -26,6 +26,155 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "finishedAt": "[date]",
       "batchStrategy": "stopped after the last task of type `upgradeDatabase` because they cannot be batched with tasks of any other type."
     },
+    {
+      "uid": 29,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.067201S",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 28,
+      "progress": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "indexDeletion": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.012727S",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 27,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "failed": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.059920S",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 26,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.088879S",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 25,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.312911S",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 24,
+      "progress": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "settingsUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.247378S",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z",
+      "batchStrategy": "unspecified"
+    },
     {
       "uid": 23,
       "progress": null,
@@ -348,179 +497,10 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "2025-01-16T17:01:14.112756687Z",
       "finishedAt": "2025-01-16T17:01:14.120064527Z",
       "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 10,
-      "progress": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 99
-        },
-        "pagination": {
-          "maxTotalHits": 15
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007391353S",
-      "startedAt": "2025-01-16T17:00:29.201180268Z",
-      "finishedAt": "2025-01-16T17:00:29.208571621Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 9,
-      "progress": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 100
-        },
-        "pagination": {
-          "maxTotalHits": 1000
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007445825S",
-      "startedAt": "2025-01-16T17:00:15.77629445Z",
-      "finishedAt": "2025-01-16T17:00:15.783740275Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 8,
-      "progress": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          },
-          "disableOnWords": [
-            "kefir"
-          ],
-          "disableOnAttributes": [
-            "surname"
-          ]
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.012020083S",
-      "startedAt": "2025-01-16T16:59:42.744086671Z",
-      "finishedAt": "2025-01-16T16:59:42.756106754Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 7,
-      "progress": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          }
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007440092S",
-      "startedAt": "2025-01-16T16:58:41.2155771Z",
-      "finishedAt": "2025-01-16T16:58:41.223017192Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 6,
-      "progress": null,
-      "details": {
-        "synonyms": {
-          "boubou": [
-            "kefir"
-          ]
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007565161S",
-      "startedAt": "2025-01-16T16:54:51.940332781Z",
-      "finishedAt": "2025-01-16T16:54:51.947897942Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 5,
-      "progress": null,
-      "details": {
-        "stopWords": [
-          "le",
-          "un"
-        ]
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.016307263S",
-      "startedAt": "2025-01-16T16:53:19.913351957Z",
-      "finishedAt": "2025-01-16T16:53:19.92965922Z",
-      "batchStrategy": "unspecified"
     }
   ],
-  "total": 23,
+  "total": 29,
   "limit": 20,
-  "from": 24,
-  "next": 4
+  "from": 30,
+  "next": 10
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/batches_filter_afterStartedAt_equal_2025-01-16T16_47_41.snap

@@ -4,7 +4,7 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 24,
+      "uid": 30,
       "progress": null,
       "details": {
         "upgradeFrom": "v1.12.0",
@@ -26,6 +26,155 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "finishedAt": "[date]",
       "batchStrategy": "stopped after the last task of type `upgradeDatabase` because they cannot be batched with tasks of any other type."
     },
+    {
+      "uid": 29,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.067201S",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 28,
+      "progress": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "indexDeletion": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.012727S",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 27,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "failed": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.059920S",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 26,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.088879S",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 25,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.312911S",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 24,
+      "progress": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "settingsUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.247378S",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z",
+      "batchStrategy": "unspecified"
+    },
     {
       "uid": 23,
       "progress": null,
@@ -348,179 +497,10 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "2025-01-16T17:01:14.112756687Z",
       "finishedAt": "2025-01-16T17:01:14.120064527Z",
       "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 10,
-      "progress": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 99
-        },
-        "pagination": {
-          "maxTotalHits": 15
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007391353S",
-      "startedAt": "2025-01-16T17:00:29.201180268Z",
-      "finishedAt": "2025-01-16T17:00:29.208571621Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 9,
-      "progress": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 100
-        },
-        "pagination": {
-          "maxTotalHits": 1000
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007445825S",
-      "startedAt": "2025-01-16T17:00:15.77629445Z",
-      "finishedAt": "2025-01-16T17:00:15.783740275Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 8,
-      "progress": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          },
-          "disableOnWords": [
-            "kefir"
-          ],
-          "disableOnAttributes": [
-            "surname"
-          ]
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.012020083S",
-      "startedAt": "2025-01-16T16:59:42.744086671Z",
-      "finishedAt": "2025-01-16T16:59:42.756106754Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 7,
-      "progress": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          }
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007440092S",
-      "startedAt": "2025-01-16T16:58:41.2155771Z",
-      "finishedAt": "2025-01-16T16:58:41.223017192Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 6,
-      "progress": null,
-      "details": {
-        "synonyms": {
-          "boubou": [
-            "kefir"
-          ]
-        }
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.007565161S",
-      "startedAt": "2025-01-16T16:54:51.940332781Z",
-      "finishedAt": "2025-01-16T16:54:51.947897942Z",
-      "batchStrategy": "unspecified"
-    },
-    {
-      "uid": 5,
-      "progress": null,
-      "details": {
-        "stopWords": [
-          "le",
-          "un"
-        ]
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "succeeded": 1
-        },
-        "types": {
-          "settingsUpdate": 1
-        },
-        "indexUids": {
-          "kefir": 1
-        }
-      },
-      "duration": "PT0.016307263S",
-      "startedAt": "2025-01-16T16:53:19.913351957Z",
-      "finishedAt": "2025-01-16T16:53:19.92965922Z",
-      "batchStrategy": "unspecified"
     }
   ],
-  "total": 23,
+  "total": 29,
   "limit": 20,
-  "from": 24,
-  "next": 4
+  "from": 30,
+  "next": 10
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/tasks_filter_afterEnqueuedAt_equal_2025-01-16T16_47_41.snap

@@ -4,8 +4,8 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 25,
-      "batchUid": 24,
+      "uid": 31,
+      "batchUid": 30,
       "indexUid": null,
       "status": "succeeded",
       "type": "upgradeDatabase",
@@ -20,6 +20,118 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "[date]",
       "finishedAt": "[date]"
     },
+    {
+      "uid": 30,
+      "batchUid": 29,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.067201S",
+      "enqueuedAt": "2025-07-07T13:43:08.772432Z",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z"
+    },
+    {
+      "uid": 29,
+      "batchUid": 28,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "indexDeletion",
+      "canceledBy": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.012727S",
+      "enqueuedAt": "2025-07-07T13:42:50.744793Z",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z"
+    },
+    {
+      "uid": 28,
+      "batchUid": 27,
+      "indexUid": "kefir",
+      "status": "failed",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "error": {
+        "message": "Index `kefir`: Bad embedder configuration in the document with id: `2`. Could not parse `._vectors.doggo_embedder`: trailing characters at line 1 column 13",
+        "code": "invalid_vectors_type",
+        "type": "invalid_request",
+        "link": "https://docs.meilisearch.com/errors#invalid_vectors_type"
+      },
+      "duration": "PT0.059920S",
+      "enqueuedAt": "2025-07-07T13:42:15.624598Z",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z"
+    },
+    {
+      "uid": 27,
+      "batchUid": 26,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.088879S",
+      "enqueuedAt": "2025-07-07T13:40:01.46081Z",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z"
+    },
+    {
+      "uid": 26,
+      "batchUid": 25,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.312911S",
+      "enqueuedAt": "2025-07-07T13:32:46.13871Z",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z"
+    },
+    {
+      "uid": 25,
+      "batchUid": 24,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "settingsUpdate",
+      "canceledBy": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "error": null,
+      "duration": "PT0.247378S",
+      "enqueuedAt": "2025-07-07T13:28:27.390054Z",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z"
+    },
     {
       "uid": 24,
       "batchUid": 23,
@@ -264,134 +376,10 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "enqueuedAt": "2025-01-16T17:02:52.527382964Z",
       "startedAt": "2025-01-16T17:02:52.539749853Z",
       "finishedAt": "2025-01-16T17:02:52.547390016Z"
-    },
-    {
-      "uid": 11,
-      "batchUid": 11,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "searchCutoffMs": 8000
-      },
-      "error": null,
-      "duration": "PT0.007307840S",
-      "enqueuedAt": "2025-01-16T17:01:14.100316617Z",
-      "startedAt": "2025-01-16T17:01:14.112756687Z",
-      "finishedAt": "2025-01-16T17:01:14.120064527Z"
-    },
-    {
-      "uid": 10,
-      "batchUid": 10,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 99
-        },
-        "pagination": {
-          "maxTotalHits": 15
-        }
-      },
-      "error": null,
-      "duration": "PT0.007391353S",
-      "enqueuedAt": "2025-01-16T17:00:29.188815062Z",
-      "startedAt": "2025-01-16T17:00:29.201180268Z",
-      "finishedAt": "2025-01-16T17:00:29.208571621Z"
-    },
-    {
-      "uid": 9,
-      "batchUid": 9,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 100
-        },
-        "pagination": {
-          "maxTotalHits": 1000
-        }
-      },
-      "error": null,
-      "duration": "PT0.007445825S",
-      "enqueuedAt": "2025-01-16T17:00:15.759501709Z",
-      "startedAt": "2025-01-16T17:00:15.77629445Z",
-      "finishedAt": "2025-01-16T17:00:15.783740275Z"
-    },
-    {
-      "uid": 8,
-      "batchUid": 8,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          },
-          "disableOnWords": [
-            "kefir"
-          ],
-          "disableOnAttributes": [
-            "surname"
-          ]
-        }
-      },
-      "error": null,
-      "duration": "PT0.012020083S",
-      "enqueuedAt": "2025-01-16T16:59:42.727292501Z",
-      "startedAt": "2025-01-16T16:59:42.744086671Z",
-      "finishedAt": "2025-01-16T16:59:42.756106754Z"
-    },
-    {
-      "uid": 7,
-      "batchUid": 7,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          }
-        }
-      },
-      "error": null,
-      "duration": "PT0.007440092S",
-      "enqueuedAt": "2025-01-16T16:58:41.203145044Z",
-      "startedAt": "2025-01-16T16:58:41.2155771Z",
-      "finishedAt": "2025-01-16T16:58:41.223017192Z"
-    },
-    {
-      "uid": 6,
-      "batchUid": 6,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "synonyms": {
-          "boubou": [
-            "kefir"
-          ]
-        }
-      },
-      "error": null,
-      "duration": "PT0.007565161S",
-      "enqueuedAt": "2025-01-16T16:54:51.927866243Z",
-      "startedAt": "2025-01-16T16:54:51.940332781Z",
-      "finishedAt": "2025-01-16T16:54:51.947897942Z"
     }
   ],
-  "total": 24,
+  "total": 30,
   "limit": 20,
-  "from": 25,
-  "next": 5
+  "from": 31,
+  "next": 11
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/tasks_filter_afterFinishedAt_equal_2025-01-16T16_47_41.snap

@@ -4,8 +4,8 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 25,
-      "batchUid": 24,
+      "uid": 31,
+      "batchUid": 30,
       "indexUid": null,
       "status": "succeeded",
       "type": "upgradeDatabase",
@@ -20,6 +20,118 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "[date]",
       "finishedAt": "[date]"
     },
+    {
+      "uid": 30,
+      "batchUid": 29,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.067201S",
+      "enqueuedAt": "2025-07-07T13:43:08.772432Z",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z"
+    },
+    {
+      "uid": 29,
+      "batchUid": 28,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "indexDeletion",
+      "canceledBy": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.012727S",
+      "enqueuedAt": "2025-07-07T13:42:50.744793Z",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z"
+    },
+    {
+      "uid": 28,
+      "batchUid": 27,
+      "indexUid": "kefir",
+      "status": "failed",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "error": {
+        "message": "Index `kefir`: Bad embedder configuration in the document with id: `2`. Could not parse `._vectors.doggo_embedder`: trailing characters at line 1 column 13",
+        "code": "invalid_vectors_type",
+        "type": "invalid_request",
+        "link": "https://docs.meilisearch.com/errors#invalid_vectors_type"
+      },
+      "duration": "PT0.059920S",
+      "enqueuedAt": "2025-07-07T13:42:15.624598Z",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z"
+    },
+    {
+      "uid": 27,
+      "batchUid": 26,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.088879S",
+      "enqueuedAt": "2025-07-07T13:40:01.46081Z",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z"
+    },
+    {
+      "uid": 26,
+      "batchUid": 25,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.312911S",
+      "enqueuedAt": "2025-07-07T13:32:46.13871Z",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z"
+    },
+    {
+      "uid": 25,
+      "batchUid": 24,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "settingsUpdate",
+      "canceledBy": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "error": null,
+      "duration": "PT0.247378S",
+      "enqueuedAt": "2025-07-07T13:28:27.390054Z",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z"
+    },
     {
       "uid": 24,
       "batchUid": 23,
@@ -264,134 +376,10 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "enqueuedAt": "2025-01-16T17:02:52.527382964Z",
       "startedAt": "2025-01-16T17:02:52.539749853Z",
       "finishedAt": "2025-01-16T17:02:52.547390016Z"
-    },
-    {
-      "uid": 11,
-      "batchUid": 11,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "searchCutoffMs": 8000
-      },
-      "error": null,
-      "duration": "PT0.007307840S",
-      "enqueuedAt": "2025-01-16T17:01:14.100316617Z",
-      "startedAt": "2025-01-16T17:01:14.112756687Z",
-      "finishedAt": "2025-01-16T17:01:14.120064527Z"
-    },
-    {
-      "uid": 10,
-      "batchUid": 10,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 99
-        },
-        "pagination": {
-          "maxTotalHits": 15
-        }
-      },
-      "error": null,
-      "duration": "PT0.007391353S",
-      "enqueuedAt": "2025-01-16T17:00:29.188815062Z",
-      "startedAt": "2025-01-16T17:00:29.201180268Z",
-      "finishedAt": "2025-01-16T17:00:29.208571621Z"
-    },
-    {
-      "uid": 9,
-      "batchUid": 9,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 100
-        },
-        "pagination": {
-          "maxTotalHits": 1000
-        }
-      },
-      "error": null,
-      "duration": "PT0.007445825S",
-      "enqueuedAt": "2025-01-16T17:00:15.759501709Z",
-      "startedAt": "2025-01-16T17:00:15.77629445Z",
-      "finishedAt": "2025-01-16T17:00:15.783740275Z"
-    },
-    {
-      "uid": 8,
-      "batchUid": 8,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          },
-          "disableOnWords": [
-            "kefir"
-          ],
-          "disableOnAttributes": [
-            "surname"
-          ]
-        }
-      },
-      "error": null,
-      "duration": "PT0.012020083S",
-      "enqueuedAt": "2025-01-16T16:59:42.727292501Z",
-      "startedAt": "2025-01-16T16:59:42.744086671Z",
-      "finishedAt": "2025-01-16T16:59:42.756106754Z"
-    },
-    {
-      "uid": 7,
-      "batchUid": 7,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          }
-        }
-      },
-      "error": null,
-      "duration": "PT0.007440092S",
-      "enqueuedAt": "2025-01-16T16:58:41.203145044Z",
-      "startedAt": "2025-01-16T16:58:41.2155771Z",
-      "finishedAt": "2025-01-16T16:58:41.223017192Z"
-    },
-    {
-      "uid": 6,
-      "batchUid": 6,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "synonyms": {
-          "boubou": [
-            "kefir"
-          ]
-        }
-      },
-      "error": null,
-      "duration": "PT0.007565161S",
-      "enqueuedAt": "2025-01-16T16:54:51.927866243Z",
-      "startedAt": "2025-01-16T16:54:51.940332781Z",
-      "finishedAt": "2025-01-16T16:54:51.947897942Z"
     }
   ],
-  "total": 24,
+  "total": 30,
   "limit": 20,
-  "from": 25,
-  "next": 5
+  "from": 31,
+  "next": 11
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/tasks_filter_afterStartedAt_equal_2025-01-16T16_47_41.snap

@@ -4,8 +4,8 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 25,
-      "batchUid": 24,
+      "uid": 31,
+      "batchUid": 30,
       "indexUid": null,
       "status": "succeeded",
       "type": "upgradeDatabase",
@@ -20,6 +20,118 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "[date]",
       "finishedAt": "[date]"
     },
+    {
+      "uid": 30,
+      "batchUid": 29,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.067201S",
+      "enqueuedAt": "2025-07-07T13:43:08.772432Z",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z"
+    },
+    {
+      "uid": 29,
+      "batchUid": 28,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "indexDeletion",
+      "canceledBy": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.012727S",
+      "enqueuedAt": "2025-07-07T13:42:50.744793Z",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z"
+    },
+    {
+      "uid": 28,
+      "batchUid": 27,
+      "indexUid": "kefir",
+      "status": "failed",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "error": {
+        "message": "Index `kefir`: Bad embedder configuration in the document with id: `2`. Could not parse `._vectors.doggo_embedder`: trailing characters at line 1 column 13",
+        "code": "invalid_vectors_type",
+        "type": "invalid_request",
+        "link": "https://docs.meilisearch.com/errors#invalid_vectors_type"
+      },
+      "duration": "PT0.059920S",
+      "enqueuedAt": "2025-07-07T13:42:15.624598Z",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z"
+    },
+    {
+      "uid": 27,
+      "batchUid": 26,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.088879S",
+      "enqueuedAt": "2025-07-07T13:40:01.46081Z",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z"
+    },
+    {
+      "uid": 26,
+      "batchUid": 25,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.312911S",
+      "enqueuedAt": "2025-07-07T13:32:46.13871Z",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z"
+    },
+    {
+      "uid": 25,
+      "batchUid": 24,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "settingsUpdate",
+      "canceledBy": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "error": null,
+      "duration": "PT0.247378S",
+      "enqueuedAt": "2025-07-07T13:28:27.390054Z",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z"
+    },
     {
       "uid": 24,
       "batchUid": 23,
@@ -264,134 +376,10 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "enqueuedAt": "2025-01-16T17:02:52.527382964Z",
       "startedAt": "2025-01-16T17:02:52.539749853Z",
       "finishedAt": "2025-01-16T17:02:52.547390016Z"
-    },
-    {
-      "uid": 11,
-      "batchUid": 11,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "searchCutoffMs": 8000
-      },
-      "error": null,
-      "duration": "PT0.007307840S",
-      "enqueuedAt": "2025-01-16T17:01:14.100316617Z",
-      "startedAt": "2025-01-16T17:01:14.112756687Z",
-      "finishedAt": "2025-01-16T17:01:14.120064527Z"
-    },
-    {
-      "uid": 10,
-      "batchUid": 10,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 99
-        },
-        "pagination": {
-          "maxTotalHits": 15
-        }
-      },
-      "error": null,
-      "duration": "PT0.007391353S",
-      "enqueuedAt": "2025-01-16T17:00:29.188815062Z",
-      "startedAt": "2025-01-16T17:00:29.201180268Z",
-      "finishedAt": "2025-01-16T17:00:29.208571621Z"
-    },
-    {
-      "uid": 9,
-      "batchUid": 9,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "faceting": {
-          "maxValuesPerFacet": 100
-        },
-        "pagination": {
-          "maxTotalHits": 1000
-        }
-      },
-      "error": null,
-      "duration": "PT0.007445825S",
-      "enqueuedAt": "2025-01-16T17:00:15.759501709Z",
-      "startedAt": "2025-01-16T17:00:15.77629445Z",
-      "finishedAt": "2025-01-16T17:00:15.783740275Z"
-    },
-    {
-      "uid": 8,
-      "batchUid": 8,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          },
-          "disableOnWords": [
-            "kefir"
-          ],
-          "disableOnAttributes": [
-            "surname"
-          ]
-        }
-      },
-      "error": null,
-      "duration": "PT0.012020083S",
-      "enqueuedAt": "2025-01-16T16:59:42.727292501Z",
-      "startedAt": "2025-01-16T16:59:42.744086671Z",
-      "finishedAt": "2025-01-16T16:59:42.756106754Z"
-    },
-    {
-      "uid": 7,
-      "batchUid": 7,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "typoTolerance": {
-          "minWordSizeForTypos": {
-            "oneTypo": 4
-          }
-        }
-      },
-      "error": null,
-      "duration": "PT0.007440092S",
-      "enqueuedAt": "2025-01-16T16:58:41.203145044Z",
-      "startedAt": "2025-01-16T16:58:41.2155771Z",
-      "finishedAt": "2025-01-16T16:58:41.223017192Z"
-    },
-    {
-      "uid": 6,
-      "batchUid": 6,
-      "indexUid": "kefir",
-      "status": "succeeded",
-      "type": "settingsUpdate",
-      "canceledBy": null,
-      "details": {
-        "synonyms": {
-          "boubou": [
-            "kefir"
-          ]
-        }
-      },
-      "error": null,
-      "duration": "PT0.007565161S",
-      "enqueuedAt": "2025-01-16T16:54:51.927866243Z",
-      "startedAt": "2025-01-16T16:54:51.940332781Z",
-      "finishedAt": "2025-01-16T16:54:51.947897942Z"
     }
   ],
-  "total": 24,
+  "total": 30,
   "limit": 20,
-  "from": 25,
-  "next": 5
+  "from": 31,
+  "next": 11
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/the_whole_batch_queue_once_everything_has_been_processed.snap

@@ -4,7 +4,7 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 24,
+      "uid": 30,
       "progress": null,
       "details": {
         "upgradeFrom": "v1.12.0",
@@ -26,6 +26,155 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "finishedAt": "[date]",
       "batchStrategy": "stopped after the last task of type `upgradeDatabase` because they cannot be batched with tasks of any other type."
     },
+    {
+      "uid": 29,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.067201S",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 28,
+      "progress": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "indexDeletion": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.012727S",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 27,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "failed": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.059920S",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 26,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "mieli": 1
+        }
+      },
+      "duration": "PT0.088879S",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 25,
+      "progress": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "documentAdditionOrUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.312911S",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z",
+      "batchStrategy": "unspecified"
+    },
+    {
+      "uid": 24,
+      "progress": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "stats": {
+        "totalNbTasks": 1,
+        "status": {
+          "succeeded": 1
+        },
+        "types": {
+          "settingsUpdate": 1
+        },
+        "indexUids": {
+          "kefir": 1
+        }
+      },
+      "duration": "PT0.247378S",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z",
+      "batchStrategy": "unspecified"
+    },
     {
       "uid": 23,
       "progress": null,
@@ -642,8 +791,8 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "batchStrategy": "unspecified"
     }
   ],
-  "total": 25,
+  "total": 31,
   "limit": 1000,
-  "from": 24,
+  "from": 30,
   "next": null
 }

crates/meilisearch/tests/upgrade/v1_12/snapshots/v1_12_0.rs/check_the_index_scheduler/the_whole_task_queue_once_everything_has_been_processed.snap

@@ -4,8 +4,8 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
 {
   "results": [
     {
-      "uid": 25,
-      "batchUid": 24,
+      "uid": 31,
+      "batchUid": 30,
       "indexUid": null,
       "status": "succeeded",
       "type": "upgradeDatabase",
@@ -20,6 +20,118 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "startedAt": "[date]",
       "finishedAt": "[date]"
     },
+    {
+      "uid": 30,
+      "batchUid": 29,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.067201S",
+      "enqueuedAt": "2025-07-07T13:43:08.772432Z",
+      "startedAt": "2025-07-07T13:43:08.772854Z",
+      "finishedAt": "2025-07-07T13:43:08.840055Z"
+    },
+    {
+      "uid": 29,
+      "batchUid": 28,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "indexDeletion",
+      "canceledBy": null,
+      "details": {
+        "deletedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.012727S",
+      "enqueuedAt": "2025-07-07T13:42:50.744793Z",
+      "startedAt": "2025-07-07T13:42:50.745461Z",
+      "finishedAt": "2025-07-07T13:42:50.758188Z"
+    },
+    {
+      "uid": 28,
+      "batchUid": 27,
+      "indexUid": "kefir",
+      "status": "failed",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 0
+      },
+      "error": {
+        "message": "Index `kefir`: Bad embedder configuration in the document with id: `2`. Could not parse `._vectors.doggo_embedder`: trailing characters at line 1 column 13",
+        "code": "invalid_vectors_type",
+        "type": "invalid_request",
+        "link": "https://docs.meilisearch.com/errors#invalid_vectors_type"
+      },
+      "duration": "PT0.059920S",
+      "enqueuedAt": "2025-07-07T13:42:15.624598Z",
+      "startedAt": "2025-07-07T13:42:15.625413Z",
+      "finishedAt": "2025-07-07T13:42:15.685333Z"
+    },
+    {
+      "uid": 27,
+      "batchUid": 26,
+      "indexUid": "mieli",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.088879S",
+      "enqueuedAt": "2025-07-07T13:40:01.46081Z",
+      "startedAt": "2025-07-07T13:40:01.461741Z",
+      "finishedAt": "2025-07-07T13:40:01.55062Z"
+    },
+    {
+      "uid": 26,
+      "batchUid": 25,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "documentAdditionOrUpdate",
+      "canceledBy": null,
+      "details": {
+        "receivedDocuments": 1,
+        "indexedDocuments": 1
+      },
+      "error": null,
+      "duration": "PT0.312911S",
+      "enqueuedAt": "2025-07-07T13:32:46.13871Z",
+      "startedAt": "2025-07-07T13:32:46.139785Z",
+      "finishedAt": "2025-07-07T13:32:46.452696Z"
+    },
+    {
+      "uid": 25,
+      "batchUid": 24,
+      "indexUid": "kefir",
+      "status": "succeeded",
+      "type": "settingsUpdate",
+      "canceledBy": null,
+      "details": {
+        "embedders": {
+          "doggo_embedder": {
+            "source": "huggingFace",
+            "model": "sentence-transformers/all-MiniLM-L6-v2",
+            "revision": "e4ce9877abf3edfe10b0d82785e83bdcb973e22e",
+            "documentTemplate": "{{doc.description}}"
+          }
+        }
+      },
+      "error": null,
+      "duration": "PT0.247378S",
+      "enqueuedAt": "2025-07-07T13:28:27.390054Z",
+      "startedAt": "2025-07-07T13:28:27.391344Z",
+      "finishedAt": "2025-07-07T13:28:27.638722Z"
+    },
     {
       "uid": 24,
       "batchUid": 23,
@@ -497,8 +609,8 @@ source: crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs
       "finishedAt": "2025-01-16T16:45:16.131303739Z"
     }
   ],
-  "total": 26,
+  "total": 32,
   "limit": 1000,
-  "from": 25,
+  "from": 31,
   "next": null
 }

crates/meilisearch/tests/upgrade/v1_12/v1_12_0.rs

@@ -114,13 +114,13 @@ async fn check_the_index_scheduler(server: &Server) {
 
     // All the indexes are still present
     let (indexes, _) = server.list_indexes(None, None).await;
-    snapshot!(indexes, @r#"
+    snapshot!(indexes, @r###"
     {
       "results": [
         {
           "uid": "kefir",
           "createdAt": "2025-01-16T16:45:16.020663157Z",
-          "updatedAt": "2025-01-23T11:36:22.634859166Z",
+          "updatedAt": "2025-07-07T13:43:08.835381Z",
           "primaryKey": "id"
         }
       ],
@@ -128,7 +128,7 @@ async fn check_the_index_scheduler(server: &Server) {
       "limit": 20,
       "total": 1
     }
-    "#);
+    "###);
     // And their metadata are still right
     let (stats, _) = server.stats().await;
     assert_json_snapshot!(stats, {
@@ -141,21 +141,21 @@ async fn check_the_index_scheduler(server: &Server) {
     {
       "databaseSize": "[bytes]",
       "usedDatabaseSize": "[bytes]",
-      "lastUpdate": "2025-01-23T11:36:22.634859166Z",
+      "lastUpdate": "2025-07-07T13:43:08.835381Z",
       "indexes": {
         "kefir": {
-          "numberOfDocuments": 1,
+          "numberOfDocuments": 2,
           "rawDocumentDbSize": "[bytes]",
           "avgDocumentSize": "[bytes]",
           "isIndexing": false,
-          "numberOfEmbeddings": 0,
-          "numberOfEmbeddedDocuments": 0,
+          "numberOfEmbeddings": 2,
+          "numberOfEmbeddedDocuments": 2,
           "fieldDistribution": {
-            "age": 1,
-            "description": 1,
-            "id": 1,
-            "name": 1,
-            "surname": 1
+            "age": 2,
+            "description": 2,
+            "id": 2,
+            "name": 2,
+            "surname": 2
           }
         }
       }
@@ -227,21 +227,21 @@ async fn check_the_index_scheduler(server: &Server) {
     {
       "databaseSize": "[bytes]",
       "usedDatabaseSize": "[bytes]",
-      "lastUpdate": "2025-01-23T11:36:22.634859166Z",
+      "lastUpdate": "2025-07-07T13:43:08.835381Z",
       "indexes": {
         "kefir": {
-          "numberOfDocuments": 1,
+          "numberOfDocuments": 2,
           "rawDocumentDbSize": "[bytes]",
           "avgDocumentSize": "[bytes]",
           "isIndexing": false,
-          "numberOfEmbeddings": 0,
-          "numberOfEmbeddedDocuments": 0,
+          "numberOfEmbeddings": 2,
+          "numberOfEmbeddedDocuments": 2,
           "fieldDistribution": {
-            "age": 1,
-            "description": 1,
-            "id": 1,
-            "name": 1,
-            "surname": 1
+            "age": 2,
+            "description": 2,
+            "id": 2,
+            "name": 2,
+            "surname": 2
           }
         }
       }
@@ -254,18 +254,18 @@ async fn check_the_index_scheduler(server: &Server) {
         ".avgDocumentSize" => "[bytes]",
     }), @r###"
     {
-      "numberOfDocuments": 1,
+      "numberOfDocuments": 2,
       "rawDocumentDbSize": "[bytes]",
       "avgDocumentSize": "[bytes]",
       "isIndexing": false,
-      "numberOfEmbeddings": 0,
-      "numberOfEmbeddedDocuments": 0,
+      "numberOfEmbeddings": 2,
+      "numberOfEmbeddedDocuments": 2,
       "fieldDistribution": {
-        "age": 1,
-        "description": 1,
-        "id": 1,
-        "name": 1,
-        "surname": 1
+        "age": 2,
+        "description": 2,
+        "id": 2,
+        "name": 2,
+        "surname": 2
       }
     }
     "###);
@@ -295,4 +295,8 @@ async fn check_the_index_features(server: &Server) {
     let (results, _status) =
         kefir.search_post(json!({ "sort": ["age:asc"], "filter": "surname = kefirounet" })).await;
     snapshot!(results, name: "search_with_sort_and_filter");
+
+    // ensuring we can get the vectors and their `regenerate` is still good.
+    let (results, _status) = kefir.search_post(json!({"retrieveVectors": true})).await;
+    snapshot!(json_string!(results["hits"], {"[]._vectors.doggo_embedder.embeddings" => "[vector]"}), name: "search_with_retrieve_vectors");
 }

crates/meilisearch/tests/vector/mod.rs

@@ -1,4 +1,5 @@
 mod binary_quantized;
+mod fragments;
 #[cfg(feature = "test-ollama")]
 mod ollama;
 mod openai;

crates/meilisearch/tests/vector/openai.rs

@@ -136,7 +136,7 @@ fn long_text() -> &'static str {
     })
 }
 
-async fn create_mock_tokenized() -> (MockServer, Value) {
+async fn create_mock_tokenized() -> (&'static MockServer, Value) {
     create_mock_with_template("{{doc.text}}", ModelDimensions::Large, false, false).await
 }
 
@@ -145,8 +145,8 @@ async fn create_mock_with_template(
     model_dimensions: ModelDimensions,
     fallible: bool,
     slow: bool,
-) -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+) -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
     const API_KEY: &str = "my-api-key";
     const API_KEY_BEARER: &str = "Bearer my-api-key";
 
@@ -299,7 +299,7 @@ async fn create_mock_with_template(
                 }
             }))
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -321,27 +321,27 @@ const DOGGO_TEMPLATE: &str = r#"{%- if doc.gender == "F" -%}Une chienne nommée
         Un chien nommé {{doc.name}}, né en {{doc.birthyear}}
         {%- endif %}, de race {{doc.breed}}."#;
 
-async fn create_mock() -> (MockServer, Value) {
+async fn create_mock() -> (&'static MockServer, Value) {
     create_mock_with_template(DOGGO_TEMPLATE, ModelDimensions::Large, false, false).await
 }
 
-async fn create_mock_dimensions() -> (MockServer, Value) {
+async fn create_mock_dimensions() -> (&'static MockServer, Value) {
     create_mock_with_template(DOGGO_TEMPLATE, ModelDimensions::Large512, false, false).await
 }
 
-async fn create_mock_small_embedding_model() -> (MockServer, Value) {
+async fn create_mock_small_embedding_model() -> (&'static MockServer, Value) {
     create_mock_with_template(DOGGO_TEMPLATE, ModelDimensions::Small, false, false).await
 }
 
-async fn create_mock_legacy_embedding_model() -> (MockServer, Value) {
+async fn create_mock_legacy_embedding_model() -> (&'static MockServer, Value) {
     create_mock_with_template(DOGGO_TEMPLATE, ModelDimensions::Ada, false, false).await
 }
 
-async fn create_fallible_mock() -> (MockServer, Value) {
+async fn create_fallible_mock() -> (&'static MockServer, Value) {
     create_mock_with_template(DOGGO_TEMPLATE, ModelDimensions::Large, true, false).await
 }
 
-async fn create_slow_mock() -> (MockServer, Value) {
+async fn create_slow_mock() -> (&'static MockServer, Value) {
     create_mock_with_template(DOGGO_TEMPLATE, ModelDimensions::Large, true, true).await
 }
 

crates/meilisearch/tests/vector/rest.rs

@@ -12,8 +12,8 @@ use crate::common::Value;
 use crate::json;
 use crate::vector::{get_server_vector, GetAllDocumentsOptions};
 
-async fn create_mock() -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_mock() -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
 
     let text_to_embedding: BTreeMap<_, _> = vec![
         // text -> embedding
@@ -32,7 +32,7 @@ async fn create_mock() -> (MockServer, Value) {
                 json!({ "data": text_to_embedding.get(text.as_str()).unwrap_or(&[99., 99., 99.]) }),
             )
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -50,8 +50,8 @@ async fn create_mock() -> (MockServer, Value) {
     (mock_server, embedder_settings)
 }
 
-async fn create_mock_default_template() -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_mock_default_template() -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
 
     let text_to_embedding: BTreeMap<_, _> = vec![
         // text -> embedding
@@ -73,7 +73,7 @@ async fn create_mock_default_template() -> (MockServer, Value) {
                     .set_body_json(json!({"error": "text not found", "text": text})),
             }
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -106,8 +106,8 @@ struct SingleResponse {
     embedding: Vec<f32>,
 }
 
-async fn create_mock_multiple() -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_mock_multiple() -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
 
     let text_to_embedding: BTreeMap<_, _> = vec![
         // text -> embedding
@@ -146,7 +146,7 @@ async fn create_mock_multiple() -> (MockServer, Value) {
 
             ResponseTemplate::new(200).set_body_json(response)
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -176,8 +176,8 @@ struct SingleRequest {
     input: String,
 }
 
-async fn create_mock_single_response_in_array() -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_mock_single_response_in_array() -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
 
     let text_to_embedding: BTreeMap<_, _> = vec![
         // text -> embedding
@@ -212,7 +212,7 @@ async fn create_mock_single_response_in_array() -> (MockServer, Value) {
 
             ResponseTemplate::new(200).set_body_json(response)
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -236,8 +236,8 @@ async fn create_mock_single_response_in_array() -> (MockServer, Value) {
     (mock_server, embedder_settings)
 }
 
-async fn create_mock_raw_with_custom_header() -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_mock_raw_with_custom_header() -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
 
     let text_to_embedding: BTreeMap<_, _> = vec![
         // text -> embedding
@@ -277,7 +277,7 @@ async fn create_mock_raw_with_custom_header() -> (MockServer, Value) {
 
             ResponseTemplate::new(200).set_body_json(output)
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -293,8 +293,8 @@ async fn create_mock_raw_with_custom_header() -> (MockServer, Value) {
     (mock_server, embedder_settings)
 }
 
-async fn create_mock_raw() -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_mock_raw() -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
 
     let text_to_embedding: BTreeMap<_, _> = vec![
         // text -> embedding
@@ -321,7 +321,7 @@ async fn create_mock_raw() -> (MockServer, Value) {
 
             ResponseTemplate::new(200).set_body_json(output)
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
     let url = mock_server.uri();
 
@@ -337,8 +337,8 @@ async fn create_mock_raw() -> (MockServer, Value) {
     (mock_server, embedder_settings)
 }
 
-async fn create_faulty_mock_raw(sender: mpsc::Sender<()>) -> (MockServer, Value) {
-    let mock_server = MockServer::start().await;
+async fn create_faulty_mock_raw(sender: mpsc::Sender<()>) -> (&'static MockServer, Value) {
+    let mock_server = Box::leak(Box::new(MockServer::start().await));
     let count = AtomicUsize::new(0);
 
     Mock::given(method("POST"))
@@ -355,7 +355,7 @@ async fn create_faulty_mock_raw(sender: mpsc::Sender<()>) -> (MockServer, Value)
                 ResponseTemplate::new(500).set_body_string("Service Unavailable")
             }
         })
-        .mount(&mock_server)
+        .mount(mock_server)
         .await;
 
     let url = mock_server.uri();

crates/meilisearch/tests/vector/settings.rs

@@ -101,14 +101,7 @@ async fn reset_embedder_documents() {
     server.wait_task(response.uid()).await;
 
     // Make sure the documents are still present
-    let (documents, _code) = index
-        .get_all_documents(GetAllDocumentsOptions {
-            limit: None,
-            offset: None,
-            retrieve_vectors: false,
-            fields: None,
-        })
-        .await;
+    let (documents, _code) = index.get_all_documents(GetAllDocumentsOptions::default()).await;
     snapshot!(json_string!(documents), @r###"
     {
       "results": [

crates/meilitool/src/main.rs

@@ -15,6 +15,7 @@ use meilisearch_types::heed::{
 };
 use meilisearch_types::milli::constants::RESERVED_VECTORS_FIELD_NAME;
 use meilisearch_types::milli::documents::{obkv_to_object, DocumentsBatchReader};
+use meilisearch_types::milli::index::EmbeddingsWithMetadata;
 use meilisearch_types::milli::vector::parsed_vectors::{ExplicitVectors, VectorOrArrayOfVectors};
 use meilisearch_types::milli::{obkv_to_json, BEU32};
 use meilisearch_types::tasks::{Status, Task};
@@ -591,12 +592,21 @@ fn export_documents(
                             .into());
                         };
 
-                        for (embedder_name, (embeddings, regenerate)) in embeddings {
+                        for (
+                            embedder_name,
+                            EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
+                        ) in embeddings
+                        {
                             let embeddings = ExplicitVectors {
                                 embeddings: Some(VectorOrArrayOfVectors::from_array_of_vectors(
                                     embeddings,
                                 )),
-                                regenerate,
+                                regenerate: regenerate &&
+                                // Meilisearch does not handle well dumps with fragments, because as the fragments
+                                // are marked as user-provided,
+                                // all embeddings would be regenerated on any settings change or document update.
+                                // To prevent this, we mark embeddings has non regenerate in this case.
+                                !has_fragments,
                             };
                             vectors
                                 .insert(embedder_name, serde_json::to_value(embeddings).unwrap());

crates/milli/Cargo.toml

@@ -40,7 +40,7 @@ indexmap = { version = "2.9.0", features = ["serde"] }
 json-depth-checker = { path = "../json-depth-checker" }
 levenshtein_automata = { version = "0.2.1", features = ["fst_automaton"] }
 memchr = "2.7.5"
-memmap2 = "0.9.5"
+memmap2 = "0.9.7"
 obkv = "0.3.0"
 once_cell = "1.21.3"
 ordered-float = "5.0.0"

crates/milli/src/asc_desc.rs

@@ -168,6 +168,16 @@ pub enum SortError {
     ReservedNameForFilter { name: String },
 }
 
+impl SortError {
+    pub fn into_search_error(self) -> Error {
+        Error::UserError(UserError::SortError { error: self, search: true })
+    }
+
+    pub fn into_document_error(self) -> Error {
+        Error::UserError(UserError::SortError { error: self, search: false })
+    }
+}
+
 impl From<AscDescError> for SortError {
     fn from(error: AscDescError) -> Self {
         match error {
@@ -190,12 +200,6 @@ impl From<AscDescError> for SortError {
     }
 }
 
-impl From<SortError> for Error {
-    fn from(error: SortError) -> Self {
-        Self::UserError(UserError::SortError(error))
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use big_s::S;

crates/milli/src/documents/geo_sort.rs

@@ -0,0 +1,294 @@
+use crate::{
+    distance_between_two_points,
+    heed_codec::facet::{FieldDocIdFacetCodec, OrderedF64Codec},
+    lat_lng_to_xyz,
+    search::new::{facet_string_values, facet_values_prefix_key},
+    GeoPoint, Index,
+};
+use heed::{
+    types::{Bytes, Unit},
+    RoPrefix, RoTxn,
+};
+use roaring::RoaringBitmap;
+use rstar::RTree;
+use std::collections::VecDeque;
+
+#[derive(Debug, Clone, Copy)]
+pub struct GeoSortParameter {
+    // Define the strategy used by the geo sort
+    pub strategy: GeoSortStrategy,
+    // Limit the number of docs in a single bucket to avoid unexpectedly large overhead
+    pub max_bucket_size: u64,
+    // Considering the errors of GPS and geographical calculations, distances less than distance_error_margin will be treated as equal
+    pub distance_error_margin: f64,
+}
+
+impl Default for GeoSortParameter {
+    fn default() -> Self {
+        Self {
+            strategy: GeoSortStrategy::default(),
+            max_bucket_size: 1000,
+            distance_error_margin: 1.0,
+        }
+    }
+}
+/// Define the strategy used by the geo sort.
+/// The parameter represents the cache size, and, in the case of the Dynamic strategy,
+/// the point where we move from using the iterative strategy to the rtree.
+#[derive(Debug, Clone, Copy)]
+pub enum GeoSortStrategy {
+    AlwaysIterative(usize),
+    AlwaysRtree(usize),
+    Dynamic(usize),
+}
+
+impl Default for GeoSortStrategy {
+    fn default() -> Self {
+        GeoSortStrategy::Dynamic(1000)
+    }
+}
+
+impl GeoSortStrategy {
+    pub fn use_rtree(&self, candidates: usize) -> bool {
+        match self {
+            GeoSortStrategy::AlwaysIterative(_) => false,
+            GeoSortStrategy::AlwaysRtree(_) => true,
+            GeoSortStrategy::Dynamic(i) => candidates >= *i,
+        }
+    }
+
+    pub fn cache_size(&self) -> usize {
+        match self {
+            GeoSortStrategy::AlwaysIterative(i)
+            | GeoSortStrategy::AlwaysRtree(i)
+            | GeoSortStrategy::Dynamic(i) => *i,
+        }
+    }
+}
+
+#[allow(clippy::too_many_arguments)]
+pub fn fill_cache(
+    index: &Index,
+    txn: &RoTxn<heed::AnyTls>,
+    strategy: GeoSortStrategy,
+    ascending: bool,
+    target_point: [f64; 2],
+    field_ids: &Option<[u16; 2]>,
+    rtree: &mut Option<RTree<GeoPoint>>,
+    geo_candidates: &RoaringBitmap,
+    cached_sorted_docids: &mut VecDeque<(u32, [f64; 2])>,
+) -> crate::Result<()> {
+    debug_assert!(cached_sorted_docids.is_empty());
+
+    // lazily initialize the rtree if needed by the strategy, and cache it in `self.rtree`
+    let rtree = if strategy.use_rtree(geo_candidates.len() as usize) {
+        if let Some(rtree) = rtree.as_ref() {
+            // get rtree from cache
+            Some(rtree)
+        } else {
+            let rtree2 = index.geo_rtree(txn)?.expect("geo candidates but no rtree");
+            // insert rtree in cache and returns it.
+            // Can't use `get_or_insert_with` because getting the rtree from the DB is a fallible operation.
+            Some(&*rtree.insert(rtree2))
+        }
+    } else {
+        None
+    };
+
+    let cache_size = strategy.cache_size();
+    if let Some(rtree) = rtree {
+        if ascending {
+            let point = lat_lng_to_xyz(&target_point);
+            for point in rtree.nearest_neighbor_iter(&point) {
+                if geo_candidates.contains(point.data.0) {
+                    cached_sorted_docids.push_back(point.data);
+                    if cached_sorted_docids.len() >= cache_size {
+                        break;
+                    }
+                }
+            }
+        } else {
+            // in the case of the desc geo sort we look for the closest point to the opposite of the queried point
+            // and we insert the points in reverse order they get reversed when emptying the cache later on
+            let point = lat_lng_to_xyz(&opposite_of(target_point));
+            for point in rtree.nearest_neighbor_iter(&point) {
+                if geo_candidates.contains(point.data.0) {
+                    cached_sorted_docids.push_front(point.data);
+                    if cached_sorted_docids.len() >= cache_size {
+                        break;
+                    }
+                }
+            }
+        }
+    } else {
+        // the iterative version
+        let [lat, lng] = field_ids.expect("fill_buffer can't be called without the lat&lng");
+
+        let mut documents = geo_candidates
+            .iter()
+            .map(|id| -> crate::Result<_> { Ok((id, geo_value(id, lat, lng, index, txn)?)) })
+            .collect::<crate::Result<Vec<(u32, [f64; 2])>>>()?;
+        // computing the distance between two points is expensive thus we cache the result
+        documents
+            .sort_by_cached_key(|(_, p)| distance_between_two_points(&target_point, p) as usize);
+        cached_sorted_docids.extend(documents);
+    };
+
+    Ok(())
+}
+
+#[allow(clippy::too_many_arguments)]
+pub fn next_bucket(
+    index: &Index,
+    txn: &RoTxn<heed::AnyTls>,
+    universe: &RoaringBitmap,
+    ascending: bool,
+    target_point: [f64; 2],
+    field_ids: &Option<[u16; 2]>,
+    rtree: &mut Option<RTree<GeoPoint>>,
+    cached_sorted_docids: &mut VecDeque<(u32, [f64; 2])>,
+    geo_candidates: &RoaringBitmap,
+    parameter: GeoSortParameter,
+) -> crate::Result<Option<(RoaringBitmap, Option<[f64; 2]>)>> {
+    let mut geo_candidates = geo_candidates & universe;
+
+    if geo_candidates.is_empty() {
+        return Ok(Some((universe.clone(), None)));
+    }
+
+    let next = |cache: &mut VecDeque<_>| {
+        if ascending {
+            cache.pop_front()
+        } else {
+            cache.pop_back()
+        }
+    };
+    let put_back = |cache: &mut VecDeque<_>, x: _| {
+        if ascending {
+            cache.push_front(x)
+        } else {
+            cache.push_back(x)
+        }
+    };
+
+    let mut current_bucket = RoaringBitmap::new();
+    // current_distance stores the first point and distance in current bucket
+    let mut current_distance: Option<([f64; 2], f64)> = None;
+    loop {
+        // The loop will only exit when we have found all points with equal distance or have exhausted the candidates.
+        if let Some((id, point)) = next(cached_sorted_docids) {
+            if geo_candidates.contains(id) {
+                let distance = distance_between_two_points(&target_point, &point);
+                if let Some((point0, bucket_distance)) = current_distance.as_ref() {
+                    if (bucket_distance - distance).abs() > parameter.distance_error_margin {
+                        // different distance, point belongs to next bucket
+                        put_back(cached_sorted_docids, (id, point));
+                        return Ok(Some((current_bucket, Some(point0.to_owned()))));
+                    } else {
+                        // same distance, point belongs to current bucket
+                        current_bucket.insert(id);
+                        // remove from candidates to prevent it from being added to the cache again
+                        geo_candidates.remove(id);
+                        // current bucket size reaches limit, force return
+                        if current_bucket.len() == parameter.max_bucket_size {
+                            return Ok(Some((current_bucket, Some(point0.to_owned()))));
+                        }
+                    }
+                } else {
+                    // first doc in current bucket
+                    current_distance = Some((point, distance));
+                    current_bucket.insert(id);
+                    geo_candidates.remove(id);
+                    // current bucket size reaches limit, force return
+                    if current_bucket.len() == parameter.max_bucket_size {
+                        return Ok(Some((current_bucket, Some(point.to_owned()))));
+                    }
+                }
+            }
+        } else {
+            // cache exhausted, we need to refill it
+            fill_cache(
+                index,
+                txn,
+                parameter.strategy,
+                ascending,
+                target_point,
+                field_ids,
+                rtree,
+                &geo_candidates,
+                cached_sorted_docids,
+            )?;
+
+            if cached_sorted_docids.is_empty() {
+                // candidates exhausted, exit
+                if let Some((point0, _)) = current_distance.as_ref() {
+                    return Ok(Some((current_bucket, Some(point0.to_owned()))));
+                } else {
+                    return Ok(Some((universe.clone(), None)));
+                }
+            }
+        }
+    }
+}
+
+/// Return an iterator over each number value in the given field of the given document.
+fn facet_number_values<'a>(
+    docid: u32,
+    field_id: u16,
+    index: &Index,
+    txn: &'a RoTxn<'a>,
+) -> crate::Result<RoPrefix<'a, FieldDocIdFacetCodec<OrderedF64Codec>, Unit>> {
+    let key = facet_values_prefix_key(field_id, docid);
+
+    let iter = index
+        .field_id_docid_facet_f64s
+        .remap_key_type::<Bytes>()
+        .prefix_iter(txn, &key)?
+        .remap_key_type();
+
+    Ok(iter)
+}
+
+/// Extracts the lat and long values from a single document.
+///
+/// If it is not able to find it in the facet number index it will extract it
+/// from the facet string index and parse it as f64 (as the geo extraction behaves).
+pub(crate) fn geo_value(
+    docid: u32,
+    field_lat: u16,
+    field_lng: u16,
+    index: &Index,
+    rtxn: &RoTxn<'_>,
+) -> crate::Result<[f64; 2]> {
+    let extract_geo = |geo_field: u16| -> crate::Result<f64> {
+        match facet_number_values(docid, geo_field, index, rtxn)?.next() {
+            Some(Ok(((_, _, geo), ()))) => Ok(geo),
+            Some(Err(e)) => Err(e.into()),
+            None => match facet_string_values(docid, geo_field, index, rtxn)?.next() {
+                Some(Ok((_, geo))) => {
+                    Ok(geo.parse::<f64>().expect("cannot parse geo field as f64"))
+                }
+                Some(Err(e)) => Err(e.into()),
+                None => panic!("A geo faceted document doesn't contain any lat or lng"),
+            },
+        }
+    };
+
+    let lat = extract_geo(field_lat)?;
+    let lng = extract_geo(field_lng)?;
+
+    Ok([lat, lng])
+}
+
+/// Compute the antipodal coordinate of `coord`
+pub(crate) fn opposite_of(mut coord: [f64; 2]) -> [f64; 2] {
+    coord[0] *= -1.;
+    // in the case of x,0 we want to return x,180
+    if coord[1] > 0. {
+        coord[1] -= 180.;
+    } else {
+        coord[1] += 180.;
+    }
+
+    coord
+}

crates/milli/src/documents/mod.rs

@@ -1,8 +1,10 @@
 mod builder;
 mod enriched;
+pub mod geo_sort;
 mod primary_key;
 mod reader;
 mod serde_impl;
+pub mod sort;
 
 use std::fmt::Debug;
 use std::io;
@@ -19,6 +21,7 @@ pub use primary_key::{
 pub use reader::{DocumentsBatchCursor, DocumentsBatchCursorError, DocumentsBatchReader};
 use serde::{Deserialize, Serialize};
 
+pub use self::geo_sort::{GeoSortParameter, GeoSortStrategy};
 use crate::error::{FieldIdMapMissingEntry, InternalError};
 use crate::{FieldId, Object, Result};
 

crates/milli/src/documents/sort.rs

@@ -0,0 +1,444 @@
+use std::collections::{BTreeSet, VecDeque};
+
+use crate::{
+    constants::RESERVED_GEO_FIELD_NAME,
+    documents::{geo_sort::next_bucket, GeoSortParameter},
+    heed_codec::{
+        facet::{FacetGroupKeyCodec, FacetGroupValueCodec},
+        BytesRefCodec,
+    },
+    is_faceted,
+    search::facet::{ascending_facet_sort, descending_facet_sort},
+    AscDesc, DocumentId, Member, UserError,
+};
+use heed::Database;
+use roaring::RoaringBitmap;
+
+#[derive(Debug, Clone, Copy)]
+enum AscDescId {
+    Facet { field_id: u16, ascending: bool },
+    Geo { field_ids: [u16; 2], target_point: [f64; 2], ascending: bool },
+}
+
+/// A [`SortedDocumentsIterator`] allows efficient access to a continuous range of sorted documents.
+/// This is ideal in the context of paginated queries in which only a small number of documents are needed at a time.
+/// Search operations will only be performed upon access.
+pub enum SortedDocumentsIterator<'ctx> {
+    Leaf {
+        /// The exact number of documents remaining
+        size: usize,
+        values: Box<dyn Iterator<Item = DocumentId> + 'ctx>,
+    },
+    Branch {
+        /// The current child, got from the children iterator
+        current_child: Option<Box<SortedDocumentsIterator<'ctx>>>,
+        /// The exact number of documents remaining, excluding documents in the current child
+        next_children_size: usize,
+        /// Iterators to become the current child once it is exhausted
+        next_children:
+            Box<dyn Iterator<Item = crate::Result<SortedDocumentsIteratorBuilder<'ctx>>> + 'ctx>,
+    },
+}
+
+impl SortedDocumentsIterator<'_> {
+    /// Takes care of updating the current child if it is `None`, and also updates the size
+    fn update_current<'ctx>(
+        current_child: &mut Option<Box<SortedDocumentsIterator<'ctx>>>,
+        next_children_size: &mut usize,
+        next_children: &mut Box<
+            dyn Iterator<Item = crate::Result<SortedDocumentsIteratorBuilder<'ctx>>> + 'ctx,
+        >,
+    ) -> crate::Result<()> {
+        if current_child.is_none() {
+            *current_child = match next_children.next() {
+                Some(Ok(builder)) => {
+                    let next_child = Box::new(builder.build()?);
+                    *next_children_size -= next_child.size_hint().0;
+                    Some(next_child)
+                }
+                Some(Err(e)) => return Err(e),
+                None => return Ok(()),
+            };
+        }
+        Ok(())
+    }
+}
+
+impl Iterator for SortedDocumentsIterator<'_> {
+    type Item = crate::Result<DocumentId>;
+
+    /// Implementing the `nth` method allows for efficient access to the nth document in the sorted order.
+    /// It's used by `skip` internally.
+    /// The default implementation of `nth` would iterate over all children, which is inefficient for large datasets.
+    /// This implementation will jump over whole chunks of children until it gets close.
+    fn nth(&mut self, n: usize) -> Option<Self::Item> {
+        if n == 0 {
+            return self.next();
+        }
+
+        // If it's at the leaf level, just forward the call to the values iterator
+        let (current_child, next_children, next_children_size) = match self {
+            SortedDocumentsIterator::Leaf { values, size } => {
+                *size = size.saturating_sub(n);
+                return values.nth(n).map(Ok);
+            }
+            SortedDocumentsIterator::Branch {
+                current_child,
+                next_children,
+                next_children_size,
+            } => (current_child, next_children, next_children_size),
+        };
+
+        // Otherwise don't directly iterate over children, skip them if we know we will go further
+        let mut to_skip = n - 1;
+        while to_skip > 0 {
+            if let Err(e) = SortedDocumentsIterator::update_current(
+                current_child,
+                next_children_size,
+                next_children,
+            ) {
+                return Some(Err(e));
+            }
+            let Some(inner) = current_child else {
+                return None; // No more inner iterators, everything has been consumed.
+            };
+
+            if to_skip >= inner.size_hint().0 {
+                // The current child isn't large enough to contain the nth element.
+                // Skip it and continue with the next one.
+                to_skip -= inner.size_hint().0;
+                *current_child = None;
+                continue;
+            } else {
+                // The current iterator is large enough, so we can forward the call to it.
+                return inner.nth(to_skip + 1);
+            }
+        }
+
+        self.next()
+    }
+
+    /// Iterators need to keep track of their size so that they can be skipped efficiently by the `nth` method.
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        let size = match self {
+            SortedDocumentsIterator::Leaf { size, .. } => *size,
+            SortedDocumentsIterator::Branch {
+                next_children_size,
+                current_child: Some(current_child),
+                ..
+            } => current_child.size_hint().0 + next_children_size,
+            SortedDocumentsIterator::Branch { next_children_size, current_child: None, .. } => {
+                *next_children_size
+            }
+        };
+
+        (size, Some(size))
+    }
+
+    fn next(&mut self) -> Option<Self::Item> {
+        match self {
+            SortedDocumentsIterator::Leaf { values, size } => {
+                let result = values.next().map(Ok);
+                if result.is_some() {
+                    *size -= 1;
+                }
+                result
+            }
+            SortedDocumentsIterator::Branch {
+                current_child,
+                next_children_size,
+                next_children,
+            } => {
+                let mut result = None;
+                while result.is_none() {
+                    // Ensure we have selected an iterator to work with
+                    if let Err(e) = SortedDocumentsIterator::update_current(
+                        current_child,
+                        next_children_size,
+                        next_children,
+                    ) {
+                        return Some(Err(e));
+                    }
+                    let Some(inner) = current_child else {
+                        return None;
+                    };
+
+                    result = inner.next();
+
+                    // If the current iterator is exhausted, we need to try the next one
+                    if result.is_none() {
+                        *current_child = None;
+                    }
+                }
+                result
+            }
+        }
+    }
+}
+
+/// Builder for a [`SortedDocumentsIterator`].
+/// Most builders won't ever be built, because pagination will skip them.
+pub struct SortedDocumentsIteratorBuilder<'ctx> {
+    index: &'ctx crate::Index,
+    rtxn: &'ctx heed::RoTxn<'ctx>,
+    number_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+    string_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+    fields: &'ctx [AscDescId],
+    candidates: RoaringBitmap,
+    geo_candidates: &'ctx RoaringBitmap,
+}
+
+impl<'ctx> SortedDocumentsIteratorBuilder<'ctx> {
+    /// Performs the sort and builds a [`SortedDocumentsIterator`].
+    fn build(self) -> crate::Result<SortedDocumentsIterator<'ctx>> {
+        let size = self.candidates.len() as usize;
+
+        match self.fields {
+            [] => Ok(SortedDocumentsIterator::Leaf {
+                size,
+                values: Box::new(self.candidates.into_iter()),
+            }),
+            [AscDescId::Facet { field_id, ascending }, next_fields @ ..] => {
+                SortedDocumentsIteratorBuilder::build_facet(
+                    self.index,
+                    self.rtxn,
+                    self.number_db,
+                    self.string_db,
+                    next_fields,
+                    self.candidates,
+                    self.geo_candidates,
+                    *field_id,
+                    *ascending,
+                )
+            }
+            [AscDescId::Geo { field_ids, target_point, ascending }, next_fields @ ..] => {
+                SortedDocumentsIteratorBuilder::build_geo(
+                    self.index,
+                    self.rtxn,
+                    self.number_db,
+                    self.string_db,
+                    next_fields,
+                    self.candidates,
+                    self.geo_candidates,
+                    *field_ids,
+                    *target_point,
+                    *ascending,
+                )
+            }
+        }
+    }
+
+    /// Builds a [`SortedDocumentsIterator`] based on the results of a facet sort.
+    #[allow(clippy::too_many_arguments)]
+    fn build_facet(
+        index: &'ctx crate::Index,
+        rtxn: &'ctx heed::RoTxn<'ctx>,
+        number_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+        string_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+        next_fields: &'ctx [AscDescId],
+        candidates: RoaringBitmap,
+        geo_candidates: &'ctx RoaringBitmap,
+        field_id: u16,
+        ascending: bool,
+    ) -> crate::Result<SortedDocumentsIterator<'ctx>> {
+        let size = candidates.len() as usize;
+
+        // Perform the sort on the first field
+        let (number_iter, string_iter) = if ascending {
+            let number_iter = ascending_facet_sort(rtxn, number_db, field_id, candidates.clone())?;
+            let string_iter = ascending_facet_sort(rtxn, string_db, field_id, candidates)?;
+
+            (itertools::Either::Left(number_iter), itertools::Either::Left(string_iter))
+        } else {
+            let number_iter = descending_facet_sort(rtxn, number_db, field_id, candidates.clone())?;
+            let string_iter = descending_facet_sort(rtxn, string_db, field_id, candidates)?;
+
+            (itertools::Either::Right(number_iter), itertools::Either::Right(string_iter))
+        };
+
+        // Create builders for the next level of the tree
+        let number_iter = number_iter.map(|r| r.map(|(d, _)| d));
+        let string_iter = string_iter.map(|r| r.map(|(d, _)| d));
+        let next_children = number_iter.chain(string_iter).map(move |r| {
+            Ok(SortedDocumentsIteratorBuilder {
+                index,
+                rtxn,
+                number_db,
+                string_db,
+                fields: next_fields,
+                candidates: r?,
+                geo_candidates,
+            })
+        });
+
+        Ok(SortedDocumentsIterator::Branch {
+            current_child: None,
+            next_children_size: size,
+            next_children: Box::new(next_children),
+        })
+    }
+
+    /// Builds a [`SortedDocumentsIterator`] based on the (lazy) results of a geo sort.
+    #[allow(clippy::too_many_arguments)]
+    fn build_geo(
+        index: &'ctx crate::Index,
+        rtxn: &'ctx heed::RoTxn<'ctx>,
+        number_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+        string_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+        next_fields: &'ctx [AscDescId],
+        candidates: RoaringBitmap,
+        geo_candidates: &'ctx RoaringBitmap,
+        field_ids: [u16; 2],
+        target_point: [f64; 2],
+        ascending: bool,
+    ) -> crate::Result<SortedDocumentsIterator<'ctx>> {
+        let mut cache = VecDeque::new();
+        let mut rtree = None;
+        let size = candidates.len() as usize;
+        let not_geo_candidates = candidates.clone() - geo_candidates;
+        let mut geo_remaining = size - not_geo_candidates.len() as usize;
+        let mut not_geo_candidates = Some(not_geo_candidates);
+
+        let next_children = std::iter::from_fn(move || {
+            // Find the next bucket of geo-sorted documents.
+            // next_bucket loops and will go back to the beginning so we use a variable to track how many are left.
+            if geo_remaining > 0 {
+                if let Ok(Some((docids, _point))) = next_bucket(
+                    index,
+                    rtxn,
+                    &candidates,
+                    ascending,
+                    target_point,
+                    &Some(field_ids),
+                    &mut rtree,
+                    &mut cache,
+                    geo_candidates,
+                    GeoSortParameter::default(),
+                ) {
+                    geo_remaining -= docids.len() as usize;
+                    return Some(Ok(SortedDocumentsIteratorBuilder {
+                        index,
+                        rtxn,
+                        number_db,
+                        string_db,
+                        fields: next_fields,
+                        candidates: docids,
+                        geo_candidates,
+                    }));
+                }
+            }
+
+            // Once all geo candidates have been processed, we can return the others
+            if let Some(not_geo_candidates) = not_geo_candidates.take() {
+                if !not_geo_candidates.is_empty() {
+                    return Some(Ok(SortedDocumentsIteratorBuilder {
+                        index,
+                        rtxn,
+                        number_db,
+                        string_db,
+                        fields: next_fields,
+                        candidates: not_geo_candidates,
+                        geo_candidates,
+                    }));
+                }
+            }
+
+            None
+        });
+
+        Ok(SortedDocumentsIterator::Branch {
+            current_child: None,
+            next_children_size: size,
+            next_children: Box::new(next_children),
+        })
+    }
+}
+
+/// A structure owning the data needed during the lifetime of a [`SortedDocumentsIterator`].
+pub struct SortedDocuments<'ctx> {
+    index: &'ctx crate::Index,
+    rtxn: &'ctx heed::RoTxn<'ctx>,
+    fields: Vec<AscDescId>,
+    number_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+    string_db: Database<FacetGroupKeyCodec<BytesRefCodec>, FacetGroupValueCodec>,
+    candidates: &'ctx RoaringBitmap,
+    geo_candidates: RoaringBitmap,
+}
+
+impl<'ctx> SortedDocuments<'ctx> {
+    pub fn iter(&'ctx self) -> crate::Result<SortedDocumentsIterator<'ctx>> {
+        let builder = SortedDocumentsIteratorBuilder {
+            index: self.index,
+            rtxn: self.rtxn,
+            number_db: self.number_db,
+            string_db: self.string_db,
+            fields: &self.fields,
+            candidates: self.candidates.clone(),
+            geo_candidates: &self.geo_candidates,
+        };
+        builder.build()
+    }
+}
+
+pub fn recursive_sort<'ctx>(
+    index: &'ctx crate::Index,
+    rtxn: &'ctx heed::RoTxn<'ctx>,
+    sort: Vec<AscDesc>,
+    candidates: &'ctx RoaringBitmap,
+) -> crate::Result<SortedDocuments<'ctx>> {
+    let sortable_fields: BTreeSet<_> = index.sortable_fields(rtxn)?.into_iter().collect();
+    let fields_ids_map = index.fields_ids_map(rtxn)?;
+
+    // Retrieve the field ids that are used for sorting
+    let mut fields = Vec::new();
+    let mut need_geo_candidates = false;
+    for asc_desc in sort {
+        let (field, geofield) = match asc_desc {
+            AscDesc::Asc(Member::Field(field)) => (Some((field, true)), None),
+            AscDesc::Desc(Member::Field(field)) => (Some((field, false)), None),
+            AscDesc::Asc(Member::Geo(target_point)) => (None, Some((target_point, true))),
+            AscDesc::Desc(Member::Geo(target_point)) => (None, Some((target_point, false))),
+        };
+        if let Some((field, ascending)) = field {
+            if is_faceted(&field, &sortable_fields) {
+                if let Some(field_id) = fields_ids_map.id(&field) {
+                    fields.push(AscDescId::Facet { field_id, ascending });
+                    continue;
+                }
+            }
+            return Err(UserError::InvalidDocumentSortableAttribute {
+                field: field.to_string(),
+                sortable_fields: sortable_fields.clone(),
+            }
+            .into());
+        }
+        if let Some((target_point, ascending)) = geofield {
+            if sortable_fields.contains(RESERVED_GEO_FIELD_NAME) {
+                if let (Some(lat), Some(lng)) =
+                    (fields_ids_map.id("_geo.lat"), fields_ids_map.id("_geo.lng"))
+                {
+                    need_geo_candidates = true;
+                    fields.push(AscDescId::Geo { field_ids: [lat, lng], target_point, ascending });
+                    continue;
+                }
+            }
+            return Err(UserError::InvalidDocumentSortableAttribute {
+                field: RESERVED_GEO_FIELD_NAME.to_string(),
+                sortable_fields: sortable_fields.clone(),
+            }
+            .into());
+        }
+    }
+
+    let geo_candidates = if need_geo_candidates {
+        index.geo_faceted_documents_ids(rtxn)?
+    } else {
+        RoaringBitmap::new()
+    };
+
+    let number_db = index.facet_id_f64_docids.remap_key_type::<FacetGroupKeyCodec<BytesRefCodec>>();
+    let string_db =
+        index.facet_id_string_docids.remap_key_type::<FacetGroupKeyCodec<BytesRefCodec>>();
+
+    Ok(SortedDocuments { index, rtxn, fields, number_db, string_db, candidates, geo_candidates })
+}

crates/milli/src/error.rs

@@ -191,7 +191,21 @@ and can not be more than 511 bytes.", .document_id.to_string()
                 ),
         }
     )]
-    InvalidSortableAttribute { field: String, valid_fields: BTreeSet<String>, hidden_fields: bool },
+    InvalidSearchSortableAttribute {
+        field: String,
+        valid_fields: BTreeSet<String>,
+        hidden_fields: bool,
+    },
+    #[error("Attribute `{}` is not sortable. {}",
+        .field,
+        match .sortable_fields.is_empty() {
+            true => "This index does not have configured sortable attributes.".to_string(),
+            false => format!("Available sortable attributes are: `{}`.",
+                    sortable_fields.iter().map(AsRef::as_ref).collect::<Vec<&str>>().join(", ")
+                ),
+        }
+    )]
+    InvalidDocumentSortableAttribute { field: String, sortable_fields: BTreeSet<String> },
     #[error("Attribute `{}` is not filterable and thus, cannot be used as distinct attribute. {}",
         .field,
         match (.valid_patterns.is_empty(), .matching_rule_index) {
@@ -272,8 +286,8 @@ and can not be more than 511 bytes.", .document_id.to_string()
     PrimaryKeyCannotBeChanged(String),
     #[error(transparent)]
     SerdeJson(serde_json::Error),
-    #[error(transparent)]
-    SortError(#[from] SortError),
+    #[error("{error}")]
+    SortError { error: SortError, search: bool },
     #[error("An unknown internal document id have been used: `{document_id}`.")]
     UnknownInternalDocumentId { document_id: DocumentId },
     #[error("`minWordSizeForTypos` setting is invalid. `oneTypo` and `twoTypos` fields should be between `0` and `255`, and `twoTypos` should be greater or equals to `oneTypo` but found `oneTypo: {0}` and twoTypos: {1}`.")]
@@ -616,7 +630,7 @@ fn conditionally_lookup_for_error_message() {
     ];
 
     for (list, suffix) in messages {
-        let err = UserError::InvalidSortableAttribute {
+        let err = UserError::InvalidSearchSortableAttribute {
             field: "name".to_string(),
             valid_fields: list,
             hidden_fields: false,

crates/milli/src/index.rs

@@ -1766,20 +1766,22 @@ impl Index {
         &self,
         rtxn: &RoTxn<'_>,
         docid: DocumentId,
-    ) -> Result<BTreeMap<String, (Vec<Embedding>, bool)>> {
+    ) -> Result<BTreeMap<String, EmbeddingsWithMetadata>> {
         let mut res = BTreeMap::new();
         let embedders = self.embedding_configs();
         for config in embedders.embedding_configs(rtxn)? {
             let embedder_info = embedders.embedder_info(rtxn, &config.name)?.unwrap();
+            let has_fragments = config.config.embedder_options.has_fragments();
             let reader = ArroyWrapper::new(
                 self.vector_arroy,
                 embedder_info.embedder_id,
                 config.config.quantized(),
             );
             let embeddings = reader.item_vectors(rtxn, docid)?;
+            let regenerate = embedder_info.embedding_status.must_regenerate(docid);
             res.insert(
                 config.name.to_owned(),
-                (embeddings, embedder_info.embedding_status.must_regenerate(docid)),
+                EmbeddingsWithMetadata { embeddings, regenerate, has_fragments },
             );
         }
         Ok(res)
@@ -1919,6 +1921,12 @@ impl Index {
     }
 }
 
+pub struct EmbeddingsWithMetadata {
+    pub embeddings: Vec<Embedding>,
+    pub regenerate: bool,
+    pub has_fragments: bool,
+}
+
 #[derive(Debug, Default, Deserialize, Serialize)]
 pub struct ChatConfig {
     pub description: String,