Update Dockerfile

Co-authored-by: Markus Machatschek <markus.machatschek@hey.com>

.cargo/config.toml

@@ -1,2 +0,0 @@
-[alias]
-xtask = "run --release --package xtask --"

.github/ISSUE_TEMPLATE/bug_report.md

@@ -23,8 +23,7 @@ A clear and concise description of what you expected to happen.
 **Screenshots**
 If applicable, add screenshots to help explain your problem.
 
-**Meilisearch version:**
-[e.g. v0.20.0]
+**Meilisearch version:** [e.g. v0.20.0]
 
 **Additional context**
 Additional information that may be relevant to the issue.

.github/ISSUE_TEMPLATE/sprint_issue.md

@@ -1,54 +0,0 @@
----
-name: New sprint issue
-about: ⚠️ Should only be used by the engine team ⚠️
-title: ''
-labels: 'missing usage in PRD, impacts docs'
-assignees: ''
-
----
-
-Related product team resources: [PRD]() (_internal only_)
-Related product discussion:
-
-## Motivation
-
-<!---Copy/paste the information in PRD or briefly detail the product motivation. Ask product team if any hesitation.-->
-
-## Usage
-
-<!---Link to the public part of the PRD, or to the related product discussion for experimental features-->
-
-## TODO
-
-<!---If necessary, create a list with technical/product steps-->
-
-### Reminders when modifying the API
-
-- [ ] Update the openAPI file with utoipa:
-  - [ ] If a new module has been introduced, create a new structure deriving [the OpenAPI proc-macro](https://docs.rs/utoipa/latest/utoipa/derive.OpenApi.html) and nest it in the main [openAPI structure](https://github.com/meilisearch/meilisearch/blob/f2185438eed60fa32d25b15480c5ee064f6fba4a/crates/meilisearch/src/routes/mod.rs#L64-L78).
-  - [ ] If a new route has been introduced, add the [path decorator](https://docs.rs/utoipa/latest/utoipa/attr.path.html) to it and add the route at the top of the file in its openAPI structure.
-  - [ ] If a structure which is deserialized or serialized in the API has been introduced or modified, it must derive the [`schema`](https://docs.rs/utoipa/latest/utoipa/macro.schema.html) or the [`IntoParams`](https://docs.rs/utoipa/latest/utoipa/derive.IntoParams.html) proc-macro.
-        If it's a **new** structure you must also add it to the big list of structures [in the main `OpenApi` structure](https://github.com/meilisearch/meilisearch/blob/f2185438eed60fa32d25b15480c5ee064f6fba4a/crates/meilisearch/src/routes/mod.rs#L88).
-  - [ ] Once everything is done, start Meilisearch with the swagger flag: `cargo run --features swagger`, open `http://localhost:7700/scalar` on your browser, and ensure everything works as expected.
-  - For more info, refer to [this presentation](https://pitch.com/v/generating-the-openapi-file-jrn3nh).
-
-### Reminders when modifying the Setting API
-
-<!--- Special steps to remind when adding a new index setting -->
-
-- [ ] Ensure the new setting route is at least tested by the [`test_setting_routes` macro](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/tests/settings/get_settings.rs#L276)
-- [ ] Ensure Analytics are fully implemented
-  - [ ] `/settings/my-new-setting` configurated in the [`make_setting_routes` macro](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/src/routes/indexes/settings.rs#L141-L165)
-  - [ ] global `/settings` route configurated in the [`update_all` function](https://github.com/meilisearch/meilisearch/blob/5204c0b60b384cbc79621b6b2176fca086069e8e/meilisearch/src/routes/indexes/settings.rs#L655-L751)
-- [ ] Ensure the dump serializing is consistent with the `/settings` route serializing, e.g., enums case can be different (`camelCase` in route and `PascalCase` in the dump)
-
-#### Special cases when adding a setting for an experimental feature
-
-- [ ] ⚠️ API stability: The setting does not appear on the main settings route when the feature has never been enabled (e.g. mark it `Unset` when returned from the index in this situation. See [an example](https://github.com/meilisearch/meilisearch/blob/7a89abd2a025606a42f8b219e539117eb2eb029f/meilisearch-types/src/settings.rs#L608))
-- [ ] The setting cannot be set when the feature is disabled, either by the main settings route or the subroute (see [`validate_settings` function](https://github.com/meilisearch/meilisearch/blob/7a89abd2a025606a42f8b219e539117eb2eb029f/meilisearch/src/routes/indexes/settings.rs#L811))
-- [ ] If possible, the setting is reset when the feature is disabled (hard if it requires reindexing)
-
-## Impacted teams
-
-<!---Ping the related teams. Ask for the engine manager if any hesitation-->
-<!---@meilisearch/docs-team when there is any API change, e.g. settings addition-->

.github/scripts/check-release.sh

@@ -1,41 +1,24 @@
-#!/usr/bin/env bash
-set -eu -o pipefail
+#!/bin/bash
 
-check_tag() {
-    local expected=$1
-    local actual=$2
-    local filename=$3
-
-    if [[ $actual != $expected ]]; then
-        echo >&2 "Error: the current tag does not match the version in $filename: found $actual, expected $expected"
-        return 1
-    fi
+# check_tag $current_tag $file_tag $file_name
+function check_tag {
+  if [[ "$1" != "$2" ]]; then
+      echo "Error: the current tag does not match the version in Cargo.toml: found $2 - expected $1"
+      ret=1
+  fi
 }
 
-read_version() {
-    grep '^version = ' | cut -d \" -f 2
-}
-
-if [[ -z "${GITHUB_REF:-}" ]]; then
-    echo >&2 "Error: GITHUB_REF is not set"
-    exit 1
-fi
-
-if [[ ! "$GITHUB_REF" =~ ^refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-[a-z0-9]+)?$ ]]; then
-    echo >&2 "Error: GITHUB_REF is not a valid tag: $GITHUB_REF"
-    exit 1
-fi
-
-current_tag=${GITHUB_REF#refs/tags/v}
 ret=0
+current_tag=${GITHUB_REF#'refs/tags/v'}
 
-toml_tag="$(cat Cargo.toml | read_version)"
-check_tag "$current_tag" "$toml_tag" Cargo.toml || ret=1
+file_tag="$(grep '^version = ' Cargo.toml | cut -d '=' -f 2 | tr -d '"' | tr -d ' ')"
+check_tag $current_tag $file_tag
 
-lock_tag=$(grep -A 1 '^name = "meilisearch-auth"' Cargo.lock | read_version)
-check_tag "$current_tag" "$lock_tag" Cargo.lock || ret=1
+lock_file='Cargo.lock'
+lock_tag=$(grep -A 1 'name = "meilisearch-auth"' $lock_file | grep version | cut -d '=' -f 2 | tr -d '"' | tr -d ' ')
+check_tag $current_tag $lock_tag $lock_file
 
-if (( ret == 0 )); then
-    echo 'OK'
+if [[ "$ret" -eq 0 ]] ; then
+  echo 'OK'
 fi
 exit $ret

.github/uffizzi/Dockerfile

@@ -0,0 +1,48 @@
+# Compile
+FROM    rust:alpine3.16 AS compiler
+
+RUN     apk add -q --update-cache --no-cache build-base openssl-dev
+
+WORKDIR /meilisearch
+
+ARG     COMMIT_SHA
+ARG     COMMIT_DATE
+ARG     GIT_TAG
+ENV     COMMIT_SHA=${COMMIT_SHA} COMMIT_DATE=${COMMIT_DATE} VERGEN_GIT_SEMVER_LIGHTWEIGHT=${GIT_TAG}
+ENV     RUSTFLAGS="-C target-feature=-crt-static"
+
+COPY    . .
+RUN     set -eux; \
+        apkArch="$(apk --print-arch)"; \
+        if [ "$apkArch" = "aarch64" ]; then \
+            export JEMALLOC_SYS_WITH_LG_PAGE=16; \
+        fi && \
+        cargo build --release
+
+# Run
+FROM    uffizzi/ttyd:alpine
+
+ENV     MEILI_HTTP_ADDR 0.0.0.0:7700
+ENV     MEILI_SERVER_PROVIDER docker
+ENV     MEILI_NO_ANALYTICS true
+
+RUN     apk update --quiet \
+        && apk add -q --no-cache libgcc tini curl
+
+# add meilisearch to the `/bin` so you can run it from anywhere and it's easy
+# to find.
+COPY    --from=compiler /meilisearch/target/release/meilisearch /bin/meilisearch
+# To stay compatible with the older version of the container (pre v0.27.0) we're
+# going to symlink the meilisearch binary in the path to `/meilisearch`
+RUN     ln -s /bin/meilisearch /meilisearch
+
+# This directory should hold all the data related to meilisearch so we're going
+# to move our PWD in there.
+# We don't want to put the meilisearch binary
+WORKDIR /meili_data
+
+
+EXPOSE  7700/tcp
+
+ENTRYPOINT ["tini", "--"]
+CMD     ["ttyd", "/bin/zsh"]

.github/uffizzi/docker-compose.uffizzi.yml

@@ -0,0 +1,26 @@
+version: "3"
+
+x-uffizzi:
+  ingress:
+    service: nginx
+    port: 8081
+
+services:
+  meilisearch:
+    image: "${MEILISEARCH_IMAGE}"
+    restart: unless-stopped
+    ports:
+    - "7681:7681"
+    - "7700:7700"
+    deploy:
+      resources:
+        limits:
+          memory: 500M
+  
+  nginx:
+    image: nginx:alpine
+    restart: unless-stopped
+    ports:
+    - "8081:8081"
+    volumes:
+    - ./.github/uffizzi/nginx:/etc/nginx

.github/uffizzi/nginx/nginx.conf

@@ -0,0 +1,28 @@
+
+events {
+  worker_connections  4096;  ## Default: 1024
+}
+
+http {
+    map $http_upgrade $connection_upgrade {
+        default upgrade;
+        '' close;
+    }
+
+    server {
+        listen 8081;
+
+        location / {
+            proxy_pass http://localhost:7681;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection $connection_upgrade;
+        }
+
+        location /meilisearch/ {
+            # rewrite /meilisearch/(.*) /$1  break;
+            proxy_pass http://localhost:7700/;
+        }
+    }
+}
+

.github/workflows/bench-manual.yml

@@ -1,28 +0,0 @@
-name: Bench (manual)
-
-on:
-    workflow_dispatch:
-        inputs:
-            workload:
-                description: 'The path to the workloads to execute (workloads/...)'
-                required: true
-                default: 'workloads/movies.json'
-
-env:
-    WORKLOAD_NAME: ${{ github.event.inputs.workload }}
-
-jobs:
-    benchmarks:
-        name: Run and upload benchmarks
-        runs-on: benchmarks
-        timeout-minutes: 180 # 3h
-        steps:
-            - uses: actions/checkout@v3
-            - uses: dtolnay/rust-toolchain@1.81
-              with:
-                profile: minimal
-
-            - name: Run benchmarks - workload ${WORKLOAD_NAME} - branch ${{ github.ref }} - commit ${{ github.sha }}
-              run: |
-               cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" --reason "Manual [Run #${{ github.run_id }}](https://github.com/meilisearch/meilisearch/actions/runs/${{ github.run_id }})" -- ${WORKLOAD_NAME}
-

.github/workflows/bench-pr.yml

@@ -1,82 +0,0 @@
-name: Bench (PR)
-on:
-    issue_comment:
-        types: [created]
-
-permissions:
-    issues: write
-
-env:
-    GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-
-jobs:
-    run-benchmarks-on-comment:
-      if: startsWith(github.event.comment.body, '/bench')
-      name: Run and upload benchmarks
-      runs-on: benchmarks
-      timeout-minutes: 180 # 3h
-      steps:
-        - name: Check permissions
-          id: permission
-          env:
-            PR_AUTHOR: ${{github.event.issue.user.login }}
-            COMMENT_AUTHOR: ${{github.event.comment.user.login }}
-            REPOSITORY: ${{github.repository}}
-            PR_ID: ${{github.event.issue.number}}
-          run: |
-            PR_REPOSITORY=$(gh api /repos/"$REPOSITORY"/pulls/"$PR_ID" --jq .head.repo.full_name)
-            if $(gh api /repos/"$REPOSITORY"/collaborators/"$PR_AUTHOR"/permission --jq .user.permissions.push)
-            then
-              echo "::notice title=Authentication success::PR author authenticated"
-            else
-              echo "::error title=Authentication error::PR author doesn't have push permission on this repository"
-              exit 1
-            fi
-            if $(gh api /repos/"$REPOSITORY"/collaborators/"$COMMENT_AUTHOR"/permission --jq .user.permissions.push)
-            then
-              echo "::notice title=Authentication success::Comment author authenticated"
-            else
-              echo "::error title=Authentication error::Comment author doesn't have push permission on this repository"
-              exit 1
-            fi
-            if [ "$PR_REPOSITORY" = "$REPOSITORY" ]
-            then
-              echo "::notice title=Authentication success::PR started from main repository"
-            else
-              echo "::error title=Authentication error::PR started from a fork"
-              exit 1
-            fi
-
-        - name: Check for Command
-          id: command
-          uses: xt0rted/slash-command-action@v2
-          with:
-              command: bench
-              reaction-type: "rocket"
-              repo-token: ${{ env.GH_TOKEN }}
-
-        - uses: xt0rted/pull-request-comment-branch@v3
-          id: comment-branch
-          with:
-            repo_token: ${{ env.GH_TOKEN }}
-
-        - uses: actions/checkout@v3
-          if: success()
-          with:
-            fetch-depth: 0 # fetch full history to be able to get main commit sha
-            ref: ${{ steps.comment-branch.outputs.head_ref }}
-
-        - uses: dtolnay/rust-toolchain@1.81
-          with:
-            profile: minimal
-
-        - name: Run benchmarks on PR ${{ github.event.issue.id }}
-          run: |
-            cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" \
-               --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" \
-               --reason "[Comment](${{ github.event.comment.html_url }}) on [#${{ github.event.issue.number }}](${{ github.event.issue.html_url }})" \
-               -- ${{ steps.command.outputs.command-arguments }} > benchlinks.txt
-
-        - name: Send comment in PR
-          run: |
-            gh pr comment ${{github.event.issue.number}} --body-file benchlinks.txt

.github/workflows/bench-push-indexing.yml

@@ -1,23 +0,0 @@
-name: Indexing bench (push)
-
-on:
-    push:
-        branches:
-            - main
-
-jobs:
-    benchmarks:
-        name: Run and upload benchmarks
-        runs-on: benchmarks
-        timeout-minutes: 180 # 3h
-        steps:
-          - uses: actions/checkout@v3
-          - uses: dtolnay/rust-toolchain@1.81
-            with:
-              profile: minimal
-
-          # Run benchmarks
-          - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch main - Commit ${{ github.sha }}
-            run: |
-              cargo xtask bench --api-key "${{ secrets.BENCHMARK_API_KEY }}" --dashboard-url "${{ vars.BENCHMARK_DASHBOARD_URL }}" --reason "Push on `main` [Run #${{ github.run_id }}](https://github.com/meilisearch/meilisearch/actions/runs/${{ github.run_id }})" -- workloads/*.json
-

.github/workflows/benchmarks-pr.yml

@@ -1,127 +0,0 @@
-name: Benchmarks (PR)
-on: issue_comment
-permissions:
-  issues: write
-
-env:
-  GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-
-jobs:
-  run-benchmarks-on-comment:
-    if: startsWith(github.event.comment.body, '/benchmark')
-    name: Run and upload benchmarks
-    runs-on: benchmarks
-    timeout-minutes: 4320 # 72h
-    steps:
-      - name: Check permissions
-        id: permission
-        env:
-          PR_AUTHOR: ${{github.event.issue.user.login }}
-          COMMENT_AUTHOR: ${{github.event.comment.user.login }}
-          REPOSITORY: ${{github.repository}}
-          PR_ID: ${{github.event.issue.number}}
-        run: |
-          PR_REPOSITORY=$(gh api /repos/"$REPOSITORY"/pulls/"$PR_ID" --jq .head.repo.full_name)
-          if $(gh api /repos/"$REPOSITORY"/collaborators/"$PR_AUTHOR"/permission --jq .user.permissions.push)
-          then
-            echo "::notice title=Authentication success::PR author authenticated"
-          else
-            echo "::error title=Authentication error::PR author doesn't have push permission on this repository"
-            exit 1
-          fi
-          if $(gh api /repos/"$REPOSITORY"/collaborators/"$COMMENT_AUTHOR"/permission --jq .user.permissions.push)
-          then
-            echo "::notice title=Authentication success::Comment author authenticated"
-          else
-            echo "::error title=Authentication error::Comment author doesn't have push permission on this repository"
-            exit 1
-          fi
-          if [ "$PR_REPOSITORY" = "$REPOSITORY" ]
-          then
-            echo "::notice title=Authentication success::PR started from main repository"
-          else
-            echo "::error title=Authentication error::PR started from a fork"
-            exit 1
-          fi
-
-      - uses: dtolnay/rust-toolchain@1.81
-        with:
-          profile: minimal
-
-      - name: Check for Command
-        id: command
-        uses: xt0rted/slash-command-action@v2
-        with:
-          command: benchmark
-          reaction-type: "eyes"
-          repo-token: ${{ env.GH_TOKEN }}
-
-      - uses: xt0rted/pull-request-comment-branch@v3
-        id: comment-branch
-        with:
-          repo_token: ${{ env.GH_TOKEN }}
-
-      - uses: actions/checkout@v3
-        if: success()
-        with:
-          fetch-depth: 0 # fetch full history to be able to get main commit sha
-          ref: ${{ steps.comment-branch.outputs.head_ref }}
-
-      # Set variables
-      - name: Set current branch name
-        shell: bash
-        run: echo "name=$(git rev-parse --abbrev-ref HEAD)" >> $GITHUB_OUTPUT
-        id: current_branch
-      - name: Set normalized current branch name # Replace `/` by `_` in branch name to avoid issues when pushing to S3
-        shell: bash
-        run: echo "name=$(git rev-parse --abbrev-ref HEAD | tr '/' '_')" >> $GITHUB_OUTPUT
-        id: normalized_current_branch
-      - name: Set shorter commit SHA
-        shell: bash
-        run: echo "short=$(echo $GITHUB_SHA | cut -c1-8)" >> $GITHUB_OUTPUT
-        id: commit_sha
-      - name: Set file basename with format "dataset_branch_commitSHA"
-        shell: bash
-        run: echo "basename=$(echo ${{ steps.command.outputs.command-arguments }}_${{ steps.normalized_current_branch.outputs.name }}_${{ steps.commit_sha.outputs.short }})" >> $GITHUB_OUTPUT
-        id: file
-
-      # Run benchmarks
-      - name: Run benchmarks - Dataset ${{ steps.command.outputs.command-arguments }} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
-        run: |
-          cd crates/benchmarks
-          cargo bench --bench ${{ steps.command.outputs.command-arguments }} -- --save-baseline ${{ steps.file.outputs.basename }}
-
-      # Generate critcmp files
-      - name: Install critcmp
-        uses: taiki-e/install-action@v2
-        with:
-          tool: critcmp
-      - name: Export cripcmp file
-        run: |
-          critcmp --export ${{ steps.file.outputs.basename }} > ${{ steps.file.outputs.basename }}.json
-
-      # Upload benchmarks
-      - name: Upload ${{ steps.file.outputs.basename }}.json to DO Spaces # DigitalOcean Spaces = S3
-        uses: BetaHuhn/do-spaces-action@v2
-        with:
-          access_key: ${{ secrets.DO_SPACES_ACCESS_KEY }}
-          secret_key: ${{ secrets.DO_SPACES_SECRET_KEY }}
-          space_name: ${{ secrets.DO_SPACES_SPACE_NAME }}
-          space_region: ${{ secrets.DO_SPACES_SPACE_REGION }}
-          source: ${{ steps.file.outputs.basename }}.json
-          out_dir: critcmp_results
-
-      # Compute the diff of the benchmarks and send a message on the GitHub PR
-      - name: Compute and send a message in the PR
-        env:
-          GITHUB_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-        run: |
-          set -x
-          export base_ref=$(git merge-base origin/main ${{ steps.comment-branch.outputs.head_ref }} | head -c8)
-          export base_filename=$(echo ${{ steps.command.outputs.command-arguments }}_main_${base_ref}.json)
-          export bench_name=$(echo ${{ steps.command.outputs.command-arguments }})
-          echo "Here are your $bench_name benchmarks diff 👊" >> body.txt
-          echo '```' >> body.txt
-          ./benchmarks/scripts/compare.sh $base_filename ${{ steps.file.outputs.basename }}.json >> body.txt
-          echo '```' >> body.txt
-          gh pr comment ${{ steps.current_branch.outputs.name }} --body-file body.txt

.github/workflows/check-valid-milestone.yml

@@ -1,100 +0,0 @@
-name: PR Milestone Check
-
-on:
-  pull_request:
-    types: [opened, reopened, edited, synchronize, milestoned, demilestoned]
-    branches:
-      - "main"
-      - "release-v*.*.*"
-
-jobs:
-  check-milestone:
-    name: Check PR Milestone
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v3
-
-      - name: Validate PR milestone
-        uses: actions/github-script@v6
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          script: |
-            // Get PR number directly from the event payload
-            const prNumber = context.payload.pull_request.number;
-
-            // Get PR details
-            const { data: prData } = await github.rest.pulls.get({
-              owner: 'meilisearch',
-              repo: 'meilisearch',
-              pull_number: prNumber
-            });
-
-            // Get base branch name
-            const baseBranch = prData.base.ref;
-            console.log(`Base branch: ${baseBranch}`);
-
-            // Get PR milestone
-            const prMilestone = prData.milestone;
-            if (!prMilestone) {
-              core.setFailed('PR must have a milestone assigned');
-              return;
-            }
-            console.log(`PR milestone: ${prMilestone.title}`);
-
-            // Validate milestone format: vx.y.z
-            const milestoneRegex = /^v\d+\.\d+\.\d+$/;
-            if (!milestoneRegex.test(prMilestone.title)) {
-              core.setFailed(`Milestone "${prMilestone.title}" does not follow the required format vx.y.z`);
-              return;
-            }
-
-            // For main branch PRs, check if the milestone is the highest one
-            if (baseBranch === 'main') {
-              // Get all milestones
-              const { data: milestones } = await github.rest.issues.listMilestones({
-                owner: 'meilisearch',
-                repo: 'meilisearch',
-                state: 'open',
-                sort: 'due_on',
-                direction: 'desc'
-              });
-
-              // Sort milestones by version number (vx.y.z)
-              const sortedMilestones = milestones
-                .filter(m => milestoneRegex.test(m.title))
-                .sort((a, b) => {
-                  const versionA = a.title.substring(1).split('.').map(Number);
-                  const versionB = b.title.substring(1).split('.').map(Number);
-
-                  // Compare major version
-                  if (versionA[0] !== versionB[0]) return versionB[0] - versionA[0];
-                  // Compare minor version
-                  if (versionA[1] !== versionB[1]) return versionB[1] - versionA[1];
-                  // Compare patch version
-                  return versionB[2] - versionA[2];
-                });
-
-              if (sortedMilestones.length === 0) {
-                core.setFailed('No valid milestones found in the repository. Please create at least one milestone with the format vx.y.z');
-                return;
-              }
-
-              const highestMilestone = sortedMilestones[0];
-              console.log(`Highest milestone: ${highestMilestone.title}`);
-
-              if (prMilestone.title !== highestMilestone.title) {
-                core.setFailed(`PRs targeting the main branch must use the highest milestone (${highestMilestone.title}), but this PR uses ${prMilestone.title}`);
-                return;
-              }
-            } else {
-              // For release branches, the milestone should match the branch version
-              const branchVersion = baseBranch.substring(8); // remove 'release-'
-              if (prMilestone.title !== branchVersion) {
-                core.setFailed(`PRs targeting release branch "${baseBranch}" must use the matching milestone "${branchVersion}", but this PR uses "${prMilestone.title}"`);
-                return;
-              }
-            }
-
-            console.log('PR milestone validation passed!');

.github/workflows/create-issue-dependencies.yml

@@ -0,0 +1,28 @@
+name: Create issue to upgrade dependencies
+on:
+  schedule:
+    # Run the first of the month, every 3 month
+    - cron: '0 0 1 */3 *'
+  workflow_dispatch:
+
+jobs:
+  create-issue:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Create an issue
+      uses: actions-ecosystem/action-create-issue@v1
+      with:
+        github_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        title: Upgrade dependencies
+        body: |
+          This issue is about updating Meilisearch dependencies:
+          - [ ] Cargo toml dependencies of Meilisearch; but also the main engine-team repositories that Meilisearch depends on (charabia, heed...)
+          - [ ] If new Rust versions have been released, update the Rust version in the Clippy job of this [GitHub Action file](./.github/workflows/rust.yml)
+
+          ⚠️ To avoid last minute bugs, this issue should only be done at the beginning of the sprint!
+
+          The GitHub action dependencies are managed by [Dependabot](./.github/dependabot.yml)
+        labels: |
+          dependencies
+          maintenance

.github/workflows/dependency-issue.yml

@@ -1,24 +0,0 @@
-name: Create issue to upgrade dependencies
-
-on:
-  schedule:
-    # Run the first of the month, every 6 month
-    - cron: '0 0 1 */6 *'
-  workflow_dispatch:
-
-jobs:
-  create-issue:
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-      GH_TOKEN: ${{ secrets.MEILI_BOT_GH_PAT }}
-    steps:
-    - uses: actions/checkout@v3
-    - name: Download the issue template
-      run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/dependency-issue.md > $ISSUE_TEMPLATE
-    - name: Create issue
-      run: |
-        gh issue create \
-          --title 'Upgrade dependencies' \
-          --label 'dependencies,maintenance' \
-          --body-file $ISSUE_TEMPLATE

.github/workflows/flaky-tests.yml

@@ -1,30 +0,0 @@
-name: Look for flaky tests
-
-on:
-  workflow_dispatch:
-  schedule:
-    - cron: "0 12 * * FRI" # Every Friday at 12:00PM
-
-jobs:
-  flaky:
-    runs-on: ubuntu-latest
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Install cargo-flaky
-        run: cargo install cargo-flaky
-      - name: Run cargo flaky in the dumps
-        run: cd crates/dump; cargo flaky -i 100 --release
-      - name: Run cargo flaky in the index-scheduler
-        run: cd crates/index-scheduler; cargo flaky -i 100 --release
-      - name: Run cargo flaky in the auth
-        run: cd crates/meilisearch-auth; cargo flaky -i 100 --release
-      - name: Run cargo flaky in meilisearch
-        run: cd crates/meilisearch; cargo flaky -i 100 --release

.github/workflows/flaky.yml

@@ -0,0 +1,32 @@
+name: Look for flaky tests
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 12 * * FRI" # Every Friday at 12:00PM
+
+jobs:
+  flaky:
+    runs-on: ubuntu-latest
+    container:
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Install cargo-flaky
+      run: cargo install cargo-flaky
+    - name: Run cargo flaky in the dumps
+      run: cd dump; cargo flaky -i 100 --release
+    - name: Run cargo flaky in the index-scheduler
+      run: cd index-scheduler; cargo flaky -i 100 --release
+    - name: Run cargo flaky in the auth
+      run: cd meilisearch-auth; cargo flaky -i 100 --release
+    - name: Run cargo flaky in meilisearch
+      run: cd meilisearch; cargo flaky -i 100 --release

.github/workflows/fuzzer-indexing.yml

@@ -1,22 +0,0 @@
-name: Run the indexing fuzzer
-
-on:
-  push:
-    branches:
-      - main
-
-jobs:
-  fuzz:
-    name: Setup the action
-    runs-on: ubuntu-latest
-    timeout-minutes: 4320 # 72h
-    steps:
-      - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
-        with:
-          profile: minimal
-
-      # Run benchmarks
-      - name: Run the fuzzer
-        run: |
-          cargo run --release --bin fuzz-indexing

.github/workflows/manual_benchmarks.yml

@@ -1,4 +1,4 @@
-name: Benchmarks (manual)
+name: Benchmarks
 
 on:
   workflow_dispatch:
@@ -18,9 +18,11 @@ jobs:
     timeout-minutes: 4320 # 72h
     steps:
       - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -43,7 +45,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files
@@ -72,4 +74,4 @@ jobs:
           echo "${{ steps.file.outputs.basename }}.json has just been pushed."
           echo 'How to compare this benchmark with another one?'
           echo '  - Check the available files with: ./benchmarks/scripts/list.sh'
-          echo "  - Run the following command: ./benchmaks/scripts/compare.sh <file-to-compare-with> ${{ steps.file.outputs.basename }}.json"
+          echo "  - Run the following command: ./benchmaks/scipts/compare.sh <file-to-compare-with> ${{ steps.file.outputs.basename }}.json"

.github/workflows/milestone-workflow.yml

@@ -110,44 +110,6 @@ jobs:
             --milestone $MILESTONE_VERSION \
             --assignee curquiza
 
-  create-update-version-issue:
-    needs: get-release-version
-    # Create the update-version issue even if the release is a patch release
-    if: github.event.action == 'created'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-version-issue.md > $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Update version in Cargo.toml for $MILESTONE_VERSION" \
-            --label 'maintenance' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
-  create-update-openapi-issue:
-    needs: get-release-version
-    # Create the openAPI issue if the release is not only a patch release
-    if: github.event.action == 'created' && needs.get-release-version.outputs.is-patch == 'false'
-    runs-on: ubuntu-latest
-    env:
-      ISSUE_TEMPLATE: issue-template.md
-    steps:
-      - uses: actions/checkout@v3
-      - name: Download the issue template
-        run: curl -s https://raw.githubusercontent.com/meilisearch/engine-team/main/issue-templates/update-openapi-issue.md > $ISSUE_TEMPLATE
-      - name: Create the issue
-        run: |
-          gh issue create \
-            --title "Update Open API file for $MILESTONE_VERSION" \
-            --label 'maintenance' \
-            --body-file $ISSUE_TEMPLATE \
-            --milestone $MILESTONE_VERSION
-
 # ----------------
 # MILESTONE CLOSED
 # ----------------

.github/workflows/publish-apt-brew-pkg.yml

@@ -1,55 +0,0 @@
-name: Publish to APT & Homebrew
-
-on:
-  release:
-    types: [released]
-
-jobs:
-  check-version:
-    name: Check the version validity
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Check release validity
-        run: bash .github/scripts/check-release.sh
-
-  debian:
-    name: Publish debian packagge
-    runs-on: ubuntu-latest
-    needs: check-version
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
-    steps:
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Install cargo-deb
-        run: cargo install cargo-deb
-      - uses: actions/checkout@v3
-      - name: Build deb package
-        run: cargo deb -p meilisearch -o target/debian/meilisearch.deb
-      - name: Upload debian pkg to release
-        uses: svenstaro/upload-release-action@2.7.0
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: target/debian/meilisearch.deb
-          asset_name: meilisearch.deb
-          tag: ${{ github.ref }}
-      - name: Upload debian pkg to apt repository
-        run: curl -F package=@target/debian/meilisearch.deb https://${{ secrets.GEMFURY_PUSH_TOKEN }}@push.fury.io/meilisearch/
-
-  homebrew:
-    name: Bump Homebrew formula
-    runs-on: ubuntu-latest
-    needs: check-version
-    steps:
-      - name: Create PR to Homebrew
-        uses: mislav/bump-homebrew-formula-action@v3
-        with:
-          formula-name: meilisearch
-          formula-path: Formula/m/meilisearch.rb
-        env:
-          COMMITTER_TOKEN: ${{ secrets.HOMEBREW_COMMITTER_TOKEN }}

.github/workflows/publish-binaries.yml

@@ -1,12 +1,12 @@
-name: Publish binaries to GitHub release
-
 on:
   workflow_dispatch:
   schedule:
-    - cron: "0 2 * * *" # Every day at 2:00am
+    - cron: '0 2 * * *' # Every day at 2:00am
   release:
     types: [published]
 
+name: Publish binaries to release
+
 jobs:
   check-version:
     name: Check the version validity
@@ -37,26 +37,29 @@ jobs:
     runs-on: ubuntu-latest
     needs: check-version
     container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
+      # Use ubuntu-18.04 to compile with glibc 2.27
+      image: ubuntu:18.04
     steps:
-      - uses: actions/checkout@v3
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Build
-        run: cargo build --release --locked
-      # No need to upload binaries for dry run (cron)
-      - name: Upload binaries to release
-        if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.7.0
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: target/release/meilisearch
-          asset_name: meilisearch-linux-amd64
-          tag: ${{ github.ref }}
+    - uses: actions/checkout@v3
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Build
+      run: cargo build --release --locked
+    # No need to upload binaries for dry run (cron)
+    - name: Upload binaries to release
+      if: github.event_name == 'release'
+      uses: svenstaro/upload-release-action@2.4.0
+      with:
+        repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        file: target/release/meilisearch
+        asset_name: meilisearch-linux-amd64
+        tag: ${{ github.ref }}
 
   publish-macos-windows:
     name: Publish binary for ${{ matrix.os }}
@@ -65,46 +68,53 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-13, windows-2022]
+        os: [macos-12, windows-2022]
         include:
-          - os: macos-13
+          - os: macos-12
             artifact_name: meilisearch
             asset_name: meilisearch-macos-amd64
           - os: windows-2022
             artifact_name: meilisearch.exe
             asset_name: meilisearch-windows-amd64.exe
     steps:
-      - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Build
-        run: cargo build --release --locked
-      # No need to upload binaries for dry run (cron)
-      - name: Upload binaries to release
-        if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.7.0
-        with:
-          repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
-          file: target/release/${{ matrix.artifact_name }}
-          asset_name: ${{ matrix.asset_name }}
-          tag: ${{ github.ref }}
+    - uses: actions/checkout@v3
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Build
+      run: cargo build --release --locked
+    # No need to upload binaries for dry run (cron)
+    - name: Upload binaries to release
+      if: github.event_name == 'release'
+      uses: svenstaro/upload-release-action@2.4.0
+      with:
+        repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        file: target/release/${{ matrix.artifact_name }}
+        asset_name: ${{ matrix.asset_name }}
+        tag: ${{ github.ref }}
 
   publish-macos-apple-silicon:
     name: Publish binary for macOS silicon
-    runs-on: macos-13
+    runs-on: ${{ matrix.os }}
     needs: check-version
     strategy:
+      fail-fast: false
       matrix:
         include:
-          - target: aarch64-apple-darwin
+          - os: macos-12
+            target: aarch64-apple-darwin
             asset_name: meilisearch-macos-apple-silicon
     steps:
       - name: Checkout repository
         uses: actions/checkout@v3
       - name: Installing Rust toolchain
-        uses: dtolnay/rust-toolchain@1.81
+        uses: actions-rs/toolchain@v1
         with:
+          toolchain: stable
           profile: minimal
           target: ${{ matrix.target }}
+          override: true
       - name: Cargo build
         uses: actions-rs/cargo@v1
         with:
@@ -113,7 +123,7 @@ jobs:
       - name: Upload the binary to release
         # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.7.0
+        uses: svenstaro/upload-release-action@2.4.0
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch
@@ -122,37 +132,37 @@ jobs:
 
   publish-aarch64:
     name: Publish binary for aarch64
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     needs: check-version
-    env:
-      DEBIAN_FRONTEND: noninteractive
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
     strategy:
+      fail-fast: false
       matrix:
         include:
-          - target: aarch64-unknown-linux-gnu
+          - build: aarch64
+            os: ubuntu-18.04
+            target: aarch64-unknown-linux-gnu
+            linker: gcc-aarch64-linux-gnu
+            use-cross: true
             asset_name: meilisearch-linux-aarch64
     steps:
       - name: Checkout repository
         uses: actions/checkout@v3
-      - name: Install needed dependencies
-        run: |
-          apt-get update -y && apt upgrade -y
-          apt-get install -y curl build-essential gcc-aarch64-linux-gnu
-      - name: Set up Docker for cross compilation
-        run: |
-          apt-get install -y curl apt-transport-https ca-certificates software-properties-common
-          curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
-          add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
-          apt-get update -y && apt-get install -y docker-ce
       - name: Installing Rust toolchain
-        uses: dtolnay/rust-toolchain@1.81
+        uses: actions-rs/toolchain@v1
         with:
+          toolchain: stable
           profile: minimal
           target: ${{ matrix.target }}
+          override: true
+      - name: APT update
+        run: |
+          sudo apt update
+      - name: Install target specific tools
+        if: matrix.use-cross
+        run: |
+          sudo apt-get install -y ${{ matrix.linker }}
       - name: Configure target aarch64 GNU
+        if: matrix.target == 'aarch64-unknown-linux-gnu'
         ## Environment variable is not passed using env:
         ## LD gold won't work with MUSL
         # env:
@@ -162,23 +172,18 @@ jobs:
           echo '[target.aarch64-unknown-linux-gnu]' >> ~/.cargo/config
           echo 'linker = "aarch64-linux-gnu-gcc"' >> ~/.cargo/config
           echo 'JEMALLOC_SYS_WITH_LG_PAGE=16' >> $GITHUB_ENV
-      - name: Install a default toolchain that will be used to build cargo cross
-        run: |
-          rustup default stable
       - name: Cargo build
         uses: actions-rs/cargo@v1
         with:
           command: build
-          use-cross: true
+          use-cross: ${{ matrix.use-cross }}
           args: --release --target ${{ matrix.target }}
-        env:
-          CROSS_DOCKER_IN_DOCKER: true
       - name: List target output files
         run: ls -lR ./target
       - name: Upload the binary to release
         # No need to upload binaries for dry run (cron)
         if: github.event_name == 'release'
-        uses: svenstaro/upload-release-action@2.7.0
+        uses: svenstaro/upload-release-action@2.4.0
         with:
           repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
           file: target/${{ matrix.target }}/release/meilisearch

.github/workflows/publish-deb-brew-pkg.yml

@@ -0,0 +1,57 @@
+name: Publish to APT repository & Homebrew
+
+on:
+  release:
+    types: [released]
+
+jobs:
+  check-version:
+    name: Check the version validity
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Check release validity
+        run: bash .github/scripts/check-release.sh
+
+  debian:
+    name: Publish debian packagge
+    runs-on: ubuntu-latest
+    needs: check-version
+    container:
+      # Use ubuntu-18.04 to compile with glibc 2.27
+      image: ubuntu:18.04
+    steps:
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Install cargo-deb
+      run: cargo install cargo-deb
+    - uses: actions/checkout@v3
+    - name: Build deb package
+      run: cargo deb -p meilisearch -o target/debian/meilisearch.deb
+    - name: Upload debian pkg to release
+      uses: svenstaro/upload-release-action@2.4.0
+      with:
+        repo_token: ${{ secrets.MEILI_BOT_GH_PAT }}
+        file: target/debian/meilisearch.deb
+        asset_name: meilisearch.deb
+        tag: ${{ github.ref }}
+    - name: Upload debian pkg to apt repository
+      run: curl -F package=@target/debian/meilisearch.deb https://${{ secrets.GEMFURY_PUSH_TOKEN }}@push.fury.io/meilisearch/
+
+  homebrew:
+    name: Bump Homebrew formula
+    runs-on: ubuntu-latest
+    needs: check-version
+    steps:
+      - name: Create PR to Homebrew
+        uses: mislav/bump-homebrew-formula-action@v2
+        with:
+          formula-name: meilisearch
+        env:
+          COMMITTER_TOKEN: ${{ secrets.HOMEBREW_COMMITTER_TOKEN }}

.github/workflows/publish-docker-images.yml

@@ -1,5 +1,4 @@
-name: Publish images to Docker Hub
-
+---
 on:
   push:
     # Will run for every tag pushed except `latest`
@@ -13,6 +12,8 @@ on:
     - cron: '0 23 * * *' # Every day at 11:00pm
   workflow_dispatch:
 
+name: Publish tagged images to Docker Hub
+
 jobs:
   docker:
     runs-on: docker
@@ -57,20 +58,20 @@ jobs:
           echo "date=$commit_date" >> $GITHUB_OUTPUT
 
       - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
+        uses: docker/setup-qemu-action@v2
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@v2
 
       - name: Login to Docker Hub
-        uses: docker/login-action@v3
+        uses: docker/login-action@v2
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Docker meta
         id: meta
-        uses: docker/metadata-action@v5
+        uses: docker/metadata-action@v4
         with:
           images: getmeili/meilisearch
           # Prevent `latest` to be updated for each new tag pushed.
@@ -80,11 +81,10 @@ jobs:
             type=ref,event=tag
             type=raw,value=nightly,enable=${{ github.event_name != 'push' }}
             type=semver,pattern=v{{major}}.{{minor}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
-            type=semver,pattern=v{{major}},enable=${{ steps.check-tag-format.outputs.stable == 'true' }}
             type=raw,value=latest,enable=${{ steps.check-tag-format.outputs.stable == 'true' && steps.check-tag-format.outputs.latest == 'true' }}
 
       - name: Build and push
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@v4
         with:
           push: true
           platforms: linux/amd64,linux/arm64
@@ -98,7 +98,7 @@ jobs:
       - name: Send CI information to Cloud team
         # Do not send if nightly build (i.e. 'schedule' or 'workflow_dispatch' event)
         if: github.event_name == 'push'
-        uses: peter-evans/repository-dispatch@v3
+        uses: peter-evans/repository-dispatch@v2
         with:
           token: ${{ secrets.MEILI_BOT_GH_PAT }}
           repository: meilisearch/meilisearch-cloud

.github/workflows/push_benchmarks_indexing.yml

@@ -1,4 +1,4 @@
-name: Benchmarks of indexing (push)
+name: Benchmarks indexing (push)
 
 on:
   push:
@@ -16,9 +16,11 @@ jobs:
     timeout-minutes: 4320 # 72h
     steps:
       - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -41,7 +43,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files

.github/workflows/push_benchmarks_search_geo.yml

@@ -1,4 +1,4 @@
-name: Benchmarks of search for geo (push)
+name: Benchmarks search geo (push)
 
 on:
   push:
@@ -15,9 +15,11 @@ jobs:
     runs-on: benchmarks
     steps:
       - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -40,7 +42,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files

.github/workflows/push_benchmarks_search_songs.yml

@@ -1,4 +1,4 @@
-name: Benchmarks of search for songs (push)
+name: Benchmarks search songs (push)
 
 on:
   push:
@@ -15,9 +15,11 @@ jobs:
     runs-on: benchmarks
     steps:
       - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -40,7 +42,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files

.github/workflows/push_benchmarks_search_wiki.yml

@@ -1,4 +1,4 @@
-name: Benchmarks of search for Wikipedia articles (push)
+name: Benchmarks search wikipedia articles (push)
 
 on:
   push:
@@ -15,9 +15,11 @@ jobs:
     runs-on: benchmarks
     steps:
       - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
 
       # Set variables
       - name: Set current branch name
@@ -40,7 +42,7 @@ jobs:
       # Run benchmarks
       - name: Run benchmarks - Dataset ${BENCH_NAME} - Branch ${{ steps.current_branch.outputs.name }} - Commit ${{ steps.commit_sha.outputs.short }}
         run: |
-          cd crates/benchmarks
+          cd benchmarks
           cargo bench --bench ${BENCH_NAME} -- --save-baseline ${{ steps.file.outputs.basename }}
 
       # Generate critcmp files

.github/workflows/rust.yml

@@ -0,0 +1,145 @@
+name: Rust
+
+on:
+  workflow_dispatch:
+  schedule:
+    # Everyday at 5:00am
+    - cron: '0 5 * * *'
+  pull_request:
+  push:
+    # trying and staging branches are for Bors config
+    branches:
+      - trying
+      - staging
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: 1
+  RUSTFLAGS: "-D warnings"
+
+jobs:
+  test-linux:
+    name: Tests on ubuntu-18.04
+    runs-on: ubuntu-latest
+    container:
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install needed dependencies
+      run: |
+        apt-get update && apt-get install -y curl
+        apt-get install build-essential -y
+    - name: Run test with Rust stable
+      if: github.event_name != 'schedule'
+      uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        override: true
+    - name: Run test with Rust nightly
+      if: github.event_name == 'schedule'
+      uses: actions-rs/toolchain@v1
+      with:
+        toolchain: nightly
+        override: true
+    # Disable cache due to disk space issues with Windows workers in CI
+    # - name: Cache dependencies
+    #   uses: Swatinem/rust-cache@v2.2.0
+    - name: Run cargo check without any default features
+      uses: actions-rs/cargo@v1
+      with:
+        command: build
+        args: --locked --release --no-default-features --all
+    - name: Run cargo test
+      uses: actions-rs/cargo@v1
+      with:
+        command: test
+        args: --locked --release --all
+
+  test-others:
+    name: Tests on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [macos-12, windows-2022]
+    steps:
+    - uses: actions/checkout@v3
+#     - name: Cache dependencies
+#       uses: Swatinem/rust-cache@v2.2.0
+    - name: Run cargo check without any default features
+      uses: actions-rs/cargo@v1
+      with:
+        command: build
+        args: --locked --release --no-default-features --all
+    - name: Run cargo test
+      uses: actions-rs/cargo@v1
+      with:
+        command: test
+        args: --locked --release --all
+
+  # We run tests in debug also, to make sure that the debug_assertions are hit
+  test-debug:
+    name: Run tests in debug
+    runs-on: ubuntu-latest
+    container:
+      # Use ubuntu-18.04 to compile with glibc 2.27, which are the production expectations
+      image: ubuntu:18.04
+    steps:
+      - uses: actions/checkout@v3
+      - name: Install needed dependencies
+        run: |
+          apt-get update && apt-get install -y curl
+          apt-get install build-essential -y
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
+      # - name: Cache dependencies
+      #   uses: Swatinem/rust-cache@v2.2.0
+      - name: Run tests in debug
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --locked --all
+
+  clippy:
+    name: Run Clippy
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: 1.67.0
+          override: true
+          components: clippy
+      # - name: Cache dependencies
+      #   uses: Swatinem/rust-cache@v2.2.0
+      - name: Run cargo clippy
+        uses: actions-rs/cargo@v1
+        with:
+          command: clippy
+          # allow unlined_format_args https://github.com/rust-lang/rust-clippy/issues/10087
+          args: --all-targets -- --deny warnings --allow clippy::uninlined_format_args
+
+  fmt:
+    name: Run Rustfmt
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: nightly
+          override: true
+          components: rustfmt
+      # - name: Cache dependencies
+      #   uses: Swatinem/rust-cache@v2.2.0
+      - name: Run cargo fmt
+        # Since we never ran the `build.rs` script in the benchmark directory we are missing one auto-generated import file.
+        # Since we want to trigger (and fail) this action as fast as possible, instead of building the benchmark crate
+        # we are going to create an empty file where rustfmt expects it.
+        run: |
+          echo -ne "\n" > benchmarks/benches/datasets_paths.rs
+          cargo fmt --all -- --check

.github/workflows/sdks-tests.yml

@@ -1,386 +0,0 @@
-# If any test fails, the engine team should ensure the "breaking" changes are expected and contact the integration team
-name: SDKs tests
-
-on:
-  workflow_dispatch:
-    inputs:
-      docker_image:
-        description: 'The Meilisearch Docker image used'
-        required: false
-        default: nightly
-  schedule:
-    - cron: "0 6 * * MON" # Every Monday at 6:00AM
-
-env:
-  MEILI_MASTER_KEY: 'masterKey'
-  MEILI_NO_ANALYTICS: 'true'
-  DISABLE_COVERAGE: 'true'
-
-jobs:
-  define-docker-image:
-    runs-on: ubuntu-latest
-    outputs:
-      docker-image: ${{ steps.define-image.outputs.docker-image }}
-    steps:
-      - uses: actions/checkout@v4
-      - name: Define the Docker image we need to use
-        id: define-image
-        run: |
-          event=${{ github.event_name }}
-          echo "docker-image=nightly" >> $GITHUB_OUTPUT
-          if [[ $event == 'workflow_dispatch' ]]; then
-            echo "docker-image=${{ github.event.inputs.docker_image }}" >> $GITHUB_OUTPUT
-          fi
-      - name: Docker image is ${{ steps.define-image.outputs.docker-image }}
-        run: echo "Docker image is ${{ steps.define-image.outputs.docker-image }}"
-
-##########
-## SDKs ##
-##########
-
-  meilisearch-dotnet-tests:
-    needs: define-docker-image
-    name: .NET SDK tests
-    runs-on: ubuntu-latest
-    env:
-      MEILISEARCH_VERSION: ${{ needs.define-docker-image.outputs.docker-image }}
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-dotnet
-      - name: Setup .NET Core
-        uses: actions/setup-dotnet@v4
-        with:
-          dotnet-version: "8.0.x"
-      - name: Install dependencies
-        run: dotnet restore
-      - name: Build
-        run: dotnet build --configuration Release --no-restore
-      - name: Meilisearch (latest version) setup with Docker
-        run: docker compose up -d
-      - name: Run tests
-        run: dotnet test --no-restore --verbosity normal
-
-  meilisearch-dart-tests:
-    needs: define-docker-image
-    name: Dart SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-dart
-      - uses: dart-lang/setup-dart@v1
-        with:
-          sdk: 'latest'
-      - name: Install dependencies
-        run: dart pub get
-      - name: Run integration tests
-        run: dart test --concurrency=4
-
-  meilisearch-go-tests:
-    needs: define-docker-image
-    name: Go SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - name: Set up Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: stable
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-go
-      - name: Get dependencies
-        run: |
-          go get -v -t -d ./...
-          if [ -f Gopkg.toml ]; then
-            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
-            dep ensure
-          fi
-      - name: Run integration tests
-        run: go test -v ./...
-
-  meilisearch-java-tests:
-    needs: define-docker-image
-    name: Java SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-java
-      - name: Set up Java
-        uses: actions/setup-java@v4
-        with:
-          java-version: 8
-          distribution: 'zulu'
-          cache: gradle
-      - name: Grant execute permission for gradlew
-        run: chmod +x gradlew
-      - name: Build and run unit and integration tests
-        run: ./gradlew build integrationTest
-
-  meilisearch-js-tests:
-    needs: define-docker-image
-    name: JS SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-js
-      - name: Setup node
-        uses: actions/setup-node@v4
-        with:
-          cache: 'yarn'
-      - name: Install dependencies
-        run: yarn --dev
-      - name: Run tests
-        run: yarn test
-      - name: Build project
-        run: yarn build
-      - name: Run ESM env
-        run: yarn test:env:esm
-      - name: Run Node.js env
-        run: yarn test:env:nodejs
-      - name: Run node typescript env
-        run: yarn test:env:node-ts
-      - name: Run Browser env
-        run: yarn test:env:browser
-
-  meilisearch-php-tests:
-    needs: define-docker-image
-    name: PHP SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-php
-      - name: Install PHP
-        uses: shivammathur/setup-php@v2
-      - name: Validate composer.json and composer.lock
-        run: composer validate
-      - name: Install dependencies
-        run: |
-          composer remove --dev friendsofphp/php-cs-fixer --no-update --no-interaction
-          composer update --prefer-dist --no-progress
-      - name: Run test suite - default HTTP client (Guzzle 7)
-        run: |
-          sh scripts/tests.sh
-          composer remove --dev guzzlehttp/guzzle http-interop/http-factory-guzzle
-
-  meilisearch-python-tests:
-    needs: define-docker-image
-    name: Python SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-python
-      - name: Set up Python
-        uses: actions/setup-python@v5
-      - name: Install pipenv
-        uses: dschep/install-pipenv-action@v1
-      - name: Install dependencies
-        run: pipenv install --dev --python=${{ matrix.python-version }}
-      - name: Test with pytest
-        run: pipenv run pytest
-
-  meilisearch-ruby-tests:
-    needs: define-docker-image
-    name: Ruby SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-ruby
-      - name: Set up Ruby 3
-        uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: 3
-      - name: Install ruby dependencies
-        run: bundle install --with test
-      - name: Run test suite
-        run: bundle exec rspec
-
-  meilisearch-rust-tests:
-    needs: define-docker-image
-    name: Rust SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-rust
-      - name: Build
-        run: cargo build --verbose
-      - name: Run tests
-        run: cargo test --verbose
-
-  meilisearch-swift-tests:
-    needs: define-docker-image
-    name: Swift SDK tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-swift
-      - name: Run tests
-        run: swift test
-
-########################
-## FRONT-END PLUGINS ##
-########################
-
-  meilisearch-js-plugins-tests:
-    needs: define-docker-image
-    name: meilisearch-js-plugins tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-js-plugins
-      - name: Setup node
-        uses: actions/setup-node@v4
-        with:
-          cache: yarn
-      - name: Install dependencies
-        run: yarn install
-      - name: Run tests
-        run: yarn test
-      - name: Build all the playgrounds and the packages
-        run: yarn build
-
-########################
-## BACK-END PLUGINS ###
-########################
-
-  meilisearch-rails-tests:
-    needs: define-docker-image
-    name: meilisearch-rails tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-rails
-      - name: Set up Ruby 3
-        uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: 3
-          bundler-cache: true
-      - name: Run tests
-        run: bundle exec rspec
-
-  meilisearch-symfony-tests:
-    needs: define-docker-image
-    name: meilisearch-symfony tests
-    runs-on: ubuntu-latest
-    services:
-      meilisearch:
-        image: getmeili/meilisearch:${{ needs.define-docker-image.outputs.docker-image }}
-        env:
-          MEILI_MASTER_KEY: ${{ env.MEILI_MASTER_KEY }}
-          MEILI_NO_ANALYTICS: ${{ env.MEILI_NO_ANALYTICS }}
-        ports:
-          - '7700:7700'
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: meilisearch/meilisearch-symfony
-      - name: Install PHP
-        uses: shivammathur/setup-php@v2
-        with:
-          tools: composer:v2, flex
-      - name: Validate composer.json and composer.lock
-        run: composer validate
-      - name: Install dependencies
-        run: composer install --prefer-dist --no-progress --quiet
-      - name: Remove doctrine/annotations
-        run: composer remove --dev doctrine/annotations
-      - name: Run test suite
-        run: composer test:unit

.github/workflows/test-suite.yml

@@ -1,201 +0,0 @@
-name: Test suite
-
-on:
-  workflow_dispatch:
-  schedule:
-    # Everyday at 5:00am
-    - cron: "0 5 * * *"
-  pull_request:
-  merge_group:
-
-env:
-  CARGO_TERM_COLOR: always
-  RUST_BACKTRACE: 1
-  RUSTFLAGS: "-D warnings"
-
-jobs:
-  test-linux:
-    name: Tests on ubuntu-22.04
-    runs-on: ubuntu-latest
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
-    steps:
-      - uses: actions/checkout@v4
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - name: Setup test with Rust stable
-        uses: dtolnay/rust-toolchain@1.81
-      - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.7.7
-      - name: Run cargo check without any default features
-        uses: actions-rs/cargo@v1
-        with:
-          command: build
-          args: --locked --release --no-default-features --all
-      - name: Run cargo test
-        uses: actions-rs/cargo@v1
-        with:
-          command: test
-          args: --locked --release --all
-
-  test-others:
-    name: Tests on ${{ matrix.os }}
-    runs-on: ${{ matrix.os }}
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [macos-13, windows-2022]
-    steps:
-      - uses: actions/checkout@v3
-      - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.7.7
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Run cargo check without any default features
-        uses: actions-rs/cargo@v1
-        with:
-          command: build
-          args: --locked --release --no-default-features --all
-      - name: Run cargo test
-        uses: actions-rs/cargo@v1
-        with:
-          command: test
-          args: --locked --release --all
-
-  test-all-features:
-    name: Tests almost all features
-    runs-on: ubuntu-latest
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install needed dependencies
-        run: |
-          apt-get update
-          apt-get install --assume-yes build-essential curl
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Run cargo build with almost all features
-        run: |
-          cargo build --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
-      - name: Run cargo test with almost all features
-        run: |
-          cargo test --workspace --locked --release --features "$(cargo xtask list-features --exclude-feature cuda,test-ollama)"
-
-  ollama-ubuntu:
-    name: Test with Ollama
-    runs-on: ubuntu-latest
-    env:
-      MEILI_TEST_OLLAMA_SERVER: "http://localhost:11434"
-    steps:
-      - uses: actions/checkout@v1
-      - name: Install Ollama
-        run: |
-          curl -fsSL https://ollama.com/install.sh | sudo -E sh
-      - name: Start serving
-        run: |
-          # Run it in the background, there is no way to daemonise at the moment
-          ollama serve &
-
-          # A short pause is required before the HTTP port is opened
-          sleep 5
-
-          # This endpoint blocks until ready
-          time curl -i http://localhost:11434
-
-      - name: Pull nomic-embed-text & all-minilm
-        run: |
-          ollama pull nomic-embed-text
-          ollama pull all-minilm
-
-      - name: Run cargo test
-        uses: actions-rs/cargo@v1
-        with:
-          command: test
-          args: --locked --release --all --features test-ollama ollama
-
-  test-disabled-tokenization:
-    name: Test disabled tokenization
-    runs-on: ubuntu-latest
-    container:
-      image: ubuntu:22.04
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install needed dependencies
-        run: |
-          apt-get update
-          apt-get install --assume-yes build-essential curl
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Run cargo tree without default features and check lindera is not present
-        run: |
-          if cargo tree -f '{p} {f}' -e normal --no-default-features | grep -qz lindera; then
-            echo "lindera has been found in the sources and it shouldn't"
-            exit 1
-          fi
-      - name: Run cargo tree with default features and check lindera is pressent
-        run: |
-          cargo tree -f '{p} {f}' -e normal | grep lindera -qz
-
-  # We run tests in debug also, to make sure that the debug_assertions are hit
-  test-debug:
-    name: Run tests in debug
-    runs-on: ubuntu-latest
-    container:
-      # Use ubuntu-22.04 to compile with glibc 2.35
-      image: ubuntu:22.04
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install needed dependencies
-        run: |
-          apt-get update && apt-get install -y curl
-          apt-get install build-essential -y
-      - uses: dtolnay/rust-toolchain@1.81
-      - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.7.7
-      - name: Run tests in debug
-        uses: actions-rs/cargo@v1
-        with:
-          command: test
-          args: --locked --all
-
-  clippy:
-    name: Run Clippy
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
-        with:
-          profile: minimal
-          components: clippy
-      - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.7.7
-      - name: Run cargo clippy
-        uses: actions-rs/cargo@v1
-        with:
-          command: clippy
-          args: --all-targets -- --deny warnings
-
-  fmt:
-    name: Run Rustfmt
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
-        with:
-          profile: minimal
-          toolchain: nightly-2024-07-09
-          override: true
-          components: rustfmt
-      - name: Cache dependencies
-        uses: Swatinem/rust-cache@v2.7.7
-      - name: Run cargo fmt
-        # Since we never ran the `build.rs` script in the benchmark directory we are missing one auto-generated import file.
-        # Since we want to trigger (and fail) this action as fast as possible, instead of building the benchmark crate
-        # we are going to create an empty file where rustfmt expects it.
-        run: |
-          echo -ne "\n" > crates/benchmarks/benches/datasets_paths.rs
-          cargo fmt --all -- --check

.github/workflows/uffizzi-build.yml

@@ -0,0 +1,100 @@
+name: Uffizzi - Build PR Image
+on:
+  pull_request:
+    types: [opened,synchronize,reopened,closed]
+
+jobs:
+  build-meilisearch:
+    name: Build and push `meilisearch`
+    runs-on: ubuntu-latest
+    outputs:
+      tags: ${{ steps.meta.outputs.tags }}
+    if: ${{ github.event.action != 'closed' }}
+    steps:
+      - name: checkout
+        uses: actions/checkout@v3
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Generate UUID image name
+        id: uuid
+        run: echo "UUID_TAG=$(uuidgen)" >> $GITHUB_ENV
+
+      - name: Docker metadata
+        id: meta
+        uses: docker/metadata-action@v3
+        with:
+          images: registry.uffizzi.com/${{ env.UUID_TAG }}
+          tags: | 
+            type=raw,value=60d
+
+      - name: Build Image
+        uses: docker/build-push-action@v3
+        with:
+          context: ./
+          file: .github/uffizzi/Dockerfile
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          push: true
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+  render-compose-file:
+    name: Render Docker Compose File
+    # Pass output of this workflow to another triggered by `workflow_run` event.
+    runs-on: ubuntu-latest
+    needs: 
+      - build-meilisearch
+    outputs:
+      compose-file-cache-key:  ${{ env.COMPOSE_FILE_HASH }}
+    steps:
+      - name: Checkout git repo
+        uses: actions/checkout@v3
+      - name: Render Compose File
+        run: |
+          MEILISEARCH_IMAGE=$(echo ${{ needs.build-meilisearch.outputs.tags }})
+          export MEILISEARCH_IMAGE
+          # Render simple template from environment variables.
+          envsubst < .github/uffizzi/docker-compose.uffizzi.yml > docker-compose.rendered.yml
+          cat docker-compose.rendered.yml
+      - name: Upload Rendered Compose File as Artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: preview-spec 
+          path: docker-compose.rendered.yml
+          retention-days: 2
+      - name: Serialize PR Event to File
+        run:  |
+          cat << EOF > event.json
+          ${{ toJSON(github.event) }} 
+          
+          EOF
+      - name: Upload PR Event as Artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: preview-spec
+          path: event.json
+          retention-days: 2
+
+  delete-preview:
+    name: Call for Preview Deletion
+    runs-on: ubuntu-latest
+    if: ${{ github.event.action == 'closed' }}
+    steps:
+      # If this PR is closing, we will not render a compose file nor pass it to the next workflow.
+      - name: Serialize PR Event to File
+        run:  |
+          cat << EOF > event.json
+          ${{ toJSON(github.event) }} 
+          
+          EOF
+      - name: Upload PR Event as Artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: preview-spec
+          path: event.json
+          retention-days: 2

.github/workflows/uffizzi-preview-deploy.yml

@@ -0,0 +1,103 @@
+name: Uffizzi - Deploy Preview
+
+on:
+  workflow_run:
+    workflows:
+      - "Uffizzi - Build PR Image"
+    types:
+      - completed
+
+jobs:
+  cache-compose-file:
+    name: Cache Compose File
+    runs-on: ubuntu-latest
+    if: ${{ github.event.workflow_run.conclusion == 'success' }}
+    outputs:
+      compose-file-cache-key: ${{ env.COMPOSE_FILE_HASH }}
+      pr-number: ${{ env.PR_NUMBER }}
+      expected-url: ${{ env.EXPECTED_URL }}
+    steps:
+      - name: 'Download artifacts'
+        # Fetch output (zip archive) from the workflow run that triggered this workflow.
+        uses: actions/github-script@v6
+        with:
+          script: |
+            let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
+               owner: context.repo.owner,
+               repo: context.repo.repo,
+               run_id: context.payload.workflow_run.id,
+            });
+            let matchArtifact = allArtifacts.data.artifacts.filter((artifact) => {
+              return artifact.name == "preview-spec"
+            })[0];
+            let download = await github.rest.actions.downloadArtifact({
+               owner: context.repo.owner,
+               repo: context.repo.repo,
+               artifact_id: matchArtifact.id,
+               archive_format: 'zip',
+            });
+            let fs = require('fs');
+            fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/preview-spec.zip`, Buffer.from(download.data));
+      
+      - name: 'Unzip artifact'
+        run: unzip preview-spec.zip
+      
+      - name: Read Event into ENV
+        run: |
+          echo 'EVENT_JSON<<EOF' >> $GITHUB_ENV
+          cat event.json >> $GITHUB_ENV
+          echo 'EOF' >> $GITHUB_ENV
+      
+      - name: Hash Rendered Compose File
+        id: hash
+        # If the previous workflow was triggered by a PR close event, we will not have a compose file artifact.
+        if: ${{ fromJSON(env.EVENT_JSON).action != 'closed' }}
+        run: echo "COMPOSE_FILE_HASH=$(md5sum docker-compose.rendered.yml | awk '{ print $1 }')" >> $GITHUB_ENV
+      
+      - name: Cache Rendered Compose File
+        if: ${{ fromJSON(env.EVENT_JSON).action != 'closed' }}
+        uses: actions/cache@v3
+        with:
+          path: docker-compose.rendered.yml
+          key: ${{ env.COMPOSE_FILE_HASH }}
+
+      - name: Read PR Number From Event Object
+        id: pr
+        run: echo "PR_NUMBER=${{ fromJSON(env.EVENT_JSON).number }}" >> $GITHUB_ENV
+      
+      - name: DEBUG - Print Job Outputs
+        if: ${{ runner.debug }}
+        run: |
+          echo "PR number: ${{ env.PR_NUMBER }}"
+          echo "Compose file hash: ${{ env.COMPOSE_FILE_HASH }}"
+          cat event.json
+
+      - name: Add expected URL env var
+        if: ${{ runner.debug }}
+        run: |
+          REPO=$(echo ${{ github.repository }} | sed 's/\./+/g')
+          echo "EXPECTED_URL=${{ inputs.server }}/github.com/$REPO/pull/${{ env.PR_NUMBER }}" >> $GITHUB_ENV
+
+  deploy-uffizzi-preview:
+    name: Use Remote Workflow to Preview on Uffizzi
+    needs:
+      - cache-compose-file
+    uses: UffizziCloud/preview-action/.github/workflows/reusable.yaml@desc
+    with:
+      # If this workflow was triggered by a PR close event, cache-key will be an empty string
+      # and this reusable workflow will delete the preview deployment.
+      compose-file-cache-key: ${{ needs.cache-compose-file.outputs.compose-file-cache-key }}
+      compose-file-cache-path: docker-compose.rendered.yml
+      server: https://app.uffizzi.com
+      pr-number: ${{ needs.cache-compose-file.outputs.pr-number }}
+      description: |
+        The meilisearch preview environment contains a web terminal from where you can run the
+        `meilisearch` command. You should be able to access this instance of meilisearch running in
+        the preview from the link Meilisearch Endpoint link given below.
+         
+        Web Terminal Endpoint : ${{ needs.cache-compose-file.outputs.expected-url }}
+        Meilisearch Endpoint : ${{ needs.cache-compose-file.outputs.expected-url }}/meilisearch
+    permissions:
+      contents: read
+      pull-requests: write
+      id-token: write

.github/workflows/update-cargo-toml-version.yml

@@ -1,4 +1,4 @@
-name: Update Meilisearch version in Cargo.toml
+name: Update Meilisearch version in all Cargo.toml files
 
 on:
   workflow_dispatch:
@@ -14,16 +14,18 @@ env:
 
 jobs:
   update-version-cargo-toml:
-    name: Update version in Cargo.toml
+    name: Update version in Cargo.toml files
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-      - uses: dtolnay/rust-toolchain@1.81
+      - uses: actions-rs/toolchain@v1
         with:
           profile: minimal
+          toolchain: stable
+          override: true
       - name: Install sd
         run: cargo install sd
-      - name: Update Cargo.toml file
+      - name: Update Cargo.toml files
         run: |
           raw_new_version=$(echo $NEW_VERSION | cut -d 'v' -f 2)
           new_string="version = \"$raw_new_version\""
@@ -33,13 +35,13 @@ jobs:
       - name: Commit and push the changes to the ${{ env.NEW_BRANCH }} branch
         uses: EndBug/add-and-commit@v9
         with:
-          message: "Update version for the next release (${{ env.NEW_VERSION }}) in Cargo.toml"
+          message: "Update version for the next release (${{ env.NEW_VERSION }}) in Cargo.toml files"
           new_branch: ${{ env.NEW_BRANCH }}
       - name: Create the PR pointing to ${{ github.ref_name }}
         run: |
           gh pr create \
-            --title "Update version for the next release ($NEW_VERSION) in Cargo.toml" \
-            --body '⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.' \
+            --title "Update version for the next release ($NEW_VERSION) in Cargo.toml files" \
+            --body '⚠️ This PR is automatically generated. Check the new version is the expected one before merging.' \
             --label 'skip changelog' \
             --milestone $NEW_VERSION \
             --base $GITHUB_REF_NAME

.gitignore

@@ -5,12 +5,10 @@
 **/*.json_lines
 **/*.rs.bk
 /*.mdb
+/query-history.txt
 /data.ms
 /snapshots
 /dumps
-/bench
-/_xtask_benchmark.ms
-/benchmarks
 
 # Snapshots
 ## ... large
@@ -19,4 +17,4 @@
 *.snap.new
 
 # Fuzzcheck data for the facet indexing fuzz test
-crates/milli/fuzz/update::facet::incremental::fuzz::fuzz/
+milli/fuzz/update::facet::incremental::fuzz::fuzz/

BENCHMARKS.md

@@ -1,392 +0,0 @@
-# Benchmarks
-
-Currently this repository hosts two kinds of benchmarks:
-
-1. The older "milli benchmarks", that use [criterion](https://github.com/bheisler/criterion.rs) and live in the "benchmarks" directory.
-2. The newer "bench" that are workload-based and so split between the [`workloads`](./workloads/) directory and the [`xtask::bench`](./xtask/src/bench/) module.
-
-This document describes the newer "bench" benchmarks. For more details on the "milli benchmarks", see [benchmarks/README.md](./benchmarks/README.md).
-
-## Design philosophy for the benchmarks
-
-The newer "bench" benchmarks are **integration** benchmarks, in the sense that they spawn an actual Meilisearch server and measure its performance end-to-end, including HTTP request overhead.
-
-Since this is prone to fluctuating, the benchmarks regain a bit of precision by measuring the runtime of the individual spans using the [logging machinery](./CONTRIBUTING.md#logging) of Meilisearch.
-
-A span roughly translates to a function call. The benchmark runner collects all the spans by name using the [logs route](https://github.com/orgs/meilisearch/discussions/721) and sums their runtime. The processed results are then sent to the [benchmark dashboard](https://bench.meilisearch.dev), which is in charge of storing and presenting the data.
-
-## Running the benchmarks
-
-Benchmarks can run locally or in CI.
-
-### Locally
-
-#### With a local benchmark dashboard
-
-The benchmarks dashboard lives in its [own repository](https://github.com/meilisearch/benchboard). We provide binaries for Ubuntu/Debian, but you can build from source for other platforms (MacOS should work as it was developed under that platform).
-
-Run the `benchboard` binary to create a fresh database of results. By default it will serve the results and the API to gather results on `http://localhost:9001`.
-
-From the Meilisearch repository, you can then run benchmarks with:
-
-```sh
-cargo xtask bench -- workloads/my_workload_1.json ..
-```
-
-This command will build and run Meilisearch locally on port 7700, so make sure that this port is available.
-To run benchmarks on a different commit, just use the usual git command to get back to the desired commit.
-
-#### Without a local benchmark dashboard
-
-To work with the raw results, you can also skip using a local benchmark dashboard.
-
-Run:
-
-```sh
-cargo xtask bench --no-dashboard -- workloads/my_workload_1.json workloads/my_workload_2.json ..
-```
-
-For processing the results, look at [Looking at benchmark results/Without dashboard](#without-dashboard).
-
-#### Sending a workload by hand
-
-Sometimes you want to visualize the metrics of a worlkoad that comes from a custom report.
-It is not quite easy to trick the benchboard in thinking that your report is legitimate but here are the commands you can run to upload your firefox report on a running benchboard.
-
-```bash
-# Name this hostname whatever you want
-echo '{ "hostname": "the-best-place" }' | xh PUT 'http://127.0.0.1:9001/api/v1/machine'
-
-# You'll receive an UUID from this command that we will call $invocation_uuid
-echo '{ "commit": { "sha1": "1234567", "commit_date": "2024-09-05 12:00:12.0 +00:00:00", "message": "A cool message" }, "machine_hostname": "the-best-place", "max_workloads": 1 }' | xh PUT 'http://127.0.0.1:9001/api/v1/invocation'
-
-# Just use UUID from the previous command
-# and you'll receive another UUID that we will call $workload_uuid
-echo '{ "invocation_uuid": "$invocation_uuid", "name": "toto", "max_runs": 1 }' | xh PUT 'http://127.0.0.1:9001/api/v1/workload'
-
-# And now use your $workload_uuid and the content of your firefox report
-# but don't forget to convert your firefox report from JSONLines into an object
-echo '{ "workload_uuid": "$workload_uuid", "data": $REPORT_JSON_DATA }' | xh PUT 'http://127.0.0.1:9001/api/v1/run'
-```
-
-### In CI
-
-We have dedicated runners to run workloads on CI. Currently, there are three ways of running the CI:
-
-1. Automatically, on every push to `main`.
-2. Manually, by clicking the [`Run workflow`](https://github.com/meilisearch/meilisearch/actions/workflows/bench-manual.yml) button and specifying the target reference (tag, commit or branch) as well as one or multiple workloads to run. The workloads must exist in the Meilisearch repository (conventionally, in the [`workloads`](./workloads/) directory) on the target reference. Globbing (e.g., `workloads/*.json`) works.
-3. Manually on a PR, by posting a comment containing a `/bench` command, followed by one or multiple workloads to run. Globbing works. The workloads must exist in the Meilisearch repository in the branch of the PR.
-  ```
-  /bench workloads/movies*.json /hackernews_1M.json
-  ```
-
-## Looking at benchmark results
-
-### On the dashboard
-
-Results are available on the global dashboard used by CI at <https://bench.meilisearch.dev> or on your [local dashboard](#with-a-local-benchmark-dashboard).
-
-The dashboard homepage presents three sections:
-
-1. The latest invocations (a call to `cargo xtask bench`, either local or by CI) with their reason (generally set to some helpful link in CI) and their status.
-2. The latest workloads ran on `main`.
-3. The latest workloads ran on other references.
-
-By default, the workload shows the total runtime delta with the latest applicable commit on `main`. The latest applicable commit is the latest commit for workload invocations that do not originate on `main`, and the latest previous commit for workload invocations that originate on `main`.
-
-You can explicitly request a detailed comparison by span with the `main` branch, the branch or origin, or any previous commit, by clicking the links at the bottom of the workload invocation.
-
-In the detailed comparison view, the spans are sorted by improvements, regressions, stable (no statistically significant change) and unstable (the span runtime is comparable to its standard deviation).
-
-You can click on the name of any span to get a box plot comparing the target commit with multiple commits of the selected branch.
-
-### Without dashboard
-
-After the workloads are done running, the reports will live in the Meilisearch repository, in the `bench/reports` directory (by default).
-
-You can then convert these reports into other formats.
-
-- To [Firefox profiler](https://profiler.firefox.com) format. Run:
-  ```sh
-  cd bench/reports
-  cargo run --release --bin trace-to-firefox -- my_workload_1-0-trace.json
-  ```
-  You can then upload the resulting `firefox-my_workload_1-0-trace.json` file to the online profiler.
-
-
-## Designing benchmark workloads
-
-Benchmark workloads conventionally live in the `workloads` directory of the Meilisearch repository.
-
-They are JSON files with the following structure (comments are not actually supported, to make your own, remove them or copy some existing workload file):
-
-```jsonc
-{
-  // Name of the workload. Must be unique to the workload, as it will be used to group results on the dashboard.
-  "name": "hackernews.ndjson_1M,no-threads",
-  // Number of consecutive runs of the commands that should be performed.
-  // Each run uses a fresh instance of Meilisearch and a fresh database.
-  // Each run produces its own report file.
-  "run_count": 3,
-  // List of arguments to add to the Meilisearch command line.
-  "extra_cli_args": ["--max-indexing-threads=1"],
-  // An expression that can be parsed as a comma-separated list of targets and levels
-  // as described in [tracing_subscriber's documentation](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/targets/struct.Targets.html#examples).
-  // The expression is used to filter the spans that are measured for profiling purposes.
-  // Optional, defaults to "indexing::=trace" (for indexing workloads), common other values is
-  // "search::=trace"
-  "target": "indexing::=trace",
-  // List of named assets that can be used in the commands.
-  "assets": {
-    // name of the asset.
-    // Must be unique at the workload level.
-    // For better results, the same asset (same sha256) should have the same name accross workloads.
-    // Having multiple assets with the same name and distinct hashes is supported accross workloads,
-    // but will lead to superfluous downloads.
-    //
-    // Assets are stored in the `bench/assets/` directory by default.
-    "hackernews-100_000.ndjson": {
-      // If the assets exists in the local filesystem (Meilisearch repository or for your local workloads)
-      // Its file path can be specified here.
-      // `null` if the asset should be downloaded from a remote location.
-      "local_location": null,
-      // URL of the remote location where the asset can be downloaded.
-      // Use the `--assets-key` of the runner to pass an API key in the `Authorization: Bearer` header of the download requests.
-      // `null` if the asset should be imported from a local location.
-      // if both local and remote locations are specified, then the local one is tried first, then the remote one
-      // if the file is locally missing or its hash differs.
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-100_000.ndjson",
-      // SHA256 of the asset.
-      // Optional, the `sha256` of the asset will be displayed during a run of the workload if it is missing.
-      // If present, the hash of the asset in the `bench/assets/` directory will be compared against this hash before
-      // running the workload. If the hashes differ, the asset will be downloaded anew.
-      "sha256": "60ecd23485d560edbd90d9ca31f0e6dba1455422f2a44e402600fbb5f7f1b213",
-      // Optional, one of "Auto", "Json", "NdJson" or "Raw".
-      // If missing, assumed to be "Auto".
-      // If "Auto", the format will be determined from the extension in the asset name.
-      "format": "NdJson"
-    },
-    "hackernews-200_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-200_000.ndjson",
-      "sha256": "785b0271fdb47cba574fab617d5d332276b835c05dd86e4a95251cf7892a1685"
-    },
-    "hackernews-300_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-300_000.ndjson",
-      "sha256": "de73c7154652eddfaf69cdc3b2f824d5c452f095f40a20a1c97bb1b5c4d80ab2"
-    },
-    "hackernews-400_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-400_000.ndjson",
-      "sha256": "c1b00a24689110f366447e434c201c086d6f456d54ed1c4995894102794d8fe7"
-    },
-    "hackernews-500_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-500_000.ndjson",
-      "sha256": "ae98f9dbef8193d750e3e2dbb6a91648941a1edca5f6e82c143e7996f4840083"
-    },
-    "hackernews-600_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-600_000.ndjson",
-      "sha256": "b495fdc72c4a944801f786400f22076ab99186bee9699f67cbab2f21f5b74dbe"
-    },
-    "hackernews-700_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-700_000.ndjson",
-      "sha256": "4b2c63974f3dabaa4954e3d4598b48324d03c522321ac05b0d583f36cb78a28b"
-    },
-    "hackernews-800_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-800_000.ndjson",
-      "sha256": "cb7b6afe0e6caa1be111be256821bc63b0771b2a0e1fad95af7aaeeffd7ba546"
-    },
-    "hackernews-900_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-900_000.ndjson",
-      "sha256": "e1154ddcd398f1c867758a93db5bcb21a07b9e55530c188a2917fdef332d3ba9"
-    },
-    "hackernews-1_000_000.ndjson": {
-      "local_location": null,
-      "remote_location": "https://milli-benchmarks.fra1.digitaloceanspaces.com/bench/datasets/hackernews/hackernews-1_000_000.ndjson",
-      "sha256": "27e25efd0b68b159b8b21350d9af76938710cb29ce0393fa71b41c4f3c630ffe"
-    }
-  },
-  // Core of the workload.
-  // A list of commands to run sequentially.
-  // Optional: A precommand is a request to the Meilisearch instance that is executed before the profiling runs.
-  "precommands": [
-    {
-      // Meilisearch route to call. `http://localhost:7700/` will be prepended.
-      "route": "indexes/movies/settings",
-      // HTTP method to call.
-      "method": "PATCH",
-      // If applicable, body of the request.
-      // Optional, if missing, the body will be empty.
-      "body": {
-        // One of "empty", "inline" or "asset".
-        // If using "empty", you can skip the entire "body" key.
-        "inline": {
-          // when "inline" is used, the body is the JSON object that is the value of the `"inline"` key.
-          "displayedAttributes": [
-            "title",
-            "by",
-            "score",
-            "time"
-          ],
-          "searchableAttributes": [
-            "title"
-          ],
-          "filterableAttributes": [
-            "by"
-          ],
-          "sortableAttributes": [
-            "score",
-            "time"
-          ]
-        }
-      },
-      // Whether to wait before running the next request.
-      // One of:
-      // - DontWait: run the next command without waiting the response to this one.
-      // - WaitForResponse: run the next command as soon as the response from the server is received.
-      // - WaitForTask: run the next command once **all** the Meilisearch tasks created up to now have finished processing.
-      "synchronous": "WaitForTask"
-    }
-  ],
-  // A command is a request to the Meilisearch instance that is executed while the profiling runs.
-  "commands": [
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        // When using "asset", use the name of an asset as value to use the content of that asset as body.
-        // the content type is derived of the format of the asset:
-        // "NdJson" => "application/x-ndjson"
-        // "Json" => "application/json"
-        // "Raw" => "application/octet-stream"
-        // See [AssetFormat::to_content_type](https://github.com/meilisearch/meilisearch/blob/7b670a4afadb132ac4a01b6403108700501a391d/xtask/src/bench/assets.rs#L30)
-        // for details and up-to-date list.
-        "asset": "hackernews-100_000.ndjson"
-      },
-      "synchronous": "WaitForTask"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-200_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-300_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-400_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-500_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-600_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-700_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-800_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-900_000.ndjson"
-      },
-      "synchronous": "WaitForResponse"
-    },
-    {
-      "route": "indexes/movies/documents",
-      "method": "POST",
-      "body": {
-        "asset": "hackernews-1_000_000.ndjson"
-      },
-      "synchronous": "WaitForTask"
-    }
-  ]
-}
-```
-
-### Adding new assets
-
-Assets reside in our DigitalOcean S3 space. Assuming you have team access to the DigitalOcean S3 space:
-
-1. go to <https://cloud.digitalocean.com/spaces/milli-benchmarks?i=d1c552&path=bench%2Fdatasets%2F>
-2. upload your dataset:
-   1. if your dataset is a single file, upload that single file using the "upload" button,
-   2. otherwise, create a folder using the "create folder" button, then inside that folder upload your individual files.
-
-## Upgrading `https://bench.meilisearch.dev`
-
-The URL of the server is in our password manager (look for "benchboard").
-
-1. Make the needed modifications on the [benchboard repository](https://github.com/meilisearch/benchboard) and merge them to main.
-2. Publish a new release to produce the Ubuntu/Debian binary.
-3. Download the binary locally, send it to the server:
-  ```
-  scp -6 ~/Downloads/benchboard root@\[<ipv6-address>\]:/bench/new-benchboard
-  ```
-  Note that the ipv6 must be between escaped square brackets for SCP.
-4. SSH to the server:
-  ```
-  ssh root@<ipv6-address>
-  ```
-  Note the ipv6 must **NOT** be between escaped square brackets for SSH 🥲
-5. On the server, set the correct permissions for the new binary:
-   ```
-   chown bench:bench /bench/new-benchboard
-   chmod 700 /bench/new-benchboard
-   ```
-6. On the server, move the new binary to the location of the running binary (if unsure, start by making a backup of the running binary):
-  ```
-  mv /bench/{new-,}benchboard
-  ```
-7. Restart the benchboard service.
-  ```
-  systemctl restart benchboard
-  ```
-8. Check that the service runs correctly.
-  ```
-  systemctl status benchboard
-  ```
-9. Check the availability of the service by going to <https://bench.meilisearch.dev> on your browser.

CONTRIBUTING.md

@@ -4,7 +4,7 @@ First, thank you for contributing to Meilisearch! The goal of this document is t
 
 Remember that there are many ways to contribute other than writing code: writing [tutorials or blog posts](https://github.com/meilisearch/awesome-meilisearch), improving [the documentation](https://github.com/meilisearch/documentation), submitting [bug reports](https://github.com/meilisearch/meilisearch/issues/new?assignees=&labels=&template=bug_report.md&title=) and [feature requests](https://github.com/meilisearch/product/discussions/categories/feedback-feature-proposal)...
 
-Meilisearch can manage multiple indexes, handle the update store, and expose an HTTP API. Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/meilisearch/tree/main/milli), while tokenization is handled by [our `charabia` library](https://github.com/meilisearch/charabia/).
+The code in this repository is only concerned with managing multiple indexes, handling the update store, and exposing an HTTP API. Search and indexation are the domain of our core engine, [`milli`](https://github.com/meilisearch/milli), while tokenization is handled by [our `charabia` library](https://github.com/meilisearch/charabia/).
 
 If Meilisearch does not offer optimized support for your language, please consider contributing to `charabia` by following the [CONTRIBUTING.md file](https://github.com/meilisearch/charabia/blob/main/CONTRIBUTING.md) and integrating your intended normalizer/segmenter.
 
@@ -18,9 +18,9 @@ If Meilisearch does not offer optimized support for your language, please consid
 
 ## Assumptions
 
-1. **You're familiar with [GitHub](https://github.com) and the [Pull Requests (PR)](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests) workflow.**
-2. **You've read the Meilisearch [documentation](https://www.meilisearch.com/docs).**
-3. **You know about the [Meilisearch community on Discord](https://discord.meilisearch.com).
+1. **You're familiar with [GitHub](https://github.com) and the [Pull Requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests)(PR) workflow.**
+2. **You've read the Meilisearch [documentation](https://docs.meilisearch.com).**
+3. **You know about the [Meilisearch community](https://docs.meilisearch.com/learn/what_is_meilisearch/contact.html).
    Please use this for help.**
 
 ## How to Contribute
@@ -52,20 +52,6 @@ cargo test
 
 This command will be triggered to each PR as a requirement for merging it.
 
-#### Faster build
-
-You can set the `LINDERA_CACHE` environment variable to speed up your successive builds by up to 2 minutes.
-It'll store some built artifacts in the directory of your choice.
-
-We recommend using the standard `$HOME/.cache/lindera` directory:
-```sh
-export LINDERA_CACHE=$HOME/.cache/lindera
-```
-
-Furthermore, you can improve incremental compilation by setting the `MEILI_NO_VERGEN` environment variable.
-Setting this variable will prevent the Meilisearch binary from being rebuilt each time the directory that hosts the Meilisearch repository changes.
-Do not enable this environment variable for production builds (as it will break the `version` route, among other things).
-
 #### Snapshot-based tests
 
 We are using [insta](https://insta.rs) to perform snapshot-based testing.
@@ -77,7 +63,7 @@ Furthermore, we provide some macros on top of insta, notably a way to use snapsh
 
 To effectively debug snapshot-based hashes, we recommend you export the `MEILI_TEST_FULL_SNAPS` environment variable so that snapshot are fully created locally:
 
-```sh
+```
 export MEILI_TEST_FULL_SNAPS=true # add this to your .bashrc, .zshrc, ...
 ```
 
@@ -89,41 +75,6 @@ If you get a "Too many open files" error you might want to increase the open fil
 ulimit -Sn 3000
 ```
 
-#### Build tools
-
-Meilisearch follows the [cargo xtask](https://github.com/matklad/cargo-xtask) workflow to provide some build tools.
-
-Run `cargo xtask --help` from the root of the repository to find out what is available.
-
-#### Update the openAPI file if the API changed
-
-To update the openAPI file in the code, see [sprint_issue.md](https://github.com/meilisearch/meilisearch/blob/main/.github/ISSUE_TEMPLATE/sprint_issue.md#reminders-when-modifying-the-api).
-If you want to update the openAPI file on the [open-api repository](https://github.com/meilisearch/open-api), see [update-openapi-issue.md](https://github.com/meilisearch/engine-team/blob/main/issue-templates/update-openapi-issue.md).
-
-### Logging
-
-Meilisearch uses [`tracing`](https://lib.rs/crates/tracing) for logging purposes. Tracing logs are structured and can be displayed as JSON to the end user, so prefer passing arguments as fields rather than interpolating them in the message.
-
-Refer to the [documentation](https://docs.rs/tracing/0.1.40/tracing/index.html#using-the-macros) for the syntax of the spans and events.
-
-Logging spans are used for 3 distinct purposes:
-
-1. Regular logging
-2. Profiling
-3. Benchmarking
-
-As a result, the spans should follow some rules:
-
-- They should not be put on functions that are called too often. That is because opening and closing a span causes some overhead. For regular logging, avoid putting spans on functions that are taking less than a few hundred nanoseconds. For profiling or benchmarking, avoid putting spans on functions that are taking less than a few microseconds.
-- For profiling and benchmarking, use the `TRACE` level.
-- For profiling and benchmarking, use the following `target` prefixes:
-  - `indexing::` for spans meant when profiling the indexing operations.
-  - `search::` for spans meant when profiling the search operations.
-
-### Benchmarking
-
-See [BENCHMARKS.md](./BENCHMARKS.md)
-
 ## Git Guidelines
 
 ### Git Branches
@@ -150,7 +101,7 @@ Some notes on GitHub PRs:
 - The PR title should be accurate and descriptive of the changes.
 - [Convert your PR as a draft](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/changing-the-stage-of-a-pull-request) if your changes are a work in progress: no one will review it until you pass your PR as ready for review.<br>
   The draft PRs are recommended when you want to show that you are working on something and make your work visible.
-- The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [GitHub Merge Queues](https://github.blog/news-insights/product-news/github-merge-queue-is-generally-available/) to automatically enforce this requirement without the PR author having to rebase manually.
+- The branch related to the PR must be **up-to-date with `main`** before merging. Fortunately, this project uses [Bors](https://github.com/bors-ng/bors-ng) to automatically enforce this requirement without the PR author having to rebase manually.
 
 ## Release Process (for internal team only)
 
@@ -158,7 +109,8 @@ Meilisearch tools follow the [Semantic Versioning Convention](https://semver.org
 
 ### Automation to rebase and Merge the PRs
 
-This project uses GitHub Merge Queues that helps us manage pull requests merging.
+This project integrates a bot that helps us manage pull requests merging.<br>
+_[Read more about this](https://github.com/meilisearch/integration-guides/blob/main/resources/bors.md)._
 
 ### How to Publish a new Release
 
@@ -168,9 +120,29 @@ The full Meilisearch release process is described in [this guide](https://github
 
 Depending on the developed feature, you might need to provide a prototyped version of Meilisearch to make it easier to test by the users.
 
-This happens in two steps:
-- [Release the prototype](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#how-to-publish-a-prototype)
-- [Communicate about it](https://github.com/meilisearch/engine-team/blob/main/resources/prototypes.md#communication)
+The prototype name must follow this convention: `prototype-X-Y` where
+- `X` is the feature name formatted in `kebab-case`. It should not end with a single number.
+- `Y` is the version of the prototype, starting from `0`.
+
+✅ Example: `prototype-auto-resize-0`. </br>
+❌ Bad example: `auto-resize-0`: lacks the `prototype` prefix. </br>
+❌ Bad example: `prototype-auto-resize`: lacks the version suffix. </br>
+❌ Bad example: `prototype-auto-resize-0-0`: feature name ends with a single number.
+
+Steps to create a prototype:
+
+1. In your terminal, go to the last commit of your branch (the one you want to provide as a prototype).
+2. Create a tag following the convention: `git tag prototype-X-Y`
+3. Run Meilisearch and check that its launch summary features a line: `Prototype: prototype-X-Y` (you may need to switch branches and back after tagging for this to work).
+3. Push the tag: `git push origin prototype-X-Y`
+4. Check the [Docker CI](https://github.com/meilisearch/meilisearch/actions/workflows/publish-docker-images.yml) is now running.
+
+🐳 Once the CI has finished to run (~1h30), a Docker image named `prototype-X-Y` will be available on [DockerHub](https://hub.docker.com/repository/docker/getmeili/meilisearch/general). People can use it with the following command: `docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-X-Y`. <br>
+More information about [how to run Meilisearch with Docker](https://docs.meilisearch.com/learn/cookbooks/docker.html#download-meilisearch-with-docker).
+
+⚙️ However, no binaries will be created. If the users do not use Docker, they can go to the `prototype-X-Y` tag in the Meilisearch repository and compile from the source code.
+
+⚠️ When sharing a prototype with users, remind them to not use it in production. Prototypes are solely for test purposes.
 
 ### Release assets
 

Cargo.toml

@@ -1,32 +1,24 @@
 [workspace]
 resolver = "2"
 members = [
-    "crates/meilisearch",
-    "crates/meilitool",
-    "crates/meilisearch-types",
-    "crates/meilisearch-auth",
-    "crates/meili-snap",
-    "crates/index-scheduler",
-    "crates/dump",
-    "crates/file-store",
-    "crates/permissive-json-pointer",
-    "crates/milli",
-    "crates/filter-parser",
-    "crates/flatten-serde-json",
-    "crates/json-depth-checker",
-    "crates/benchmarks",
-    "crates/fuzzers",
-    "crates/tracing-trace",
-    "crates/xtask",
-    "crates/build-info",
+    "meilisearch",
+    "meilisearch-types",
+    "meilisearch-auth",
+    "meili-snap",
+    "index-scheduler",
+    "dump",
+    "file-store",
+    "permissive-json-pointer",
+    "milli",
+    "filter-parser",
+    "flatten-serde-json",
+    "json-depth-checker",
+    "benchmarks"
 ]
 
 [workspace.package]
-version = "1.14.0"
-authors = [
-    "Quentin de Quelen <quentin@dequelen.me>",
-    "Clément Renault <clement@meilisearch.com>",
-]
+version = "1.0.0"
+authors = ["Quentin de Quelen <quentin@dequelen.me>", "Clément Renault <clement@meilisearch.com>"]
 description = "Meilisearch HTTP server"
 homepage = "https://meilisearch.com"
 readme = "README.md"
@@ -36,12 +28,6 @@ license = "MIT"
 [profile.release]
 codegen-units = 1
 
-# We now compile heed without the NDEBUG define for better performance.
-# However, we still enable debug assertions for a better detection of
-# disk corruption on the cloud or in OSS.
-[profile.release.package.heed]
-debug-assertions = true
-
 [profile.dev.package.flate2]
 opt-level = 3
 
@@ -49,3 +35,24 @@ opt-level = 3
 opt-level = 3
 [profile.dev.package.roaring]
 opt-level = 3
+
+[profile.dev.package.lindera-ipadic-builder]
+opt-level = 3
+[profile.dev.package.encoding]
+opt-level = 3
+[profile.dev.package.yada]
+opt-level = 3
+
+[profile.release.package.lindera-ipadic-builder]
+opt-level = 3
+[profile.release.package.encoding]
+opt-level = 3
+[profile.release.package.yada]
+opt-level = 3
+
+[profile.bench.package.lindera-ipadic-builder]
+opt-level = 3
+[profile.bench.package.encoding]
+opt-level = 3
+[profile.bench.package.yada]
+opt-level = 3

Dockerfile

@@ -1,37 +1,36 @@
 # Compile
-FROM    rust:1.81.0-alpine3.20 AS compiler
+FROM    rust:bullseye AS compiler
 
-RUN     apk add -q --no-cache build-base openssl-dev
-
-WORKDIR /
+WORKDIR /meilisearch
 
 ARG     COMMIT_SHA
 ARG     COMMIT_DATE
 ARG     GIT_TAG
-ENV     VERGEN_GIT_SHA=${COMMIT_SHA} VERGEN_GIT_COMMIT_TIMESTAMP=${COMMIT_DATE} VERGEN_GIT_DESCRIBE=${GIT_TAG}
+ENV     VERGEN_GIT_SHA=${COMMIT_SHA} VERGEN_GIT_COMMIT_TIMESTAMP=${COMMIT_DATE} VERGEN_GIT_SEMVER_LIGHTWEIGHT=${GIT_TAG}
 ENV     RUSTFLAGS="-C target-feature=-crt-static"
 
 COPY    . .
 RUN     set -eux; \
-        apkArch="$(apk --print-arch)"; \
-        if [ "$apkArch" = "aarch64" ]; then \
+        arch="$(dpkg --print-architecture)"; \
+        if [ "$arch" = "arm64" ]; then \
             export JEMALLOC_SYS_WITH_LG_PAGE=16; \
         fi && \
-        cargo build --release -p meilisearch -p meilitool
+        cargo build --release
 
 # Run
-FROM    alpine:3.20
-LABEL   org.opencontainers.image.source="https://github.com/meilisearch/meilisearch"
+FROM    debian:11.6
 
 ENV     MEILI_HTTP_ADDR 0.0.0.0:7700
 ENV     MEILI_SERVER_PROVIDER docker
 
-RUN     apk add -q --no-cache libgcc tini curl
+RUN     set -ex; \
+        apt-get update -q; \
+        apt-get install -q -y --no-install-recommends tini; \
+        rm -rf /var/lib/apt/lists/*
 
-# add meilisearch and meilitool to the `/bin` so you can run it from anywhere
-# and it's easy to find.
-COPY    --from=compiler /target/release/meilisearch /bin/meilisearch
-COPY    --from=compiler /target/release/meilitool /bin/meilitool
+# add meilisearch to the `/bin` so you can run it from anywhere and it's easy
+# to find.
+COPY    --from=compiler /meilisearch/target/release/meilisearch /bin/meilisearch
 # To stay compatible with the older version of the container (pre v0.27.0) we're
 # going to symlink the meilisearch binary in the path to `/meilisearch`
 RUN     ln -s /bin/meilisearch /meilisearch

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019-2025 Meili SAS
+Copyright (c) 2019-2022 Meili SAS
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

PROFILING.md

@@ -1,19 +0,0 @@
-# Profiling Meilisearch
-
-Search engine technologies are complex pieces of software that require thorough profiling tools. We chose to use [Puffin](https://github.com/EmbarkStudios/puffin), which the Rust gaming industry uses extensively. You can export and import the profiling reports using the top bar's _File_ menu options [in Puffin Viewer](https://github.com/embarkstudios/puffin#ui).
-
-![An example profiling with Puffin viewer](assets/profiling-example.png)
-
-## Profiling the Indexing Process
-
-When you enable [the `exportPuffinReports` experimental feature](https://www.meilisearch.com/docs/learn/experimental/overview) of Meilisearch, Puffin reports with the `.puffin` extension will be automatically exported to disk. When this option is enabled, the engine will automatically create a "frame" whenever it executes the `IndexScheduler::tick` method.
-
-[Puffin Viewer](https://github.com/EmbarkStudios/puffin/tree/main/puffin_viewer) is used to analyze the reports. Those reports show areas where Meilisearch spent time during indexing.
-
-Another piece of advice on the Puffin viewer UI interface is to consider the _Merge children with same ID_ option. It can hide the exact actual timings at which events were sent. Please turn it off when you see strange gaps on the Flamegraph. It can help.
-
-## Profiling the Search Process
-
-We still need to take the time to profile the search side of the engine with Puffin. It would require time to profile the filtering phase, query parsing, creation, and execution. We could even profile the Actix HTTP server.
-
-The only issue we see is the framing system. Puffin requires a global frame-based profiling phase, which collides with Meilisearch's ability to accept and answer multiple requests on different threads simultaneously.

README.md

@@ -1,105 +1,94 @@
 <p align="center">
-  <a href="https://www.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=logo#gh-light-mode-only" target="_blank">
-    <img src="assets/meilisearch-logo-light.svg?sanitize=true#gh-light-mode-only">
-  </a>
-  <a href="https://www.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=logo#gh-dark-mode-only" target="_blank">
-    <img src="assets/meilisearch-logo-dark.svg?sanitize=true#gh-dark-mode-only">
-  </a>
+  <img src="assets/meilisearch-logo-light.svg?sanitize=true#gh-light-mode-only">
+  <img src="assets/meilisearch-logo-dark.svg?sanitize=true#gh-dark-mode-only">
 </p>
 
 <h4 align="center">
-  <a href="https://www.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=nav">Website</a> |
+  <a href="https://www.meilisearch.com">Website</a> |
   <a href="https://roadmap.meilisearch.com/tabs/1-under-consideration">Roadmap</a> |
-  <a href="https://www.meilisearch.com/pricing?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=nav">Meilisearch Cloud</a> |
-  <a href="https://blog.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=nav">Blog</a> |
-  <a href="https://www.meilisearch.com/docs?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=nav">Documentation</a> |
-  <a href="https://www.meilisearch.com/docs/faq?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=nav">FAQ</a> |
-  <a href="https://discord.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=nav">Discord</a>
+  <a href="https://blog.meilisearch.com">Blog</a> |
+  <a href="https://docs.meilisearch.com">Documentation</a> |
+  <a href="https://docs.meilisearch.com/faq/">FAQ</a> |
+  <a href="https://discord.meilisearch.com">Discord</a>
 </h4>
 
 <p align="center">
   <a href="https://deps.rs/repo/github/meilisearch/meilisearch"><img src="https://deps.rs/repo/github/meilisearch/meilisearch/status.svg" alt="Dependency status"></a>
   <a href="https://github.com/meilisearch/meilisearch/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-informational" alt="License"></a>
-  <a href="https://github.com/meilisearch/meilisearch/queue"><img alt="Merge Queues enabled" src="https://img.shields.io/badge/Merge_Queues-enabled-%2357cf60?logo=github"></a>
+  <a href="https://app.bors.tech/repositories/26457"><img src="https://bors.tech/images/badge_small.svg" alt="Bors enabled"></a>
 </p>
 
 <p align="center">⚡ A lightning-fast search engine that fits effortlessly into your apps, websites, and workflow 🔍</p>
 
-[Meilisearch](https://www.meilisearch.com?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=intro) helps you shape a delightful search experience in a snap, offering features that work out of the box to speed up your workflow.
+Meilisearch helps you shape a delightful search experience in a snap, offering features that work out-of-the-box to speed up your workflow.
 
 <p align="center" name="demo">
-  <a href="https://where2watch.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demo-gif#gh-light-mode-only" target="_blank">
+  <a href="https://where2watch.meilisearch.com/#gh-light-mode-only" target="_blank">
     <img src="assets/demo-light.gif#gh-light-mode-only" alt="A bright colored application for finding movies screening near the user">
   </a>
-  <a href="https://where2watch.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demo-gif#gh-dark-mode-only" target="_blank">
+  <a href="https://where2watch.meilisearch.com/#gh-dark-mode-only" target="_blank">
     <img src="assets/demo-dark.gif#gh-dark-mode-only" alt="A dark colored application for finding movies screening near the user">
   </a>
 </p>
 
-## 🖥 Examples
-
-- [**Movies**](https://where2watch.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=organization) — An application to help you find streaming platforms to watch movies using [hybrid search](https://www.meilisearch.com/solutions/hybrid-search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos).
-- [**Ecommerce**](https://ecommerce.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) — Ecommerce website using disjunctive [facets](https://www.meilisearch.com/docs/learn/fine_tuning_results/faceted_search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos), range and rating filtering, and pagination.
-- [**Songs**](https://music.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) — Search through 47 million of songs.
-- [**SaaS**](https://saas.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) — Search for contacts, deals, and companies in this [multi-tenant](https://www.meilisearch.com/docs/learn/security/multitenancy_tenant_tokens?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=demos) CRM application.
-
-See the list of all our example apps in our [demos repository](https://github.com/meilisearch/demos).
+🔥 [**Try it!**](https://where2watch.meilisearch.com/) 🔥
 
 ## ✨ Features
-- **Hybrid search:** Combine the best of both [semantic](https://www.meilisearch.com/docs/learn/experimental/vector_search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features) & full-text search to get the most relevant results
-- **Search-as-you-type:** Find & display results in less than 50 milliseconds to provide an intuitive experience
-- **[Typo tolerance](https://www.meilisearch.com/docs/learn/relevancy/typo_tolerance_settings?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** get relevant matches even when queries contain typos and misspellings
-- **[Filtering](https://www.meilisearch.com/docs/learn/fine_tuning_results/filtering?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features) and [faceted search](https://www.meilisearch.com/docs/learn/fine_tuning_results/faceted_search?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** enhance your users' search experience with custom filters and build a faceted search interface in a few lines of code
-- **[Sorting](https://www.meilisearch.com/docs/learn/fine_tuning_results/sorting?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** sort results based on price, date, or pretty much anything else your users need
-- **[Synonym support](https://www.meilisearch.com/docs/learn/relevancy/synonyms?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** configure synonyms to include more relevant content in your search results
-- **[Geosearch](https://www.meilisearch.com/docs/learn/fine_tuning_results/geosearch?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** filter and sort documents based on geographic data
-- **[Extensive language support](https://www.meilisearch.com/docs/learn/what_is_meilisearch/language?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** search datasets in any language, with optimized support for Chinese, Japanese, Hebrew, and languages using the Latin alphabet
-- **[Security management](https://www.meilisearch.com/docs/learn/security/master_api_keys?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** control which users can access what data with API keys that allow fine-grained permissions handling
-- **[Multi-Tenancy](https://www.meilisearch.com/docs/learn/security/multitenancy_tenant_tokens?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** personalize search results for any number of application tenants
+
+- **Search-as-you-type:** find search results in less than 50 milliseconds
+- **[Typo tolerance](https://docs.meilisearch.com/learn/getting_started/customizing_relevancy.html#typo-tolerance):** get relevant matches even when queries contain typos and misspellings
+- **[Filtering and faceted search](https://docs.meilisearch.com/learn/advanced/filtering_and_faceted_search.html):** enhance your user's search experience with custom filters and build a faceted search interface in a few lines of code
+- **[Sorting](https://docs.meilisearch.com/learn/advanced/sorting.html):** sort results based on price, date, or pretty much anything else your users need
+- **[Synonym support](https://docs.meilisearch.com/learn/getting_started/customizing_relevancy.html#synonyms):** configure synonyms to include more relevant content in your search results
+- **[Geosearch](https://docs.meilisearch.com/learn/advanced/geosearch.html):** filter and sort documents based on geographic data
+- **[Extensive language support](https://docs.meilisearch.com/learn/what_is_meilisearch/language.html):** search datasets in any language, with optimized support for Chinese, Japanese, Hebrew, and languages using the Latin alphabet
+- **[Security management](https://docs.meilisearch.com/learn/security/master_api_keys.html):** control which users can access what data with API keys that allow fine-grained permissions handling
+- **[Multi-Tenancy](https://docs.meilisearch.com/learn/security/tenant_tokens.html):** personalize search results for any number of application tenants
 - **Highly Customizable:** customize Meilisearch to your specific needs or use our out-of-the-box and hassle-free presets
-- **[RESTful API](https://www.meilisearch.com/docs/reference/api/overview?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=features):** integrate Meilisearch in your technical stack with our plugins and SDKs
-- **AI-ready:** works out of the box with [langchain](https://www.meilisearch.com/with/langchain) and the [model context protocol](https://github.com/meilisearch/meilisearch-mcp)
+- **[RESTful API](https://docs.meilisearch.com/reference/api/overview.html):** integrate Meilisearch in your technical stack with our plugins and SDKs
 - **Easy to install, deploy, and maintain**
 
 ## 📖 Documentation
 
-You can consult Meilisearch's documentation at [meilisearch.com/docs](https://www.meilisearch.com/docs/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=docs).
+You can consult Meilisearch's documentation at [https://docs.meilisearch.com](https://docs.meilisearch.com/).
 
 ## 🚀 Getting started
 
-For basic instructions on how to set up Meilisearch, add documents to an index, and search for documents, take a look at our [documentation](https://www.meilisearch.com/docs?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=get-started) guide.
+For basic instructions on how to set up Meilisearch, add documents to an index, and search for documents, take a look at our [Quick Start](https://docs.meilisearch.com/learn/getting_started/quick_start.html) guide.
 
-## 🌍 Supercharge your Meilisearch experience
+You may also want to check out [Meilisearch 101](https://docs.meilisearch.com/learn/getting_started/filtering_and_sorting.html) for an introduction to some of Meilisearch's most popular features.
 
-Say goodbye to server deployment and manual updates with [Meilisearch Cloud](https://www.meilisearch.com/cloud?utm_campaign=oss&utm_source=github&utm_medium=meilisearch). Additional features include analytics & monitoring in many regions around the world. No credit card is required.
+## ☁️ Meilisearch cloud
+
+Let us manage your infrastructure so you can focus on integrating a great search experience. Try [Meilisearch Cloud](https://meilisearch.com/pricing) today.
 
 ## 🧰 SDKs & integration tools
 
 Install one of our SDKs in your project for seamless integration between Meilisearch and your favorite language or framework!
 
-Take a look at the complete [Meilisearch integration list](https://www.meilisearch.com/docs/learn/what_is_meilisearch/sdks?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=sdks-link).
+Take a look at the complete [Meilisearch integration list](https://docs.meilisearch.com/learn/what_is_meilisearch/sdks.html).
 
-[![Logos belonging to different languages and frameworks supported by Meilisearch, including React, Ruby on Rails, Go, Rust, and PHP](assets/integrations.png)](https://www.meilisearch.com/docs/learn/what_is_meilisearch/sdks?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=sdks-logos)
+[![Logos belonging to different languages and frameworks supported by Meilisearch, including React, Ruby on Rails, Go, Rust, and PHP](assets/integrations.png)](https://docs.meilisearch.com/learn/what_is_meilisearch/sdks.html)
 
 ## ⚙️ Advanced usage
 
-Experienced users will want to keep our [API Reference](https://www.meilisearch.com/docs/reference/api/overview?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced) close at hand.
+Experienced users will want to keep our [API Reference](https://docs.meilisearch.com/reference/api) close at hand.
 
-We also offer a wide range of dedicated guides to all Meilisearch features, such as [filtering](https://www.meilisearch.com/docs/learn/fine_tuning_results/filtering?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced), [sorting](https://www.meilisearch.com/docs/learn/fine_tuning_results/sorting?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced), [geosearch](https://www.meilisearch.com/docs/learn/fine_tuning_results/geosearch?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced), [API keys](https://www.meilisearch.com/docs/learn/security/master_api_keys?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced), and [tenant tokens](https://www.meilisearch.com/docs/learn/security/tenant_tokens?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced).
+We also offer a wide range of dedicated guides to all Meilisearch features, such as [filtering](https://docs.meilisearch.com/learn/advanced/filtering_and_faceted_search.html), [sorting](https://docs.meilisearch.com/learn/advanced/sorting.html), [geosearch](https://docs.meilisearch.com/learn/advanced/geosearch.html), [API keys](https://docs.meilisearch.com/learn/security/master_api_keys.html), and [tenant tokens](https://docs.meilisearch.com/learn/security/tenant_tokens.html).
 
-Finally, for more in-depth information, refer to our articles explaining fundamental Meilisearch concepts such as [documents](https://www.meilisearch.com/docs/learn/core_concepts/documents?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced) and [indexes](https://www.meilisearch.com/docs/learn/core_concepts/indexes?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=advanced).
+Finally, for more in-depth information, refer to our articles explaining fundamental Meilisearch concepts such as [documents](https://docs.meilisearch.com/learn/core_concepts/documents.html) and [indexes](https://docs.meilisearch.com/learn/core_concepts/indexes.html).
 
 ## 📊 Telemetry
 
-Meilisearch collects **anonymized** user data to help us improve our product. You can [deactivate this](https://www.meilisearch.com/docs/learn/what_is_meilisearch/telemetry?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=telemetry#how-to-disable-data-collection) whenever you want.
+Meilisearch collects **anonymized** data from users to help us improve our product. You can [deactivate this](https://docs.meilisearch.com/learn/what_is_meilisearch/telemetry.html#how-to-disable-data-collection) whenever you want.
 
-To request deletion of collected data, please write to us at [privacy@meilisearch.com](mailto:privacy@meilisearch.com). Remember to include your `Instance UID` in the message, as this helps us quickly find and delete your data.
+To request deletion of collected data, please write to us at [privacy@meilisearch.com](mailto:privacy@meilisearch.com). Don't forget to include your `Instance UID` in the message, as this helps us quickly find and delete your data.
 
-If you want to know more about the kind of data we collect and what we use it for, check the [telemetry section](https://www.meilisearch.com/docs/learn/what_is_meilisearch/telemetry?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=telemetry#how-to-disable-data-collection) of our documentation.
+If you want to know more about the kind of data we collect and what we use it for, check the [telemetry section](https://docs.meilisearch.com/learn/what_is_meilisearch/telemetry.html) of our documentation.
 
 ## 📫 Get in touch!
 
-Meilisearch is a search engine created by [Meili]([https://www.welcometothejungle.com/en/companies/meilisearch](https://www.meilisearch.com/careers)), a software development company headquartered in France and with team members all over the world. Want to know more about us? [Check out our blog!](https://blog.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=contact)
+Meilisearch is a search engine created by [Meili](https://www.welcometothejungle.com/en/companies/meilisearch), a software development company based in France and with team members all over the world. Want to know more about us? [Check out our blog!](https://blog.meilisearch.com/)
 
 🗞 [Subscribe to our newsletter](https://meilisearch.us2.list-manage.com/subscribe?u=27870f7b71c908a8b359599fb&id=79582d828e) if you don't want to miss any updates! We promise we won't clutter your mailbox: we only send one edition every two months.
 
@@ -107,17 +96,18 @@ Meilisearch is a search engine created by [Meili]([https://www.welcometothejungl
 
 - For feature requests, please visit our [product repository](https://github.com/meilisearch/product/discussions)
 - Found a bug? Open an [issue](https://github.com/meilisearch/meilisearch/issues)!
-- Want to be part of our Discord community? [Join us!](https://discord.meilisearch.com/?utm_campaign=oss&utm_source=github&utm_medium=meilisearch&utm_content=contact)
+- Want to be part of our Discord community? [Join us!](https://discord.gg/meilisearch)
+- For everything else, please check [this page listing some of the other places where you can find us](https://docs.meilisearch.com/learn/what_is_meilisearch/contact.html)
 
 Thank you for your support!
 
 ## 👩‍💻 Contributing
 
-Meilisearch is, and will always be, open-source! If you want to contribute to the project, please look at [our contribution guidelines](CONTRIBUTING.md).
+Meilisearch is, and will always be, open-source! If you want to contribute to the project, please take a look at [our contribution guidelines](CONTRIBUTING.md).
 
 ## 📦 Versioning
 
-Meilisearch releases and their associated binaries are available on the project's [releases page](https://github.com/meilisearch/meilisearch/releases).
+Meilisearch releases and their associated binaries are available [in this GitHub page](https://github.com/meilisearch/meilisearch/releases).
 
 The binaries are versioned following [SemVer conventions](https://semver.org/). To know more, read our [versioning policy](https://github.com/meilisearch/engine-team/blob/main/resources/versioning-policy.md).
 

assets/prometheus-basic-scraper.yml

@@ -1,19 +0,0 @@
-global:
-  scrape_interval:     15s # By default, scrape targets every 15 seconds.
-
-  # Attach these labels to any time series or alerts when communicating with
-  # external systems (federation, remote storage, Alertmanager).
-  external_labels:
-    monitor: 'codelab-monitor'
-
-# A scrape configuration containing exactly one endpoint to scrape:
-# Here it's Prometheus itself.
-scrape_configs:
-  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
-  - job_name: 'meilisearch'
-
-    # Override the global default and scrape targets from this job every 5 seconds.
-    scrape_interval: 5s
-
-    static_configs:
-      - targets: ['localhost:7700']

benchmarks/Cargo.toml

@@ -11,30 +11,27 @@ edition.workspace = true
 license.workspace = true
 
 [dependencies]
-anyhow = "1.0.95"
-bumpalo = "3.16.0"
-csv = "1.3.1"
-memmap2 = "0.9.5"
-milli = { path = "../milli" }
-mimalloc = { version = "0.1.43", default-features = false }
-serde_json = { version = "1.0.135", features = ["preserve_order"] }
-tempfile = "3.15.0"
+anyhow = "1.0.65"
+csv = "1.1.6"
+milli = { path = "../milli", default-features = false }
+mimalloc = { version = "0.1.29", default-features = false }
+serde_json = { version = "1.0.85", features = ["preserve_order"] }
 
 [dev-dependencies]
-criterion = { version = "0.5.1", features = ["html_reports"] }
+criterion = { version = "0.4.0", features = ["html_reports"] }
 rand = "0.8.5"
 rand_chacha = "0.3.1"
-roaring = "0.10.10"
+roaring = "0.10.1"
 
 [build-dependencies]
-anyhow = "1.0.95"
-bytes = "1.9.0"
+anyhow = "1.0.65"
+bytes = "1.2.1"
 convert_case = "0.6.0"
-flate2 = "1.0.35"
-reqwest = { version = "0.12.12", features = ["blocking", "rustls-tls"], default-features = false }
+flate2 = "1.0.24"
+reqwest = { version = "0.11.12", features = ["blocking", "rustls-tls"], default-features = false }
 
 [features]
-default = ["milli/all-tokenizations"]
+default = ["milli/default"]
 
 [[bench]]
 name = "search_songs"
@@ -51,3 +48,7 @@ harness = false
 [[bench]]
 name = "indexing"
 harness = false
+
+[[bench]]
+name = "formatting"
+harness = false

benchmarks/README.md

@@ -119,9 +119,9 @@ _[Download the `smol-wiki` dataset](https://milli-benchmarks.fra1.digitaloceansp
 
 ### Movies
 
-`movies` is a really small dataset we uses as our example in the [getting started](https://www.meilisearch.com/docs/learn/getting_started/quick_start)
+`movies` is a really small dataset we uses as our example in the [getting started](https://docs.meilisearch.com/learn/getting_started/)
 
-_[Download the `movies` dataset](https://www.meilisearch.com/movies.json)._
+_[Download the `movies` dataset](https://docs.meilisearch.com/movies.json)._
 
 
 ### All Countries

benchmarks/benches/formatting.rs

@@ -0,0 +1,67 @@
+use std::rc::Rc;
+
+use criterion::{criterion_group, criterion_main};
+use milli::tokenizer::TokenizerBuilder;
+use milli::{FormatOptions, MatcherBuilder, MatchingWord, MatchingWords};
+
+#[global_allocator]
+static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
+
+struct Conf<'a> {
+    name: &'a str,
+    text: &'a str,
+    matching_words: MatcherBuilder<'a, Vec<u8>>,
+}
+
+fn bench_formatting(c: &mut criterion::Criterion) {
+    #[rustfmt::skip]
+    let confs = &[
+    	Conf {
+    		name: "'the door d'",
+			text: r#"He used to do the door sounds in "Star Trek" with his mouth, phssst, phssst. The MD-11 passenger and cargo doors also tend to behave like electromagnetic apertures, because the doors do not have continuous electrical contact with the door frames around the door perimeter. But Theodor said that the doors don't work."#,
+			matching_words: MatcherBuilder::new(MatchingWords::new(vec![
+	            (vec![Rc::new(MatchingWord::new("t".to_string(), 0, false).unwrap()), Rc::new(MatchingWord::new("he".to_string(), 0, false).unwrap())], vec![0]),
+	            (vec![Rc::new(MatchingWord::new("the".to_string(), 0, false).unwrap())], vec![0]),
+	            (vec![Rc::new(MatchingWord::new("door".to_string(), 1, false).unwrap())], vec![1]),
+	            (vec![Rc::new(MatchingWord::new("do".to_string(), 0, false).unwrap()), Rc::new(MatchingWord::new("or".to_string(), 0, false).unwrap())], vec![0]),
+	            (vec![Rc::new(MatchingWord::new("thedoor".to_string(), 1, false).unwrap())], vec![0, 1]),
+	            (vec![Rc::new(MatchingWord::new("d".to_string(), 0, true).unwrap())], vec![2]),
+	            (vec![Rc::new(MatchingWord::new("thedoord".to_string(), 1, true).unwrap())], vec![0, 1, 2]),
+	            (vec![Rc::new(MatchingWord::new("doord".to_string(), 1, true).unwrap())], vec![1, 2]),
+        	]
+            ), TokenizerBuilder::default().build()),
+		},
+    ];
+
+    let format_options = &[
+        FormatOptions { highlight: false, crop: None },
+        FormatOptions { highlight: true, crop: None },
+        FormatOptions { highlight: false, crop: Some(10) },
+        FormatOptions { highlight: true, crop: Some(10) },
+        FormatOptions { highlight: false, crop: Some(20) },
+        FormatOptions { highlight: true, crop: Some(20) },
+    ];
+
+    for option in format_options {
+        let highlight = if option.highlight { "highlight" } else { "no-highlight" };
+
+        let name = match option.crop {
+            Some(size) => format!("{}-crop({})", highlight, size),
+            None => format!("{}-no-crop", highlight),
+        };
+
+        let mut group = c.benchmark_group(&name);
+        for conf in confs {
+            group.bench_function(conf.name, |b| {
+                b.iter(|| {
+                    let mut matcher = conf.matching_words.build(conf.text);
+                    matcher.format(*option);
+                })
+            });
+        }
+        group.finish();
+    }
+}
+
+criterion_group!(benches, bench_formatting);
+criterion_main!(benches);

benchmarks/benches/search_geo.rs

@@ -2,10 +2,9 @@ mod datasets_paths;
 mod utils;
 
 use criterion::{criterion_group, criterion_main};
-use milli::{update::Settings, FilterableAttributesRule};
+use milli::update::Settings;
 use utils::Conf;
 
-#[cfg(not(windows))]
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
@@ -21,10 +20,8 @@ fn base_conf(builder: &mut Settings) {
         ["name", "alternatenames", "elevation"].iter().map(|s| s.to_string()).collect();
     builder.set_searchable_fields(searchable_fields);
 
-    let filterable_fields = ["_geo", "population", "elevation"]
-        .iter()
-        .map(|s| FilterableAttributesRule::Field(s.to_string()))
-        .collect();
+    let filterable_fields =
+        ["_geo", "population", "elevation"].iter().map(|s| s.to_string()).collect();
     builder.set_filterable_fields(filterable_fields);
 
     let sortable_fields =

benchmarks/benches/search_songs.rs

@@ -2,10 +2,9 @@ mod datasets_paths;
 mod utils;
 
 use criterion::{criterion_group, criterion_main};
-use milli::{update::Settings, FilterableAttributesRule};
+use milli::update::Settings;
 use utils::Conf;
 
-#[cfg(not(windows))]
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
@@ -22,7 +21,7 @@ fn base_conf(builder: &mut Settings) {
 
     let faceted_fields = ["released-timestamp", "duration-float", "genre", "country", "artist"]
         .iter()
-        .map(|s| FilterableAttributesRule::Field(s.to_string()))
+        .map(|s| s.to_string())
         .collect();
     builder.set_filterable_fields(faceted_fields);
 }

benchmarks/benches/search_wiki.rs

@@ -5,7 +5,6 @@ use criterion::{criterion_group, criterion_main};
 use milli::update::Settings;
 use utils::Conf;
 
-#[cfg(not(windows))]
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 

benchmarks/benches/utils.rs

@@ -1,19 +1,17 @@
 #![allow(dead_code)]
 
 use std::fs::{create_dir_all, remove_dir_all, File};
-use std::io::{self, BufReader, BufWriter, Read};
+use std::io::{self, BufRead, BufReader, Cursor, Read, Seek};
+use std::num::ParseFloatError;
 use std::path::Path;
-use std::str::FromStr as _;
+use std::str::FromStr;
 
-use anyhow::Context;
-use bumpalo::Bump;
 use criterion::BenchmarkId;
-use memmap2::Mmap;
+use milli::documents::{DocumentsBatchBuilder, DocumentsBatchReader};
 use milli::heed::EnvOpenOptions;
-use milli::progress::Progress;
-use milli::update::new::indexer;
-use milli::update::{IndexerConfig, Settings};
-use milli::vector::EmbeddingConfigs;
+use milli::update::{
+    IndexDocuments, IndexDocumentsConfig, IndexDocumentsMethod, IndexerConfig, Settings,
+};
 use milli::{Criterion, Filter, Index, Object, TermsMatchingStrategy};
 use serde_json::Value;
 
@@ -65,11 +63,10 @@ pub fn base_setup(conf: &Conf) -> Index {
     }
     create_dir_all(conf.database_name).unwrap();
 
-    let options = EnvOpenOptions::new();
-    let mut options = options.read_txn_without_tls();
+    let mut options = EnvOpenOptions::new();
     options.map_size(100 * 1024 * 1024 * 1024); // 100 GB
-    options.max_readers(100);
-    let index = Index::new(options, conf.database_name, true).unwrap();
+    options.max_readers(10);
+    let index = Index::new(options, conf.database_name).unwrap();
 
     let config = IndexerConfig::default();
     let mut wtxn = index.write_txn().unwrap();
@@ -95,44 +92,18 @@ pub fn base_setup(conf: &Conf) -> Index {
 
     let config = IndexerConfig::default();
     let mut wtxn = index.write_txn().unwrap();
-    let rtxn = index.read_txn().unwrap();
-    let db_fields_ids_map = index.fields_ids_map(&rtxn).unwrap();
-    let mut new_fields_ids_map = db_fields_ids_map.clone();
-
+    let indexing_config = IndexDocumentsConfig {
+        autogenerate_docids: conf.primary_key.is_none(),
+        update_method: IndexDocumentsMethod::ReplaceDocuments,
+        ..Default::default()
+    };
+    let builder =
+        IndexDocuments::new(&mut wtxn, &index, &config, indexing_config, |_| (), || false).unwrap();
     let documents = documents_from(conf.dataset, conf.dataset_format);
-    let mut indexer = indexer::DocumentOperation::new();
-    indexer.replace_documents(&documents).unwrap();
-
-    let indexer_alloc = Bump::new();
-    let (document_changes, _operation_stats, primary_key) = indexer
-        .into_changes(
-            &indexer_alloc,
-            &index,
-            &rtxn,
-            None,
-            &mut new_fields_ids_map,
-            &|| false,
-            Progress::default(),
-        )
-        .unwrap();
-
-    indexer::index(
-        &mut wtxn,
-        &index,
-        &milli::ThreadPoolNoAbortBuilder::new().build().unwrap(),
-        config.grenad_parameters(),
-        &db_fields_ids_map,
-        new_fields_ids_map,
-        primary_key,
-        &document_changes,
-        EmbeddingConfigs::default(),
-        &|| false,
-        &Progress::default(),
-    )
-    .unwrap();
-
+    let (builder, user_error) = builder.add_documents(documents).unwrap();
+    user_error.unwrap();
+    builder.execute().unwrap();
     wtxn.commit().unwrap();
-    drop(rtxn);
 
     index
 }
@@ -169,96 +140,49 @@ pub fn run_benches(c: &mut criterion::Criterion, confs: &[Conf]) {
     }
 }
 
-pub fn documents_from(filename: &str, filetype: &str) -> Mmap {
-    let file = File::open(filename)
-        .unwrap_or_else(|_| panic!("could not find the dataset in: {filename}"));
-    match filetype {
-        "csv" => documents_from_csv(file).unwrap(),
-        "json" => documents_from_json(file).unwrap(),
-        "jsonl" => documents_from_jsonl(file).unwrap(),
-        otherwise => panic!("invalid update format {otherwise:?}"),
-    }
+pub fn documents_from(filename: &str, filetype: &str) -> DocumentsBatchReader<impl BufRead + Seek> {
+    let reader = File::open(filename)
+        .unwrap_or_else(|_| panic!("could not find the dataset in: {}", filename));
+    let reader = BufReader::new(reader);
+    let documents = match filetype {
+        "csv" => documents_from_csv(reader).unwrap(),
+        "json" => documents_from_json(reader).unwrap(),
+        "jsonl" => documents_from_jsonl(reader).unwrap(),
+        otherwise => panic!("invalid update format {:?}", otherwise),
+    };
+    DocumentsBatchReader::from_reader(Cursor::new(documents)).unwrap()
 }
 
-fn documents_from_jsonl(file: File) -> anyhow::Result<Mmap> {
-    unsafe { Mmap::map(&file).map_err(Into::into) }
-}
+fn documents_from_jsonl(reader: impl BufRead) -> anyhow::Result<Vec<u8>> {
+    let mut documents = DocumentsBatchBuilder::new(Vec::new());
 
-fn documents_from_json(file: File) -> anyhow::Result<Mmap> {
-    let reader = BufReader::new(file);
-    let documents: Vec<milli::Object> = serde_json::from_reader(reader)?;
-    let mut output = tempfile::tempfile().map(BufWriter::new)?;
-
-    for document in documents {
-        serde_json::to_writer(&mut output, &document)?;
+    for result in serde_json::Deserializer::from_reader(reader).into_iter::<Object>() {
+        let object = result?;
+        documents.append_json_object(&object)?;
     }
 
-    let file = output.into_inner()?;
-    unsafe { Mmap::map(&file).map_err(Into::into) }
+    documents.into_inner().map_err(Into::into)
 }
 
-fn documents_from_csv(file: File) -> anyhow::Result<Mmap> {
-    let output = tempfile::tempfile()?;
-    let mut output = BufWriter::new(output);
-    let mut reader = csv::ReaderBuilder::new().from_reader(file);
+fn documents_from_json(reader: impl BufRead) -> anyhow::Result<Vec<u8>> {
+    let mut documents = DocumentsBatchBuilder::new(Vec::new());
 
-    let headers = reader.headers().context("while retrieving headers")?.clone();
-    let typed_fields: Vec<_> = headers.iter().map(parse_csv_header).collect();
-    let mut object: serde_json::Map<_, _> =
-        typed_fields.iter().map(|(k, _)| (k.to_string(), Value::Null)).collect();
+    documents.append_json_array(reader)?;
 
-    let mut line = 0;
-    let mut record = csv::StringRecord::new();
-    while reader.read_record(&mut record).context("while reading a record")? {
-        // We increment here and not at the end of the loop
-        // to take the header offset into account.
-        line += 1;
+    documents.into_inner().map_err(Into::into)
+}
 
-        // Reset the document values
-        object.iter_mut().for_each(|(_, v)| *v = Value::Null);
+fn documents_from_csv(reader: impl BufRead) -> anyhow::Result<Vec<u8>> {
+    let csv = csv::Reader::from_reader(reader);
 
-        for (i, (name, atype)) in typed_fields.iter().enumerate() {
-            let value = &record[i];
-            let trimmed_value = value.trim();
-            let value = match atype {
-                AllowedType::Number if trimmed_value.is_empty() => Value::Null,
-                AllowedType::Number => {
-                    match trimmed_value.parse::<i64>() {
-                        Ok(integer) => Value::from(integer),
-                        Err(_) => match trimmed_value.parse::<f64>() {
-                            Ok(float) => Value::from(float),
-                            Err(error) => {
-                                anyhow::bail!("document format error on line {line}: {error}. For value: {value}")
-                            }
-                        },
-                    }
-                }
-                AllowedType::Boolean if trimmed_value.is_empty() => Value::Null,
-                AllowedType::Boolean => match trimmed_value.parse::<bool>() {
-                    Ok(bool) => Value::from(bool),
-                    Err(error) => {
-                        anyhow::bail!(
-                            "document format error on line {line}: {error}. For value: {value}"
-                        )
-                    }
-                },
-                AllowedType::String if value.is_empty() => Value::Null,
-                AllowedType::String => Value::from(value),
-            };
+    let mut documents = DocumentsBatchBuilder::new(Vec::new());
+    documents.append_csv(csv)?;
 
-            *object.get_mut(name).expect("encountered an unknown field") = value;
-        }
-
-        serde_json::to_writer(&mut output, &object).context("while writing to disk")?;
-    }
-
-    let output = output.into_inner()?;
-    unsafe { Mmap::map(&output).map_err(Into::into) }
+    documents.into_inner().map_err(Into::into)
 }
 
 enum AllowedType {
     String,
-    Boolean,
     Number,
 }
 
@@ -267,9 +191,8 @@ fn parse_csv_header(header: &str) -> (String, AllowedType) {
     match header.rsplit_once(':') {
         Some((field_name, field_type)) => match field_type {
             "string" => (field_name.to_string(), AllowedType::String),
-            "boolean" => (field_name.to_string(), AllowedType::Boolean),
             "number" => (field_name.to_string(), AllowedType::Number),
-            // if the pattern isn't recognized, we keep the whole field.
+            // we may return an error in this case.
             _otherwise => (header.to_string(), AllowedType::String),
         },
         None => (header.to_string(), AllowedType::String),
@@ -307,13 +230,10 @@ impl<R: Read> Iterator for CSVDocumentDeserializer<R> {
                 for ((field_name, field_type), value) in
                     self.headers.iter().zip(csv_document.into_iter())
                 {
-                    let parsed_value: anyhow::Result<Value> = match field_type {
+                    let parsed_value: Result<Value, ParseFloatError> = match field_type {
                         AllowedType::Number => {
                             value.parse::<f64>().map(Value::from).map_err(Into::into)
                         }
-                        AllowedType::Boolean => {
-                            value.parse::<bool>().map(Value::from).map_err(Into::into)
-                        }
                         AllowedType::String => Ok(Value::String(value.to_string())),
                     };
 

bors.toml

@@ -0,0 +1,11 @@
+status = [
+    'Tests on ubuntu-18.04',
+    'Tests on macos-12',
+    'Tests on windows-2022',
+    'Run Clippy',
+    'Run Rustfmt',
+    'Run tests in debug',
+]
+pr_status = ['Milestone Check']
+# 3 hours timeout
+timeout-sec = 10800

config.toml

@@ -1,134 +1,120 @@
 # This file shows the default configuration of Meilisearch.
-# All variables are defined here: https://www.meilisearch.com/docs/learn/configuration/instance_options#environment-variables
+# All variables are defined here: https://docs.meilisearch.com/learn/configuration/instance_options.html#environment-variables
 
-# Designates the location where database files will be created and retrieved.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#database-path
 db_path = "./data.ms"
+# Designates the location where database files will be created and retrieved.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#database-path
 
-# Configures the instance's environment. Value must be either `production` or `development`.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#environment
 env = "development"
+# Configures the instance's environment. Value must be either `production` or `development`.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#environment
 
-# The address on which the HTTP server will listen.
 http_addr = "localhost:7700"
+# The address on which the HTTP server will listen.
 
-# Sets the instance's master key, automatically protecting all routes except GET /health.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#master-key
 # master_key = "YOUR_MASTER_KEY_VALUE"
+# Sets the instance's master key, automatically protecting all routes except GET /health.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#master-key
 
+# no_analytics = true
 # Deactivates Meilisearch's built-in telemetry when provided.
 # Meilisearch automatically collects data from all instances that do not opt out using this flag.
 # All gathered data is used solely for the purpose of improving Meilisearch, and can be deleted at any time.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#disable-analytics
-# no_analytics = true
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#disable-analytics
 
-# Sets the maximum size of accepted payloads.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#payload-limit-size
 http_payload_size_limit = "100 MB"
+# Sets the maximum size of accepted payloads.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#payload-limit-size
 
+log_level = "INFO"
 # Defines how much detail should be present in Meilisearch's logs.
 # Meilisearch currently supports six log levels, listed in order of increasing verbosity:  `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#log-level
-log_level = "INFO"
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#log-level
 
-# Sets the maximum amount of RAM Meilisearch can use when indexing.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#max-indexing-memory
 # max_indexing_memory = "2 GiB"
+# Sets the maximum amount of RAM Meilisearch can use when indexing.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#max-indexing-memory
 
-# Sets the maximum number of threads Meilisearch can use during indexing.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#max-indexing-threads
 # max_indexing_threads = 4
+# Sets the maximum number of threads Meilisearch can use during indexing.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#max-indexing-threads
 
 #############
 ### DUMPS ###
 #############
 
-# Sets the directory where Meilisearch will create dump files.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#dump-directory
 dump_dir = "dumps/"
+# Sets the directory where Meilisearch will create dump files.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#dump-directory
 
-# Imports the dump file located at the specified path. Path must point to a .dump file.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#import-dump
 # import_dump = "./path/to/my/file.dump"
+# Imports the dump file located at the specified path. Path must point to a .dump file.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#import-dump
 
-# Prevents Meilisearch from throwing an error when `import_dump` does not point to a valid dump file.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ignore-missing-dump
 ignore_missing_dump = false
+# Prevents Meilisearch from throwing an error when `import_dump` does not point to a valid dump file.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ignore-missing-dump
 
-# Prevents a Meilisearch instance with an existing database from throwing an error when using `import_dump`.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ignore-dump-if-db-exists
 ignore_dump_if_db_exists = false
+# Prevents a Meilisearch instance with an existing database from throwing an error when using `import_dump`.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ignore-dump-if-db-exists
 
 
 #################
 ### SNAPSHOTS ###
 #################
 
+schedule_snapshot = false
 # Enables scheduled snapshots when true, disable when false (the default).
 # If the value is given as an integer, then enables the scheduled snapshot with the passed value as the interval
 # between each snapshot, in seconds.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#schedule-snapshot-creation
-schedule_snapshot = false
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#schedule-snapshot-creation
 
-# Sets the directory where Meilisearch will store snapshots.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#snapshot-destination
 snapshot_dir = "snapshots/"
+# Sets the directory where Meilisearch will store snapshots.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#snapshot-destination
 
-# Launches Meilisearch after importing a previously-generated snapshot at the given filepath.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#import-snapshot
 # import_snapshot = "./path/to/my/snapshot"
+# Launches Meilisearch after importing a previously-generated snapshot at the given filepath.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#import-snapshot
 
-# Prevents a Meilisearch instance from throwing an error when `import_snapshot` does not point to a valid snapshot file.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ignore-missing-snapshot
 ignore_missing_snapshot = false
+# Prevents a Meilisearch instance from throwing an error when `import_snapshot` does not point to a valid snapshot file.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ignore-missing-snapshot
 
-# Prevents a Meilisearch instance with an existing database from throwing an error when using `import_snapshot`.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ignore-snapshot-if-db-exists
 ignore_snapshot_if_db_exists = false
+# Prevents a Meilisearch instance with an existing database from throwing an error when using `import_snapshot`.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ignore-snapshot-if-db-exists
 
 
 ###########
 ### SSL ###
 ###########
 
-# Enables client authentication in the specified path.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-authentication-path
 # ssl_auth_path = "./path/to/root"
+# Enables client authentication in the specified path.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-authentication-path
 
-# Sets the server's SSL certificates.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-certificates-path
 # ssl_cert_path = "./path/to/certfile"
+# Sets the server's SSL certificates.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-certificates-path
 
-# Sets the server's SSL key files.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-key-path
 # ssl_key_path = "./path/to/private-key"
+# Sets the server's SSL key files.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-key-path
 
-# Sets the server's OCSP file.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-ocsp-path
 # ssl_ocsp_path = "./path/to/ocsp-file"
+# Sets the server's OCSP file.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-ocsp-path
 
-# Makes SSL authentication mandatory.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-require-auth
 ssl_require_auth = false
+# Makes SSL authentication mandatory.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-require-auth
 
-# Activates SSL session resumption.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-resumption
 ssl_resumption = false
+# Activates SSL session resumption.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-resumption
 
-# Activates SSL tickets.
-# https://www.meilisearch.com/docs/learn/configuration/instance_options#ssl-tickets
 ssl_tickets = false
-
-#############################
-### Experimental features ###
-#############################
-
-# Experimental metrics feature. For more information, see: <https://github.com/meilisearch/meilisearch/discussions/3518>
-# Enables the Prometheus metrics on the `GET /metrics` endpoint.
-experimental_enable_metrics = false
-
-# Experimental RAM reduction during indexing, do not use in production, see: <https://github.com/meilisearch/product/discussions/652>
-experimental_reduce_indexing_memory_usage = false
-
-# Experimentally reduces the maximum number of tasks that will be processed at once, see: <https://github.com/orgs/meilisearch/discussions/713>
-# experimental_max_number_of_batched_tasks = 100
+# Activates SSL tickets.
+# https://docs.meilisearch.com/learn/configuration/instance_options.html#ssl-tickets

crates/build-info/Cargo.toml

@@ -1,18 +0,0 @@
-[package]
-name = "build-info"
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-homepage.workspace = true
-readme.workspace = true
-edition.workspace = true
-license.workspace = true
-
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
-
-[dependencies]
-time = { version = "0.3.37", features = ["parsing"] }
-
-[build-dependencies]
-anyhow = "1.0.95"
-vergen-git2 = "1.0.2"

crates/build-info/build.rs

@@ -1,29 +0,0 @@
-fn main() {
-    if let Err(err) = emit_git_variables() {
-        println!("cargo:warning=vergen: {}", err);
-    }
-}
-
-fn emit_git_variables() -> anyhow::Result<()> {
-    println!("cargo::rerun-if-env-changed=MEILI_NO_VERGEN");
-
-    let has_vergen =
-        !matches!(std::env::var_os("MEILI_NO_VERGEN"), Some(x) if x != "false" && x != "0");
-
-    anyhow::ensure!(has_vergen, "disabled via `MEILI_NO_VERGEN`");
-
-    // Note: any code that needs VERGEN_ environment variables should take care to define them manually in the Dockerfile and pass them
-    // in the corresponding GitHub workflow (publish_docker.yml).
-    // This is due to the Dockerfile building the binary outside of the git directory.
-    let mut builder = vergen_git2::Git2Builder::default();
-
-    builder.branch(true);
-    builder.commit_timestamp(true);
-    builder.commit_message(true);
-    builder.describe(true, true, None);
-    builder.sha(false);
-
-    let git2 = builder.build()?;
-
-    vergen_git2::Emitter::default().fail_on_error().add_instructions(&git2)?.emit()
-}

crates/build-info/src/lib.rs

@@ -1,203 +0,0 @@
-use time::format_description::well_known::Iso8601;
-
-#[derive(Debug, Clone)]
-pub struct BuildInfo {
-    pub branch: Option<&'static str>,
-    pub describe: Option<DescribeResult>,
-    pub commit_sha1: Option<&'static str>,
-    pub commit_msg: Option<&'static str>,
-    pub commit_timestamp: Option<time::OffsetDateTime>,
-}
-
-impl BuildInfo {
-    pub fn from_build() -> Self {
-        let branch: Option<&'static str> = option_env!("VERGEN_GIT_BRANCH");
-        let describe = DescribeResult::from_build();
-        let commit_sha1 = option_env!("VERGEN_GIT_SHA");
-        let commit_msg = option_env!("VERGEN_GIT_COMMIT_MESSAGE");
-        let commit_timestamp = option_env!("VERGEN_GIT_COMMIT_TIMESTAMP");
-
-        let commit_timestamp = commit_timestamp.and_then(|commit_timestamp| {
-            time::OffsetDateTime::parse(commit_timestamp, &Iso8601::DEFAULT).ok()
-        });
-
-        Self { branch, describe, commit_sha1, commit_msg, commit_timestamp }
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
-pub enum DescribeResult {
-    Prototype { name: &'static str },
-    Release { version: &'static str, major: u64, minor: u64, patch: u64 },
-    Prerelease { version: &'static str, major: u64, minor: u64, patch: u64, rc: u64 },
-    NotATag { describe: &'static str },
-}
-
-impl DescribeResult {
-    pub fn new(describe: &'static str) -> Self {
-        if let Some(name) = prototype_name(describe) {
-            Self::Prototype { name }
-        } else if let Some(release) = release_version(describe) {
-            release
-        } else if let Some(prerelease) = prerelease_version(describe) {
-            prerelease
-        } else {
-            Self::NotATag { describe }
-        }
-    }
-
-    pub fn from_build() -> Option<Self> {
-        let describe: &'static str = option_env!("VERGEN_GIT_DESCRIBE")?;
-        Some(Self::new(describe))
-    }
-
-    pub fn as_tag(&self) -> Option<&'static str> {
-        match self {
-            DescribeResult::Prototype { name } => Some(name),
-            DescribeResult::Release { version, .. } => Some(version),
-            DescribeResult::Prerelease { version, .. } => Some(version),
-            DescribeResult::NotATag { describe: _ } => None,
-        }
-    }
-
-    pub fn as_prototype(&self) -> Option<&'static str> {
-        match self {
-            DescribeResult::Prototype { name } => Some(name),
-            DescribeResult::Release { .. }
-            | DescribeResult::Prerelease { .. }
-            | DescribeResult::NotATag { .. } => None,
-        }
-    }
-}
-
-/// Parses the input as a prototype name.
-///
-/// Returns `Some(prototype_name)` if the following conditions are met on this value:
-///
-/// 1. starts with `prototype-`,
-/// 2. ends with `-<some_number>`,
-/// 3. does not end with `<some_number>-<some_number>`.
-///
-/// Otherwise, returns `None`.
-fn prototype_name(describe: &'static str) -> Option<&'static str> {
-    if !describe.starts_with("prototype-") {
-        return None;
-    }
-
-    let mut rsplit_prototype = describe.rsplit('-');
-    // last component MUST be a number
-    rsplit_prototype.next()?.parse::<u64>().ok()?;
-    // before than last component SHALL NOT be a number
-    rsplit_prototype.next()?.parse::<u64>().err()?;
-
-    Some(describe)
-}
-
-fn release_version(describe: &'static str) -> Option<DescribeResult> {
-    if !describe.starts_with('v') {
-        return None;
-    }
-
-    // full release version don't contain a `-`
-    if describe.contains('-') {
-        return None;
-    }
-
-    // full release version parse as vX.Y.Z, with X, Y, Z numbers.
-    let mut dots = describe[1..].split('.');
-    let major: u64 = dots.next()?.parse().ok()?;
-    let minor: u64 = dots.next()?.parse().ok()?;
-    let patch: u64 = dots.next()?.parse().ok()?;
-
-    if dots.next().is_some() {
-        return None;
-    }
-
-    Some(DescribeResult::Release { version: describe, major, minor, patch })
-}
-
-fn prerelease_version(describe: &'static str) -> Option<DescribeResult> {
-    // prerelease version is in the shape vM.N.P-rc.C
-    let mut hyphen = describe.rsplit('-');
-    let prerelease = hyphen.next()?;
-    if !prerelease.starts_with("rc.") {
-        return None;
-    }
-
-    let rc: u64 = prerelease[3..].parse().ok()?;
-
-    let release = hyphen.next()?;
-
-    let DescribeResult::Release { version: _, major, minor, patch } = release_version(release)?
-    else {
-        return None;
-    };
-
-    Some(DescribeResult::Prerelease { version: describe, major, minor, patch, rc })
-}
-
-#[cfg(test)]
-mod test {
-    use super::DescribeResult;
-
-    fn assert_not_a_tag(describe: &'static str) {
-        assert_eq!(DescribeResult::NotATag { describe }, DescribeResult::new(describe))
-    }
-
-    fn assert_proto(describe: &'static str) {
-        assert_eq!(DescribeResult::Prototype { name: describe }, DescribeResult::new(describe))
-    }
-
-    fn assert_release(describe: &'static str, major: u64, minor: u64, patch: u64) {
-        assert_eq!(
-            DescribeResult::Release { version: describe, major, minor, patch },
-            DescribeResult::new(describe)
-        )
-    }
-
-    fn assert_prerelease(describe: &'static str, major: u64, minor: u64, patch: u64, rc: u64) {
-        assert_eq!(
-            DescribeResult::Prerelease { version: describe, major, minor, patch, rc },
-            DescribeResult::new(describe)
-        )
-    }
-
-    #[test]
-    fn not_a_tag() {
-        assert_not_a_tag("whatever-fuzzy");
-        assert_not_a_tag("whatever-fuzzy-5-ggg-dirty");
-        assert_not_a_tag("whatever-fuzzy-120-ggg-dirty");
-
-        // technically a tag, but not a proto nor a version, so not parsed as a tag
-        assert_not_a_tag("whatever");
-
-        // dirty version
-        assert_not_a_tag("v1.7.0-1-ggga-dirty");
-        assert_not_a_tag("v1.7.0-rc.1-1-ggga-dirty");
-
-        // after version
-        assert_not_a_tag("v1.7.0-1-ggga");
-        assert_not_a_tag("v1.7.0-rc.1-1-ggga");
-
-        // after proto
-        assert_not_a_tag("protoype-tag-0-1-ggga");
-        assert_not_a_tag("protoype-tag-0-1-ggga-dirty");
-    }
-
-    #[test]
-    fn prototype() {
-        assert_proto("prototype-tag-0");
-        assert_proto("prototype-tag-10");
-        assert_proto("prototype-long-name-tag-10");
-    }
-
-    #[test]
-    fn release() {
-        assert_release("v1.7.2", 1, 7, 2);
-    }
-
-    #[test]
-    fn prerelease() {
-        assert_prerelease("v1.7.2-rc.3", 1, 7, 2, 3);
-    }
-}

crates/dump/Cargo.toml

@@ -1,34 +0,0 @@
-[package]
-name = "dump"
-publish = false
-
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-edition.workspace = true
-homepage.workspace = true
-readme.workspace = true
-license.workspace = true
-
-[dependencies]
-anyhow = "1.0.95"
-flate2 = "1.0.35"
-http = "1.2.0"
-meilisearch-types = { path = "../meilisearch-types" }
-once_cell = "1.20.2"
-regex = "1.11.1"
-roaring = { version = "0.10.10", features = ["serde"] }
-serde = { version = "1.0.217", features = ["derive"] }
-serde_json = { version = "1.0.135", features = ["preserve_order"] }
-tar = "0.4.43"
-tempfile = "3.15.0"
-thiserror = "2.0.9"
-time = { version = "0.3.37", features = ["serde-well-known", "formatting", "parsing", "macros"] }
-tracing = "0.1.41"
-uuid = { version = "1.11.0", features = ["serde", "v4"] }
-
-[dev-dependencies]
-big_s = "1.0.2"
-maplit = "1.0.2"
-meili-snap = { path = "../meili-snap" }
-meilisearch-types = { path = "../meilisearch-types" }

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-5.snap

@@ -1,57 +0,0 @@
----
-source: crates/dump/src/reader/mod.rs
-expression: vector_index.settings().unwrap()
----
-{
-  "displayedAttributes": [
-    "*"
-  ],
-  "searchableAttributes": [
-    "*"
-  ],
-  "filterableAttributes": [],
-  "sortableAttributes": [],
-  "rankingRules": [
-    "words",
-    "typo",
-    "proximity",
-    "attribute",
-    "sort",
-    "exactness"
-  ],
-  "stopWords": [],
-  "nonSeparatorTokens": [],
-  "separatorTokens": [],
-  "dictionary": [],
-  "synonyms": {},
-  "distinctAttribute": null,
-  "proximityPrecision": "byWord",
-  "typoTolerance": {
-    "enabled": true,
-    "minWordSizeForTypos": {
-      "oneTypo": 5,
-      "twoTypos": 9
-    },
-    "disableOnWords": [],
-    "disableOnAttributes": []
-  },
-  "faceting": {
-    "maxValuesPerFacet": 100,
-    "sortFacetValuesBy": {
-      "*": "alpha"
-    }
-  },
-  "pagination": {
-    "maxTotalHits": 1000
-  },
-  "embedders": {
-    "default": {
-      "source": "huggingFace",
-      "model": "BAAI/bge-base-en-v1.5",
-      "revision": "617ca489d9e86b49b8167676d8220688b99db36e",
-      "pooling": "forceMean",
-      "documentTemplate": "{% for field in fields %} {{ field.name }}: {{ field.value }}\n{% endfor %}"
-    }
-  },
-  "searchCutoffMs": null
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-6.snap

@@ -1,783 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e3",
-  "desc": "overriden vector + map",
-  "_vectors": {
-    "default": [
-      0.2,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1
-    ],
-    "toto": [
-      0.1
-    ]
-  }
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-7.snap

@@ -1,786 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e2",
-  "desc": "natural vector + map",
-  "_vectors": {
-    "toto": [],
-    "default": {
-      "embeddings": [
-        [
-          -0.05189208313822746,
-          -0.9273212552070618,
-          0.1443813145160675,
-          0.0932632014155388,
-          0.2665371894836426,
-          0.36266782879829407,
-          0.6402910947799683,
-          0.32014018297195435,
-          0.030915971845388412,
-          -0.9312191605567932,
-          -0.3718109726905823,
-          -0.2700554132461548,
-          -1.1014580726623535,
-          0.9154956936836244,
-          -0.3406888246536255,
-          1.0077725648880005,
-          0.6577560901641846,
-          -0.3955195546150207,
-          -0.4148270785808563,
-          0.1855088472366333,
-          0.5062315464019775,
-          -0.3632686734199524,
-          -0.2277890294790268,
-          0.2560805082321167,
-          -0.3853609561920166,
-          -0.1604762226343155,
-          -0.13947471976280212,
-          -0.20147813856601715,
-          -0.4466346800327301,
-          -0.3761846721172333,
-          0.1443382054567337,
-          0.18205296993255615,
-          0.49359792470932007,
-          -0.22538000345230105,
-          -0.4996317625045776,
-          -0.22734887897968292,
-          -0.6034309267997742,
-          -0.7857939600944519,
-          -0.34923747181892395,
-          -0.3466345965862274,
-          0.21176661550998688,
-          -0.5101462006568909,
-          -0.3403083384037018,
-          0.000315118464641273,
-          0.236465722322464,
-          -0.10246097296476364,
-          -1.3013339042663574,
-          0.3419138789176941,
-          -0.32963496446609497,
-          -0.0901619717478752,
-          -0.5426247119903564,
-          0.22656650841236117,
-          -0.44758284091949463,
-          0.14151698350906372,
-          -0.1089438870549202,
-          0.5500766634941101,
-          -0.670711100101471,
-          -0.6227269768714905,
-          0.3894464075565338,
-          -0.27609574794769287,
-          0.7028202414512634,
-          -0.19697771966457367,
-          0.328511506319046,
-          0.5063360929489136,
-          0.4065195322036743,
-          0.2614171802997589,
-          -0.30274391174316406,
-          1.0393824577331543,
-          -0.7742937207221985,
-          -0.7874112129211426,
-          -0.6749666929244995,
-          0.5190866589546204,
-          0.004123548045754433,
-          -0.28312963247299194,
-          -0.038731709122657776,
-          -1.0142987966537476,
-          -0.09519586712121964,
-          0.8755272626876831,
-          0.4876938760280609,
-          0.7811151742935181,
-          0.85174959897995,
-          0.11826585978269576,
-          0.5373436808586121,
-          0.3649002015590668,
-          0.19064077734947205,
-          -0.00287026260048151,
-          -0.7305403351783752,
-          -0.015206154435873032,
-          -0.7899249196052551,
-          0.19407285749912265,
-          0.08596625179052353,
-          -0.28976231813430786,
-          -0.1525907665491104,
-          0.3798313438892365,
-          0.050306469202041626,
-          -0.5697937607765198,
-          0.4219021201133728,
-          0.276252806186676,
-          0.1559903472661972,
-          0.10030482709407806,
-          -0.4043720066547394,
-          -0.1969818025827408,
-          0.5739826560020447,
-          0.2116064727306366,
-          -1.4620544910430908,
-          -0.7802462577819824,
-          -0.24739810824394223,
-          -0.09791352599859238,
-          -0.4413802027702331,
-          0.21549351513385773,
-          -0.9520436525344848,
-          -0.08762510865926743,
-          0.08154498040676117,
-          -0.6154940724372864,
-          -1.01079523563385,
-          0.885427713394165,
-          0.6967288851737976,
-          0.27186504006385803,
-          -0.43194177746772766,
-          -0.11248451471328735,
-          0.7576630711555481,
-          0.4998855590820313,
-          0.0264343973249197,
-          0.9872855544090272,
-          0.5634694695472717,
-          0.053698331117630005,
-          0.19410227239131927,
-          0.3570743501186371,
-          -0.23670297861099243,
-          -0.9114483594894408,
-          0.07884842902421951,
-          0.7318344116210938,
-          0.44630110263824463,
-          0.08745364099740982,
-          -0.347101628780365,
-          -0.4314247667789459,
-          -0.5060274004936218,
-          0.003706763498485088,
-          0.44320008158683777,
-          -0.00788921769708395,
-          -0.1368623524904251,
-          -0.17391923069953918,
-          0.14473655819892883,
-          0.10927865654230118,
-          0.6974599361419678,
-          0.005052129738032818,
-          -0.016953065991401672,
-          -0.1256176233291626,
-          -0.036742497235536575,
-          0.5591985583305359,
-          -0.37619709968566895,
-          0.22429119050502777,
-          0.5403043031692505,
-          -0.8603790998458862,
-          -0.3456307053565979,
-          0.9292937517166138,
-          0.5074859261512756,
-          0.6310645937919617,
-          -0.3091641068458557,
-          0.46902573108673096,
-          0.7891915440559387,
-          0.4499550759792328,
-          0.2744995653629303,
-          0.2712305784225464,
-          -0.04349074140191078,
-          -0.3638863265514374,
-          0.7839881777763367,
-          0.7352104783058167,
-          -0.19457511603832245,
-          -0.5957832932472229,
-          -0.43704694509506226,
-          -1.084769368171692,
-          0.4904985725879669,
-          0.5385226011276245,
-          0.1891629993915558,
-          0.12338479608297348,
-          0.8315675258636475,
-          -0.07830192148685455,
-          1.0916285514831543,
-          -0.28066861629486084,
-          -1.3585069179534912,
-          0.5203898549079895,
-          0.08678033947944641,
-          -0.2566044330596924,
-          0.09484415501356123,
-          -0.0180208683013916,
-          1.0264745950698853,
-          -0.023572135716676712,
-          0.5864979028701782,
-          0.7625196576118469,
-          -0.2543414533138275,
-          -0.8877770900726318,
-          0.7611982822418213,
-          -0.06220436468720436,
-          0.937336564064026,
-          0.2704363465309143,
-          -0.37733694911003113,
-          0.5076137781143188,
-          -0.30641937255859375,
-          0.6252772808074951,
-          -0.0823579877614975,
-          -0.03736555948853493,
-          0.4131673276424408,
-          -0.6514252424240112,
-          0.12918265163898468,
-          -0.4483584463596344,
-          0.6750786304473877,
-          -0.37008383870124817,
-          -0.02324833907186985,
-          0.38027650117874146,
-          -0.26374951004981995,
-          0.4346931278705597,
-          0.42882832884788513,
-          -0.48798441886901855,
-          1.1882442235946655,
-          0.5132288336753845,
-          0.5284568667411804,
-          -0.03538886830210686,
-          0.29620853066444397,
-          -1.0683696269989014,
-          0.25936177372932434,
-          0.10404160618782043,
-          -0.25796034932136536,
-          0.027896970510482788,
-          -0.09225251525640488,
-          1.4811025857925415,
-          0.641173779964447,
-          -0.13838383555412292,
-          -0.3437179923057556,
-          0.5667019486427307,
-          -0.5400741696357727,
-          0.31090837717056274,
-          0.6470608115196228,
-          -0.3747067153453827,
-          -0.7364534735679626,
-          -0.07431528717279434,
-          0.5173454880714417,
-          -0.6578747034072876,
-          0.7107478976249695,
-          -0.7918999791145325,
-          -0.0648345872759819,
-          0.609937846660614,
-          -0.7329513430595398,
-          0.9741371870040894,
-          0.17912346124649048,
-          -0.02658769302070141,
-          0.5162150859832764,
-          -0.3978803157806397,
-          -0.7833885550498962,
-          -0.6497276425361633,
-          -0.3898126780986786,
-          -0.0952848568558693,
-          0.2663288116455078,
-          -0.1604052186012268,
-          0.373076468706131,
-          -0.8357769250869751,
-          -0.05217683315277099,
-          -0.2680160701274872,
-          0.8389158248901367,
-          0.6833611130714417,
-          -0.6712407469749451,
-          0.7406917214393616,
-          -0.44522786140441895,
-          -0.34645363688468933,
-          -0.27384576201438904,
-          -0.9878405928611756,
-          -0.8166060447692871,
-          0.06268279999494553,
-          0.38567957282066345,
-          -0.3274703919887543,
-          0.5296315550804138,
-          -0.11810623109340668,
-          0.23029841482639313,
-          0.08616159111261368,
-          -0.2195747196674347,
-          0.09430307894945145,
-          0.4057176411151886,
-          0.4892159104347229,
-          -0.1636916548013687,
-          -0.6071445345878601,
-          0.41256585717201233,
-          0.622254490852356,
-          -0.41223976016044617,
-          -0.6686707139015198,
-          -0.7474371790885925,
-          -0.8509522080421448,
-          -0.16754287481307983,
-          -0.9078601002693176,
-          -0.29653599858283997,
-          -0.5020652413368225,
-          0.4692700505256653,
-          0.01281109917908907,
-          -0.16071580350399017,
-          0.03388889133930206,
-          -0.020511148497462273,
-          0.5027827024459839,
-          -0.20729811489582065,
-          0.48107290267944336,
-          0.33669769763946533,
-          -0.5275911688804626,
-          0.48271527886390686,
-          0.2738940715789795,
-          -0.033152539283037186,
-          -0.13629786670207977,
-          -0.05965912342071533,
-          -0.26200807094573975,
-          0.04002794995903969,
-          -0.34095603227615356,
-          -3.986898899078369,
-          -0.46819332242012024,
-          -0.422744482755661,
-          -0.169097900390625,
-          0.6008929014205933,
-          0.058016058057546616,
-          -0.11401277780532836,
-          -0.3077819049358368,
-          -0.09595538675785063,
-          0.6723822355270386,
-          0.19367831945419312,
-          0.28304359316825867,
-          0.1609862744808197,
-          0.7567598819732666,
-          0.6889985799789429,
-          0.06907720118761063,
-          -0.04188092052936554,
-          -0.7434936165809631,
-          0.13321782648563385,
-          0.8456063270568848,
-          -0.10364038497209548,
-          -0.45084846019744873,
-          -0.4758241474628449,
-          0.43882066011428833,
-          -0.6432598829269409,
-          0.7217311859130859,
-          -0.24189773201942444,
-          0.12737572193145752,
-          -1.1008601188659668,
-          -0.3305315673351288,
-          0.14614742994308472,
-          -0.7819333076477051,
-          0.5287120342254639,
-          -0.055538054555654526,
-          0.1877404749393463,
-          -0.6907662153244019,
-          0.5616975426673889,
-          -0.4611121714115143,
-          -0.26109233498573303,
-          -0.12898315489292145,
-          -0.3724522292613983,
-          -0.7191406488418579,
-          -0.4425233602523804,
-          -0.644108235836029,
-          0.8424481153488159,
-          0.17532426118850708,
-          -0.5121750235557556,
-          -0.6467239260673523,
-          -0.0008507720194756985,
-          0.7866212129592896,
-          -0.02644744887948036,
-          -0.005045140627771616,
-          0.015782782807946205,
-          0.16334445774555206,
-          -0.1913367658853531,
-          -0.13697923719882965,
-          -0.6684983372688293,
-          0.18346354365348816,
-          -0.341105580329895,
-          0.5427411198616028,
-          0.3779832422733307,
-          -0.6778115034103394,
-          -0.2931850254535675,
-          -0.8805161714553833,
-          -0.4212774932384491,
-          -0.5368952751159668,
-          -1.3937891721725464,
-          -1.225494146347046,
-          0.4276703894138336,
-          1.1205668449401855,
-          -0.6005299687385559,
-          0.15732505917549133,
-          -0.3914784789085388,
-          -1.357046604156494,
-          -0.4707142114639282,
-          -0.1497287154197693,
-          -0.25035548210144043,
-          -0.34328439831733704,
-          0.39083412289619446,
-          0.1623048633337021,
-          -0.9275814294815063,
-          -0.6430015563964844,
-          0.2973862886428833,
-          0.5580436587333679,
-          -0.6232585310935974,
-          -0.6611042022705078,
-          0.4015969038009643,
-          -1.0232892036437988,
-          -0.2585645020008087,
-          -0.5431421399116516,
-          0.5021264553070068,
-          -0.48601630330085754,
-          -0.010242084041237833,
-          0.5862035155296326,
-          0.7316920161247253,
-          0.4036808013916016,
-          0.4269520044326782,
-          -0.705938458442688,
-          0.7747307419776917,
-          0.10164368897676468,
-          0.7887958884239197,
-          -0.9612497091293336,
-          0.12755516171455383,
-          0.06812842190265656,
-          -0.022603651508688927,
-          0.14722754061222076,
-          -0.5588505268096924,
-          -0.20689940452575684,
-          0.3557641804218292,
-          -0.6812759637832642,
-          0.2860803008079529,
-          -0.38954633474349976,
-          0.1759403496980667,
-          -0.5678874850273132,
-          -0.1692986786365509,
-          -0.14578519761562347,
-          0.5711379051208496,
-          1.0208125114440918,
-          0.7759483456611633,
-          -0.372348427772522,
-          -0.5460885763168335,
-          0.7190321683883667,
-          -0.6914990544319153,
-          0.13365162909030914,
-          -0.4854792356491089,
-          0.4054908752441406,
-          0.4502798914909363,
-          -0.3041122555732727,
-          -0.06726965308189392,
-          -0.05570871382951737,
-          -0.0455719493329525,
-          0.4785125255584717,
-          0.8867972493171692,
-          0.4107886850833893,
-          0.6121342182159424,
-          -0.20477132499217987,
-          -0.5598517656326294,
-          -0.6443566679954529,
-          -0.5905212759971619,
-          -0.5571200251579285,
-          0.17573799192905426,
-          -0.28621870279312134,
-          0.1685224026441574,
-          0.09719007462263109,
-          -0.04223639518022537,
-          -0.28623101115226746,
-          -0.1449810117483139,
-          -0.3789580464363098,
-          -0.5227636098861694,
-          -0.049728814512491226,
-          0.7849089503288269,
-          0.16792525351047516,
-          0.9849340915679932,
-          -0.6559549570083618,
-          0.35723909735679626,
-          -0.6822739243507385,
-          1.2873116731643677,
-          0.19993330538272855,
-          0.03512010723352432,
-          -0.6972134113311768,
-          0.18453484773635864,
-          -0.2437680810689926,
-          0.2156416028738022,
-          0.5230382680892944,
-          0.22020135819911957,
-          0.8314080238342285,
-          0.15627102553844452,
-          -0.7330264449119568,
-          0.3888184726238251,
-          -0.22034703195095065,
-          0.5457669496536255,
-          -0.48084837198257446,
-          -0.45576658844947815,
-          -0.09287727624177931,
-          -0.06968110054731369,
-          0.35125672817230225,
-          -0.4278119504451752,
-          0.2038476765155792,
-          0.11392722278833388,
-          0.9433983564376832,
-          -0.4097744226455689,
-          0.035297419875860214,
-          -0.4274404048919678,
-          -0.25100165605545044,
-          1.0943366289138794,
-          -0.07634022831916809,
-          -0.2925529479980469,
-          -0.7512530088424683,
-          0.2649727463722229,
-          -0.4078235328197479,
-          -0.3372223973274231,
-          0.05190162733197212,
-          0.005654910113662481,
-          -0.0001571219472680241,
-          -0.35445958375930786,
-          -0.7837416529655457,
-          0.1500556766986847,
-          0.4383024573326111,
-          0.6099548935890198,
-          0.05951934307813645,
-          -0.21325334906578064,
-          0.0199207104742527,
-          -0.22704418003559113,
-          -0.6481077671051025,
-          0.37442275881767273,
-          -1.015955924987793,
-          0.38637226819992065,
-          -0.06489371508359909,
-          -0.494120329618454,
-          0.3469836115837097,
-          0.15402406454086304,
-          -0.7660972476005554,
-          -0.7053225040435791,
-          -0.25964751839637756,
-          0.014004424214363098,
-          -0.2860170006752014,
-          -0.17565494775772095,
-          -0.45117494463920593,
-          -0.0031954257283359766,
-          0.09676837921142578,
-          -0.514464259147644,
-          0.41698193550109863,
-          -0.21642713248729703,
-          -0.5398141145706177,
-          -0.3647628426551819,
-          0.37005379796028137,
-          0.239425927400589,
-          -0.08833975344896317,
-          0.934946596622467,
-          -0.48340797424316406,
-          0.6241437792778015,
-          -0.7253676652908325,
-          -0.04303571209311485,
-          1.1125205755233765,
-          -0.15692919492721558,
-          -0.2914651036262512,
-          -0.5117168426513672,
-          0.21365483105182648,
-          0.4924402534961701,
-          0.5269662141799927,
-          0.0352792888879776,
-          -0.149167999625206,
-          -0.6019760370254517,
-          0.08245442807674408,
-          0.4900692105293274,
-          0.518824577331543,
-          -0.00005570516441366635,
-          -0.553304135799408,
-          0.22217543423175812,
-          0.5047767758369446,
-          0.135724738240242,
-          1.1511540412902832,
-          -0.3541218340396881,
-          -0.9712511897087096,
-          0.8353699445724487,
-          -0.39227569103240967,
-          -0.9117669463157654,
-          -0.26349931955337524,
-          0.05597023293375969,
-          0.20695461332798004,
-          0.3178807199001312,
-          1.0663238763809204,
-          0.5062212347984314,
-          0.7288597822189331,
-          0.09899299591779707,
-          0.553720235824585,
-          0.675009548664093,
-          -0.20067055523395536,
-          0.3138423264026642,
-          -0.6886593103408813,
-          -0.2910398542881012,
-          -1.3186300992965698,
-          -0.4684459865093231,
-          -0.095743365585804,
-          -0.1257995069026947,
-          -0.4858281314373016,
-          -0.4935407340526581,
-          -0.3266896903514862,
-          -0.3928797245025635,
-          -0.40803104639053345,
-          -0.9975396394729614,
-          0.4229583740234375,
-          0.37309643626213074,
-          0.4431034922599793,
-          0.30364808440208435,
-          -0.3765178918838501,
-          0.5616499185562134,
-          0.16904796659946442,
-          -0.7343707084655762,
-          0.2560209631919861,
-          0.6166825294494629,
-          0.3200829327106476,
-          -0.4483652710914612,
-          0.16224201023578644,
-          -0.31495288014411926,
-          -0.42713335156440735,
-          0.7270734906196594,
-          0.7049484848976135,
-          -0.0571461021900177,
-          0.04477125033736229,
-          -0.6647796034812927,
-          1.183672308921814,
-          0.36199676990509033,
-          0.046881116926670074,
-          0.4515796303749085,
-          0.9278061985969543,
-          0.31471705436706543,
-          -0.7073333859443665,
-          -0.3443860113620758,
-          0.5440067052841187,
-          -0.15020819008350372,
-          -0.541202962398529,
-          0.5203295946121216,
-          1.2192286252975464,
-          -0.9983593225479126,
-          -0.18758884072303772,
-          0.2758221924304962,
-          -0.6511523723602295,
-          -0.1584404855966568,
-          -0.236241415143013,
-          0.2692437767982483,
-          -0.4941152036190033,
-          0.4987454116344452,
-          -0.3331359028816223,
-          0.3163745701313019,
-          0.745529294013977,
-          -0.2905873656272888,
-          0.13602906465530396,
-          0.4679684340953827,
-          1.0555986166000366,
-          1.075700044631958,
-          0.5368486046791077,
-          -0.5118206739425659,
-          0.8668332099914551,
-          -0.5726966857910156,
-          -0.7811751961708069,
-          0.1938626915216446,
-          -0.1929349899291992,
-          0.1757766306400299,
-          0.6384295225143433,
-          0.26462844014167786,
-          0.9542630314826964,
-          0.19313029944896695,
-          1.264248013496399,
-          -0.6304428577423096,
-          0.0487106591463089,
-          -0.16211535036563873,
-          -0.7894763350486755,
-          0.3582514822483063,
-          -0.04153040423989296,
-          0.635784387588501,
-          0.6554391980171204,
-          -0.47010496258735657,
-          -0.8302040696144104,
-          -0.1350124627351761,
-          0.2568812072277069,
-          0.13614831864833832,
-          -0.2563649117946625,
-          -1.0434694290161133,
-          0.3232482671737671,
-          0.47882452607154846,
-          0.4298652410507202,
-          1.0563770532608032,
-          -0.28917592763900757,
-          -0.8533256649971008,
-          0.10648339986801147,
-          0.6376127004623413,
-          -0.20832888782024384,
-          0.2370245456695557,
-          0.0018312990432605147,
-          -0.2034837007522583,
-          0.01051164511591196,
-          -1.105310082435608,
-          0.29724350571632385,
-          0.15604574978351593,
-          0.1973688006401062,
-          0.44394731521606445,
-          0.3974513411521912,
-          -0.13625948131084442,
-          0.9571986198425292,
-          0.2257384955883026,
-          0.2323588728904724,
-          -0.5583669543266296,
-          -0.7854922413825989,
-          0.1647188365459442,
-          -1.6098142862319946,
-          0.318587988615036,
-          -0.13399995863437653,
-          -0.2172701060771942,
-          -0.767514705657959,
-          -0.5813586711883545,
-          -0.3195130527019501,
-          -0.04894036799669266,
-          0.2929930090904236,
-          -0.8213384747505188,
-          0.07181350141763687,
-          0.7469993829727173,
-          0.6407455801963806,
-          0.16365697979927063,
-          0.7870153188705444,
-          0.6524736881256104,
-          0.6399973630905151,
-          -0.04992736503481865,
-          -0.03959266096353531,
-          -0.2512352466583252,
-          0.8448855876922607,
-          -0.1422702670097351,
-          0.1216789186000824,
-          -1.2647287845611572,
-          0.5931149125099182,
-          0.7186052203178406,
-          -0.06118432432413101,
-          -1.1942816972732544,
-          -0.17677085101604462,
-          0.31543800234794617,
-          -0.32252824306488037,
-          0.8255583047866821,
-          -0.14529970288276672,
-          -0.2695446312427521,
-          -0.33378756046295166,
-          -0.1653425395488739,
-          0.1454019844532013,
-          -0.3920115828514099,
-          0.912214994430542,
-          -0.7279734015464783,
-          0.7374742031097412,
-          0.933980405330658,
-          0.13429680466651917,
-          -0.514870285987854,
-          0.3989711999893189,
-          -0.11613689363002776,
-          0.4022413492202759,
-          -0.9990655779838562,
-          -0.33749932050704956,
-          -0.4334589838981629,
-          -1.376373291015625,
-          -0.2993924915790558,
-          -0.09454808384180068,
-          -0.01314175222069025,
-          -0.001090060803107917,
-          0.2137461006641388,
-          0.2938512861728668,
-          0.17508235573768616,
-          0.8260607123374939,
-          -0.7218498587608337,
-          0.2414487451314926,
-          -0.47296759486198425,
-          -0.3002610504627228,
-          -1.238540768623352,
-          0.08663805574178696,
-          0.6805586218833923,
-          0.5909030437469482,
-          -0.42807504534721375,
-          -0.22887496650218964,
-          0.47537800669670105,
-          -1.0474627017974854,
-          0.6338009238243103,
-          0.06548397243022919,
-          0.4971011281013489,
-          1.3484878540039063
-        ]
-      ],
-      "regenerate": true
-    }
-  }
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-8.snap

@@ -1,785 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e1",
-  "desc": "natural vector",
-  "_vectors": {
-    "default": {
-      "embeddings": [
-        [
-          -0.2979458272457123,
-          -0.5288640856742859,
-          -0.019957859069108963,
-          -0.18495318293571472,
-          0.7429973483085632,
-          0.5238497257232666,
-          0.432366281747818,
-          0.32744166254997253,
-          0.0020762972999364138,
-          -0.9507834911346436,
-          -0.35097137093544006,
-          0.08469701558351517,
-          -1.4176613092422483,
-          0.4647577106952667,
-          -0.69340580701828,
-          1.0372896194458008,
-          0.3716741800308227,
-          0.06031008064746857,
-          -0.6152024269104004,
-          0.007914665155112743,
-          0.7954924702644348,
-          -0.20773003995418549,
-          0.09376765787601472,
-          0.04508133605122566,
-          -0.2084471583366394,
-          -0.1518009901046753,
-          0.018195509910583496,
-          -0.07044368237257004,
-          -0.18119366466999057,
-          -0.4480230510234833,
-          0.3822529911994934,
-          0.1911812424659729,
-          0.4674372375011444,
-          0.06963984668254852,
-          -0.09341949224472046,
-          0.005675444379448891,
-          -0.6774799227714539,
-          -0.7066726684570313,
-          -0.39256376028060913,
-          0.04005039855837822,
-          0.2084812968969345,
-          -0.7872875928878784,
-          -0.8205880522727966,
-          0.2919981777667999,
-          -0.06004738807678223,
-          -0.4907574355602264,
-          -1.5937862396240234,
-          0.24249385297298431,
-          -0.14709846675395966,
-          -0.11860740929841997,
-          -0.8299489617347717,
-          0.472964346408844,
-          -0.497518390417099,
-          -0.22205302119255063,
-          -0.4196169078350067,
-          0.32697558403015137,
-          -0.360930860042572,
-          -0.9789686799049376,
-          0.1887447088956833,
-          -0.403737336397171,
-          0.18524253368377688,
-          0.3768732249736786,
-          0.3666233420372009,
-          0.3511938452720642,
-          0.6985810995101929,
-          0.41721710562705994,
-          0.09754953533411026,
-          0.6204307079315186,
-          -1.0762996673583984,
-          -0.06263761967420578,
-          -0.7376511693000793,
-          0.6849768161773682,
-          -0.1745152473449707,
-          -0.40449759364128113,
-          0.20757411420345304,
-          -0.8424443006515503,
-          0.330015629529953,
-          0.3489064872264862,
-          1.0954371690750122,
-          0.8487558960914612,
-          1.1076823472976685,
-          0.61430823802948,
-          0.4155903458595276,
-          0.4111340939998626,
-          0.05753209814429283,
-          -0.06429877132177353,
-          -0.765606164932251,
-          -0.41703930497169495,
-          -0.508820652961731,
-          0.19859947264194489,
-          -0.16607828438282013,
-          -0.28112146258354187,
-          0.11032675206661224,
-          0.38809511065483093,
-          -0.36498191952705383,
-          -0.48671194911003113,
-          0.6755134463310242,
-          0.03958442434668541,
-          0.4478721618652344,
-          -0.10335399955511092,
-          -0.9546685814857484,
-          -0.6087718605995178,
-          0.17498846352100372,
-          0.08320838958024979,
-          -1.4478336572647097,
-          -0.605027437210083,
-          -0.5867993235588074,
-          -0.14711688458919525,
-          -0.5447602272033691,
-          -0.026259321719408035,
-          -0.6997418403625488,
-          -0.07349082082509995,
-          0.10638900846242905,
-          -0.7133527398109436,
-          -0.9396815299987792,
-          1.087092399597168,
-          1.1885089874267578,
-          0.4011896848678589,
-          -0.4089202582836151,
-          -0.10938972979784012,
-          0.6726722121238708,
-          0.24576938152313232,
-          -0.24247920513153076,
-          1.1499971151351929,
-          0.47813335061073303,
-          -0.05331678315997124,
-          0.32338133454322815,
-          0.4870913326740265,
-          -0.23144258558750153,
-          -1.2023426294326782,
-          0.2349330335855484,
-          1.080536961555481,
-          0.29334118962287903,
-          0.391574501991272,
-          -0.15818795561790466,
-          -0.2948290705680847,
-          -0.024689948186278343,
-          0.06602869182825089,
-          0.5937030911445618,
-          -0.047901444137096405,
-          -0.512734591960907,
-          -0.35780075192451477,
-          0.28751692175865173,
-          0.4298716187477112,
-          0.9242428541183472,
-          -0.17208744585514069,
-          0.11515070497989656,
-          -0.0335976779460907,
-          -0.3422986567020416,
-          0.5344581604003906,
-          0.19895796477794647,
-          0.33001241087913513,
-          0.6390730142593384,
-          -0.6074934005737305,
-          -0.2553696632385254,
-          0.9644920229911804,
-          0.2699219584465027,
-          0.6403993368148804,
-          -0.6380003690719604,
-          -0.027310986071825027,
-          0.638815701007843,
-          0.27719101309776306,
-          -0.13553589582443237,
-          0.750195324420929,
-          0.1224869191646576,
-          -0.20613941550254825,
-          0.8444448709487915,
-          0.16200250387191772,
-          -0.24750925600528717,
-          -0.739950954914093,
-          -0.28443849086761475,
-          -1.176282525062561,
-          0.516107976436615,
-          0.3774825632572174,
-          0.10906043648719788,
-          0.07962015271186829,
-          0.7384604215621948,
-          -0.051241904497146606,
-          1.1730090379714966,
-          -0.4828610122203827,
-          -1.404372215270996,
-          0.8811132311820984,
-          -0.3839482367038727,
-          0.022516896948218346,
-          -0.0491158664226532,
-          -0.43027013540267944,
-          1.2049334049224854,
-          -0.27309560775756836,
-          0.6883630752563477,
-          0.8264574408531189,
-          -0.5020735263824463,
-          -0.4874092042446137,
-          0.6007202863693237,
-          -0.4965405762195587,
-          1.1302915811538696,
-          0.032572727650403976,
-          -0.3731859028339386,
-          0.658271849155426,
-          -0.9023059010505676,
-          0.7400162220001221,
-          0.014550759457051754,
-          -0.19699542224407196,
-          0.2319706380367279,
-          -0.789058268070221,
-          -0.14905710518360138,
-          -0.5826214551925659,
-          0.207652747631073,
-          -0.4507439732551574,
-          -0.3163885474205017,
-          0.3604124188423157,
-          -0.45119962096214294,
-          0.3428427278995514,
-          0.3005594313144684,
-          -0.36026081442832947,
-          1.1014249324798584,
-          0.40884315967559814,
-          0.34991952776908875,
-          -0.1806638240814209,
-          0.27440476417541504,
-          -0.7118373513221741,
-          0.4645499587059021,
-          0.214790478348732,
-          -0.2343102991580963,
-          0.10500429570674896,
-          -0.28034430742263794,
-          1.2267805337905884,
-          1.0561333894729614,
-          -0.497364342212677,
-          -0.6143305897712708,
-          0.24963727593421936,
-          -0.33136463165283203,
-          -0.01473914459347725,
-          0.495918869972229,
-          -0.6985538005828857,
-          -1.0033197402954102,
-          0.35937801003456116,
-          0.6325868368148804,
-          -0.6808838844299316,
-          1.0354058742523191,
-          -0.7214401960372925,
-          -0.33318862318992615,
-          0.874398410320282,
-          -0.6594992280006409,
-          0.6830640435218811,
-          -0.18534131348133087,
-          0.024834271520376205,
-          0.19901277124881744,
-          -0.5992477536201477,
-          -1.2126628160476685,
-          -0.9245557188987732,
-          -0.3898217976093292,
-          -0.1286519467830658,
-          0.4217943847179413,
-          -0.1143646091222763,
-          0.5630772709846497,
-          -0.5240639448165894,
-          0.21152715384960177,
-          -0.3792001008987427,
-          0.8266305327415466,
-          1.170984387397766,
-          -0.8072142004966736,
-          0.11382893472909927,
-          -0.17953898012638092,
-          -0.1789460331201553,
-          -0.15078622102737427,
-          -1.2082908153533936,
-          -0.7812382578849792,
-          -0.10903695970773696,
-          0.7303897142410278,
-          -0.39054441452026367,
-          0.19511254131793976,
-          -0.09121843427419662,
-          0.22400228679180145,
-          0.30143046379089355,
-          0.1141919493675232,
-          0.48112115263938904,
-          0.7307931780815125,
-          0.09701362252235413,
-          -0.2795647978782654,
-          -0.3997688889503479,
-          0.5540812611579895,
-          0.564578115940094,
-          -0.40065160393714905,
-          -0.3629159033298493,
-          -0.3789091110229492,
-          -0.7298538088798523,
-          -0.6996853351593018,
-          -0.4477842152118683,
-          -0.289089560508728,
-          -0.6430277824401855,
-          0.2344944179058075,
-          0.3742927014827728,
-          -0.5079357028007507,
-          0.28841453790664673,
-          0.06515737622976303,
-          0.707315981388092,
-          0.09498685598373412,
-          0.8365515470504761,
-          0.10002726316452026,
-          -0.7695478200912476,
-          0.6264724135398865,
-          0.7562043070793152,
-          -0.23112858831882477,
-          -0.2871039807796478,
-          -0.25010058283805847,
-          0.2783474028110504,
-          -0.03224996477365494,
-          -0.9119359850883484,
-          -3.6940200328826904,
-          -0.5099936127662659,
-          -0.1604711413383484,
-          0.17453284561634064,
-          0.41759559512138367,
-          0.1419190913438797,
-          -0.11362407356500626,
-          -0.33312007784843445,
-          0.11511333286762238,
-          0.4667884409427643,
-          -0.0031647447030991316,
-          0.15879854559898376,
-          0.3042248487472534,
-          0.5404849052429199,
-          0.8515422344207764,
-          0.06286454200744629,
-          0.43790125846862793,
-          -0.8682025074958801,
-          -0.06363756954669952,
-          0.5547921657562256,
-          -0.01483887154608965,
-          -0.07361344993114471,
-          -0.929947018623352,
-          0.3502565622329712,
-          -0.5080993175506592,
-          1.0380364656448364,
-          -0.2017953395843506,
-          0.21319580078125,
-          -1.0763001441955566,
-          -0.556368887424469,
-          0.1949922740459442,
-          -0.6445739269256592,
-          0.6791343688964844,
-          0.21188358962535855,
-          0.3736183941364288,
-          -0.21800459921360016,
-          0.7597446441650391,
-          -0.3732394874095917,
-          -0.4710160195827484,
-          0.025146087631583217,
-          0.05341297015547752,
-          -0.9522109627723694,
-          -0.6000866889953613,
-          -0.08469046652317047,
-          0.5966026186943054,
-          0.3444081246852875,
-          -0.461188405752182,
-          -0.5279349088668823,
-          0.10296865552663804,
-          0.5175143480300903,
-          -0.20671147108078003,
-          0.13392412662506104,
-          0.4812754988670349,
-          0.2993808686733246,
-          -0.3005635440349579,
-          0.5141698122024536,
-          -0.6239235401153564,
-          0.2877119481563568,
-          -0.4452739953994751,
-          0.5621107816696167,
-          0.5047508478164673,
-          -0.4226335883140564,
-          -0.18578553199768064,
-          -1.1967322826385498,
-          0.28178197145462036,
-          -0.8692031502723694,
-          -1.1812998056411743,
-          -1.4526212215423584,
-          0.4645712077617645,
-          0.9327932000160216,
-          -0.6560136675834656,
-          0.461549699306488,
-          -0.5621527433395386,
-          -1.328449010848999,
-          -0.08676894754171371,
-          0.00021918353741057217,
-          -0.18864136934280396,
-          0.1259666532278061,
-          0.18240638077259064,
-          -0.14919660985469818,
-          -0.8965857625007629,
-          -0.7539900541305542,
-          0.013973715715110302,
-          0.504276692867279,
-          -0.704748272895813,
-          -0.6428424119949341,
-          0.6303996443748474,
-          -0.5404738187789917,
-          -0.31176653504371643,
-          -0.21262824535369873,
-          0.18736739456653595,
-          -0.7998970746994019,
-          0.039946746081113815,
-          0.7390344738960266,
-          0.4283199906349182,
-          0.3795057237148285,
-          0.07204607129096985,
-          -0.9230587482452391,
-          0.9440426230430604,
-          0.26272690296173096,
-          0.5598306655883789,
-          -1.0520871877670288,
-          -0.2677186131477356,
-          -0.1888762265443802,
-          0.30426350235939026,
-          0.4746131896972656,
-          -0.5746733546257019,
-          -0.4197768568992615,
-          0.8565112948417664,
-          -0.6767723560333252,
-          0.23448683321475983,
-          -0.2010004222393036,
-          0.4112907350063324,
-          -0.6497949957847595,
-          -0.418667733669281,
-          -0.4950824975967407,
-          0.44438859820365906,
-          1.026281714439392,
-          0.482397586107254,
-          -0.26220494508743286,
-          -0.3640787005424499,
-          0.5907743573188782,
-          -0.8771642446517944,
-          0.09708411991596222,
-          -0.3671700060367584,
-          0.4331349730491638,
-          0.619417667388916,
-          -0.2684665620326996,
-          -0.5123821496963501,
-          -0.1502324342727661,
-          -0.012190685607492924,
-          0.3580845892429352,
-          0.8617186546325684,
-          0.3493645489215851,
-          1.0270192623138428,
-          0.18297909200191495,
-          -0.5881339311599731,
-          -0.1733516901731491,
-          -0.5040576457977295,
-          -0.340370237827301,
-          -0.26767754554748535,
-          -0.28570041060447693,
-          -0.032928116619586945,
-          0.6029254794120789,
-          0.17397655546665192,
-          0.09346921741962431,
-          0.27815181016921997,
-          -0.46699589490890503,
-          -0.8148876428604126,
-          -0.3964351713657379,
-          0.3812595009803772,
-          0.13547226786613464,
-          0.7126688361167908,
-          -0.3473474085330963,
-          -0.06573959439992905,
-          -0.6483767032623291,
-          1.4808889627456665,
-          0.30924928188323975,
-          -0.5085946917533875,
-          -0.8613000512123108,
-          0.3048902451992035,
-          -0.4241599142551422,
-          0.15909206867218018,
-          0.5764641761779785,
-          -0.07879110425710678,
-          1.015336513519287,
-          0.07599356025457382,
-          -0.7025855779647827,
-          0.30047643184661865,
-          -0.35094937682151794,
-          0.2522146999835968,
-          -0.2338722199201584,
-          -0.8326804637908936,
-          -0.13695412874221802,
-          -0.03452421352267265,
-          0.47974953055381775,
-          -0.18385636806488037,
-          0.32438594102859497,
-          0.1797013282775879,
-          0.787494957447052,
-          -0.12579888105392456,
-          -0.07507286965847015,
-          -0.4389670491218567,
-          0.2720070779323578,
-          0.8138866424560547,
-          0.01974171027541161,
-          -0.3057698905467987,
-          -0.6709924936294556,
-          0.0885881632566452,
-          -0.2862754464149475,
-          0.03475658595561981,
-          -0.1285519152879715,
-          0.3838353455066681,
-          -0.2944154739379883,
-          -0.4204859137535095,
-          -0.4416137933731079,
-          0.13426260650157928,
-          0.36733248829841614,
-          0.573428750038147,
-          -0.14928072690963745,
-          -0.026076916605234143,
-          0.33286052942276,
-          -0.5340145826339722,
-          -0.17279052734375,
-          -0.01154550164937973,
-          -0.6620771884918213,
-          0.18390542268753052,
-          -0.08265615254640579,
-          -0.2489682286977768,
-          0.2429984211921692,
-          -0.044153645634651184,
-          -0.986578404903412,
-          -0.33574509620666504,
-          -0.5387663841247559,
-          0.19767941534519196,
-          0.12540718913078308,
-          -0.3403128981590271,
-          -0.4154576361179352,
-          0.17275673151016235,
-          0.09407442808151244,
-          -0.5414086580276489,
-          0.4393929839134216,
-          0.1725579798221588,
-          -0.4998118281364441,
-          -0.6926208138465881,
-          0.16552448272705078,
-          0.6659538149833679,
-          -0.10949844866991044,
-          0.986426830291748,
-          0.01748848147690296,
-          0.4003709554672241,
-          -0.5430638194084167,
-          0.35347291827201843,
-          0.6887399554252625,
-          0.08274628221988678,
-          0.13407137989997864,
-          -0.591465950012207,
-          0.3446292281150818,
-          0.6069018244743347,
-          0.1935492902994156,
-          -0.0989871397614479,
-          0.07008486241102219,
-          -0.8503749370574951,
-          -0.09507356584072112,
-          0.6259510517120361,
-          0.13934025168418884,
-          0.06392545253038406,
-          -0.4112265408039093,
-          -0.08475656062364578,
-          0.4974113404750824,
-          -0.30606114864349365,
-          1.111435890197754,
-          -0.018766529858112335,
-          -0.8422622680664063,
-          0.4325508773326874,
-          -0.2832120656967163,
-          -0.4859798848628998,
-          -0.41498348116874695,
-          0.015977520495653152,
-          0.5292825698852539,
-          0.4538311660289765,
-          1.1328668594360352,
-          0.22632671892642975,
-          0.7918671369552612,
-          0.33401933312416077,
-          0.7306135296821594,
-          0.3548600673675537,
-          0.12506209313869476,
-          0.8573207855224609,
-          -0.5818327069282532,
-          -0.6953738927841187,
-          -1.6171947717666626,
-          -0.1699674427509308,
-          0.6318262815475464,
-          -0.05671752244234085,
-          -0.28145185112953186,
-          -0.3976689279079437,
-          -0.2041076272726059,
-          -0.5495951175689697,
-          -0.5152917504310608,
-          -0.9309796094894408,
-          0.101932130753994,
-          0.1367802917957306,
-          0.1490798443555832,
-          0.5304336547851563,
-          -0.5082434415817261,
-          0.06688683480024338,
-          0.14657628536224365,
-          -0.782435953617096,
-          0.2962816655635834,
-          0.6965363621711731,
-          0.8496337532997131,
-          -0.3042965829372406,
-          0.04343798756599426,
-          0.0330701619386673,
-          -0.5662598013877869,
-          1.1086925268173218,
-          0.756072998046875,
-          -0.204134538769722,
-          0.2404300570487976,
-          -0.47848284244537354,
-          1.3659011125564575,
-          0.5645433068275452,
-          -0.15836156904697418,
-          0.43395575881004333,
-          0.5944653749465942,
-          1.0043466091156006,
-          -0.49446743726730347,
-          -0.5954391360282898,
-          0.5341240763664246,
-          0.020598189905285835,
-          -0.4036853015422821,
-          0.4473709762096405,
-          1.1998231410980225,
-          -0.9317775368690492,
-          -0.23321466147899628,
-          0.2052552700042725,
-          -0.7423108816146851,
-          -0.19917210936546328,
-          -0.1722569614648819,
-          -0.034072667360305786,
-          -0.00671181408688426,
-          0.46396249532699585,
-          -0.1372445821762085,
-          0.053376372903585434,
-          0.7392690777778625,
-          -0.38447609543800354,
-          0.07497968524694443,
-          0.5197252631187439,
-          1.3746477365493774,
-          0.9060075879096984,
-          0.20000585913658145,
-          -0.4053704142570496,
-          0.7497360110282898,
-          -0.34087055921554565,
-          -1.101803183555603,
-          0.273650586605072,
-          -0.5125769376754761,
-          0.22472351789474487,
-          0.480757474899292,
-          -0.19845178723335263,
-          0.8857700824737549,
-          0.30752456188201904,
-          1.1109285354614258,
-          -0.6768012642860413,
-          0.524367094039917,
-          -0.22495046257972717,
-          -0.4224412739276886,
-          0.40753406286239624,
-          -0.23133376240730288,
-          0.3297771215438843,
-          0.4905449151992798,
-          -0.6813114285469055,
-          -0.7543983459472656,
-          -0.5599071383476257,
-          0.14351597428321838,
-          -0.029278717935085297,
-          -0.3970443606376648,
-          -0.303079217672348,
-          0.24161772429943085,
-          0.008353390730917454,
-          -0.0062365154735744,
-          1.0824860334396362,
-          -0.3704061508178711,
-          -1.0337258577346802,
-          0.04638749733567238,
-          1.163011074066162,
-          -0.31737643480300903,
-          0.013986887410283089,
-          0.19223114848136905,
-          -0.2260770797729492,
-          -0.210910826921463,
-          -1.0191949605941772,
-          0.22356095910072327,
-          0.09353553503751756,
-          0.18096882104873657,
-          0.14867214858531952,
-          0.43408671021461487,
-          -0.33312076330184937,
-          0.8173948526382446,
-          0.6428242921829224,
-          0.20215003192424777,
-          -0.6634518504142761,
-          -0.4132290482521057,
-          0.29815030097961426,
-          -1.579406976699829,
-          -0.0981958732008934,
-          -0.03941014781594277,
-          0.1709178239107132,
-          -0.5481140613555908,
-          -0.5338194966316223,
-          -0.3528362512588501,
-          -0.11561278253793716,
-          -0.21793591976165771,
-          -1.1570470333099363,
-          0.2157980799674988,
-          0.42083489894866943,
-          0.9639263153076172,
-          0.09747201204299928,
-          0.15671424567699432,
-          0.4034591615200043,
-          0.6728067994117737,
-          -0.5216875672340393,
-          0.09657668322324751,
-          -0.2416689097881317,
-          0.747975766658783,
-          0.1021689772605896,
-          0.11652665585279463,
-          -1.0484966039657593,
-          0.8489304780960083,
-          0.7169828414916992,
-          -0.09012343734502792,
-          -1.3173753023147583,
-          0.057890523225069046,
-          -0.006231260951608419,
-          -0.1018214002251625,
-          0.936040461063385,
-          -0.0502331368625164,
-          -0.4284322261810303,
-          -0.38209280371665955,
-          -0.22668412327766416,
-          0.0782942995429039,
-          -0.4881664514541626,
-          0.9268959760665894,
-          0.001867273123934865,
-          0.42261114716529846,
-          0.8283362984657288,
-          0.4256294071674347,
-          -0.7965338826179504,
-          0.4840078353881836,
-          -0.19861412048339844,
-          0.33977967500686646,
-          -0.4604192078113556,
-          -0.3107339143753052,
-          -0.2839638590812683,
-          -1.5734281539916992,
-          0.005220232997089624,
-          0.09239906817674635,
-          -0.7828494906425476,
-          -0.1397123783826828,
-          0.2576255202293396,
-          0.21372435986995697,
-          -0.23169949650764465,
-          0.4016408920288086,
-          -0.462497353553772,
-          -0.2186472862958908,
-          -0.5617868900299072,
-          -0.3649831712245941,
-          -1.1585862636566162,
-          -0.08222806453704834,
-          0.931126832962036,
-          0.4327389597892761,
-          -0.46451422572135925,
-          -0.5430706143379211,
-          -0.27434298396110535,
-          -0.9479129314422609,
-          0.1845661848783493,
-          0.3972720205783844,
-          0.4883299469947815,
-          1.04031240940094
-        ]
-      ],
-      "regenerate": true
-    }
-  }
-}

crates/dump/src/reader/snapshots/dump__reader__test__import_dump_v6_with_vectors-9.snap

@@ -1,780 +0,0 @@
----
-source: dump/src/reader/mod.rs
-expression: document
----
-{
-  "id": "e0",
-  "desc": "overriden vector",
-  "_vectors": {
-    "default": [
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1,
-      0.1
-    ]
-  }
-}

crates/filter-parser/src/condition.rs

@@ -1,178 +0,0 @@
-//! BNF grammar:
-//!
-//! ```text
-//! condition      = value ("==" | ">" ...) value
-//! to             = value value TO value
-//! ```
-
-use nom::branch::alt;
-use nom::bytes::complete::tag;
-use nom::character::complete::multispace1;
-use nom::combinator::cut;
-use nom::sequence::{terminated, tuple};
-use Condition::*;
-
-use crate::{parse_value, FilterCondition, IResult, Span, Token};
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub enum Condition<'a> {
-    GreaterThan(Token<'a>),
-    GreaterThanOrEqual(Token<'a>),
-    Equal(Token<'a>),
-    NotEqual(Token<'a>),
-    Null,
-    Empty,
-    Exists,
-    LowerThan(Token<'a>),
-    LowerThanOrEqual(Token<'a>),
-    Between { from: Token<'a>, to: Token<'a> },
-    Contains { keyword: Token<'a>, word: Token<'a> },
-    StartsWith { keyword: Token<'a>, word: Token<'a> },
-}
-
-impl Condition<'_> {
-    pub fn operator(&self) -> &str {
-        match self {
-            Condition::GreaterThan(_) => ">",
-            Condition::GreaterThanOrEqual(_) => ">=",
-            Condition::Equal(_) => "=",
-            Condition::NotEqual(_) => "!=",
-            Condition::Null => "IS NULL",
-            Condition::Empty => "IS EMPTY",
-            Condition::Exists => "EXISTS",
-            Condition::LowerThan(_) => "<",
-            Condition::LowerThanOrEqual(_) => "<=",
-            Condition::Between { .. } => "TO",
-            Condition::Contains { .. } => "CONTAINS",
-            Condition::StartsWith { .. } => "STARTS WITH",
-        }
-    }
-}
-
-/// condition      = value ("==" | ">" ...) value
-pub fn parse_condition(input: Span) -> IResult<FilterCondition> {
-    let operator = alt((tag("<="), tag(">="), tag("!="), tag("<"), tag(">"), tag("=")));
-    let (input, (fid, op, value)) = tuple((parse_value, operator, cut(parse_value)))(input)?;
-
-    let condition = match *op.fragment() {
-        "<=" => FilterCondition::Condition { fid, op: LowerThanOrEqual(value) },
-        ">=" => FilterCondition::Condition { fid, op: GreaterThanOrEqual(value) },
-        "!=" => FilterCondition::Condition { fid, op: NotEqual(value) },
-        "<" => FilterCondition::Condition { fid, op: LowerThan(value) },
-        ">" => FilterCondition::Condition { fid, op: GreaterThan(value) },
-        "=" => FilterCondition::Condition { fid, op: Equal(value) },
-        _ => unreachable!(),
-    };
-
-    Ok((input, condition))
-}
-
-/// null          = value "IS" WS+ "NULL"
-pub fn parse_is_null(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("NULL")))(input)?;
-    Ok((input, FilterCondition::Condition { fid: key, op: Null }))
-}
-
-/// null          = value "IS" WS+ "NOT" WS+ "NULL"
-pub fn parse_is_not_null(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("NOT"), multispace1, tag("NULL")))(input)?;
-    Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Null }))))
-}
-
-/// empty          = value "IS" WS+ "EMPTY"
-pub fn parse_is_empty(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("EMPTY")))(input)?;
-    Ok((input, FilterCondition::Condition { fid: key, op: Empty }))
-}
-
-/// empty          = value "IS" WS+ "NOT" WS+ "EMPTY"
-pub fn parse_is_not_empty(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("IS"), multispace1, tag("NOT"), multispace1, tag("EMPTY")))(input)?;
-    Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Empty }))))
-}
-
-/// exist          = value "EXISTS"
-pub fn parse_exists(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = terminated(parse_value, tag("EXISTS"))(input)?;
-
-    Ok((input, FilterCondition::Condition { fid: key, op: Exists }))
-}
-/// exist          = value "NOT" WS+ "EXISTS"
-pub fn parse_not_exists(input: Span) -> IResult<FilterCondition> {
-    let (input, key) = parse_value(input)?;
-
-    let (input, _) = tuple((tag("NOT"), multispace1, tag("EXISTS")))(input)?;
-    Ok((input, FilterCondition::Not(Box::new(FilterCondition::Condition { fid: key, op: Exists }))))
-}
-
-/// contains        = value "CONTAINS" value
-pub fn parse_contains(input: Span) -> IResult<FilterCondition> {
-    let (input, (fid, contains, value)) =
-        tuple((parse_value, tag("CONTAINS"), cut(parse_value)))(input)?;
-    Ok((
-        input,
-        FilterCondition::Condition {
-            fid,
-            op: Contains { keyword: Token { span: contains, value: None }, word: value },
-        },
-    ))
-}
-
-/// contains        = value "NOT" WS+ "CONTAINS" value
-pub fn parse_not_contains(input: Span) -> IResult<FilterCondition> {
-    let keyword = tuple((tag("NOT"), multispace1, tag("CONTAINS")));
-    let (input, (fid, (_not, _spaces, contains), value)) =
-        tuple((parse_value, keyword, cut(parse_value)))(input)?;
-
-    Ok((
-        input,
-        FilterCondition::Not(Box::new(FilterCondition::Condition {
-            fid,
-            op: Contains { keyword: Token { span: contains, value: None }, word: value },
-        })),
-    ))
-}
-
-/// starts with        = value "CONTAINS" value
-pub fn parse_starts_with(input: Span) -> IResult<FilterCondition> {
-    let (input, (fid, starts_with, value)) =
-        tuple((parse_value, tag("STARTS WITH"), cut(parse_value)))(input)?;
-    Ok((
-        input,
-        FilterCondition::Condition {
-            fid,
-            op: StartsWith { keyword: Token { span: starts_with, value: None }, word: value },
-        },
-    ))
-}
-
-/// starts with        = value "NOT" WS+ "CONTAINS" value
-pub fn parse_not_starts_with(input: Span) -> IResult<FilterCondition> {
-    let keyword = tuple((tag("NOT"), multispace1, tag("STARTS WITH")));
-    let (input, (fid, (_not, _spaces, starts_with), value)) =
-        tuple((parse_value, keyword, cut(parse_value)))(input)?;
-
-    Ok((
-        input,
-        FilterCondition::Not(Box::new(FilterCondition::Condition {
-            fid,
-            op: StartsWith { keyword: Token { span: starts_with, value: None }, word: value },
-        })),
-    ))
-}
-
-/// to             = value value "TO" WS+ value
-pub fn parse_to(input: Span) -> IResult<FilterCondition> {
-    let (input, (key, from, _, _, to)) =
-        tuple((parse_value, parse_value, tag("TO"), multispace1, cut(parse_value)))(input)?;
-
-    Ok((input, FilterCondition::Condition { fid: key, op: Between { from, to } }))
-}

crates/fuzzers/Cargo.toml

@@ -1,22 +0,0 @@
-[package]
-name = "fuzzers"
-publish = false
-
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-homepage.workspace = true
-readme.workspace = true
-edition.workspace = true
-license.workspace = true
-
-[dependencies]
-arbitrary = { version = "1.4.1", features = ["derive"] }
-bumpalo = "3.16.0"
-clap = { version = "4.5.24", features = ["derive"] }
-either = "1.13.0"
-fastrand = "2.3.0"
-milli = { path = "../milli" }
-serde = { version = "1.0.217", features = ["derive"] }
-serde_json = { version = "1.0.135", features = ["preserve_order"] }
-tempfile = "3.15.0"

crates/fuzzers/README.md

@@ -1,3 +0,0 @@
-# Fuzzers
-
-The purpose of this crate is to contains all the handmade "fuzzer" we may need.

crates/fuzzers/src/bin/fuzz-indexing.rs

@@ -1,191 +0,0 @@
-use std::num::NonZeroUsize;
-use std::path::PathBuf;
-use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
-use std::time::Duration;
-
-use arbitrary::{Arbitrary, Unstructured};
-use bumpalo::Bump;
-use clap::Parser;
-use either::Either;
-use fuzzers::Operation;
-use milli::documents::mmap_from_objects;
-use milli::heed::EnvOpenOptions;
-use milli::progress::Progress;
-use milli::update::new::indexer;
-use milli::update::IndexerConfig;
-use milli::vector::EmbeddingConfigs;
-use milli::Index;
-use serde_json::Value;
-use tempfile::TempDir;
-
-#[derive(Debug, Arbitrary)]
-struct Batch([Operation; 5]);
-
-#[derive(Debug, Clone, Parser)]
-struct Opt {
-    /// The number of fuzzer to run in parallel.
-    #[clap(long)]
-    par: Option<NonZeroUsize>,
-    // We need to put a lot of newlines in the following documentation or else everything gets collapsed on one line
-    /// The path in which the databases will be created.
-    /// Using a ramdisk is recommended.
-    ///
-    /// Linux:
-    ///
-    /// sudo mount -t tmpfs -o size=2g tmpfs ramdisk # to create it
-    ///
-    /// sudo umount ramdisk # to remove it
-    ///
-    /// MacOS:
-    ///
-    /// diskutil erasevolume HFS+ 'RAM Disk' `hdiutil attach -nobrowse -nomount ram://4194304 # create it
-    ///
-    /// hdiutil detach /dev/:the_disk
-    #[clap(long)]
-    path: Option<PathBuf>,
-}
-
-fn main() {
-    let opt = Opt::parse();
-    let progression: &'static AtomicUsize = Box::leak(Box::new(AtomicUsize::new(0)));
-    let stop: &'static AtomicBool = Box::leak(Box::new(AtomicBool::new(false)));
-
-    let par = opt.par.unwrap_or_else(|| std::thread::available_parallelism().unwrap()).get();
-    let mut handles = Vec::with_capacity(par);
-
-    for _ in 0..par {
-        let opt = opt.clone();
-
-        let handle = std::thread::spawn(move || {
-            let options = EnvOpenOptions::new();
-            let mut options = options.read_txn_without_tls();
-            options.map_size(1024 * 1024 * 1024 * 1024);
-            let tempdir = match opt.path {
-                Some(path) => TempDir::new_in(path).unwrap(),
-                None => TempDir::new().unwrap(),
-            };
-            let index = Index::new(options, tempdir.path(), true).unwrap();
-            let indexer_config = IndexerConfig::default();
-
-            std::thread::scope(|s| {
-                loop {
-                    if stop.load(Ordering::Relaxed) {
-                        return;
-                    }
-                    let v: Vec<u8> =
-                        std::iter::repeat_with(|| fastrand::u8(..)).take(1000).collect();
-
-                    let mut data = Unstructured::new(&v);
-                    let batches = <[Batch; 5]>::arbitrary(&mut data).unwrap();
-                    // will be used to display the error once a thread crashes
-                    let dbg_input = format!("{:#?}", batches);
-
-                    let handle = s.spawn(|| {
-                        let mut wtxn = index.write_txn().unwrap();
-                        let rtxn = index.read_txn().unwrap();
-
-                        for batch in batches {
-                            let db_fields_ids_map = index.fields_ids_map(&rtxn).unwrap();
-                            let mut new_fields_ids_map = db_fields_ids_map.clone();
-
-                            let indexer_alloc = Bump::new();
-                            let embedders = EmbeddingConfigs::default();
-                            let mut indexer = indexer::DocumentOperation::new();
-
-                            let mut operations = Vec::new();
-                            for op in batch.0 {
-                                match op {
-                                    Operation::AddDoc(doc) => {
-                                        let object = match doc.to_d() {
-                                            Value::Object(object) => object,
-                                            _ => unreachable!(),
-                                        };
-                                        let documents = mmap_from_objects(vec![object]);
-                                        operations.push(Either::Left(documents));
-                                    }
-                                    Operation::DeleteDoc(id) => {
-                                        let id = indexer_alloc.alloc_str(&id.to_s());
-                                        let ids = indexer_alloc.alloc_slice_copy(&[&*id]);
-                                        operations.push(Either::Right(ids));
-                                    }
-                                }
-                            }
-
-                            for op in &operations {
-                                match op {
-                                    Either::Left(documents) => {
-                                        indexer.replace_documents(documents).unwrap()
-                                    }
-                                    Either::Right(ids) => indexer.delete_documents(ids),
-                                }
-                            }
-
-                            let (document_changes, _operation_stats, primary_key) = indexer
-                                .into_changes(
-                                    &indexer_alloc,
-                                    &index,
-                                    &rtxn,
-                                    None,
-                                    &mut new_fields_ids_map,
-                                    &|| false,
-                                    Progress::default(),
-                                )
-                                .unwrap();
-
-                            indexer::index(
-                                &mut wtxn,
-                                &index,
-                                &milli::ThreadPoolNoAbortBuilder::new().build().unwrap(),
-                                indexer_config.grenad_parameters(),
-                                &db_fields_ids_map,
-                                new_fields_ids_map,
-                                primary_key,
-                                &document_changes,
-                                embedders,
-                                &|| false,
-                                &Progress::default(),
-                            )
-                            .unwrap();
-
-                            // after executing a batch we check if the database is corrupted
-                            let res = index.search(&wtxn).execute().unwrap();
-                            index.documents(&wtxn, res.documents_ids).unwrap();
-                            progression.fetch_add(1, Ordering::Relaxed);
-                        }
-                        wtxn.abort();
-                    });
-                    if let err @ Err(_) = handle.join() {
-                        stop.store(true, Ordering::Relaxed);
-                        err.expect(&dbg_input);
-                    }
-                }
-            });
-        });
-        handles.push(handle);
-    }
-
-    std::thread::spawn(|| {
-        let mut last_value = 0;
-        let start = std::time::Instant::now();
-        loop {
-            let total = progression.load(Ordering::Relaxed);
-            let elapsed = start.elapsed().as_secs();
-            if elapsed > 3600 {
-                // after 1 hour, stop the fuzzer, success
-                std::process::exit(0);
-            }
-            println!(
-                "Has been running for {:?} seconds. Tested {} new values for a total of {}.",
-                elapsed,
-                total - last_value,
-                total
-            );
-            last_value = total;
-            std::thread::sleep(Duration::from_secs(1));
-        }
-    });
-
-    for handle in handles {
-        handle.join().unwrap();
-    }
-}

crates/fuzzers/src/lib.rs

@@ -1,46 +0,0 @@
-use arbitrary::Arbitrary;
-use serde_json::{json, Value};
-
-#[derive(Debug, Arbitrary)]
-pub enum Document {
-    One,
-    Two,
-    Three,
-    Four,
-    Five,
-    Six,
-}
-
-impl Document {
-    pub fn to_d(&self) -> Value {
-        match self {
-            Document::One => json!({ "id": 0, "doggo": "bernese" }),
-            Document::Two => json!({ "id": 0, "doggo": "golden" }),
-            Document::Three => json!({ "id": 0, "catto": "jorts" }),
-            Document::Four => json!({ "id": 1, "doggo": "bernese" }),
-            Document::Five => json!({ "id": 1, "doggo": "golden" }),
-            Document::Six => json!({ "id": 1, "catto": "jorts" }),
-        }
-    }
-}
-
-#[derive(Debug, Arbitrary)]
-pub enum DocId {
-    Zero,
-    One,
-}
-
-impl DocId {
-    pub fn to_s(&self) -> String {
-        match self {
-            DocId::Zero => "0".to_string(),
-            DocId::One => "1".to_string(),
-        }
-    }
-}
-
-#[derive(Debug, Arbitrary)]
-pub enum Operation {
-    AddDoc(Document),
-    DeleteDoc(DocId),
-}

crates/index-scheduler/Cargo.toml

@@ -1,54 +0,0 @@
-[package]
-name = "index-scheduler"
-publish = false
-
-version.workspace = true
-authors.workspace = true
-description.workspace = true
-homepage.workspace = true
-readme.workspace = true
-edition.workspace = true
-license.workspace = true
-
-[dependencies]
-anyhow = "1.0.95"
-bincode = "1.3.3"
-byte-unit = "5.1.6"
-bumpalo = "3.16.0"
-bumparaw-collections = "0.1.4"
-convert_case = "0.6.0"
-csv = "1.3.1"
-derive_builder = "0.20.2"
-dump = { path = "../dump" }
-enum-iterator = "2.1.0"
-file-store = { path = "../file-store" }
-flate2 = "1.0.35"
-indexmap = "2.7.0"
-meilisearch-auth = { path = "../meilisearch-auth" }
-meilisearch-types = { path = "../meilisearch-types" }
-memmap2 = "0.9.5"
-page_size = "0.6.0"
-rayon = "1.10.0"
-roaring = { version = "0.10.10", features = ["serde"] }
-serde = { version = "1.0.217", features = ["derive"] }
-serde_json = { version = "1.0.138", features = ["preserve_order"] }
-synchronoise = "1.0.1"
-tempfile = "3.15.0"
-thiserror = "2.0.9"
-time = { version = "0.3.37", features = [
-    "serde-well-known",
-    "formatting",
-    "parsing",
-    "macros",
-] }
-tracing = "0.1.41"
-ureq = "2.12.1"
-uuid = { version = "1.11.0", features = ["serde", "v4"] }
-
-[dev-dependencies]
-big_s = "1.0.2"
-crossbeam-channel = "0.5.14"
-# fixed version due to format breakages in v1.40
-insta = { version = "=1.39.0", features = ["json", "redactions"] }
-maplit = "1.0.2"
-meili-snap = { path = "../meili-snap" }

crates/index-scheduler/src/dump.rs

@@ -1,304 +0,0 @@
-use std::collections::HashMap;
-use std::io;
-
-use dump::{KindDump, TaskDump, UpdateFile};
-use meilisearch_types::batches::{Batch, BatchId};
-use meilisearch_types::heed::RwTxn;
-use meilisearch_types::milli;
-use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
-use roaring::RoaringBitmap;
-use uuid::Uuid;
-
-use crate::{utils, Error, IndexScheduler, Result};
-
-pub struct Dump<'a> {
-    index_scheduler: &'a IndexScheduler,
-    wtxn: RwTxn<'a>,
-
-    batch_to_task_mapping: HashMap<BatchId, RoaringBitmap>,
-
-    indexes: HashMap<String, RoaringBitmap>,
-    statuses: HashMap<Status, RoaringBitmap>,
-    kinds: HashMap<Kind, RoaringBitmap>,
-
-    batch_indexes: HashMap<String, RoaringBitmap>,
-    batch_statuses: HashMap<Status, RoaringBitmap>,
-    batch_kinds: HashMap<Kind, RoaringBitmap>,
-}
-
-impl<'a> Dump<'a> {
-    pub(crate) fn new(index_scheduler: &'a mut IndexScheduler) -> Result<Self> {
-        // While loading a dump no one should be able to access the scheduler thus I can block everything.
-        let wtxn = index_scheduler.env.write_txn()?;
-
-        Ok(Dump {
-            index_scheduler,
-            wtxn,
-            batch_to_task_mapping: HashMap::new(),
-            indexes: HashMap::new(),
-            statuses: HashMap::new(),
-            kinds: HashMap::new(),
-            batch_indexes: HashMap::new(),
-            batch_statuses: HashMap::new(),
-            batch_kinds: HashMap::new(),
-        })
-    }
-
-    /// Register a new batch coming from a dump in the scheduler.
-    /// By taking a mutable ref we're pretty sure no one will ever import a dump while actix is running.
-    pub fn register_dumped_batch(&mut self, batch: Batch) -> Result<()> {
-        self.index_scheduler.queue.batches.all_batches.put(&mut self.wtxn, &batch.uid, &batch)?;
-        if let Some(enqueued_at) = batch.enqueued_at {
-            utils::insert_task_datetime(
-                &mut self.wtxn,
-                self.index_scheduler.queue.batches.enqueued_at,
-                enqueued_at.earliest,
-                batch.uid,
-            )?;
-            utils::insert_task_datetime(
-                &mut self.wtxn,
-                self.index_scheduler.queue.batches.enqueued_at,
-                enqueued_at.oldest,
-                batch.uid,
-            )?;
-        }
-        utils::insert_task_datetime(
-            &mut self.wtxn,
-            self.index_scheduler.queue.batches.started_at,
-            batch.started_at,
-            batch.uid,
-        )?;
-        if let Some(finished_at) = batch.finished_at {
-            utils::insert_task_datetime(
-                &mut self.wtxn,
-                self.index_scheduler.queue.batches.finished_at,
-                finished_at,
-                batch.uid,
-            )?;
-        }
-
-        for index in batch.stats.index_uids.keys() {
-            match self.batch_indexes.get_mut(index) {
-                Some(bitmap) => {
-                    bitmap.insert(batch.uid);
-                }
-                None => {
-                    let mut bitmap = RoaringBitmap::new();
-                    bitmap.insert(batch.uid);
-                    self.batch_indexes.insert(index.to_string(), bitmap);
-                }
-            };
-        }
-
-        for status in batch.stats.status.keys() {
-            self.batch_statuses.entry(*status).or_default().insert(batch.uid);
-        }
-        for kind in batch.stats.types.keys() {
-            self.batch_kinds.entry(*kind).or_default().insert(batch.uid);
-        }
-
-        Ok(())
-    }
-
-    /// Register a new task coming from a dump in the scheduler.
-    /// By taking a mutable ref we're pretty sure no one will ever import a dump while actix is running.
-    pub fn register_dumped_task(
-        &mut self,
-        task: TaskDump,
-        content_file: Option<Box<UpdateFile>>,
-    ) -> Result<Task> {
-        let task_has_no_docs = matches!(task.kind, KindDump::DocumentImport { documents_count, .. } if documents_count == 0);
-
-        let content_uuid = match content_file {
-            Some(content_file) if task.status == Status::Enqueued => {
-                let (uuid, file) = self.index_scheduler.queue.create_update_file(false)?;
-                let mut writer = io::BufWriter::new(file);
-                for doc in content_file {
-                    let doc = doc?;
-                    serde_json::to_writer(&mut writer, &doc).map_err(|e| {
-                        Error::from_milli(milli::InternalError::SerdeJson(e).into(), None)
-                    })?;
-                }
-                let file = writer.into_inner().map_err(|e| e.into_error())?;
-                file.persist()?;
-
-                Some(uuid)
-            }
-            // If the task isn't `Enqueued` then just generate a recognisable `Uuid`
-            // in case we try to open it later.
-            _ if task.status != Status::Enqueued => Some(Uuid::nil()),
-            None if task.status == Status::Enqueued && task_has_no_docs => {
-                let (uuid, file) = self.index_scheduler.queue.create_update_file(false)?;
-                file.persist()?;
-
-                Some(uuid)
-            }
-            _ => None,
-        };
-
-        let task = Task {
-            uid: task.uid,
-            batch_uid: task.batch_uid,
-            enqueued_at: task.enqueued_at,
-            started_at: task.started_at,
-            finished_at: task.finished_at,
-            error: task.error,
-            canceled_by: task.canceled_by,
-            details: task.details,
-            status: task.status,
-            kind: match task.kind {
-                KindDump::DocumentImport {
-                    primary_key,
-                    method,
-                    documents_count,
-                    allow_index_creation,
-                } => KindWithContent::DocumentAdditionOrUpdate {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    primary_key,
-                    method,
-                    content_file: content_uuid.ok_or(Error::CorruptedDump)?,
-                    documents_count,
-                    allow_index_creation,
-                },
-                KindDump::DocumentDeletion { documents_ids } => KindWithContent::DocumentDeletion {
-                    documents_ids,
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                },
-                KindDump::DocumentDeletionByFilter { filter } => {
-                    KindWithContent::DocumentDeletionByFilter {
-                        filter_expr: filter,
-                        index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    }
-                }
-                KindDump::DocumentEdition { filter, context, function } => {
-                    KindWithContent::DocumentEdition {
-                        index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                        filter_expr: filter,
-                        context,
-                        function,
-                    }
-                }
-                KindDump::DocumentClear => KindWithContent::DocumentClear {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                },
-                KindDump::Settings { settings, is_deletion, allow_index_creation } => {
-                    KindWithContent::SettingsUpdate {
-                        index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                        new_settings: settings,
-                        is_deletion,
-                        allow_index_creation,
-                    }
-                }
-                KindDump::IndexDeletion => KindWithContent::IndexDeletion {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                },
-                KindDump::IndexCreation { primary_key } => KindWithContent::IndexCreation {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    primary_key,
-                },
-                KindDump::IndexUpdate { primary_key } => KindWithContent::IndexUpdate {
-                    index_uid: task.index_uid.ok_or(Error::CorruptedDump)?,
-                    primary_key,
-                },
-                KindDump::IndexSwap { swaps } => KindWithContent::IndexSwap { swaps },
-                KindDump::TaskCancelation { query, tasks } => {
-                    KindWithContent::TaskCancelation { query, tasks }
-                }
-                KindDump::TasksDeletion { query, tasks } => {
-                    KindWithContent::TaskDeletion { query, tasks }
-                }
-                KindDump::DumpCreation { keys, instance_uid } => {
-                    KindWithContent::DumpCreation { keys, instance_uid }
-                }
-                KindDump::SnapshotCreation => KindWithContent::SnapshotCreation,
-                KindDump::UpgradeDatabase { from } => KindWithContent::UpgradeDatabase { from },
-            },
-        };
-
-        self.index_scheduler.queue.tasks.all_tasks.put(&mut self.wtxn, &task.uid, &task)?;
-        if let Some(batch_id) = task.batch_uid {
-            self.batch_to_task_mapping.entry(batch_id).or_default().insert(task.uid);
-        }
-
-        for index in task.indexes() {
-            match self.indexes.get_mut(index) {
-                Some(bitmap) => {
-                    bitmap.insert(task.uid);
-                }
-                None => {
-                    let mut bitmap = RoaringBitmap::new();
-                    bitmap.insert(task.uid);
-                    self.indexes.insert(index.to_string(), bitmap);
-                }
-            };
-        }
-
-        utils::insert_task_datetime(
-            &mut self.wtxn,
-            self.index_scheduler.queue.tasks.enqueued_at,
-            task.enqueued_at,
-            task.uid,
-        )?;
-
-        // we can't override the started_at & finished_at, so we must only set it if the tasks is finished and won't change
-        if matches!(task.status, Status::Succeeded | Status::Failed | Status::Canceled) {
-            if let Some(started_at) = task.started_at {
-                utils::insert_task_datetime(
-                    &mut self.wtxn,
-                    self.index_scheduler.queue.tasks.started_at,
-                    started_at,
-                    task.uid,
-                )?;
-            }
-            if let Some(finished_at) = task.finished_at {
-                utils::insert_task_datetime(
-                    &mut self.wtxn,
-                    self.index_scheduler.queue.tasks.finished_at,
-                    finished_at,
-                    task.uid,
-                )?;
-            }
-        }
-
-        self.statuses.entry(task.status).or_default().insert(task.uid);
-        self.kinds.entry(task.kind.as_kind()).or_default().insert(task.uid);
-
-        Ok(task)
-    }
-
-    /// Commit all the changes and exit the importing dump state
-    pub fn finish(mut self) -> Result<()> {
-        for (batch_id, task_ids) in self.batch_to_task_mapping {
-            self.index_scheduler.queue.batch_to_tasks_mapping.put(
-                &mut self.wtxn,
-                &batch_id,
-                &task_ids,
-            )?;
-        }
-
-        for (index, bitmap) in self.indexes {
-            self.index_scheduler.queue.tasks.index_tasks.put(&mut self.wtxn, &index, &bitmap)?;
-        }
-        for (status, bitmap) in self.statuses {
-            self.index_scheduler.queue.tasks.put_status(&mut self.wtxn, status, &bitmap)?;
-        }
-        for (kind, bitmap) in self.kinds {
-            self.index_scheduler.queue.tasks.put_kind(&mut self.wtxn, kind, &bitmap)?;
-        }
-
-        for (index, bitmap) in self.batch_indexes {
-            self.index_scheduler.queue.batches.index_tasks.put(&mut self.wtxn, &index, &bitmap)?;
-        }
-        for (status, bitmap) in self.batch_statuses {
-            self.index_scheduler.queue.batches.put_status(&mut self.wtxn, status, &bitmap)?;
-        }
-        for (kind, bitmap) in self.batch_kinds {
-            self.index_scheduler.queue.batches.put_kind(&mut self.wtxn, kind, &bitmap)?;
-        }
-
-        self.wtxn.commit()?;
-        self.index_scheduler.scheduler.wake_up.signal();
-
-        Ok(())
-    }
-}

crates/index-scheduler/src/features.rs

@@ -1,212 +0,0 @@
-use std::sync::{Arc, RwLock};
-
-use meilisearch_types::features::{InstanceTogglableFeatures, Network, RuntimeTogglableFeatures};
-use meilisearch_types::heed::types::{SerdeJson, Str};
-use meilisearch_types::heed::{Database, Env, RwTxn, WithoutTls};
-
-use crate::error::FeatureNotEnabledError;
-use crate::Result;
-
-/// The number of database used by features
-const NUMBER_OF_DATABASES: u32 = 1;
-/// Database const names for the `FeatureData`.
-mod db_name {
-    pub const EXPERIMENTAL_FEATURES: &str = "experimental-features";
-}
-
-mod db_keys {
-    pub const EXPERIMENTAL_FEATURES: &str = "experimental-features";
-    pub const NETWORK: &str = "network";
-}
-
-#[derive(Clone)]
-pub(crate) struct FeatureData {
-    persisted: Database<Str, SerdeJson<RuntimeTogglableFeatures>>,
-    runtime: Arc<RwLock<RuntimeTogglableFeatures>>,
-    network: Arc<RwLock<Network>>,
-}
-
-#[derive(Debug, Clone, Copy)]
-pub struct RoFeatures {
-    runtime: RuntimeTogglableFeatures,
-}
-
-impl RoFeatures {
-    fn new(data: &FeatureData) -> Self {
-        let runtime = data.runtime_features();
-        Self { runtime }
-    }
-
-    pub fn runtime_features(&self) -> RuntimeTogglableFeatures {
-        self.runtime
-    }
-
-    pub fn check_metrics(&self) -> Result<()> {
-        if self.runtime.metrics {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Getting metrics",
-                feature: "metrics",
-                issue_link: "https://github.com/meilisearch/product/discussions/625",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_logs_route(&self) -> Result<()> {
-        if self.runtime.logs_route {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Modifying logs through the `/logs/*` routes",
-                feature: "logs route",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/721",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_edit_documents_by_function(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.edit_documents_by_function {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "edit documents by function",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/762",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_contains_filter(&self) -> Result<()> {
-        if self.runtime.contains_filter {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Using `CONTAINS` or `STARTS WITH` in a filter",
-                feature: "contains filter",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/763",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_network(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.network {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "network",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/805",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_get_task_documents_route(&self) -> Result<()> {
-        if self.runtime.get_task_documents_route {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action: "Getting the documents of an enqueued task",
-                feature: "get task documents route",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/808",
-            }
-            .into())
-        }
-    }
-
-    pub fn check_composite_embedders(&self, disabled_action: &'static str) -> Result<()> {
-        if self.runtime.composite_embedders {
-            Ok(())
-        } else {
-            Err(FeatureNotEnabledError {
-                disabled_action,
-                feature: "composite embedders",
-                issue_link: "https://github.com/orgs/meilisearch/discussions/816",
-            }
-            .into())
-        }
-    }
-}
-
-impl FeatureData {
-    pub(crate) const fn nb_db() -> u32 {
-        NUMBER_OF_DATABASES
-    }
-
-    pub fn new(
-        env: &Env<WithoutTls>,
-        wtxn: &mut RwTxn,
-        instance_features: InstanceTogglableFeatures,
-    ) -> Result<Self> {
-        let runtime_features_db =
-            env.create_database(wtxn, Some(db_name::EXPERIMENTAL_FEATURES))?;
-
-        let persisted_features: RuntimeTogglableFeatures =
-            runtime_features_db.get(wtxn, db_keys::EXPERIMENTAL_FEATURES)?.unwrap_or_default();
-        let InstanceTogglableFeatures { metrics, logs_route, contains_filter } = instance_features;
-        let runtime = Arc::new(RwLock::new(RuntimeTogglableFeatures {
-            metrics: metrics || persisted_features.metrics,
-            logs_route: logs_route || persisted_features.logs_route,
-            contains_filter: contains_filter || persisted_features.contains_filter,
-            ..persisted_features
-        }));
-
-        let network_db = runtime_features_db.remap_data_type::<SerdeJson<Network>>();
-        let network: Network = network_db.get(wtxn, db_keys::NETWORK)?.unwrap_or_default();
-
-        Ok(Self {
-            persisted: runtime_features_db,
-            runtime,
-            network: Arc::new(RwLock::new(network)),
-        })
-    }
-
-    pub fn put_runtime_features(
-        &self,
-        mut wtxn: RwTxn,
-        features: RuntimeTogglableFeatures,
-    ) -> Result<()> {
-        self.persisted.put(&mut wtxn, db_keys::EXPERIMENTAL_FEATURES, &features)?;
-        wtxn.commit()?;
-
-        // safe to unwrap, the lock will only fail if:
-        // 1. requested by the same thread concurrently -> it is called and released in methods that don't call each other
-        // 2. there's a panic while the thread is held -> it is only used for an assignment here.
-        let mut toggled_features = self.runtime.write().unwrap();
-        *toggled_features = features;
-        Ok(())
-    }
-
-    fn runtime_features(&self) -> RuntimeTogglableFeatures {
-        // sound to unwrap, the lock will only fail if:
-        // 1. requested by the same thread concurrently -> it is called and released in methods that don't call each other
-        // 2. there's a panic while the thread is held -> it is only used for copying the data here
-        *self.runtime.read().unwrap()
-    }
-
-    pub fn features(&self) -> RoFeatures {
-        RoFeatures::new(self)
-    }
-
-    pub fn put_network(&self, mut wtxn: RwTxn, new_network: Network) -> Result<()> {
-        self.persisted.remap_data_type::<SerdeJson<Network>>().put(
-            &mut wtxn,
-            db_keys::NETWORK,
-            &new_network,
-        )?;
-        wtxn.commit()?;
-
-        let mut network = self.network.write().unwrap();
-        *network = new_network;
-        Ok(())
-    }
-
-    pub fn network(&self) -> Network {
-        Network::clone(&*self.network.read().unwrap())
-    }
-}

crates/index-scheduler/src/index_mapper/index_map.rs

@@ -1,405 +0,0 @@
-use std::collections::BTreeMap;
-use std::env::VarError;
-use std::path::Path;
-use std::str::FromStr;
-use std::time::Duration;
-
-use meilisearch_types::heed::{EnvClosingEvent, EnvFlags, EnvOpenOptions};
-use meilisearch_types::milli::{Index, Result};
-use time::OffsetDateTime;
-use uuid::Uuid;
-
-use super::IndexStatus::{self, Available, BeingDeleted, Closing, Missing};
-use crate::clamp_to_page_size;
-use crate::lru::{InsertionOutcome, LruMap};
-/// Keep an internally consistent view of the open indexes in memory.
-///
-/// This view is made of an LRU cache that will evict the least frequently used indexes when new indexes are opened.
-/// Indexes that are being closed (for resizing or due to cache eviction) or deleted cannot be evicted from the cache and
-/// are stored separately.
-///
-/// This view provides operations to change the state of the index as it is known in memory:
-/// open an index (making it available for queries), close an index (specifying the new size it should be opened with),
-/// delete an index.
-///
-/// External consistency with the other bits of data of an index is provided by the `IndexMapper` parent structure.
-pub struct IndexMap {
-    /// A LRU map of indexes that are in the open state and available for queries.
-    available: LruMap<Uuid, Index>,
-    /// A map of indexes that are not available for queries, either because they are being deleted
-    /// or because they are being closed.
-    ///
-    /// If they are being deleted, the UUID points to `None`.
-    unavailable: BTreeMap<Uuid, Option<ClosingIndex>>,
-
-    /// A monotonically increasing generation number, used to differentiate between multiple successive index closing requests.
-    ///
-    /// Because multiple readers could be waiting on an index to close, the following could theoretically happen:
-    ///
-    /// 1. Multiple readers wait for the index closing to occur.
-    /// 2. One of them "wins the race", takes the lock and then removes the index that finished closing from the map.
-    /// 3. The index is reopened, but must be closed again (such as being resized again).
-    /// 4. One reader that "lost the race" in (2) wakes up and tries to take the lock and remove the index from the map.
-    ///
-    /// In that situation, the index may or may not have finished closing. The `generation` field allows to remember which
-    /// closing request was made, so the reader that "lost the race" has the old generation and will need to wait again for the index
-    /// to close.
-    generation: usize,
-}
-
-#[derive(Clone)]
-pub struct ClosingIndex {
-    uuid: Uuid,
-    closing_event: EnvClosingEvent,
-    enable_mdb_writemap: bool,
-    map_size: usize,
-    generation: usize,
-}
-
-impl ClosingIndex {
-    /// Waits for the index to be definitely closed.
-    ///
-    /// To avoid blocking, users should relinquish their locks to the IndexMap before calling this function.
-    ///
-    /// After the index is physically closed, the in memory map must still be updated to take this into account.
-    /// To do so, a `ReopenableIndex` is returned, that can be used to either definitely close or definitely open
-    /// the index without waiting anymore.
-    pub fn wait_timeout(self, timeout: Duration) -> Option<ReopenableIndex> {
-        self.closing_event.wait_timeout(timeout).then_some(ReopenableIndex {
-            uuid: self.uuid,
-            enable_mdb_writemap: self.enable_mdb_writemap,
-            map_size: self.map_size,
-            generation: self.generation,
-        })
-    }
-}
-
-pub struct ReopenableIndex {
-    uuid: Uuid,
-    enable_mdb_writemap: bool,
-    map_size: usize,
-    generation: usize,
-}
-
-impl ReopenableIndex {
-    /// Attempts to reopen the index, which can result in the index being reopened again or not
-    /// (e.g. if another thread already opened and closed the index again).
-    ///
-    /// Use get again on the IndexMap to get the updated status.
-    ///
-    /// Fails if the underlying index creation fails.
-    ///
-    /// # Status table
-    ///
-    /// | Previous Status | New Status                                   |
-    /// |-----------------|----------------------------------------------|
-    /// | Missing         | Missing                                      |
-    /// | BeingDeleted    | BeingDeleted                                 |
-    /// | Closing         | Available or Closing depending on generation |
-    /// | Available       | Available                                    |
-    ///
-    pub fn reopen(self, map: &mut IndexMap, path: &Path) -> Result<()> {
-        if let Closing(reopen) = map.get(&self.uuid) {
-            if reopen.generation != self.generation {
-                return Ok(());
-            }
-            map.unavailable.remove(&self.uuid);
-            map.create(&self.uuid, path, None, self.enable_mdb_writemap, self.map_size, false)?;
-        }
-        Ok(())
-    }
-
-    /// Attempts to close the index, which may or may not result in the index being closed
-    /// (e.g. if another thread already reopened the index again).
-    ///
-    /// Use get again on the IndexMap to get the updated status.
-    ///
-    /// # Status table
-    ///
-    /// | Previous Status | New Status                                 |
-    /// |-----------------|--------------------------------------------|
-    /// | Missing         | Missing                                    |
-    /// | BeingDeleted    | BeingDeleted                               |
-    /// | Closing         | Missing or Closing depending on generation |
-    /// | Available       | Available                                  |
-    pub fn close(self, map: &mut IndexMap) {
-        if let Closing(reopen) = map.get(&self.uuid) {
-            if reopen.generation != self.generation {
-                return;
-            }
-            map.unavailable.remove(&self.uuid);
-        }
-    }
-}
-
-impl IndexMap {
-    pub fn new(cap: usize) -> IndexMap {
-        Self { unavailable: Default::default(), available: LruMap::new(cap), generation: 0 }
-    }
-
-    /// Gets the current status of an index in the map.
-    ///
-    /// If the index is available it can be accessed from the returned status.
-    pub fn get(&self, uuid: &Uuid) -> IndexStatus {
-        self.available
-            .get(uuid)
-            .map(|index| Available(index.clone()))
-            .unwrap_or_else(|| self.get_unavailable(uuid))
-    }
-
-    fn get_unavailable(&self, uuid: &Uuid) -> IndexStatus {
-        match self.unavailable.get(uuid) {
-            Some(Some(reopen)) => Closing(reopen.clone()),
-            Some(None) => BeingDeleted,
-            None => Missing,
-        }
-    }
-
-    /// Attempts to create a new index that wasn't existing before.
-    ///
-    /// # Status table
-    ///
-    /// | Previous Status | New Status |
-    /// |-----------------|------------|
-    /// | Missing         | Available  |
-    /// | BeingDeleted    | panics     |
-    /// | Closing         | panics     |
-    /// | Available       | panics     |
-    ///
-    pub fn create(
-        &mut self,
-        uuid: &Uuid,
-        path: &Path,
-        date: Option<(OffsetDateTime, OffsetDateTime)>,
-        enable_mdb_writemap: bool,
-        map_size: usize,
-        creation: bool,
-    ) -> Result<Index> {
-        if !matches!(self.get_unavailable(uuid), Missing) {
-            panic!("Attempt to open an index that was unavailable");
-        }
-        let index = create_or_open_index(path, date, enable_mdb_writemap, map_size, creation)?;
-        match self.available.insert(*uuid, index.clone()) {
-            InsertionOutcome::InsertedNew => (),
-            InsertionOutcome::Evicted(evicted_uuid, evicted_index) => {
-                self.close(evicted_uuid, evicted_index, enable_mdb_writemap, 0);
-            }
-            InsertionOutcome::Replaced(_) => {
-                panic!("Attempt to open an index that was already opened")
-            }
-        }
-        Ok(index)
-    }
-
-    /// Increases the current generation. See documentation for this field.
-    ///
-    /// In the unlikely event that the 2^64 generations would have been exhausted, we simply wrap-around.
-    ///
-    /// For this to cause an issue, one should be able to stop a reader in time after it got a `ReopenableIndex` and before it takes the lock
-    /// to remove it from the unavailable map, and keep the reader in this frozen state for 2^64 closing of other indexes.
-    ///
-    /// This seems overwhelmingly impossible to achieve in practice.
-    fn next_generation(&mut self) -> usize {
-        self.generation = self.generation.wrapping_add(1);
-        self.generation
-    }
-
-    /// Attempts to close an index.
-    ///
-    /// # Status table
-    ///
-    /// | Previous Status | New Status    |
-    /// |-----------------|---------------|
-    /// | Missing         | Missing       |
-    /// | BeingDeleted    | BeingDeleted  |
-    /// | Closing         | Closing       |
-    /// | Available       | Closing       |
-    ///
-    pub fn close_for_resize(
-        &mut self,
-        uuid: &Uuid,
-        enable_mdb_writemap: bool,
-        map_size_growth: usize,
-    ) {
-        let Some(index) = self.available.remove(uuid) else {
-            return;
-        };
-        self.close(*uuid, index, enable_mdb_writemap, map_size_growth);
-    }
-
-    fn close(
-        &mut self,
-        uuid: Uuid,
-        index: Index,
-        enable_mdb_writemap: bool,
-        map_size_growth: usize,
-    ) {
-        let map_size = index.map_size() + map_size_growth;
-        let closing_event = index.prepare_for_closing();
-        let generation = self.next_generation();
-        self.unavailable.insert(
-            uuid,
-            Some(ClosingIndex { uuid, closing_event, enable_mdb_writemap, map_size, generation }),
-        );
-    }
-
-    /// Attempts to delete and index.
-    ///
-    ///  `end_deletion` must be called just after.
-    ///
-    /// # Status table
-    ///
-    /// | Previous Status | New Status   | Return value                |
-    /// |-----------------|--------------|-----------------------------|
-    /// | Missing         | BeingDeleted | Ok(None)                    |
-    /// | BeingDeleted    | BeingDeleted | Err(None)                   |
-    /// | Closing         | Closing      | Err(Some(reopen))           |
-    /// | Available       | BeingDeleted | Ok(Some(env_closing_event)) |
-    pub fn start_deletion(
-        &mut self,
-        uuid: &Uuid,
-    ) -> std::result::Result<Option<EnvClosingEvent>, Option<ClosingIndex>> {
-        if let Some(index) = self.available.remove(uuid) {
-            self.unavailable.insert(*uuid, None);
-            return Ok(Some(index.prepare_for_closing()));
-        }
-        match self.unavailable.remove(uuid) {
-            Some(Some(reopen)) => Err(Some(reopen)),
-            Some(None) => Err(None),
-            None => Ok(None),
-        }
-    }
-
-    /// Marks that an index deletion finished.
-    ///
-    /// Must be used after calling `start_deletion`.
-    ///
-    /// # Status table
-    ///
-    /// | Previous Status | New Status |
-    /// |-----------------|------------|
-    /// | Missing         | Missing    |
-    /// | BeingDeleted    | Missing    |
-    /// | Closing         | panics     |
-    /// | Available       | panics     |
-    pub fn end_deletion(&mut self, uuid: &Uuid) {
-        assert!(
-            self.available.get(uuid).is_none(),
-            "Attempt to finish deletion of an index that was not being deleted"
-        );
-        // Do not panic if the index was Missing or BeingDeleted
-        assert!(
-            !matches!(self.unavailable.remove(uuid), Some(Some(_))),
-            "Attempt to finish deletion of an index that was being closed"
-        );
-    }
-}
-
-/// Create or open an index in the specified path.
-/// The path *must* exist or an error will be thrown.
-fn create_or_open_index(
-    path: &Path,
-    date: Option<(OffsetDateTime, OffsetDateTime)>,
-    enable_mdb_writemap: bool,
-    map_size: usize,
-    creation: bool,
-) -> Result<Index> {
-    let options = EnvOpenOptions::new();
-    let mut options = options.read_txn_without_tls();
-    options.map_size(clamp_to_page_size(map_size));
-
-    // You can find more details about this experimental
-    // environment variable on the following GitHub discussion:
-    // <https://github.com/orgs/meilisearch/discussions/806>
-    let max_readers = match std::env::var("MEILI_EXPERIMENTAL_INDEX_MAX_READERS") {
-        Ok(value) => u32::from_str(&value).unwrap(),
-        Err(VarError::NotPresent) => 1024,
-        Err(VarError::NotUnicode(value)) => panic!(
-            "Invalid unicode for the `MEILI_EXPERIMENTAL_INDEX_MAX_READERS` env var: {value:?}"
-        ),
-    };
-    options.max_readers(max_readers);
-    if enable_mdb_writemap {
-        unsafe { options.flags(EnvFlags::WRITE_MAP) };
-    }
-
-    if let Some((created, updated)) = date {
-        Ok(Index::new_with_creation_dates(options, path, created, updated, creation)?)
-    } else {
-        Ok(Index::new(options, path, creation)?)
-    }
-}
-
-/// Putting the tests of the LRU down there so we have access to the cache's private members
-#[cfg(test)]
-mod tests {
-
-    use meilisearch_types::heed::{Env, WithoutTls};
-    use meilisearch_types::Index;
-    use uuid::Uuid;
-
-    use super::super::IndexMapper;
-    use crate::test_utils::IndexSchedulerHandle;
-    use crate::utils::clamp_to_page_size;
-    use crate::IndexScheduler;
-
-    impl IndexMapper {
-        fn test() -> (Self, Env<WithoutTls>, IndexSchedulerHandle) {
-            let (index_scheduler, handle) = IndexScheduler::test(true, vec![]);
-            (index_scheduler.index_mapper, index_scheduler.env, handle)
-        }
-    }
-
-    fn check_first_unavailable(mapper: &IndexMapper, expected_uuid: Uuid, is_closing: bool) {
-        let index_map = mapper.index_map.read().unwrap();
-        let (uuid, state) = index_map.unavailable.first_key_value().unwrap();
-        assert_eq!(uuid, &expected_uuid);
-        assert_eq!(state.is_some(), is_closing);
-    }
-
-    #[test]
-    fn evict_indexes() {
-        let (mapper, env, _handle) = IndexMapper::test();
-        let mut uuids = vec![];
-        // LRU cap + 1
-        for i in 0..(5 + 1) {
-            let index_name = format!("index-{i}");
-            let wtxn = env.write_txn().unwrap();
-            mapper.create_index(wtxn, &index_name, None).unwrap();
-            let txn = env.read_txn().unwrap();
-            uuids.push(mapper.index_mapping.get(&txn, &index_name).unwrap().unwrap());
-        }
-        // index-0 was evicted
-        check_first_unavailable(&mapper, uuids[0], true);
-
-        // get back the evicted index
-        let wtxn = env.write_txn().unwrap();
-        mapper.create_index(wtxn, "index-0", None).unwrap();
-
-        // Least recently used is now index-1
-        check_first_unavailable(&mapper, uuids[1], true);
-    }
-
-    #[test]
-    fn resize_index() {
-        let (mapper, env, _handle) = IndexMapper::test();
-        let index = mapper.create_index(env.write_txn().unwrap(), "index", None).unwrap();
-        assert_index_size(index, mapper.index_base_map_size);
-
-        mapper.resize_index(&env.read_txn().unwrap(), "index").unwrap();
-
-        let index = mapper.create_index(env.write_txn().unwrap(), "index", None).unwrap();
-        assert_index_size(index, mapper.index_base_map_size + mapper.index_growth_amount);
-
-        mapper.resize_index(&env.read_txn().unwrap(), "index").unwrap();
-
-        let index = mapper.create_index(env.write_txn().unwrap(), "index", None).unwrap();
-        assert_index_size(index, mapper.index_base_map_size + mapper.index_growth_amount * 2);
-    }
-
-    fn assert_index_size(index: Index, expected: usize) {
-        let expected = clamp_to_page_size(expected);
-        let index_map_size = index.map_size();
-        assert_eq!(index_map_size, expected);
-    }
-}

crates/index-scheduler/src/index_mapper/mod.rs

@@ -1,534 +0,0 @@
-use std::path::PathBuf;
-use std::sync::{Arc, RwLock};
-use std::time::Duration;
-use std::{fs, thread};
-
-use meilisearch_types::heed::types::{SerdeJson, Str};
-use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
-use meilisearch_types::milli;
-use meilisearch_types::milli::database_stats::DatabaseStats;
-use meilisearch_types::milli::update::IndexerConfig;
-use meilisearch_types::milli::{FieldDistribution, Index};
-use serde::{Deserialize, Serialize};
-use time::OffsetDateTime;
-use tracing::error;
-use uuid::Uuid;
-
-use self::index_map::IndexMap;
-use self::IndexStatus::{Available, BeingDeleted, Closing, Missing};
-use crate::uuid_codec::UuidCodec;
-use crate::{Error, IndexBudget, IndexSchedulerOptions, Result};
-
-mod index_map;
-
-/// The number of database used by index mapper
-const NUMBER_OF_DATABASES: u32 = 2;
-/// Database const names for the `IndexMapper`.
-mod db_name {
-    pub const INDEX_MAPPING: &str = "index-mapping";
-    pub const INDEX_STATS: &str = "index-stats";
-}
-
-/// Structure managing meilisearch's indexes.
-///
-/// It is responsible for:
-/// 1. Creating new indexes
-/// 2. Opening indexes and storing references to these opened indexes
-/// 3. Accessing indexes through their uuid
-/// 4. Mapping a user-defined name to each index uuid.
-///
-/// # Implementation notes
-///
-/// An index exists as 3 bits of data:
-/// 1. The index data on disk, that can exist in 3 states: Missing, Present, or BeingDeleted.
-/// 2. The persistent database containing the association between the index' name and its UUID,
-///    that can exist in 2 states: Missing or Present.
-/// 3. The state of the index in the in-memory `IndexMap`, that can exist in multiple states:
-///   - Missing
-///   - Available
-///   - Closing (because an index needs resizing or was evicted from the cache)
-///   - BeingDeleted
-///
-/// All of this data should be kept consistent between index operations, which is achieved by the `IndexMapper`
-/// with the use of the following primitives:
-/// - A RwLock on the `IndexMap`.
-/// - Transactions on the association database.
-/// - ClosingEvent signals emitted when closing an environment.
-#[derive(Clone)]
-pub struct IndexMapper {
-    /// Keep track of the opened indexes. Used mainly by the index resolver.
-    index_map: Arc<RwLock<IndexMap>>,
-
-    /// Map an index name with an index uuid currently available on disk.
-    pub(crate) index_mapping: Database<Str, UuidCodec>,
-    /// Map an index UUID with the cached stats associated to the index.
-    ///
-    /// Using an UUID forces to use the index_mapping table to recover the index behind a name, ensuring
-    /// consistency wrt index swapping.
-    pub(crate) index_stats: Database<UuidCodec, SerdeJson<IndexStats>>,
-
-    /// Path to the folder where the LMDB environments of each index are.
-    base_path: PathBuf,
-    /// The map size an index is opened with on the first time.
-    index_base_map_size: usize,
-    /// The quantity by which the map size of an index is incremented upon reopening, in bytes.
-    index_growth_amount: usize,
-    /// Whether we open a meilisearch index with the MDB_WRITEMAP option or not.
-    enable_mdb_writemap: bool,
-    pub indexer_config: Arc<IndexerConfig>,
-
-    /// A few types of long running batches of tasks that act on a single index set this field
-    /// so that a handle to the index is available from other threads (search) in an optimized manner.
-    currently_updating_index: Arc<RwLock<Option<(String, Index)>>>,
-}
-
-/// Whether the index is available for use or is forbidden to be inserted back in the index map
-#[allow(clippy::large_enum_variant)]
-#[derive(Clone)]
-pub enum IndexStatus {
-    /// Not currently in the index map.
-    Missing,
-    /// Do not insert it back in the index map as it is currently being deleted.
-    BeingDeleted,
-    /// Temporarily do not insert the index in the index map as it is currently being resized/evicted from the map.
-    Closing(index_map::ClosingIndex),
-    /// You can use the index without worrying about anything.
-    Available(Index),
-}
-
-/// The statistics that can be computed from an `Index` object.
-#[derive(Serialize, Deserialize, Debug)]
-pub struct IndexStats {
-    /// Stats of the documents database.
-    #[serde(default)]
-    pub documents_database_stats: DatabaseStats,
-
-    #[serde(default, skip_serializing)]
-    pub number_of_documents: Option<u64>,
-
-    /// Size taken up by the index' DB, in bytes.
-    ///
-    /// This includes the size taken by both the used and free pages of the DB, and as the free pages
-    /// are not returned to the disk after a deletion, this number is typically larger than
-    /// `used_database_size` that only includes the size of the used pages.
-    pub database_size: u64,
-    /// Number of embeddings in the index.
-    /// Option: retrocompatible with the stats of the pre-v1.13.0 versions of meilisearch
-    pub number_of_embeddings: Option<u64>,
-    /// Number of embedded documents in the index.
-    /// Option: retrocompatible with the stats of the pre-v1.13.0 versions of meilisearch
-    pub number_of_embedded_documents: Option<u64>,
-    /// Size taken by the used pages of the index' DB, in bytes.
-    ///
-    /// As the DB backend does not return to the disk the pages that are not currently used by the DB,
-    /// this value is typically smaller than `database_size`.
-    pub used_database_size: u64,
-    /// The primary key of the index
-    pub primary_key: Option<String>,
-    /// Association of every field name with the number of times it occurs in the documents.
-    pub field_distribution: FieldDistribution,
-    /// Creation date of the index.
-    #[serde(with = "time::serde::rfc3339")]
-    pub created_at: OffsetDateTime,
-    /// Date of the last update of the index.
-    #[serde(with = "time::serde::rfc3339")]
-    pub updated_at: OffsetDateTime,
-}
-
-impl IndexStats {
-    /// Compute the stats of an index
-    ///
-    /// # Parameters
-    ///
-    /// - rtxn: a RO transaction for the index, obtained from `Index::read_txn()`.
-    pub fn new(index: &Index, rtxn: &RoTxn) -> milli::Result<Self> {
-        let arroy_stats = index.arroy_stats(rtxn)?;
-        Ok(IndexStats {
-            number_of_embeddings: Some(arroy_stats.number_of_embeddings),
-            number_of_embedded_documents: Some(arroy_stats.documents.len()),
-            documents_database_stats: index.documents_stats(rtxn)?.unwrap_or_default(),
-            number_of_documents: None,
-            database_size: index.on_disk_size()?,
-            used_database_size: index.used_size()?,
-            primary_key: index.primary_key(rtxn)?.map(|s| s.to_string()),
-            field_distribution: index.field_distribution(rtxn)?,
-            created_at: index.created_at(rtxn)?,
-            updated_at: index.updated_at(rtxn)?,
-        })
-    }
-}
-
-impl IndexMapper {
-    pub(crate) const fn nb_db() -> u32 {
-        NUMBER_OF_DATABASES
-    }
-
-    pub fn new(
-        env: &Env<WithoutTls>,
-        wtxn: &mut RwTxn,
-        options: &IndexSchedulerOptions,
-        budget: IndexBudget,
-    ) -> Result<Self> {
-        Ok(Self {
-            index_map: Arc::new(RwLock::new(IndexMap::new(budget.index_count))),
-            index_mapping: env.create_database(wtxn, Some(db_name::INDEX_MAPPING))?,
-            index_stats: env.create_database(wtxn, Some(db_name::INDEX_STATS))?,
-            base_path: options.indexes_path.clone(),
-            index_base_map_size: budget.map_size,
-            index_growth_amount: options.index_growth_amount,
-            enable_mdb_writemap: options.enable_mdb_writemap,
-            indexer_config: options.indexer_config.clone(),
-            currently_updating_index: Default::default(),
-        })
-    }
-
-    /// Get or create the index.
-    pub fn create_index(
-        &self,
-        mut wtxn: RwTxn,
-        name: &str,
-        date: Option<(OffsetDateTime, OffsetDateTime)>,
-    ) -> Result<Index> {
-        match self.index(&wtxn, name) {
-            Ok(index) => {
-                wtxn.commit()?;
-                Ok(index)
-            }
-            Err(Error::IndexNotFound(_)) => {
-                let uuid = Uuid::new_v4();
-                self.index_mapping.put(&mut wtxn, name, &uuid)?;
-
-                let index_path = self.base_path.join(uuid.to_string());
-                fs::create_dir_all(&index_path)?;
-
-                // Error if the UUIDv4 somehow already exists in the map, since it should be fresh.
-                // This is very unlikely to happen in practice.
-                // TODO: it would be better to lazily create the index. But we need an Index::open function for milli.
-                let index = self
-                    .index_map
-                    .write()
-                    .unwrap()
-                    .create(
-                        &uuid,
-                        &index_path,
-                        date,
-                        self.enable_mdb_writemap,
-                        self.index_base_map_size,
-                        true,
-                    )
-                    .map_err(|e| Error::from_milli(e, Some(uuid.to_string())))?;
-                let index_rtxn = index.read_txn()?;
-                let stats = crate::index_mapper::IndexStats::new(&index, &index_rtxn)
-                    .map_err(|e| Error::from_milli(e, Some(name.to_string())))?;
-                self.store_stats_of(&mut wtxn, name, &stats)?;
-                drop(index_rtxn);
-
-                wtxn.commit()?;
-
-                Ok(index)
-            }
-            error => error,
-        }
-    }
-
-    /// Removes the index from the mapping table and the in-memory index map
-    /// but keeps the associated tasks.
-    pub fn delete_index(&self, mut wtxn: RwTxn, name: &str) -> Result<()> {
-        let uuid = self
-            .index_mapping
-            .get(&wtxn, name)?
-            .ok_or_else(|| Error::IndexNotFound(name.to_string()))?;
-
-        // Not an error if the index had no stats in cache.
-        self.index_stats.delete(&mut wtxn, &uuid)?;
-
-        // Once we retrieved the UUID of the index we remove it from the mapping table.
-        assert!(self.index_mapping.delete(&mut wtxn, name)?);
-
-        wtxn.commit()?;
-
-        let mut tries = 0;
-        // Attempts to remove the index from the in-memory index map in a loop.
-        //
-        // If the index is currently being closed, we will wait for it to be closed and retry getting it in a subsequent
-        // loop iteration.
-        //
-        // We make 100 attempts before giving up.
-        // This could happen in the following situations:
-        //
-        // 1. There is a bug preventing the index from being correctly closed, or us from detecting this.
-        // 2. A user of the index is keeping it open for more than 600 seconds. This could happen e.g. during a pathological search.
-        //    This can not be caused by indexation because deleting an index happens in the scheduler itself, so cannot be concurrent with indexation.
-        //
-        // In these situations, reporting the error through a panic is in order.
-        let closing_event = loop {
-            let mut lock = self.index_map.write().unwrap();
-            match lock.start_deletion(&uuid) {
-                Ok(env_closing) => break env_closing,
-                Err(Some(reopen)) => {
-                    // drop the lock here so that we don't synchronously wait for the index to close.
-                    drop(lock);
-                    tries += 1;
-                    if tries >= 100 {
-                        panic!("Too many attempts to close index {name} prior to deletion.")
-                    }
-                    let reopen = if let Some(reopen) = reopen.wait_timeout(Duration::from_secs(6)) {
-                        reopen
-                    } else {
-                        continue;
-                    };
-                    reopen.close(&mut self.index_map.write().unwrap());
-                    continue;
-                }
-                Err(None) => return Ok(()),
-            }
-        };
-
-        let index_map = self.index_map.clone();
-        let index_path = self.base_path.join(uuid.to_string());
-        let index_name = name.to_string();
-        thread::Builder::new()
-            .name(String::from("index_deleter"))
-            .spawn(move || {
-                // We first wait to be sure that the previously opened index is effectively closed.
-                // This can take a lot of time, this is why we do that in a separate thread.
-                if let Some(closing_event) = closing_event {
-                    closing_event.wait();
-                }
-
-                // Then we remove the content from disk.
-                if let Err(e) = fs::remove_dir_all(&index_path) {
-                    error!(
-                        "An error happened when deleting the index {} ({}): {}",
-                        index_name, uuid, e
-                    );
-                }
-
-                // Finally we remove the entry from the index map.
-                index_map.write().unwrap().end_deletion(&uuid);
-            })
-            .unwrap();
-
-        Ok(())
-    }
-
-    pub fn exists(&self, rtxn: &RoTxn, name: &str) -> Result<bool> {
-        Ok(self.index_mapping.get(rtxn, name)?.is_some())
-    }
-
-    /// Resizes the maximum size of the specified index to the double of its current maximum size.
-    ///
-    /// This operation involves closing the underlying environment and so can take a long time to complete.
-    ///
-    /// # Panics
-    ///
-    /// - If the Index corresponding to the passed name is concurrently being deleted/resized or cannot be found in the
-    ///   in memory hash map.
-    pub fn resize_index(&self, rtxn: &RoTxn, name: &str) -> Result<()> {
-        let uuid = self
-            .index_mapping
-            .get(rtxn, name)?
-            .ok_or_else(|| Error::IndexNotFound(name.to_string()))?;
-
-        // We remove the index from the in-memory index map.
-        self.index_map.write().unwrap().close_for_resize(
-            &uuid,
-            self.enable_mdb_writemap,
-            self.index_growth_amount,
-        );
-
-        Ok(())
-    }
-
-    /// Return an index, may open it if it wasn't already opened.
-    pub fn index(&self, rtxn: &RoTxn, name: &str) -> Result<Index> {
-        if let Some((current_name, current_index)) =
-            self.currently_updating_index.read().unwrap().as_ref()
-        {
-            if current_name == name {
-                return Ok(current_index.clone());
-            }
-        }
-
-        let uuid = self
-            .index_mapping
-            .get(rtxn, name)?
-            .ok_or_else(|| Error::IndexNotFound(name.to_string()))?;
-
-        let mut tries = 0;
-        // attempts to open the index in a loop.
-        //
-        // If the index is currently being closed, we will wait for it to be closed and retry getting it in a subsequent
-        // loop iteration.
-        //
-        // We make 100 attempts before giving up.
-        // This could happen in the following situations:
-        //
-        // 1. There is a bug preventing the index from being correctly closed, or us from detecting it was.
-        // 2. A user of the index is keeping it open for more than 600 seconds. This could happen e.g. during a long indexation,
-        //    a pathological search, and so on.
-        //
-        // In these situations, reporting the error through a panic is in order.
-        let index = loop {
-            tries += 1;
-            if tries > 100 {
-                panic!("Too many spurious wake ups while trying to open the index {name}");
-            }
-
-            // we get the index here to drop the lock before entering the match
-            let index = self.index_map.read().unwrap().get(&uuid);
-
-            match index {
-                Available(index) => break index,
-                Closing(reopen) => {
-                    // Avoiding deadlocks: no lock taken while doing this operation.
-                    let reopen = if let Some(reopen) = reopen.wait_timeout(Duration::from_secs(6)) {
-                        reopen
-                    } else {
-                        continue;
-                    };
-                    let index_path = self.base_path.join(uuid.to_string());
-                    // take the lock to reopen the environment.
-                    reopen
-                        .reopen(&mut self.index_map.write().unwrap(), &index_path)
-                        .map_err(|e| Error::from_milli(e, Some(uuid.to_string())))?;
-                    continue;
-                }
-                BeingDeleted => return Err(Error::IndexNotFound(name.to_string())),
-                // since we're lazy, it's possible that the index has not been opened yet.
-                Missing => {
-                    let mut index_map = self.index_map.write().unwrap();
-                    // between the read lock and the write lock it's not impossible
-                    // that someone already opened the index (eg if two searches happen
-                    // at the same time), thus before opening it we check a second time
-                    // if it's not already there.
-                    match index_map.get(&uuid) {
-                        Missing => {
-                            let index_path = self.base_path.join(uuid.to_string());
-
-                            break index_map
-                                .create(
-                                    &uuid,
-                                    &index_path,
-                                    None,
-                                    self.enable_mdb_writemap,
-                                    self.index_base_map_size,
-                                    false,
-                                )
-                                .map_err(|e| Error::from_milli(e, Some(uuid.to_string())))?;
-                        }
-                        Available(index) => break index,
-                        Closing(_) => {
-                            // the reopening will be handled in the next loop operation
-                            continue;
-                        }
-                        BeingDeleted => return Err(Error::IndexNotFound(name.to_string())),
-                    }
-                }
-            }
-        };
-
-        Ok(index)
-    }
-
-    /// Attempts `f` for each index that exists in the index mapper.
-    ///
-    /// It is preferable to use this function rather than a loop that opens all indexes, as a way to avoid having all indexes opened,
-    /// which is unsupported in general.
-    ///
-    /// Since `f` is allowed to return a result, and `Index` is cloneable, it is still possible to wrongly build e.g. a vector of
-    /// all the indexes, but this function makes it harder and so less likely to do accidentally.
-    pub fn try_for_each_index<U, V>(
-        &self,
-        rtxn: &RoTxn,
-        mut f: impl FnMut(&str, &Index) -> Result<U>,
-    ) -> Result<V>
-    where
-        V: FromIterator<U>,
-    {
-        self.index_mapping
-            .iter(rtxn)?
-            .map(|res| {
-                res.map_err(Error::from)
-                    .and_then(|(name, _)| self.index(rtxn, name).and_then(|index| f(name, &index)))
-            })
-            .collect()
-    }
-
-    /// Return the name of all indexes without opening them.
-    pub fn index_names(&self, rtxn: &RoTxn) -> Result<Vec<String>> {
-        self.index_mapping
-            .iter(rtxn)?
-            .map(|res| res.map_err(Error::from).map(|(name, _)| name.to_string()))
-            .collect()
-    }
-
-    /// Swap two index names.
-    pub fn swap(&self, wtxn: &mut RwTxn, lhs: &str, rhs: &str) -> Result<()> {
-        let lhs_uuid = self
-            .index_mapping
-            .get(wtxn, lhs)?
-            .ok_or_else(|| Error::IndexNotFound(lhs.to_string()))?;
-        let rhs_uuid = self
-            .index_mapping
-            .get(wtxn, rhs)?
-            .ok_or_else(|| Error::IndexNotFound(rhs.to_string()))?;
-
-        self.index_mapping.put(wtxn, lhs, &rhs_uuid)?;
-        self.index_mapping.put(wtxn, rhs, &lhs_uuid)?;
-
-        Ok(())
-    }
-
-    /// The stats of an index.
-    ///
-    /// If available in the cache, they are directly returned.
-    /// Otherwise, the `Index` is opened to compute the stats on the fly (the result is not cached).
-    /// The stats for an index are cached after each `Index` update.
-    pub fn stats_of(&self, rtxn: &RoTxn, index_uid: &str) -> Result<IndexStats> {
-        let uuid = self
-            .index_mapping
-            .get(rtxn, index_uid)?
-            .ok_or_else(|| Error::IndexNotFound(index_uid.to_string()))?;
-
-        match self.index_stats.get(rtxn, &uuid)? {
-            Some(stats) => Ok(stats),
-            None => {
-                let index = self.index(rtxn, index_uid)?;
-                let index_rtxn = index.read_txn()?;
-                IndexStats::new(&index, &index_rtxn)
-                    .map_err(|e| Error::from_milli(e, Some(uuid.to_string())))
-            }
-        }
-    }
-
-    /// Stores the new stats for an index.
-    ///
-    /// Expected usage is to compute the stats the index using `IndexStats::new`, the pass it to this function.
-    pub fn store_stats_of(
-        &self,
-        wtxn: &mut RwTxn,
-        index_uid: &str,
-        stats: &IndexStats,
-    ) -> Result<()> {
-        let uuid = self
-            .index_mapping
-            .get(wtxn, index_uid)?
-            .ok_or_else(|| Error::IndexNotFound(index_uid.to_string()))?;
-
-        self.index_stats.put(wtxn, &uuid, stats)?;
-        Ok(())
-    }
-
-    pub fn index_exists(&self, rtxn: &RoTxn, name: &str) -> Result<bool> {
-        Ok(self.index_mapping.get(rtxn, name)?.is_some())
-    }
-
-    pub fn indexer_config(&self) -> &IndexerConfig {
-        &self.indexer_config
-    }
-
-    pub fn set_currently_updating_index(&self, index: Option<(String, Index)>) {
-        *self.currently_updating_index.write().unwrap() = index;
-    }
-}

crates/index-scheduler/src/lib.rs

@@ -1,879 +0,0 @@
-/*!
-This crate defines the index scheduler, which is responsible for:
-1. Keeping references to meilisearch's indexes and mapping them to their
-   user-defined names.
-2. Scheduling tasks given by the user and executing them, in batch if possible.
-
-When an `IndexScheduler` is created, a new thread containing a reference to the
-scheduler is created. This thread runs the scheduler's run loop, where the
-scheduler waits to be woken up to process new tasks. It wakes up when:
-
-1. it is launched for the first time
-2. a new task is registered
-3. a batch of tasks has been processed
-
-It is only within this thread that the scheduler is allowed to process tasks.
-On the other hand, the publicly accessible methods of the scheduler can be
-called asynchronously from any thread. These methods can either query the
-content of the scheduler or enqueue new tasks.
-*/
-
-mod dump;
-pub mod error;
-mod features;
-mod index_mapper;
-#[cfg(test)]
-mod insta_snapshot;
-mod lru;
-mod processing;
-mod queue;
-mod scheduler;
-#[cfg(test)]
-mod test_utils;
-pub mod upgrade;
-mod utils;
-pub mod uuid_codec;
-pub mod versioning;
-
-pub type Result<T, E = Error> = std::result::Result<T, E>;
-pub type TaskId = u32;
-
-use std::collections::{BTreeMap, HashMap};
-use std::io::{self, BufReader, Read};
-use std::panic::{catch_unwind, AssertUnwindSafe};
-use std::path::{Path, PathBuf};
-use std::sync::{Arc, RwLock};
-use std::time::Duration;
-
-use dump::Dump;
-pub use error::Error;
-pub use features::RoFeatures;
-use flate2::bufread::GzEncoder;
-use flate2::Compression;
-use meilisearch_types::batches::Batch;
-use meilisearch_types::features::{InstanceTogglableFeatures, Network, RuntimeTogglableFeatures};
-use meilisearch_types::heed::byteorder::BE;
-use meilisearch_types::heed::types::I128;
-use meilisearch_types::heed::{self, Env, RoTxn, WithoutTls};
-use meilisearch_types::milli::index::IndexEmbeddingConfig;
-use meilisearch_types::milli::update::IndexerConfig;
-use meilisearch_types::milli::vector::{Embedder, EmbedderOptions, EmbeddingConfigs};
-use meilisearch_types::milli::{self, Index};
-use meilisearch_types::task_view::TaskView;
-use meilisearch_types::tasks::{KindWithContent, Task};
-use processing::ProcessingTasks;
-pub use queue::Query;
-use queue::Queue;
-use roaring::RoaringBitmap;
-use scheduler::Scheduler;
-use time::OffsetDateTime;
-use versioning::Versioning;
-
-use crate::index_mapper::IndexMapper;
-use crate::utils::clamp_to_page_size;
-
-pub(crate) type BEI128 = I128<BE>;
-
-#[derive(Debug)]
-pub struct IndexSchedulerOptions {
-    /// The path to the version file of Meilisearch.
-    pub version_file_path: PathBuf,
-    /// The path to the folder containing the auth LMDB env.
-    pub auth_path: PathBuf,
-    /// The path to the folder containing the task databases.
-    pub tasks_path: PathBuf,
-    /// The path to the file store containing the files associated to the tasks.
-    pub update_file_path: PathBuf,
-    /// The path to the folder containing meilisearch's indexes.
-    pub indexes_path: PathBuf,
-    /// The path to the folder containing the snapshots.
-    pub snapshots_path: PathBuf,
-    /// The path to the folder containing the dumps.
-    pub dumps_path: PathBuf,
-    /// The URL on which we must send the tasks statuses
-    pub webhook_url: Option<String>,
-    /// The value we will send into the Authorization HTTP header on the webhook URL
-    pub webhook_authorization_header: Option<String>,
-    /// The maximum size, in bytes, of the task index.
-    pub task_db_size: usize,
-    /// The size, in bytes, with which a meilisearch index is opened the first time of each meilisearch index.
-    pub index_base_map_size: usize,
-    /// Whether we open a meilisearch index with the MDB_WRITEMAP option or not.
-    pub enable_mdb_writemap: bool,
-    /// The size, in bytes, by which the map size of an index is increased when it resized due to being full.
-    pub index_growth_amount: usize,
-    /// The number of indexes that can be concurrently opened in memory.
-    pub index_count: usize,
-    /// Configuration used during indexing for each meilisearch index.
-    pub indexer_config: Arc<IndexerConfig>,
-    /// Set to `true` iff the index scheduler is allowed to automatically
-    /// batch tasks together, to process multiple tasks at once.
-    pub autobatching_enabled: bool,
-    /// Set to `true` iff the index scheduler is allowed to automatically
-    /// delete the finished tasks when there are too many tasks.
-    pub cleanup_enabled: bool,
-    /// The maximum number of tasks stored in the task queue before starting
-    /// to auto schedule task deletions.
-    pub max_number_of_tasks: usize,
-    /// If the autobatcher is allowed to automatically batch tasks
-    /// it will only batch this defined number of tasks at once.
-    pub max_number_of_batched_tasks: usize,
-    /// If the autobatcher is allowed to automatically batch tasks
-    /// it will only batch this defined maximum size (in bytes) of tasks at once.
-    pub batched_tasks_size_limit: u64,
-    /// The experimental features enabled for this instance.
-    pub instance_features: InstanceTogglableFeatures,
-    /// The experimental features enabled for this instance.
-    pub auto_upgrade: bool,
-    /// The maximal number of entries in the search query cache of an embedder.
-    ///
-    /// 0 disables the cache.
-    pub embedding_cache_cap: usize,
-}
-
-/// Structure which holds meilisearch's indexes and schedules the tasks
-/// to be performed on them.
-pub struct IndexScheduler {
-    /// The LMDB environment which the DBs are associated with.
-    pub(crate) env: Env<WithoutTls>,
-
-    /// The list of tasks currently processing
-    pub(crate) processing_tasks: Arc<RwLock<ProcessingTasks>>,
-
-    /// A database containing only the version of the index-scheduler
-    pub version: versioning::Versioning,
-    /// The queue containing both the tasks and the batches.
-    pub queue: queue::Queue,
-    /// In charge of creating, opening, storing and returning indexes.
-    pub(crate) index_mapper: IndexMapper,
-    /// In charge of fetching and setting the status of experimental features.
-    features: features::FeatureData,
-
-    /// Everything related to the processing of the tasks
-    pub scheduler: scheduler::Scheduler,
-
-    /// Whether we should automatically cleanup the task queue or not.
-    pub(crate) cleanup_enabled: bool,
-
-    /// The webhook url we should send tasks to after processing every batches.
-    pub(crate) webhook_url: Option<String>,
-    /// The Authorization header to send to the webhook URL.
-    pub(crate) webhook_authorization_header: Option<String>,
-
-    /// A map to retrieve the runtime representation of an embedder depending on its configuration.
-    ///
-    /// This map may return the same embedder object for two different indexes or embedder settings,
-    /// but it will only do this if the embedder configuration options are the same, leading
-    /// to the same embeddings for the same input text.
-    embedders: Arc<RwLock<HashMap<EmbedderOptions, Arc<Embedder>>>>,
-
-    // ================= test
-    // The next entry is dedicated to the tests.
-    /// Provide a way to set a breakpoint in multiple part of the scheduler.
-    ///
-    /// See [self.breakpoint()](`IndexScheduler::breakpoint`) for an explanation.
-    #[cfg(test)]
-    test_breakpoint_sdr: crossbeam_channel::Sender<(test_utils::Breakpoint, bool)>,
-
-    /// A list of planned failures within the [`tick`](IndexScheduler::tick) method of the index scheduler.
-    ///
-    /// The first field is the iteration index and the second field identifies a location in the code.
-    #[cfg(test)]
-    planned_failures: Vec<(usize, test_utils::FailureLocation)>,
-
-    /// A counter that is incremented before every call to [`tick`](IndexScheduler::tick)
-    #[cfg(test)]
-    run_loop_iteration: Arc<RwLock<usize>>,
-}
-
-impl IndexScheduler {
-    fn private_clone(&self) -> IndexScheduler {
-        IndexScheduler {
-            env: self.env.clone(),
-            processing_tasks: self.processing_tasks.clone(),
-            version: self.version.clone(),
-            queue: self.queue.private_clone(),
-            scheduler: self.scheduler.private_clone(),
-
-            index_mapper: self.index_mapper.clone(),
-            cleanup_enabled: self.cleanup_enabled,
-            webhook_url: self.webhook_url.clone(),
-            webhook_authorization_header: self.webhook_authorization_header.clone(),
-            embedders: self.embedders.clone(),
-            #[cfg(test)]
-            test_breakpoint_sdr: self.test_breakpoint_sdr.clone(),
-            #[cfg(test)]
-            planned_failures: self.planned_failures.clone(),
-            #[cfg(test)]
-            run_loop_iteration: self.run_loop_iteration.clone(),
-            features: self.features.clone(),
-        }
-    }
-
-    pub(crate) const fn nb_db() -> u32 {
-        Versioning::nb_db() + Queue::nb_db() + IndexMapper::nb_db() + features::FeatureData::nb_db()
-    }
-
-    /// Create an index scheduler and start its run loop.
-    #[allow(private_interfaces)] // because test_utils is private
-    pub fn new(
-        options: IndexSchedulerOptions,
-        auth_env: Env<WithoutTls>,
-        from_db_version: (u32, u32, u32),
-        #[cfg(test)] test_breakpoint_sdr: crossbeam_channel::Sender<(test_utils::Breakpoint, bool)>,
-        #[cfg(test)] planned_failures: Vec<(usize, test_utils::FailureLocation)>,
-    ) -> Result<Self> {
-        std::fs::create_dir_all(&options.tasks_path)?;
-        std::fs::create_dir_all(&options.update_file_path)?;
-        std::fs::create_dir_all(&options.indexes_path)?;
-        std::fs::create_dir_all(&options.dumps_path)?;
-
-        if cfg!(windows) && options.enable_mdb_writemap {
-            // programmer error if this happens: in normal use passing the option on Windows is an error in main
-            panic!("Windows doesn't support the MDB_WRITEMAP LMDB option");
-        }
-
-        let task_db_size = clamp_to_page_size(options.task_db_size);
-        let budget = if options.indexer_config.skip_index_budget {
-            IndexBudget {
-                map_size: options.index_base_map_size,
-                index_count: options.index_count,
-                task_db_size,
-            }
-        } else {
-            Self::index_budget(
-                &options.tasks_path,
-                options.index_base_map_size,
-                task_db_size,
-                options.index_count,
-            )
-        };
-
-        let env = unsafe {
-            let env_options = heed::EnvOpenOptions::new();
-            let mut env_options = env_options.read_txn_without_tls();
-            env_options
-                .max_dbs(Self::nb_db())
-                .map_size(budget.task_db_size)
-                .open(&options.tasks_path)
-        }?;
-
-        // We **must** starts by upgrading the version because it'll also upgrade the required database before we can open them
-        let version = versioning::Versioning::new(&env, from_db_version)?;
-
-        let mut wtxn = env.write_txn()?;
-        let features = features::FeatureData::new(&env, &mut wtxn, options.instance_features)?;
-        let queue = Queue::new(&env, &mut wtxn, &options)?;
-        let index_mapper = IndexMapper::new(&env, &mut wtxn, &options, budget)?;
-        wtxn.commit()?;
-
-        // allow unreachable_code to get rids of the warning in the case of a test build.
-        let this = Self {
-            processing_tasks: Arc::new(RwLock::new(ProcessingTasks::new())),
-            version,
-            queue,
-            scheduler: Scheduler::new(&options, auth_env),
-
-            index_mapper,
-            env,
-            cleanup_enabled: options.cleanup_enabled,
-            webhook_url: options.webhook_url,
-            webhook_authorization_header: options.webhook_authorization_header,
-            embedders: Default::default(),
-
-            #[cfg(test)]
-            test_breakpoint_sdr,
-            #[cfg(test)]
-            planned_failures,
-            #[cfg(test)]
-            run_loop_iteration: Arc::new(RwLock::new(0)),
-            features,
-        };
-
-        this.run();
-        Ok(this)
-    }
-
-    /// Return `Ok(())` if the index scheduler is able to access one of its database.
-    pub fn health(&self) -> Result<()> {
-        let rtxn = self.env.read_txn()?;
-        self.queue.batch_to_tasks_mapping.first(&rtxn)?;
-        Ok(())
-    }
-
-    fn index_budget(
-        tasks_path: &Path,
-        base_map_size: usize,
-        mut task_db_size: usize,
-        max_index_count: usize,
-    ) -> IndexBudget {
-        #[cfg(windows)]
-        const DEFAULT_BUDGET: usize = 6 * 1024 * 1024 * 1024 * 1024; // 6 TiB, 1 index
-        #[cfg(not(windows))]
-        const DEFAULT_BUDGET: usize = 80 * 1024 * 1024 * 1024 * 1024; // 80 TiB, 18 indexes
-
-        let budget = if Self::is_good_heed(tasks_path, DEFAULT_BUDGET) {
-            DEFAULT_BUDGET
-        } else {
-            tracing::debug!("determining budget with dichotomic search");
-            utils::dichotomic_search(DEFAULT_BUDGET / 2, |map_size| {
-                Self::is_good_heed(tasks_path, map_size)
-            })
-        };
-
-        tracing::debug!("memmap budget: {budget}B");
-        let mut budget = budget / 2;
-        if task_db_size > (budget / 2) {
-            task_db_size = clamp_to_page_size(budget * 2 / 5);
-            tracing::debug!(
-                "Decreasing max size of task DB to {task_db_size}B due to constrained memory space"
-            );
-        }
-        budget -= task_db_size;
-
-        // won't be mutated again
-        let budget = budget;
-        let task_db_size = task_db_size;
-
-        tracing::debug!("index budget: {budget}B");
-        let mut index_count = budget / base_map_size;
-        if index_count < 2 {
-            // take a bit less than half than the budget to make sure we can always afford to open an index
-            let map_size = (budget * 2) / 5;
-            // single index of max budget
-            tracing::debug!("1 index of {map_size}B can be opened simultaneously.");
-            return IndexBudget { map_size, index_count: 1, task_db_size };
-        }
-        // give us some space for an additional index when the cache is already full
-        // decrement is OK because index_count >= 2.
-        index_count -= 1;
-        if index_count > max_index_count {
-            index_count = max_index_count;
-        }
-        tracing::debug!("Up to {index_count} indexes of {base_map_size}B opened simultaneously.");
-        IndexBudget { map_size: base_map_size, index_count, task_db_size }
-    }
-
-    fn is_good_heed(tasks_path: &Path, map_size: usize) -> bool {
-        if let Ok(env) = unsafe {
-            heed::EnvOpenOptions::new().map_size(clamp_to_page_size(map_size)).open(tasks_path)
-        } {
-            env.prepare_for_closing().wait();
-            true
-        } else {
-            // We're treating all errors equally here, not only allocation errors.
-            // This means there's a possiblity for the budget to lower due to errors different from allocation errors.
-            // For persistent errors, this is OK as long as the task db is then reopened normally without ignoring the error this time.
-            // For transient errors, this could lead to an instance with too low a budget.
-            // However transient errors are: 1) less likely than persistent errors 2) likely to cause other issues down the line anyway.
-            false
-        }
-    }
-
-    pub fn read_txn(&self) -> Result<RoTxn<WithoutTls>> {
-        self.env.read_txn().map_err(|e| e.into())
-    }
-
-    /// Start the run loop for the given index scheduler.
-    ///
-    /// This function will execute in a different thread and must be called
-    /// only once per index scheduler.
-    fn run(&self) {
-        let run = self.private_clone();
-        std::thread::Builder::new()
-            .name(String::from("scheduler"))
-            .spawn(move || {
-                #[cfg(test)]
-                run.breakpoint(test_utils::Breakpoint::Init);
-
-                run.scheduler.wake_up.wait_timeout(std::time::Duration::from_secs(60));
-
-                loop {
-                    let ret = catch_unwind(AssertUnwindSafe(|| run.tick()));
-                    match ret {
-                        Ok(Ok(TickOutcome::TickAgain(_))) => (),
-                        Ok(Ok(TickOutcome::WaitForSignal)) => run.scheduler.wake_up.wait(),
-                        Ok(Ok(TickOutcome::StopProcessingForever)) => break,
-                        Ok(Err(e)) => {
-                            tracing::error!("{e}");
-                            // Wait one second when an irrecoverable error occurs.
-                            if !e.is_recoverable() {
-                                std::thread::sleep(Duration::from_secs(1));
-                            }
-                        }
-                        Err(_panic) => {
-                            tracing::error!("Internal error: Unexpected panic in the `IndexScheduler::run` method.");
-
-                        }
-                    }
-                }
-            })
-            .unwrap();
-    }
-
-    pub fn indexer_config(&self) -> &IndexerConfig {
-        &self.index_mapper.indexer_config
-    }
-
-    /// Return the real database size (i.e.: The size **with** the free pages)
-    pub fn size(&self) -> Result<u64> {
-        Ok(self.env.real_disk_size()?)
-    }
-
-    /// Return the used database size (i.e.: The size **without** the free pages)
-    pub fn used_size(&self) -> Result<u64> {
-        Ok(self.env.non_free_pages_size()?)
-    }
-
-    /// Return the index corresponding to the name.
-    ///
-    /// * If the index wasn't opened before, the index will be opened.
-    /// * If the index doesn't exist on disk, the `IndexNotFoundError` is thrown.
-    ///
-    /// ### Note
-    ///
-    /// As an `Index` requires a large swath of the virtual memory address space, correct usage of an `Index` does not
-    /// keep its handle for too long.
-    ///
-    /// Some configurations also can't reasonably open multiple indexes at once.
-    /// If you need to fetch information from or perform an action on all indexes,
-    /// see the `try_for_each_index` function.
-    pub fn index(&self, name: &str) -> Result<Index> {
-        let rtxn = self.env.read_txn()?;
-        self.index_mapper.index(&rtxn, name)
-    }
-
-    /// Return the boolean referring if index exists.
-    pub fn index_exists(&self, name: &str) -> Result<bool> {
-        let rtxn = self.env.read_txn()?;
-        self.index_mapper.index_exists(&rtxn, name)
-    }
-
-    /// Return the name of all indexes without opening them.
-    pub fn index_names(&self) -> Result<Vec<String>> {
-        let rtxn = self.env.read_txn()?;
-        self.index_mapper.index_names(&rtxn)
-    }
-
-    /// Attempts `f` for each index that exists known to the index scheduler.
-    ///
-    /// It is preferable to use this function rather than a loop that opens all indexes, as a way to avoid having all indexes opened,
-    /// which is unsupported in general.
-    ///
-    /// Since `f` is allowed to return a result, and `Index` is cloneable, it is still possible to wrongly build e.g. a vector of
-    /// all the indexes, but this function makes it harder and so less likely to do accidentally.
-    ///
-    /// If many indexes exist, this operation can take time to complete (in the order of seconds for a 1000 of indexes) as it needs to open
-    /// all the indexes.
-    pub fn try_for_each_index<U, V>(&self, f: impl FnMut(&str, &Index) -> Result<U>) -> Result<V>
-    where
-        V: FromIterator<U>,
-    {
-        let rtxn = self.env.read_txn()?;
-        self.index_mapper.try_for_each_index(&rtxn, f)
-    }
-
-    /// Returns the total number of indexes available for the specified filter.
-    /// And a `Vec` of the index_uid + its stats
-    pub fn get_paginated_indexes_stats(
-        &self,
-        filters: &meilisearch_auth::AuthFilter,
-        from: usize,
-        limit: usize,
-    ) -> Result<(usize, Vec<(String, index_mapper::IndexStats)>)> {
-        let rtxn = self.read_txn()?;
-
-        let mut total = 0;
-        let mut iter = self
-            .index_mapper
-            .index_mapping
-            .iter(&rtxn)?
-            // in case of an error we want to keep the value to return it
-            .filter(|ret| {
-                ret.as_ref().map_or(true, |(name, _uuid)| filters.is_index_authorized(name))
-            })
-            .inspect(|_| total += 1)
-            .skip(from);
-        let ret = iter
-            .by_ref()
-            .take(limit)
-            .map(|ret| ret.map_err(Error::from))
-            .map(|ret| {
-                ret.and_then(|(name, uuid)| {
-                    self.index_mapper.index_stats.get(&rtxn, &uuid).map_err(Error::from).and_then(
-                        |stat| {
-                            stat.map(|stat| (name.to_string(), stat))
-                                .ok_or(Error::CorruptedTaskQueue)
-                        },
-                    )
-                })
-            })
-            .collect::<Result<Vec<(String, index_mapper::IndexStats)>>>();
-
-        // We must iterate on the rest of the indexes to compute the total
-        iter.for_each(drop);
-
-        ret.map(|ret| (total, ret))
-    }
-
-    /// The returned structure contains:
-    /// 1. The name of the property being observed can be `statuses`, `types`, or `indexes`.
-    /// 2. The name of the specific data related to the property can be `enqueued` for the `statuses`, `settingsUpdate` for the `types`, or the name of the index for the `indexes`, for example.
-    /// 3. The number of times the properties appeared.
-    pub fn get_stats(&self) -> Result<BTreeMap<String, BTreeMap<String, u64>>> {
-        let rtxn = self.read_txn()?;
-        self.queue.get_stats(&rtxn, &self.processing_tasks.read().unwrap())
-    }
-
-    // Return true if there is at least one task that is processing.
-    pub fn is_task_processing(&self) -> Result<bool> {
-        Ok(!self.processing_tasks.read().unwrap().processing.is_empty())
-    }
-
-    /// Return true iff there is at least one task associated with this index
-    /// that is processing.
-    pub fn is_index_processing(&self, index: &str) -> Result<bool> {
-        let rtxn = self.env.read_txn()?;
-        let processing_tasks = self.processing_tasks.read().unwrap().processing.clone();
-        let index_tasks = self.queue.tasks.index_tasks(&rtxn, index)?;
-        let nbr_index_processing_tasks = processing_tasks.intersection_len(&index_tasks);
-        Ok(nbr_index_processing_tasks > 0)
-    }
-
-    /// Return the tasks matching the query from the user's point of view along
-    /// with the total number of tasks matching the query, ignoring from and limit.
-    ///
-    /// There are two differences between an internal query and a query executed by
-    /// the user.
-    ///
-    /// 1. IndexSwap tasks are not publicly associated with any index, but they are associated
-    ///    with many indexes internally.
-    /// 2. The user may not have the rights to access the tasks (internally) associated with all indexes.
-    pub fn get_tasks_from_authorized_indexes(
-        &self,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-    ) -> Result<(Vec<Task>, u64)> {
-        let rtxn = self.read_txn()?;
-        let processing = self.processing_tasks.read().unwrap();
-        self.queue.get_tasks_from_authorized_indexes(&rtxn, query, filters, &processing)
-    }
-
-    /// Return the task ids matching the query along with the total number of tasks
-    /// by ignoring the from and limit parameters from the user's point of view.
-    ///
-    /// There are two differences between an internal query and a query executed by
-    /// the user.
-    ///
-    /// 1. IndexSwap tasks are not publicly associated with any index, but they are associated
-    ///    with many indexes internally.
-    /// 2. The user may not have the rights to access the tasks (internally) associated with all indexes.
-    pub fn get_task_ids_from_authorized_indexes(
-        &self,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-    ) -> Result<(RoaringBitmap, u64)> {
-        let rtxn = self.read_txn()?;
-        let processing = self.processing_tasks.read().unwrap();
-        self.queue.get_task_ids_from_authorized_indexes(&rtxn, query, filters, &processing)
-    }
-
-    /// Return the batches matching the query from the user's point of view along
-    /// with the total number of batches matching the query, ignoring from and limit.
-    ///
-    /// There are two differences between an internal query and a query executed by
-    /// the user.
-    ///
-    /// 1. IndexSwap tasks are not publicly associated with any index, but they are associated
-    ///    with many indexes internally.
-    /// 2. The user may not have the rights to access the tasks (internally) associated with all indexes.
-    pub fn get_batches_from_authorized_indexes(
-        &self,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-    ) -> Result<(Vec<Batch>, u64)> {
-        let rtxn = self.read_txn()?;
-        let processing = self.processing_tasks.read().unwrap();
-        self.queue.get_batches_from_authorized_indexes(&rtxn, query, filters, &processing)
-    }
-
-    /// Return the batch ids matching the query along with the total number of batches
-    /// by ignoring the from and limit parameters from the user's point of view.
-    ///
-    /// There are two differences between an internal query and a query executed by
-    /// the user.
-    ///
-    /// 1. IndexSwap tasks are not publicly associated with any index, but they are associated
-    ///    with many indexes internally.
-    /// 2. The user may not have the rights to access the tasks (internally) associated with all indexes.
-    pub fn get_batch_ids_from_authorized_indexes(
-        &self,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-    ) -> Result<(RoaringBitmap, u64)> {
-        let rtxn = self.read_txn()?;
-        let processing = self.processing_tasks.read().unwrap();
-        self.queue.get_batch_ids_from_authorized_indexes(&rtxn, query, filters, &processing)
-    }
-
-    /// Register a new task in the scheduler.
-    ///
-    /// If it fails and data was associated with the task, it tries to delete the associated data.
-    pub fn register(
-        &self,
-        kind: KindWithContent,
-        task_id: Option<TaskId>,
-        dry_run: bool,
-    ) -> Result<Task> {
-        // if the task doesn't delete anything and 50% of the task queue is full, we must refuse to enqueue the incomming task
-        if !matches!(&kind, KindWithContent::TaskDeletion { tasks, .. } if !tasks.is_empty())
-            && (self.env.non_free_pages_size()? * 100) / self.env.info().map_size as u64 > 40
-        {
-            return Err(Error::NoSpaceLeftInTaskQueue);
-        }
-
-        let mut wtxn = self.env.write_txn()?;
-        let task = self.queue.register(&mut wtxn, &kind, task_id, dry_run)?;
-
-        // If the registered task is a task cancelation
-        // we inform the processing tasks to stop (if necessary).
-        if let KindWithContent::TaskCancelation { tasks, .. } = kind {
-            let tasks_to_cancel = RoaringBitmap::from_iter(tasks);
-            if self.processing_tasks.read().unwrap().must_cancel_processing_tasks(&tasks_to_cancel)
-            {
-                self.scheduler.must_stop_processing.must_stop();
-            }
-        }
-
-        if let Err(e) = wtxn.commit() {
-            self.queue.delete_persisted_task_data(&task)?;
-            return Err(e.into());
-        }
-
-        // notify the scheduler loop to execute a new tick
-        self.scheduler.wake_up.signal();
-        Ok(task)
-    }
-
-    /// Register a new task coming from a dump in the scheduler.
-    /// By taking a mutable ref we're pretty sure no one will ever import a dump while actix is running.
-    pub fn register_dumped_task(&mut self) -> Result<Dump> {
-        Dump::new(self)
-    }
-
-    /// Create a new index without any associated task.
-    pub fn create_raw_index(
-        &self,
-        name: &str,
-        date: Option<(OffsetDateTime, OffsetDateTime)>,
-    ) -> Result<Index> {
-        let wtxn = self.env.write_txn()?;
-        let index = self.index_mapper.create_index(wtxn, name, date)?;
-        Ok(index)
-    }
-
-    pub fn refresh_index_stats(&self, name: &str) -> Result<()> {
-        let mut mapper_wtxn = self.env.write_txn()?;
-        let index = self.index_mapper.index(&mapper_wtxn, name)?;
-        let index_rtxn = index.read_txn()?;
-
-        let stats = crate::index_mapper::IndexStats::new(&index, &index_rtxn)
-            .map_err(|e| Error::from_milli(e, Some(name.to_string())))?;
-
-        self.index_mapper.store_stats_of(&mut mapper_wtxn, name, &stats)?;
-        mapper_wtxn.commit()?;
-        Ok(())
-    }
-
-    /// Once the tasks changes have been committed we must send all the tasks that were updated to our webhook if there is one.
-    fn notify_webhook(&self, updated: &RoaringBitmap) -> Result<()> {
-        if let Some(ref url) = self.webhook_url {
-            struct TaskReader<'a, 'b> {
-                rtxn: &'a RoTxn<'a>,
-                index_scheduler: &'a IndexScheduler,
-                tasks: &'b mut roaring::bitmap::Iter<'b>,
-                buffer: Vec<u8>,
-                written: usize,
-            }
-
-            impl<'a, 'b> Read for TaskReader<'a, 'b> {
-                fn read(&mut self, mut buf: &mut [u8]) -> std::io::Result<usize> {
-                    if self.buffer.is_empty() {
-                        match self.tasks.next() {
-                            None => return Ok(0),
-                            Some(task_id) => {
-                                let task = self
-                                    .index_scheduler
-                                    .queue
-                                    .tasks
-                                    .get_task(self.rtxn, task_id)
-                                    .map_err(|err| io::Error::new(io::ErrorKind::Other, err))?
-                                    .ok_or_else(|| {
-                                        io::Error::new(
-                                            io::ErrorKind::Other,
-                                            Error::CorruptedTaskQueue,
-                                        )
-                                    })?;
-
-                                serde_json::to_writer(
-                                    &mut self.buffer,
-                                    &TaskView::from_task(&task),
-                                )?;
-                                self.buffer.push(b'\n');
-                            }
-                        }
-                    }
-
-                    let mut to_write = &self.buffer[self.written..];
-                    let wrote = io::copy(&mut to_write, &mut buf)?;
-                    self.written += wrote as usize;
-
-                    // we wrote everything and must refresh our buffer on the next call
-                    if self.written == self.buffer.len() {
-                        self.written = 0;
-                        self.buffer.clear();
-                    }
-
-                    Ok(wrote as usize)
-                }
-            }
-
-            let rtxn = self.env.read_txn()?;
-
-            let task_reader = TaskReader {
-                rtxn: &rtxn,
-                index_scheduler: self,
-                tasks: &mut updated.into_iter(),
-                buffer: Vec::with_capacity(50), // on average a task is around ~100 bytes
-                written: 0,
-            };
-
-            // let reader = GzEncoder::new(BufReader::new(task_reader), Compression::default());
-            let reader = GzEncoder::new(BufReader::new(task_reader), Compression::default());
-            let request = ureq::post(url)
-                .timeout(Duration::from_secs(30))
-                .set("Content-Encoding", "gzip")
-                .set("Content-Type", "application/x-ndjson");
-            let request = match &self.webhook_authorization_header {
-                Some(header) => request.set("Authorization", header),
-                None => request,
-            };
-
-            if let Err(e) = request.send(reader) {
-                tracing::error!("While sending data to the webhook: {e}");
-            }
-        }
-
-        Ok(())
-    }
-
-    pub fn index_stats(&self, index_uid: &str) -> Result<IndexStats> {
-        let is_indexing = self.is_index_processing(index_uid)?;
-        let rtxn = self.read_txn()?;
-        let index_stats = self.index_mapper.stats_of(&rtxn, index_uid)?;
-
-        Ok(IndexStats { is_indexing, inner_stats: index_stats })
-    }
-
-    pub fn features(&self) -> RoFeatures {
-        self.features.features()
-    }
-
-    pub fn put_runtime_features(&self, features: RuntimeTogglableFeatures) -> Result<()> {
-        let wtxn = self.env.write_txn().map_err(Error::HeedTransaction)?;
-        self.features.put_runtime_features(wtxn, features)?;
-        Ok(())
-    }
-
-    pub fn put_network(&self, network: Network) -> Result<()> {
-        let wtxn = self.env.write_txn().map_err(Error::HeedTransaction)?;
-        self.features.put_network(wtxn, network)?;
-        Ok(())
-    }
-
-    pub fn network(&self) -> Network {
-        self.features.network()
-    }
-
-    pub fn embedders(
-        &self,
-        index_uid: String,
-        embedding_configs: Vec<IndexEmbeddingConfig>,
-    ) -> Result<EmbeddingConfigs> {
-        let res: Result<_> = embedding_configs
-            .into_iter()
-            .map(
-                |IndexEmbeddingConfig {
-                     name,
-                     config: milli::vector::EmbeddingConfig { embedder_options, prompt, quantized },
-                     ..
-                 }| {
-                    let prompt = Arc::new(
-                        prompt
-                            .try_into()
-                            .map_err(meilisearch_types::milli::Error::from)
-                            .map_err(|err| Error::from_milli(err, Some(index_uid.clone())))?,
-                    );
-                    // optimistically return existing embedder
-                    {
-                        let embedders = self.embedders.read().unwrap();
-                        if let Some(embedder) = embedders.get(&embedder_options) {
-                            return Ok((
-                                name,
-                                (embedder.clone(), prompt, quantized.unwrap_or_default()),
-                            ));
-                        }
-                    }
-
-                    // add missing embedder
-                    let embedder = Arc::new(
-                        Embedder::new(embedder_options.clone(), self.scheduler.embedding_cache_cap)
-                            .map_err(meilisearch_types::milli::vector::Error::from)
-                            .map_err(|err| {
-                                Error::from_milli(err.into(), Some(index_uid.clone()))
-                            })?,
-                    );
-                    {
-                        let mut embedders = self.embedders.write().unwrap();
-                        embedders.insert(embedder_options, embedder.clone());
-                    }
-                    Ok((name, (embedder, prompt, quantized.unwrap_or_default())))
-                },
-            )
-            .collect();
-        res.map(EmbeddingConfigs::new)
-    }
-}
-
-/// The outcome of calling the [`IndexScheduler::tick`] function.
-pub enum TickOutcome {
-    /// The scheduler should immediately attempt another `tick`.
-    ///
-    /// The `usize` field contains the number of processed tasks.
-    TickAgain(u64),
-    /// The scheduler should wait for an external signal before attempting another `tick`.
-    WaitForSignal,
-    /// The scheduler exits the run-loop and will never process tasks again
-    StopProcessingForever,
-}
-
-/// How many indexes we can afford to have open simultaneously.
-struct IndexBudget {
-    /// Map size of an index.
-    map_size: usize,
-    /// Maximum number of simultaneously opened indexes.
-    index_count: usize,
-    /// For very constrained systems we might need to reduce the base task_db_size so we can accept at least one index.
-    task_db_size: usize,
-}
-
-/// The statistics that can be computed from an `Index` object and the scheduler.
-///
-/// Compared with `index_mapper::IndexStats`, it adds the scheduling status.
-#[derive(Debug)]
-pub struct IndexStats {
-    /// Whether this index is currently performing indexation, according to the scheduler.
-    pub is_indexing: bool,
-    /// Internal stats computed from the index.
-    pub inner_stats: index_mapper::IndexStats,
-}

crates/index-scheduler/src/lru.rs

@@ -1,203 +0,0 @@
-//! Thread-safe `Vec`-backend LRU cache using [`std::sync::atomic::AtomicU64`] for synchronization.
-
-use std::sync::atomic::{AtomicU64, Ordering};
-
-/// Thread-safe `Vec`-backend LRU cache
-#[derive(Debug)]
-pub struct Lru<T> {
-    data: Vec<(AtomicU64, T)>,
-    generation: AtomicU64,
-    cap: usize,
-}
-
-impl<T> Lru<T> {
-    /// Creates a new LRU cache with the specified capacity.
-    ///
-    /// The capacity is allocated up-front, and will never change through a [`Self::put`] operation.
-    ///
-    /// # Panics
-    ///
-    /// - If the capacity is 0.
-    /// - If the capacity exceeds `isize::MAX` bytes.
-    pub fn new(cap: usize) -> Self {
-        assert_ne!(cap, 0, "The capacity of a cache cannot be 0");
-        Self {
-            // Note: since the element of the vector contains an AtomicU64, it is definitely not zero-sized so cap will never be usize::MAX.
-            data: Vec::with_capacity(cap),
-            generation: AtomicU64::new(0),
-            cap,
-        }
-    }
-
-    /// The capacity of this LRU cache, that is the maximum number of elements it can hold before evicting elements from the cache.
-    ///
-    /// The cache will contain at most this number of elements at any given time.
-    pub fn capacity(&self) -> usize {
-        self.cap
-    }
-
-    fn next_generation(&self) -> u64 {
-        // Acquire so this "happens-before" any potential store to a data cell (with Release ordering)
-        let generation = self.generation.fetch_add(1, Ordering::Acquire);
-        generation + 1
-    }
-
-    fn next_generation_mut(&mut self) -> u64 {
-        let generation = self.generation.get_mut();
-        *generation += 1;
-        *generation
-    }
-
-    /// Add a value in the cache, evicting an older value if necessary.
-    ///
-    /// If a value was evicted from the cache, it is returned.
-    ///
-    /// # Complexity
-    ///
-    /// - If the cache is full, then linear in the capacity.
-    /// - Otherwise constant.
-    pub fn put(&mut self, value: T) -> Option<T> {
-        // no need for a memory fence: we assume that whichever mechanism provides us synchronization
-        // (very probably, a RwLock) takes care of fencing for us.
-
-        let next_generation = self.next_generation_mut();
-        let evicted = if self.is_full() { self.pop() } else { None };
-        self.data.push((AtomicU64::new(next_generation), value));
-        evicted
-    }
-
-    /// Evict the oldest value from the cache.
-    ///
-    /// If the cache is empty, `None` will be returned.
-    ///
-    /// # Complexity
-    ///
-    /// - Linear in the capacity of the cache.
-    pub fn pop(&mut self) -> Option<T> {
-        // Don't use `Iterator::min_by_key` that provides shared references to its elements,
-        // so that we can get an exclusive one.
-        // This allows to handles the `AtomicU64`s as normal integers without using atomic instructions.
-        let mut min_generation_index = None;
-        for (index, (generation, _)) in self.data.iter_mut().enumerate() {
-            let generation = *generation.get_mut();
-            if let Some((_, min_generation)) = min_generation_index {
-                if min_generation > generation {
-                    min_generation_index = Some((index, generation));
-                }
-            } else {
-                min_generation_index = Some((index, generation))
-            }
-        }
-        min_generation_index.map(|(min_index, _)| self.data.swap_remove(min_index).1)
-    }
-
-    /// The current number of elements in the cache.
-    ///
-    /// This value is guaranteed to be less than or equal to [`Self::capacity`].
-    pub fn len(&self) -> usize {
-        self.data.len()
-    }
-
-    /// Returns `true` if putting any additional element in the cache would cause the eviction of an element.
-    pub fn is_full(&self) -> bool {
-        self.len() == self.capacity()
-    }
-}
-
-pub struct LruMap<K, V>(Lru<(K, V)>);
-
-impl<K, V> LruMap<K, V>
-where
-    K: Eq,
-{
-    /// Creates a new LRU cache map with the specified capacity.
-    ///
-    /// The capacity is allocated up-front, and will never change through a [`Self::insert`] operation.
-    ///
-    /// # Panics
-    ///
-    /// - If the capacity is 0.
-    /// - If the capacity exceeds `isize::MAX` bytes.
-    pub fn new(cap: usize) -> Self {
-        Self(Lru::new(cap))
-    }
-
-    /// Gets a value in the cache map by its key.
-    ///
-    /// If no value matches, `None` will be returned.
-    ///
-    /// # Complexity
-    ///
-    /// - Linear in the capacity of the cache.
-    pub fn get(&self, key: &K) -> Option<&V> {
-        for (generation, (candidate, value)) in self.0.data.iter() {
-            if key == candidate {
-                generation.store(self.0.next_generation(), Ordering::Release);
-                return Some(value);
-            }
-        }
-        None
-    }
-
-    /// Gets a value in the cache map by its key.
-    ///
-    /// If no value matches, `None` will be returned.
-    ///
-    /// # Complexity
-    ///
-    /// - Linear in the capacity of the cache.
-    pub fn get_mut(&mut self, key: &K) -> Option<&mut V> {
-        let next_generation = self.0.next_generation_mut();
-        for (generation, (candidate, value)) in self.0.data.iter_mut() {
-            if key == candidate {
-                *generation.get_mut() = next_generation;
-                return Some(value);
-            }
-        }
-        None
-    }
-
-    /// Inserts a value in the cache map by its key, replacing any existing value and returning any evicted value.
-    ///
-    /// # Complexity
-    ///
-    /// - Linear in the capacity of the cache.
-    pub fn insert(&mut self, key: K, mut value: V) -> InsertionOutcome<K, V> {
-        match self.get_mut(&key) {
-            Some(old_value) => {
-                std::mem::swap(old_value, &mut value);
-                InsertionOutcome::Replaced(value)
-            }
-            None => match self.0.put((key, value)) {
-                Some((key, value)) => InsertionOutcome::Evicted(key, value),
-                None => InsertionOutcome::InsertedNew,
-            },
-        }
-    }
-
-    /// Removes an element from the cache map by its key, returning its value.
-    ///
-    /// Returns `None` if there was no element with this key in the cache.
-    ///
-    /// # Complexity
-    ///
-    /// - Linear in the capacity of the cache.
-    pub fn remove(&mut self, key: &K) -> Option<V> {
-        for (index, (_, (candidate, _))) in self.0.data.iter_mut().enumerate() {
-            if key == candidate {
-                return Some(self.0.data.swap_remove(index).1 .1);
-            }
-        }
-        None
-    }
-}
-
-/// The result of an insertion in a LRU map.
-pub enum InsertionOutcome<K, V> {
-    /// The key was not in the cache, the key-value pair has been inserted.
-    InsertedNew,
-    /// The key was not in the cache and an old key-value pair was evicted from the cache to make room for its insertions.
-    Evicted(K, V),
-    /// The key was already in the cache map, its value has been updated.
-    Replaced(V),
-}

crates/index-scheduler/src/processing.rs

@@ -1,296 +0,0 @@
-use std::sync::Arc;
-
-use meilisearch_types::milli::progress::{AtomicSubStep, NamedStep, Progress, ProgressView};
-use meilisearch_types::milli::{make_atomic_progress, make_enum_progress};
-use roaring::RoaringBitmap;
-
-use crate::utils::ProcessingBatch;
-
-#[derive(Clone, Default)]
-pub struct ProcessingTasks {
-    pub batch: Option<Arc<ProcessingBatch>>,
-    /// The list of tasks ids that are currently running.
-    pub processing: Arc<RoaringBitmap>,
-    /// The progress on processing tasks
-    pub progress: Option<Progress>,
-}
-
-impl ProcessingTasks {
-    /// Creates an empty `ProcessingAt` struct.
-    pub fn new() -> ProcessingTasks {
-        ProcessingTasks::default()
-    }
-
-    pub fn get_progress_view(&self) -> Option<ProgressView> {
-        Some(self.progress.as_ref()?.as_progress_view())
-    }
-
-    /// Stores the currently processing tasks, and the date time at which it started.
-    pub fn start_processing(
-        &mut self,
-        processing_batch: ProcessingBatch,
-        processing: RoaringBitmap,
-    ) -> Progress {
-        self.batch = Some(Arc::new(processing_batch));
-        self.processing = Arc::new(processing);
-        let progress = Progress::default();
-        progress.update_progress(BatchProgress::ProcessingTasks);
-        self.progress = Some(progress.clone());
-
-        progress
-    }
-
-    /// Set the processing tasks to an empty list
-    pub fn stop_processing(&mut self) -> Self {
-        self.progress = None;
-
-        Self {
-            batch: std::mem::take(&mut self.batch),
-            processing: std::mem::take(&mut self.processing),
-            progress: None,
-        }
-    }
-
-    /// Returns `true` if there, at least, is one task that is currently processing that we must stop.
-    pub fn must_cancel_processing_tasks(&self, canceled_tasks: &RoaringBitmap) -> bool {
-        !self.processing.is_disjoint(canceled_tasks)
-    }
-}
-
-make_enum_progress! {
-    pub enum BatchProgress {
-        ProcessingTasks,
-        WritingTasksToDisk,
-    }
-}
-
-make_enum_progress! {
-    pub enum FinalizingIndexStep {
-        Committing,
-        ComputingStats,
-    }
-}
-
-make_enum_progress! {
-    pub enum TaskCancelationProgress {
-        RetrievingTasks,
-        UpdatingTasks,
-    }
-}
-
-make_enum_progress! {
-    pub enum TaskDeletionProgress {
-        DeletingTasksDateTime,
-        DeletingTasksMetadata,
-        DeletingTasks,
-        DeletingBatches,
-    }
-}
-
-make_enum_progress! {
-    pub enum SnapshotCreationProgress {
-        StartTheSnapshotCreation,
-        SnapshotTheIndexScheduler,
-        SnapshotTheUpdateFiles,
-        SnapshotTheIndexes,
-        SnapshotTheApiKeys,
-        CreateTheTarball,
-    }
-}
-
-make_enum_progress! {
-    pub enum DumpCreationProgress {
-        StartTheDumpCreation,
-        DumpTheApiKeys,
-        DumpTheTasks,
-        DumpTheBatches,
-        DumpTheIndexes,
-        DumpTheExperimentalFeatures,
-        CompressTheDump,
-    }
-}
-
-make_enum_progress! {
-    pub enum CreateIndexProgress {
-        CreatingTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum UpdateIndexProgress {
-        UpdatingTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum DeleteIndexProgress {
-        DeletingTheIndex,
-    }
-}
-
-make_enum_progress! {
-    pub enum SwappingTheIndexes {
-        EnsuringCorrectnessOfTheSwap,
-        SwappingTheIndexes,
-    }
-}
-
-make_enum_progress! {
-    pub enum InnerSwappingTwoIndexes {
-        RetrieveTheTasks,
-        UpdateTheTasks,
-        UpdateTheIndexesMetadata,
-    }
-}
-
-make_enum_progress! {
-    pub enum DocumentOperationProgress {
-        RetrievingConfig,
-        ComputingDocumentChanges,
-        Indexing,
-    }
-}
-
-make_enum_progress! {
-    pub enum DocumentEditionProgress {
-        RetrievingConfig,
-        ComputingDocumentChanges,
-        Indexing,
-    }
-}
-
-make_enum_progress! {
-    pub enum DocumentDeletionProgress {
-        RetrievingConfig,
-        DeleteDocuments,
-        Indexing,
-    }
-}
-
-make_enum_progress! {
-    pub enum SettingsProgress {
-        RetrievingAndMergingTheSettings,
-        ApplyTheSettings,
-    }
-}
-
-make_atomic_progress!(Task alias AtomicTaskStep => "task" );
-make_atomic_progress!(Document alias AtomicDocumentStep => "document" );
-make_atomic_progress!(Batch alias AtomicBatchStep => "batch" );
-make_atomic_progress!(UpdateFile alias AtomicUpdateFileStep => "update file" );
-
-#[cfg(test)]
-mod test {
-    use std::sync::atomic::Ordering;
-
-    use meili_snap::{json_string, snapshot};
-
-    use super::*;
-
-    #[test]
-    fn one_level() {
-        let mut processing = ProcessingTasks::new();
-        processing.start_processing(ProcessingBatch::new(0), RoaringBitmap::new());
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "processing tasks",
-              "finished": 0,
-              "total": 2
-            }
-          ],
-          "percentage": 0.0
-        }
-        "#);
-        processing.progress.as_ref().unwrap().update_progress(BatchProgress::WritingTasksToDisk);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "writing tasks to disk",
-              "finished": 1,
-              "total": 2
-            }
-          ],
-          "percentage": 50.0
-        }
-        "#);
-    }
-
-    #[test]
-    fn task_progress() {
-        let mut processing = ProcessingTasks::new();
-        processing.start_processing(ProcessingBatch::new(0), RoaringBitmap::new());
-        let (atomic, tasks) = AtomicTaskStep::new(10);
-        processing.progress.as_ref().unwrap().update_progress(tasks);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "processing tasks",
-              "finished": 0,
-              "total": 2
-            },
-            {
-              "currentStep": "task",
-              "finished": 0,
-              "total": 10
-            }
-          ],
-          "percentage": 0.0
-        }
-        "#);
-        atomic.fetch_add(6, Ordering::Relaxed);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "processing tasks",
-              "finished": 0,
-              "total": 2
-            },
-            {
-              "currentStep": "task",
-              "finished": 6,
-              "total": 10
-            }
-          ],
-          "percentage": 30.000002
-        }
-        "#);
-        processing.progress.as_ref().unwrap().update_progress(BatchProgress::WritingTasksToDisk);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "writing tasks to disk",
-              "finished": 1,
-              "total": 2
-            }
-          ],
-          "percentage": 50.0
-        }
-        "#);
-        let (atomic, tasks) = AtomicTaskStep::new(5);
-        processing.progress.as_ref().unwrap().update_progress(tasks);
-        atomic.fetch_add(4, Ordering::Relaxed);
-        snapshot!(json_string!(processing.get_progress_view()), @r#"
-        {
-          "steps": [
-            {
-              "currentStep": "writing tasks to disk",
-              "finished": 1,
-              "total": 2
-            },
-            {
-              "currentStep": "task",
-              "finished": 4,
-              "total": 5
-            }
-          ],
-          "percentage": 90.0
-        }
-        "#);
-    }
-}

crates/index-scheduler/src/queue/batches.rs

@@ -1,603 +0,0 @@
-use std::collections::HashSet;
-use std::ops::{Bound, RangeBounds};
-
-use meilisearch_types::batches::{Batch, BatchId};
-use meilisearch_types::heed::types::{DecodeIgnore, SerdeBincode, SerdeJson, Str};
-use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
-use meilisearch_types::milli::{CboRoaringBitmapCodec, RoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Kind, Status};
-use roaring::{MultiOps, RoaringBitmap};
-use time::OffsetDateTime;
-
-use super::{Query, Queue};
-use crate::processing::ProcessingTasks;
-use crate::utils::{
-    insert_task_datetime, keep_ids_within_datetimes, map_bound,
-    remove_n_tasks_datetime_earlier_than, remove_task_datetime, ProcessingBatch,
-};
-use crate::{Error, Result, BEI128};
-
-/// The number of database used by the batch queue
-const NUMBER_OF_DATABASES: u32 = 7;
-/// Database const names for the `IndexScheduler`.
-mod db_name {
-    pub const ALL_BATCHES: &str = "all-batches";
-
-    pub const BATCH_STATUS: &str = "batch-status";
-    pub const BATCH_KIND: &str = "batch-kind";
-    pub const BATCH_INDEX_TASKS: &str = "batch-index-tasks";
-    pub const BATCH_ENQUEUED_AT: &str = "batch-enqueued-at";
-    pub const BATCH_STARTED_AT: &str = "batch-started-at";
-    pub const BATCH_FINISHED_AT: &str = "batch-finished-at";
-}
-
-pub struct BatchQueue {
-    /// Contains all the batches accessible by their Id.
-    pub(crate) all_batches: Database<BEU32, SerdeJson<Batch>>,
-
-    /// All the batches containing a task matching the selected status.
-    pub(crate) status: Database<SerdeBincode<Status>, RoaringBitmapCodec>,
-    /// All the batches ids grouped by the kind of their task.
-    pub(crate) kind: Database<SerdeBincode<Kind>, RoaringBitmapCodec>,
-    /// Store the batches associated to an index.
-    pub(crate) index_tasks: Database<Str, RoaringBitmapCodec>,
-    /// Store the batches containing tasks which were enqueued at a specific date
-    pub(crate) enqueued_at: Database<BEI128, CboRoaringBitmapCodec>,
-    /// Store the batches containing finished tasks started at a specific date
-    pub(crate) started_at: Database<BEI128, CboRoaringBitmapCodec>,
-    /// Store the batches containing tasks finished at a specific date
-    pub(crate) finished_at: Database<BEI128, CboRoaringBitmapCodec>,
-}
-
-impl BatchQueue {
-    pub(crate) fn private_clone(&self) -> BatchQueue {
-        BatchQueue {
-            all_batches: self.all_batches,
-            status: self.status,
-            kind: self.kind,
-            index_tasks: self.index_tasks,
-            enqueued_at: self.enqueued_at,
-            started_at: self.started_at,
-            finished_at: self.finished_at,
-        }
-    }
-
-    pub(crate) const fn nb_db() -> u32 {
-        NUMBER_OF_DATABASES
-    }
-
-    pub(super) fn new(env: &Env<WithoutTls>, wtxn: &mut RwTxn) -> Result<Self> {
-        Ok(Self {
-            all_batches: env.create_database(wtxn, Some(db_name::ALL_BATCHES))?,
-            status: env.create_database(wtxn, Some(db_name::BATCH_STATUS))?,
-            kind: env.create_database(wtxn, Some(db_name::BATCH_KIND))?,
-            index_tasks: env.create_database(wtxn, Some(db_name::BATCH_INDEX_TASKS))?,
-            enqueued_at: env.create_database(wtxn, Some(db_name::BATCH_ENQUEUED_AT))?,
-            started_at: env.create_database(wtxn, Some(db_name::BATCH_STARTED_AT))?,
-            finished_at: env.create_database(wtxn, Some(db_name::BATCH_FINISHED_AT))?,
-        })
-    }
-
-    pub(crate) fn all_batch_ids(&self, rtxn: &RoTxn) -> Result<RoaringBitmap> {
-        enum_iterator::all().map(|s| self.get_status(rtxn, s)).union()
-    }
-
-    pub(crate) fn next_batch_id(&self, rtxn: &RoTxn) -> Result<BatchId> {
-        Ok(self
-            .all_batches
-            .remap_data_type::<DecodeIgnore>()
-            .last(rtxn)?
-            .map(|(k, _)| k + 1)
-            .unwrap_or_default())
-    }
-
-    pub(crate) fn get_batch(&self, rtxn: &RoTxn, batch_id: BatchId) -> Result<Option<Batch>> {
-        Ok(self.all_batches.get(rtxn, &batch_id)?)
-    }
-
-    /// Returns the whole set of batches that belongs to this index.
-    pub(crate) fn index_batches(&self, rtxn: &RoTxn, index: &str) -> Result<RoaringBitmap> {
-        Ok(self.index_tasks.get(rtxn, index)?.unwrap_or_default())
-    }
-
-    pub(crate) fn update_index(
-        &self,
-        wtxn: &mut RwTxn,
-        index: &str,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut batches = self.index_batches(wtxn, index)?;
-        f(&mut batches);
-        if batches.is_empty() {
-            self.index_tasks.delete(wtxn, index)?;
-        } else {
-            self.index_tasks.put(wtxn, index, &batches)?;
-        }
-
-        Ok(())
-    }
-
-    pub(crate) fn get_status(&self, rtxn: &RoTxn, status: Status) -> Result<RoaringBitmap> {
-        Ok(self.status.get(rtxn, &status)?.unwrap_or_default())
-    }
-
-    pub(crate) fn put_status(
-        &self,
-        wtxn: &mut RwTxn,
-        status: Status,
-        bitmap: &RoaringBitmap,
-    ) -> Result<()> {
-        Ok(self.status.put(wtxn, &status, bitmap)?)
-    }
-
-    pub(crate) fn update_status(
-        &self,
-        wtxn: &mut RwTxn,
-        status: Status,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.get_status(wtxn, status)?;
-        f(&mut tasks);
-        self.put_status(wtxn, status, &tasks)?;
-
-        Ok(())
-    }
-
-    pub(crate) fn get_kind(&self, rtxn: &RoTxn, kind: Kind) -> Result<RoaringBitmap> {
-        Ok(self.kind.get(rtxn, &kind)?.unwrap_or_default())
-    }
-
-    pub(crate) fn put_kind(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: Kind,
-        bitmap: &RoaringBitmap,
-    ) -> Result<()> {
-        Ok(self.kind.put(wtxn, &kind, bitmap)?)
-    }
-
-    pub(crate) fn update_kind(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: Kind,
-        f: impl Fn(&mut RoaringBitmap),
-    ) -> Result<()> {
-        let mut tasks = self.get_kind(wtxn, kind)?;
-        f(&mut tasks);
-        self.put_kind(wtxn, kind, &tasks)?;
-        Ok(())
-    }
-
-    pub(crate) fn write_batch(&self, wtxn: &mut RwTxn, batch: ProcessingBatch) -> Result<()> {
-        let old_batch = self.all_batches.get(wtxn, &batch.uid)?;
-
-        self.all_batches.put(
-            wtxn,
-            &batch.uid,
-            &Batch {
-                uid: batch.uid,
-                progress: None,
-                details: batch.details,
-                stats: batch.stats,
-                started_at: batch.started_at,
-                finished_at: batch.finished_at,
-                enqueued_at: batch.enqueued_at,
-            },
-        )?;
-
-        // Update the statuses
-        if let Some(ref old_batch) = old_batch {
-            for status in old_batch.stats.status.keys() {
-                self.update_status(wtxn, *status, |bitmap| {
-                    bitmap.remove(batch.uid);
-                })?;
-            }
-        }
-        for status in batch.statuses {
-            self.update_status(wtxn, status, |bitmap| {
-                bitmap.insert(batch.uid);
-            })?;
-        }
-
-        // Update the kinds / types
-        if let Some(ref old_batch) = old_batch {
-            let kinds: HashSet<_> = old_batch.stats.types.keys().cloned().collect();
-            for kind in kinds.difference(&batch.kinds) {
-                self.update_kind(wtxn, *kind, |bitmap| {
-                    bitmap.remove(batch.uid);
-                })?;
-            }
-        }
-        for kind in batch.kinds {
-            self.update_kind(wtxn, kind, |bitmap| {
-                bitmap.insert(batch.uid);
-            })?;
-        }
-
-        // Update the indexes
-        if let Some(ref old_batch) = old_batch {
-            let indexes: HashSet<_> = old_batch.stats.index_uids.keys().cloned().collect();
-            for index in indexes.difference(&batch.indexes) {
-                self.update_index(wtxn, index, |bitmap| {
-                    bitmap.remove(batch.uid);
-                })?;
-            }
-        }
-        for index in batch.indexes {
-            self.update_index(wtxn, &index, |bitmap| {
-                bitmap.insert(batch.uid);
-            })?;
-        }
-
-        // Update the enqueued_at: we cannot retrieve the previous enqueued at from the previous batch, and
-        // must instead go through the db looking for it. We cannot look at the task contained in this batch either
-        // because they may have been removed.
-        // What we know, though, is that the task date is from before the enqueued_at, and max two timestamps have been written
-        // to the DB per batches.
-        if let Some(ref old_batch) = old_batch {
-            if let Some(enqueued_at) = old_batch.enqueued_at {
-                remove_task_datetime(wtxn, self.enqueued_at, enqueued_at.earliest, old_batch.uid)?;
-                remove_task_datetime(wtxn, self.enqueued_at, enqueued_at.oldest, old_batch.uid)?;
-            } else {
-                // If we don't have the enqueued at in the batch it means the database comes from the v1.12
-                // and we still need to find the date by scrolling the database
-                remove_n_tasks_datetime_earlier_than(
-                    wtxn,
-                    self.enqueued_at,
-                    old_batch.started_at,
-                    old_batch.stats.total_nb_tasks.clamp(1, 2) as usize,
-                    old_batch.uid,
-                )?;
-            }
-        }
-        // A finished batch MUST contains at least one task and have an enqueued_at
-        let enqueued_at = batch.enqueued_at.as_ref().unwrap();
-        insert_task_datetime(wtxn, self.enqueued_at, enqueued_at.earliest, batch.uid)?;
-        insert_task_datetime(wtxn, self.enqueued_at, enqueued_at.oldest, batch.uid)?;
-
-        // Update the started at and finished at
-        if let Some(ref old_batch) = old_batch {
-            remove_task_datetime(wtxn, self.started_at, old_batch.started_at, old_batch.uid)?;
-            if let Some(finished_at) = old_batch.finished_at {
-                remove_task_datetime(wtxn, self.finished_at, finished_at, old_batch.uid)?;
-            }
-        }
-        insert_task_datetime(wtxn, self.started_at, batch.started_at, batch.uid)?;
-        insert_task_datetime(wtxn, self.finished_at, batch.finished_at.unwrap(), batch.uid)?;
-
-        Ok(())
-    }
-
-    /// Convert an iterator to a `Vec` of batches. The batches MUST exist or a
-    /// `CorruptedTaskQueue` error will be thrown.
-    pub(crate) fn get_existing_batches(
-        &self,
-        rtxn: &RoTxn,
-        tasks: impl IntoIterator<Item = BatchId>,
-        processing: &ProcessingTasks,
-    ) -> Result<Vec<Batch>> {
-        tasks
-            .into_iter()
-            .map(|batch_id| {
-                if Some(batch_id) == processing.batch.as_ref().map(|batch| batch.uid) {
-                    let mut batch = processing.batch.as_ref().unwrap().to_batch();
-                    batch.progress = processing.get_progress_view();
-                    Ok(batch)
-                } else {
-                    self.get_batch(rtxn, batch_id)
-                        .and_then(|task| task.ok_or(Error::CorruptedTaskQueue))
-                }
-            })
-            .collect::<Result<_>>()
-    }
-}
-
-impl Queue {
-    /// Return the batch ids matched by the given query from the index scheduler's point of view.
-    pub(crate) fn get_batch_ids(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        processing: &ProcessingTasks,
-    ) -> Result<RoaringBitmap> {
-        let Query {
-            limit,
-            from,
-            reverse,
-            uids,
-            batch_uids,
-            statuses,
-            types,
-            index_uids,
-            canceled_by,
-            before_enqueued_at,
-            after_enqueued_at,
-            before_started_at,
-            after_started_at,
-            before_finished_at,
-            after_finished_at,
-        } = query;
-
-        let mut batches = self.batches.all_batch_ids(rtxn)?;
-        if let Some(batch_id) = processing.batch.as_ref().map(|batch| batch.uid) {
-            batches.insert(batch_id);
-        }
-
-        if let Some(from) = from {
-            let range = if reverse.unwrap_or_default() {
-                u32::MIN..*from
-            } else {
-                from.saturating_add(1)..u32::MAX
-            };
-            batches.remove_range(range);
-        }
-
-        if let Some(batch_uids) = &batch_uids {
-            let batches_uids = RoaringBitmap::from_iter(batch_uids);
-            batches &= batches_uids;
-        }
-
-        if let Some(status) = &statuses {
-            let mut status_batches = RoaringBitmap::new();
-            for status in status {
-                match status {
-                    // special case for Processing batches
-                    Status::Processing => {
-                        if let Some(batch_id) = processing.batch.as_ref().map(|batch| batch.uid) {
-                            status_batches.insert(batch_id);
-                        }
-                    }
-                    // Enqueued tasks are not stored in batches
-                    Status::Enqueued => (),
-                    status => status_batches |= &self.batches.get_status(rtxn, *status)?,
-                };
-            }
-            if !status.contains(&Status::Processing) {
-                if let Some(ref batch) = processing.batch {
-                    batches.remove(batch.uid);
-                }
-            }
-            batches &= status_batches;
-        }
-
-        if let Some(task_uids) = &uids {
-            let mut batches_by_task_uids = RoaringBitmap::new();
-            for task_uid in task_uids {
-                if let Some(task) = self.tasks.get_task(rtxn, *task_uid)? {
-                    if let Some(batch_uid) = task.batch_uid {
-                        batches_by_task_uids.insert(batch_uid);
-                    }
-                }
-            }
-            batches &= batches_by_task_uids;
-        }
-
-        // There is no database for this query, we must retrieve the task queried by the client and ensure it's valid
-        if let Some(canceled_by) = &canceled_by {
-            let mut all_canceled_batches = RoaringBitmap::new();
-            for cancel_uid in canceled_by {
-                if let Some(task) = self.tasks.get_task(rtxn, *cancel_uid)? {
-                    if task.kind.as_kind() == Kind::TaskCancelation
-                        && task.status == Status::Succeeded
-                    {
-                        if let Some(batch_uid) = task.batch_uid {
-                            all_canceled_batches.insert(batch_uid);
-                        }
-                    }
-                }
-            }
-
-            // if the canceled_by has been specified but no batch
-            // matches then we prefer matching zero than all batches.
-            if all_canceled_batches.is_empty() {
-                return Ok(RoaringBitmap::new());
-            } else {
-                batches &= all_canceled_batches;
-            }
-        }
-
-        if let Some(kind) = &types {
-            let mut kind_batches = RoaringBitmap::new();
-            for kind in kind {
-                kind_batches |= self.batches.get_kind(rtxn, *kind)?;
-                if let Some(uid) = processing
-                    .batch
-                    .as_ref()
-                    .and_then(|batch| batch.kinds.contains(kind).then_some(batch.uid))
-                {
-                    kind_batches.insert(uid);
-                }
-            }
-            batches &= &kind_batches;
-        }
-
-        if let Some(index) = &index_uids {
-            let mut index_batches = RoaringBitmap::new();
-            for index in index {
-                index_batches |= self.batches.index_batches(rtxn, index)?;
-                if let Some(uid) = processing
-                    .batch
-                    .as_ref()
-                    .and_then(|batch| batch.indexes.contains(index).then_some(batch.uid))
-                {
-                    index_batches.insert(uid);
-                }
-            }
-            batches &= &index_batches;
-        }
-
-        // For the started_at filter, we need to treat the part of the batches that are processing from the part of the
-        // batches that are not processing. The non-processing ones are filtered normally while the processing ones
-        // are entirely removed unless the in-memory startedAt variable falls within the date filter.
-        // Once we have filtered the two subsets, we put them back together and assign it back to `batches`.
-        batches = {
-            let (mut filtered_non_processing_batches, mut filtered_processing_batches) =
-                (&batches - &*processing.processing, &batches & &*processing.processing);
-
-            // special case for Processing batches
-            // A closure that clears the filtered_processing_batches if their started_at date falls outside the given bounds
-            let mut clear_filtered_processing_batches =
-                |start: Bound<OffsetDateTime>, end: Bound<OffsetDateTime>| {
-                    let start = map_bound(start, |b| b.unix_timestamp_nanos());
-                    let end = map_bound(end, |b| b.unix_timestamp_nanos());
-                    let is_within_dates = RangeBounds::contains(
-                        &(start, end),
-                        &processing
-                            .batch
-                            .as_ref()
-                            .map_or_else(OffsetDateTime::now_utc, |batch| batch.started_at)
-                            .unix_timestamp_nanos(),
-                    );
-                    if !is_within_dates {
-                        filtered_processing_batches.clear();
-                    }
-                };
-            match (after_started_at, before_started_at) {
-                (None, None) => (),
-                (None, Some(before)) => {
-                    clear_filtered_processing_batches(Bound::Unbounded, Bound::Excluded(*before))
-                }
-                (Some(after), None) => {
-                    clear_filtered_processing_batches(Bound::Excluded(*after), Bound::Unbounded)
-                }
-                (Some(after), Some(before)) => clear_filtered_processing_batches(
-                    Bound::Excluded(*after),
-                    Bound::Excluded(*before),
-                ),
-            };
-
-            keep_ids_within_datetimes(
-                rtxn,
-                &mut filtered_non_processing_batches,
-                self.batches.started_at,
-                *after_started_at,
-                *before_started_at,
-            )?;
-            filtered_non_processing_batches | filtered_processing_batches
-        };
-
-        keep_ids_within_datetimes(
-            rtxn,
-            &mut batches,
-            self.batches.enqueued_at,
-            *after_enqueued_at,
-            *before_enqueued_at,
-        )?;
-
-        keep_ids_within_datetimes(
-            rtxn,
-            &mut batches,
-            self.batches.finished_at,
-            *after_finished_at,
-            *before_finished_at,
-        )?;
-
-        if let Some(limit) = limit {
-            batches = if query.reverse.unwrap_or_default() {
-                batches.into_iter().take(*limit as usize).collect()
-            } else {
-                batches.into_iter().rev().take(*limit as usize).collect()
-            };
-        }
-
-        Ok(batches)
-    }
-
-    /// Return the batch ids matching the query along with the total number of batches
-    /// by ignoring the from and limit parameters from the user's point of view.
-    ///
-    /// There are two differences between an internal query and a query executed by
-    /// the user.
-    ///
-    /// 1. IndexSwap tasks are not publicly associated with any index, but they are associated
-    ///    with many indexes internally.
-    /// 2. The user may not have the rights to access the tasks (internally) associated with all indexes.
-    pub(crate) fn get_batch_ids_from_authorized_indexes(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-        processing: &ProcessingTasks,
-    ) -> Result<(RoaringBitmap, u64)> {
-        // compute all batches matching the filter by ignoring the limits, to find the number of batches matching
-        // the filter.
-        // As this causes us to compute the filter twice it is slightly inefficient, but doing it this way spares
-        // us from modifying the underlying implementation, and the performance remains sufficient.
-        // Should this change, we would modify `get_batch_ids` to directly return the number of matching batches.
-        let total_batches =
-            self.get_batch_ids(rtxn, &query.clone().without_limits(), processing)?;
-        let mut batches = self.get_batch_ids(rtxn, query, processing)?;
-
-        // If the query contains a list of index uid or there is a finite list of authorized indexes,
-        // then we must exclude all the batches that only contains tasks associated to multiple indexes.
-        // This works because we don't autobatch tasks associated to multiple indexes with tasks associated
-        // to a single index. e.g: IndexSwap cannot be batched with IndexCreation.
-        if query.index_uids.is_some() || !filters.all_indexes_authorized() {
-            for kind in enum_iterator::all::<Kind>().filter(|kind| !kind.related_to_one_index()) {
-                batches -= self.tasks.get_kind(rtxn, kind)?;
-                if let Some(batch) = processing.batch.as_ref() {
-                    if batch.kinds.contains(&kind) {
-                        batches.remove(batch.uid);
-                    }
-                }
-            }
-        }
-
-        // Any batch that is internally associated with at least one authorized index
-        // must be returned.
-        if !filters.all_indexes_authorized() {
-            let mut valid_indexes = RoaringBitmap::new();
-            let mut forbidden_indexes = RoaringBitmap::new();
-
-            let all_indexes_iter = self.batches.index_tasks.iter(rtxn)?;
-            for result in all_indexes_iter {
-                let (index, index_tasks) = result?;
-                if filters.is_index_authorized(index) {
-                    valid_indexes |= index_tasks;
-                } else {
-                    forbidden_indexes |= index_tasks;
-                }
-            }
-            if let Some(batch) = processing.batch.as_ref() {
-                for index in &batch.indexes {
-                    if filters.is_index_authorized(index) {
-                        valid_indexes.insert(batch.uid);
-                    } else {
-                        forbidden_indexes.insert(batch.uid);
-                    }
-                }
-            }
-
-            // If a batch had ONE valid task then it should be returned
-            let invalid_batches = forbidden_indexes - valid_indexes;
-
-            batches -= invalid_batches;
-        }
-
-        Ok((batches, total_batches.len()))
-    }
-
-    pub(crate) fn get_batches_from_authorized_indexes(
-        &self,
-        rtxn: &RoTxn,
-        query: &Query,
-        filters: &meilisearch_auth::AuthFilter,
-        processing: &ProcessingTasks,
-    ) -> Result<(Vec<Batch>, u64)> {
-        let (batches, total) =
-            self.get_batch_ids_from_authorized_indexes(rtxn, query, filters, processing)?;
-        let batches = if query.reverse.unwrap_or_default() {
-            Box::new(batches.into_iter()) as Box<dyn Iterator<Item = u32>>
-        } else {
-            Box::new(batches.into_iter().rev()) as Box<dyn Iterator<Item = u32>>
-        };
-
-        let batches = self.batches.get_existing_batches(
-            rtxn,
-            batches.take(query.limit.unwrap_or(u32::MAX) as usize),
-            processing,
-        )?;
-
-        Ok((batches, total))
-    }
-}

crates/index-scheduler/src/queue/batches_test.rs

@@ -1,476 +0,0 @@
-use meili_snap::snapshot;
-use meilisearch_auth::AuthFilter;
-use meilisearch_types::index_uid_pattern::IndexUidPattern;
-use meilisearch_types::tasks::{IndexSwap, KindWithContent, Status};
-use time::{Duration, OffsetDateTime};
-
-use crate::insta_snapshot::{snapshot_bitmap, snapshot_index_scheduler};
-use crate::test_utils::Breakpoint::*;
-use crate::test_utils::{index_creation_task, FailureLocation};
-use crate::{IndexScheduler, Query};
-
-#[test]
-fn query_batches_from_and_limit() {
-    let (index_scheduler, mut handle) = IndexScheduler::test(true, vec![]);
-
-    let kind = index_creation_task("doggo", "bone");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_first_task");
-    let kind = index_creation_task("whalo", "plankton");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_second_task");
-    let kind = index_creation_task("catto", "his_own_vomit");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "registered_the_third_task");
-
-    handle.advance_n_successful_batches(3);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "processed_all_tasks");
-
-    let proc = index_scheduler.processing_tasks.read().unwrap().clone();
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let query = Query { limit: Some(0), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query { limit: Some(1), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query { limit: Some(2), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[1,2,]");
-
-    let query = Query { from: Some(1), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-
-    let query = Query { from: Some(2), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,2,]");
-
-    let query = Query { from: Some(1), limit: Some(1), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-
-    let query = Query { from: Some(1), limit: Some(2), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-}
-
-#[test]
-fn query_batches_simple() {
-    let start_time = OffsetDateTime::now_utc();
-
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("whalo", "fish");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    handle.advance_till([Start, BatchCreated]);
-
-    let query = Query { statuses: Some(vec![Status::Processing]), ..Default::default() };
-    let (mut batches, _) = index_scheduler
-        .get_batches_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    assert_eq!(batches.len(), 1);
-    batches[0].started_at = OffsetDateTime::UNIX_EPOCH;
-    assert!(batches[0].enqueued_at.is_some());
-    batches[0].enqueued_at = None;
-    // Insta cannot snapshot our batches because the batch stats contains an enum as key: https://github.com/mitsuhiko/insta/issues/689
-    let batch = serde_json::to_string_pretty(&batches[0]).unwrap();
-    snapshot!(batch, @r#"
-    {
-      "uid": 0,
-      "details": {
-        "primaryKey": "mouse"
-      },
-      "stats": {
-        "totalNbTasks": 1,
-        "status": {
-          "processing": 1
-        },
-        "types": {
-          "indexCreation": 1
-        },
-        "indexUids": {
-          "catto": 1
-        }
-      },
-      "startedAt": "1970-01-01T00:00:00Z",
-      "finishedAt": null,
-      "enqueuedAt": null
-    }
-    "#);
-
-    let query = Query { statuses: Some(vec![Status::Enqueued]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[]"); // The batches don't contains any enqueued tasks
-
-    let query =
-        Query { statuses: Some(vec![Status::Enqueued, Status::Processing]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    snapshot!(snapshot_bitmap(&batches), @"[0,]"); // both enqueued and processing tasks in the first tick
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        after_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test, which should excludes the enqueued tasks
-    snapshot!(snapshot_bitmap(&batches), @"[0,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        before_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes before the start of the test, which should excludes all of them
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Processing]),
-        after_started_at: Some(start_time),
-        before_started_at: Some(start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both enqueued and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test and before one minute after the start of the test,
-    // which should exclude the enqueued tasks and include the only processing task
-    snapshot!(snapshot_bitmap(&batches), @"[0,]");
-
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-        Start,
-        BatchCreated,
-    ]);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "after-advancing-a-bit");
-
-    let second_start_time = OffsetDateTime::now_utc();
-
-    let query = Query {
-        statuses: Some(vec![Status::Succeeded, Status::Processing]),
-        after_started_at: Some(start_time),
-        before_started_at: Some(start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the test and before one minute after the start of the test,
-    // which should include all tasks
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Succeeded, Status::Processing]),
-        before_started_at: Some(start_time),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes before the start of the test, which should exclude all tasks
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Succeeded, Status::Processing]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // both succeeded and processing tasks in the first tick, but limited to those with a started_at
-    // that comes after the start of the second part of the test and before one minute after the
-    // second start of the test, which should exclude all tasks
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    // now we make one more batch, the started_at field of the new tasks will be past `second_start_time`
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-        Start,
-        BatchCreated,
-    ]);
-
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // we run the same query to verify that, and indeed find that the last task is matched
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Enqueued, Status::Succeeded, Status::Processing]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // enqueued, succeeded, or processing tasks started after the second part of the test, should
-    // again only return the last task
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    handle.advance_till([ProcessBatchFailed, AfterProcessing]);
-
-    // now the last task should have failed
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "end");
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // so running the last query should return nothing
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // but the same query on failed tasks should return the last task
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // but the same query on failed tasks should return the last task
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        uids: Some(vec![1]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // same query but with an invalid uid
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    let query = Query {
-        statuses: Some(vec![Status::Failed]),
-        uids: Some(vec![2]),
-        after_started_at: Some(second_start_time),
-        before_started_at: Some(second_start_time + Duration::minutes(1)),
-        ..Default::default()
-    };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // same query but with a valid uid
-    snapshot!(snapshot_bitmap(&batches), @"[2,]");
-}
-
-#[test]
-fn query_batches_special_rules() {
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()) }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "whalo".to_owned()) }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    handle.advance_till([Start, BatchCreated]);
-
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-    let proc = index_scheduler.processing_tasks.read().unwrap().clone();
-
-    let query = Query { index_uids: Some(vec!["catto".to_owned()]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // only the first task associated with catto is returned, the indexSwap tasks are excluded!
-    snapshot!(snapshot_bitmap(&batches), @"[0,]");
-
-    let query = Query { index_uids: Some(vec!["catto".to_owned()]), ..Default::default() };
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we have asked for only the tasks associated with catto, but are only authorized to retrieve the tasks
-    // associated with doggo -> empty result
-    snapshot!(snapshot_bitmap(&batches), @"[]");
-
-    drop(rtxn);
-    // We're going to advance and process all the batches for the next query to actually hit the db
-    handle.advance_till([
-        InsideProcessBatch,
-        InsideProcessBatch,
-        ProcessBatchSucceeded,
-        AfterProcessing,
-    ]);
-    handle.advance_one_successful_batch();
-    handle.advance_n_failed_batches(2);
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "after-processing-everything");
-    let rtxn = index_scheduler.env.read_txn().unwrap();
-
-    let query = Query::default();
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we asked for all the tasks, but we are only authorized to retrieve the doggo tasks
-    // -> only the index creation of doggo should be returned
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-
-    let query = Query::default();
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(
-            &rtxn,
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![
-                    IndexUidPattern::new_unchecked("catto"),
-                    IndexUidPattern::new_unchecked("doggo"),
-                ]
-                .into_iter()
-                .collect(),
-            ),
-            &proc,
-        )
-        .unwrap();
-    // we asked for all the tasks, but we are only authorized to retrieve the doggo and catto tasks
-    // -> all tasks except the swap of catto with whalo are returned
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,]");
-
-    let query = Query::default();
-    let (batches, _) = index_scheduler
-        .queue
-        .get_batch_ids_from_authorized_indexes(&rtxn, &query, &AuthFilter::default(), &proc)
-        .unwrap();
-    // we asked for all the tasks with all index authorized -> all tasks returned
-    snapshot!(snapshot_bitmap(&batches), @"[0,1,2,3,]");
-}
-
-#[test]
-fn query_batches_canceled_by() {
-    let (index_scheduler, mut handle) =
-        IndexScheduler::test(true, vec![(3, FailureLocation::InsideProcessBatch)]);
-
-    let kind = index_creation_task("catto", "mouse");
-    let _ = index_scheduler.register(kind, None, false).unwrap();
-    let kind = index_creation_task("doggo", "sheep");
-    let _ = index_scheduler.register(kind, None, false).unwrap();
-    let kind = KindWithContent::IndexSwap {
-        swaps: vec![IndexSwap { indexes: ("catto".to_owned(), "doggo".to_owned()) }],
-    };
-    let _task = index_scheduler.register(kind, None, false).unwrap();
-
-    handle.advance_n_successful_batches(1);
-    let kind = KindWithContent::TaskCancelation {
-        query: "test_query".to_string(),
-        tasks: [0, 1, 2, 3].into_iter().collect(),
-    };
-    let task_cancelation = index_scheduler.register(kind, None, false).unwrap();
-    handle.advance_n_successful_batches(1);
-
-    snapshot!(snapshot_index_scheduler(&index_scheduler), name: "start");
-
-    let query = Query { canceled_by: Some(vec![task_cancelation.uid]), ..Query::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(&query, &AuthFilter::default())
-        .unwrap();
-    // The batch zero was the index creation task, the 1 is the task cancellation
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-
-    let query = Query { canceled_by: Some(vec![task_cancelation.uid]), ..Query::default() };
-    let (batches, _) = index_scheduler
-        .get_batch_ids_from_authorized_indexes(
-            &query,
-            &AuthFilter::with_allowed_indexes(
-                vec![IndexUidPattern::new_unchecked("doggo")].into_iter().collect(),
-            ),
-        )
-        .unwrap();
-    // Return only 1 because the user is not authorized to see task 2
-    snapshot!(snapshot_bitmap(&batches), @"[1,]");
-}

crates/index-scheduler/src/queue/mod.rs

@@ -1,391 +0,0 @@
-mod batches;
-#[cfg(test)]
-mod batches_test;
-mod tasks;
-#[cfg(test)]
-mod tasks_test;
-#[cfg(test)]
-mod test;
-
-use std::collections::BTreeMap;
-use std::fs::File as StdFile;
-use std::time::Duration;
-
-use file_store::FileStore;
-use meilisearch_types::batches::BatchId;
-use meilisearch_types::heed::{Database, Env, RoTxn, RwTxn, WithoutTls};
-use meilisearch_types::milli::{CboRoaringBitmapCodec, BEU32};
-use meilisearch_types::tasks::{Kind, KindWithContent, Status, Task};
-use roaring::RoaringBitmap;
-use time::format_description::well_known::Rfc3339;
-use time::OffsetDateTime;
-use uuid::Uuid;
-
-pub(crate) use self::batches::BatchQueue;
-pub(crate) use self::tasks::TaskQueue;
-use crate::processing::ProcessingTasks;
-use crate::utils::{
-    check_index_swap_validity, filter_out_references_to_newer_tasks, ProcessingBatch,
-};
-use crate::{Error, IndexSchedulerOptions, Result, TaskId};
-
-/// The number of database used by queue itself
-const NUMBER_OF_DATABASES: u32 = 1;
-/// Database const names for the `IndexScheduler`.
-mod db_name {
-    pub const BATCH_TO_TASKS_MAPPING: &str = "batch-to-tasks-mapping";
-}
-
-/// Defines a subset of tasks to be retrieved from the [`IndexScheduler`].
-///
-/// An empty/default query (where each field is set to `None`) matches all tasks.
-/// Each non-null field restricts the set of tasks further.
-#[derive(Default, Debug, Clone, PartialEq, Eq)]
-pub struct Query {
-    /// The maximum number of tasks to be matched
-    pub limit: Option<u32>,
-    /// The minimum [task id](`meilisearch_types::tasks::Task::uid`) to be matched
-    pub from: Option<u32>,
-    /// The order used to return the tasks. By default the newest tasks are returned first and the boolean is `false`.
-    pub reverse: Option<bool>,
-    /// The [task ids](`meilisearch_types::tasks::Task::uid`) to be matched
-    pub uids: Option<Vec<TaskId>>,
-    /// The [batch ids](`meilisearch_types::batches::Batch::uid`) to be matched
-    pub batch_uids: Option<Vec<BatchId>>,
-    /// The allowed [statuses](`meilisearch_types::tasks::Task::status`) of the matched tasls
-    pub statuses: Option<Vec<Status>>,
-    /// The allowed [kinds](meilisearch_types::tasks::Kind) of the matched tasks.
-    ///
-    /// The kind of a task is given by:
-    /// ```
-    /// # use meilisearch_types::tasks::{Task, Kind};
-    /// # fn doc_func(task: Task) -> Kind {
-    /// task.kind.as_kind()
-    /// # }
-    /// ```
-    pub types: Option<Vec<Kind>>,
-    /// The allowed [index ids](meilisearch_types::tasks::Task::index_uid) of the matched tasks
-    pub index_uids: Option<Vec<String>>,
-    /// The [task ids](`meilisearch_types::tasks::Task::uid`) of the [`TaskCancelation`](meilisearch_types::tasks::Task::Kind::TaskCancelation) tasks
-    /// that canceled the matched tasks.
-    pub canceled_by: Option<Vec<TaskId>>,
-    /// Exclusive upper bound of the matched tasks' [`enqueued_at`](meilisearch_types::tasks::Task::enqueued_at) field.
-    pub before_enqueued_at: Option<OffsetDateTime>,
-    /// Exclusive lower bound of the matched tasks' [`enqueued_at`](meilisearch_types::tasks::Task::enqueued_at) field.
-    pub after_enqueued_at: Option<OffsetDateTime>,
-    /// Exclusive upper bound of the matched tasks' [`started_at`](meilisearch_types::tasks::Task::started_at) field.
-    pub before_started_at: Option<OffsetDateTime>,
-    /// Exclusive lower bound of the matched tasks' [`started_at`](meilisearch_types::tasks::Task::started_at) field.
-    pub after_started_at: Option<OffsetDateTime>,
-    /// Exclusive upper bound of the matched tasks' [`finished_at`](meilisearch_types::tasks::Task::finished_at) field.
-    pub before_finished_at: Option<OffsetDateTime>,
-    /// Exclusive lower bound of the matched tasks' [`finished_at`](meilisearch_types::tasks::Task::finished_at) field.
-    pub after_finished_at: Option<OffsetDateTime>,
-}
-
-impl Query {
-    /// Return `true` if every field of the query is set to `None`, such that the query
-    /// matches all tasks.
-    pub fn is_empty(&self) -> bool {
-        matches!(
-            self,
-            Query {
-                limit: None,
-                from: None,
-                reverse: None,
-                uids: None,
-                batch_uids: None,
-                statuses: None,
-                types: None,
-                index_uids: None,
-                canceled_by: None,
-                before_enqueued_at: None,
-                after_enqueued_at: None,
-                before_started_at: None,
-                after_started_at: None,
-                before_finished_at: None,
-                after_finished_at: None,
-            }
-        )
-    }
-
-    /// Add an [index id](meilisearch_types::tasks::Task::index_uid) to the list of permitted indexes.
-    pub fn with_index(self, index_uid: String) -> Self {
-        let mut index_vec = self.index_uids.unwrap_or_default();
-        index_vec.push(index_uid);
-        Self { index_uids: Some(index_vec), ..self }
-    }
-
-    // Removes the `from` and `limit` restrictions from the query.
-    // Useful to get the total number of tasks matching a filter.
-    pub fn without_limits(self) -> Self {
-        Query { limit: None, from: None, ..self }
-    }
-}
-
-/// Structure which holds meilisearch's indexes and schedules the tasks
-/// to be performed on them.
-pub struct Queue {
-    pub(crate) tasks: tasks::TaskQueue,
-    pub(crate) batches: batches::BatchQueue,
-
-    /// Matches a batch id with the associated task ids.
-    pub(crate) batch_to_tasks_mapping: Database<BEU32, CboRoaringBitmapCodec>,
-
-    /// The list of files referenced by the tasks.
-    pub(crate) file_store: FileStore,
-
-    /// The max number of tasks allowed before the scheduler starts to delete
-    /// the finished tasks automatically.
-    pub(crate) max_number_of_tasks: usize,
-}
-
-impl Queue {
-    pub(crate) fn private_clone(&self) -> Queue {
-        Queue {
-            tasks: self.tasks.private_clone(),
-            batches: self.batches.private_clone(),
-            batch_to_tasks_mapping: self.batch_to_tasks_mapping,
-            file_store: self.file_store.clone(),
-            max_number_of_tasks: self.max_number_of_tasks,
-        }
-    }
-
-    pub(crate) const fn nb_db() -> u32 {
-        tasks::TaskQueue::nb_db() + batches::BatchQueue::nb_db() + NUMBER_OF_DATABASES
-    }
-
-    /// Create an index scheduler and start its run loop.
-    pub(crate) fn new(
-        env: &Env<WithoutTls>,
-        wtxn: &mut RwTxn,
-        options: &IndexSchedulerOptions,
-    ) -> Result<Self> {
-        // allow unreachable_code to get rids of the warning in the case of a test build.
-        Ok(Self {
-            file_store: FileStore::new(&options.update_file_path)?,
-            batch_to_tasks_mapping: env
-                .create_database(wtxn, Some(db_name::BATCH_TO_TASKS_MAPPING))?,
-            tasks: TaskQueue::new(env, wtxn)?,
-            batches: BatchQueue::new(env, wtxn)?,
-            max_number_of_tasks: options.max_number_of_tasks,
-        })
-    }
-
-    /// Returns the whole set of tasks that belongs to this batch.
-    pub(crate) fn tasks_in_batch(&self, rtxn: &RoTxn, batch_id: BatchId) -> Result<RoaringBitmap> {
-        Ok(self.batch_to_tasks_mapping.get(rtxn, &batch_id)?.unwrap_or_default())
-    }
-
-    /// Convert an iterator to a `Vec` of tasks and edit the `ProcessingBatch` to add the given tasks.
-    ///
-    /// The tasks MUST exist, or a `CorruptedTaskQueue` error will be thrown.
-    pub(crate) fn get_existing_tasks_for_processing_batch(
-        &self,
-        rtxn: &RoTxn,
-        processing_batch: &mut ProcessingBatch,
-        tasks: impl IntoIterator<Item = TaskId>,
-    ) -> Result<Vec<Task>> {
-        tasks
-            .into_iter()
-            .map(|task_id| {
-                let mut task = self
-                    .tasks
-                    .get_task(rtxn, task_id)
-                    .and_then(|task| task.ok_or(Error::CorruptedTaskQueue));
-                processing_batch.processing(&mut task);
-                task
-            })
-            .collect::<Result<_>>()
-    }
-
-    pub(crate) fn write_batch(
-        &self,
-        wtxn: &mut RwTxn,
-        batch: ProcessingBatch,
-        tasks: &RoaringBitmap,
-    ) -> Result<()> {
-        self.batch_to_tasks_mapping.put(wtxn, &batch.uid, tasks)?;
-        self.batches.write_batch(wtxn, batch)?;
-        Ok(())
-    }
-
-    pub(crate) fn delete_persisted_task_data(&self, task: &Task) -> Result<()> {
-        match task.content_uuid() {
-            Some(content_file) => self.delete_update_file(content_file),
-            None => Ok(()),
-        }
-    }
-
-    /// Open and returns the task's content File.
-    pub fn update_file(&self, uuid: Uuid) -> file_store::Result<StdFile> {
-        self.file_store.get_update(uuid)
-    }
-
-    /// Delete a file from the index scheduler.
-    ///
-    /// Counterpart to the [`create_update_file`](IndexScheduler::create_update_file) method.
-    pub fn delete_update_file(&self, uuid: Uuid) -> Result<()> {
-        Ok(self.file_store.delete(uuid)?)
-    }
-
-    /// Create a file and register it in the index scheduler.
-    ///
-    /// The returned file and uuid can be used to associate
-    /// some data to a task. The file will be kept until
-    /// the task has been fully processed.
-    pub fn create_update_file(&self, dry_run: bool) -> Result<(Uuid, file_store::File)> {
-        if dry_run {
-            Ok((Uuid::nil(), file_store::File::dry_file()?))
-        } else {
-            Ok(self.file_store.new_update()?)
-        }
-    }
-
-    #[cfg(test)]
-    pub fn create_update_file_with_uuid(&self, uuid: u128) -> Result<(Uuid, file_store::File)> {
-        Ok(self.file_store.new_update_with_uuid(uuid)?)
-    }
-
-    /// The size on disk taken by all the updates files contained in the `IndexScheduler`, in bytes.
-    pub fn compute_update_file_size(&self) -> Result<u64> {
-        Ok(self.file_store.compute_total_size()?)
-    }
-
-    pub fn register(
-        &self,
-        wtxn: &mut RwTxn,
-        kind: &KindWithContent,
-        task_id: Option<TaskId>,
-        dry_run: bool,
-    ) -> Result<Task> {
-        let next_task_id = self.tasks.next_task_id(wtxn)?;
-
-        if let Some(uid) = task_id {
-            if uid < next_task_id {
-                return Err(Error::BadTaskId { received: uid, expected: next_task_id });
-            }
-        }
-
-        let mut task = Task {
-            uid: task_id.unwrap_or(next_task_id),
-            // The batch is defined once we starts processing the task
-            batch_uid: None,
-            enqueued_at: OffsetDateTime::now_utc(),
-            started_at: None,
-            finished_at: None,
-            error: None,
-            canceled_by: None,
-            details: kind.default_details(),
-            status: Status::Enqueued,
-            kind: kind.clone(),
-        };
-        // For deletion and cancelation tasks, we want to make extra sure that they
-        // don't attempt to delete/cancel tasks that are newer than themselves.
-        filter_out_references_to_newer_tasks(&mut task);
-        // If the register task is an index swap task, verify that it is well-formed
-        // (that it does not contain duplicate indexes).
-        check_index_swap_validity(&task)?;
-
-        // At this point the task is going to be registered and no further checks will be done
-        if dry_run {
-            return Ok(task);
-        }
-
-        // Get rid of the mutability.
-        let task = task;
-        self.tasks.register(wtxn, &task)?;
-
-        Ok(task)
-    }
-
-    /// Register a task to cleanup the task queue if needed
-    pub fn cleanup_task_queue(&self, wtxn: &mut RwTxn) -> Result<()> {
-        let nb_tasks = self.tasks.all_task_ids(wtxn)?.len();
-        // if we have less than 1M tasks everything is fine
-        if nb_tasks < self.max_number_of_tasks as u64 {
-            return Ok(());
-        }
-
-        let finished = self.tasks.status.get(wtxn, &Status::Succeeded)?.unwrap_or_default()
-            | self.tasks.status.get(wtxn, &Status::Failed)?.unwrap_or_default()
-            | self.tasks.status.get(wtxn, &Status::Canceled)?.unwrap_or_default();
-
-        let to_delete = RoaringBitmap::from_iter(finished.into_iter().rev().take(100_000));
-
-        // /!\ the len must be at least 2 or else we might enter an infinite loop where we only delete
-        //     the deletion tasks we enqueued ourselves.
-        if to_delete.len() < 2 {
-            tracing::warn!("The task queue is almost full, but no task can be deleted yet.");
-            // the only thing we can do is hope that the user tasks are going to finish
-            return Ok(());
-        }
-
-        tracing::info!(
-            "The task queue is almost full. Deleting the oldest {} finished tasks.",
-            to_delete.len()
-        );
-
-        // it's safe to unwrap here because we checked the len above
-        let newest_task_id = to_delete.iter().last().unwrap();
-        let last_task_to_delete =
-            self.tasks.get_task(wtxn, newest_task_id)?.ok_or(Error::CorruptedTaskQueue)?;
-
-        // increase time by one nanosecond so that the enqueuedAt of the last task to delete is also lower than that date.
-        let delete_before = last_task_to_delete.enqueued_at + Duration::from_nanos(1);
-
-        self.register(
-            wtxn,
-            &KindWithContent::TaskDeletion {
-                query: format!(
-                    "?beforeEnqueuedAt={}&statuses=succeeded,failed,canceled",
-                    delete_before.format(&Rfc3339).map_err(|_| Error::CorruptedTaskQueue)?,
-                ),
-                tasks: to_delete,
-            },
-            None,
-            false,
-        )?;
-
-        Ok(())
-    }
-
-    pub fn get_stats(
-        &self,
-        rtxn: &RoTxn,
-        processing: &ProcessingTasks,
-    ) -> Result<BTreeMap<String, BTreeMap<String, u64>>> {
-        let mut res = BTreeMap::new();
-        let processing_tasks = processing.processing.len();
-
-        res.insert(
-            "statuses".to_string(),
-            enum_iterator::all::<Status>()
-                .map(|s| {
-                    let tasks = self.tasks.get_status(rtxn, s)?.len();
-                    match s {
-                        Status::Enqueued => Ok((s.to_string(), tasks - processing_tasks)),
-                        Status::Processing => Ok((s.to_string(), processing_tasks)),
-                        s => Ok((s.to_string(), tasks)),
-                    }
-                })
-                .collect::<Result<BTreeMap<String, u64>>>()?,
-        );
-        res.insert(
-            "types".to_string(),
-            enum_iterator::all::<Kind>()
-                .map(|s| Ok((s.to_string(), self.tasks.get_kind(rtxn, s)?.len())))
-                .collect::<Result<BTreeMap<String, u64>>>()?,
-        );
-        res.insert(
-            "indexes".to_string(),
-            self.tasks
-                .index_tasks
-                .iter(rtxn)?
-                .map(|res| Ok(res.map(|(name, bitmap)| (name.to_string(), bitmap.len()))?))
-                .collect::<Result<BTreeMap<String, u64>>>()?,
-        );
-
-        Ok(res)
-    }
-}

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_canceled_by/start.snap

@@ -1,87 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,3,]
-canceled [1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,]
-"taskCancelation" [3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,]
-doggo [1,2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-3 [1,2,]
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, }
-1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"]}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,2,3,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-canceled [1,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,]
-"indexSwap" [1,]
-"taskCancelation" [1,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,1,]
-doggo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/processed_all_tasks.snap

@@ -1,88 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, batch_uid: 2, status: succeeded, details: { primary_key: Some("his_own_vomit") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-whalo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"bone"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, }
-1 {uid: 1, details: {"primaryKey":"plankton"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"whalo":1}}, }
-2 {uid: 2, details: {"primaryKey":"his_own_vomit"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,2,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_first_task.snap

@@ -1,52 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,]
-----------------------------------------------------------------------
-### Index Tasks:
-doggo [0,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_second_task.snap

@@ -1,55 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-----------------------------------------------------------------------
-### Index Tasks:
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_from_and_limit/registered_the_third_task.snap

@@ -1,58 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("bone") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("bone") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("plankton") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("plankton") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("his_own_vomit") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("his_own_vomit") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [2,]
-doggo [0,]
-whalo [1,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/after-advancing-a-bit.snap

@@ -1,71 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch Some(1):
-[1,]
-{uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"processing":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, }
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [1,2,]
-succeeded [0,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/end.snap

@@ -1,89 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,]
-failed [2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, }
-1 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, }
-2 {uid: 2, details: {"primaryKey":"fish"}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexCreation":1},"indexUids":{"whalo":1}}, }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-failed [2,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_simple/start.snap

@@ -1,58 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { primary_key: Some("fish") }, kind: IndexCreation { index_uid: "whalo", primary_key: Some("fish") }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,2,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,]
-doggo [1,]
-whalo [2,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_special_rules/after-processing-everything.snap

@@ -1,100 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: succeeded, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 2, status: failed, error: ResponseError { code: 200, message: "Planned failure for tests.", error_code: "internal", error_type: "internal", error_link: "https://docs.meilisearch.com/errors#internal" }, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, batch_uid: 3, status: failed, error: ResponseError { code: 200, message: "Index `whalo` not found.", error_code: "index_not_found", error_type: "invalid_request", error_link: "https://docs.meilisearch.com/errors#index_not_found" }, details: { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,1,]
-failed [2,3,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,3,]
-doggo [1,2,]
-whalo [3,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-doggo: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, }
-1 {uid: 1, details: {"primaryKey":"sheep"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"doggo":1}}, }
-2 {uid: 2, details: {"swaps":[{"indexes":["catto","doggo"]}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, }
-3 {uid: 3, details: {"swaps":[{"indexes":["catto","whalo"]}]}, stats: {"totalNbTasks":1,"status":{"failed":1},"types":{"indexSwap":1},"indexUids":{}}, }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,]
-2 [2,]
-3 [3,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-failed [2,3,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,3,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,2,3,]
-doggo [1,2,]
-whalo [3,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/batches_test.rs/query_batches_special_rules/start.snap

@@ -1,61 +0,0 @@
----
-source: crates/index-scheduler/src/queue/batches_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, status: enqueued, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, status: enqueued, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, status: enqueued, details: { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "whalo") }] }}
-----------------------------------------------------------------------
-### Status:
-enqueued [0,1,2,3,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,3,]
-doggo [1,2,]
-whalo [3,]
-----------------------------------------------------------------------
-### Index Mapper:
-
-----------------------------------------------------------------------
-### Canceled By:
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-----------------------------------------------------------------------
-### Finished At:
-----------------------------------------------------------------------
-### All Batches:
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-----------------------------------------------------------------------
-### Batches Status:
-----------------------------------------------------------------------
-### Batches Kind:
-----------------------------------------------------------------------
-### Batches Index Tasks:
-----------------------------------------------------------------------
-### Batches Enqueued At:
-----------------------------------------------------------------------
-### Batches Started At:
-----------------------------------------------------------------------
-### Batches Finished At:
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------

crates/index-scheduler/src/queue/snapshots/tasks_test.rs/query_tasks_canceled_by/start.snap

@@ -1,87 +0,0 @@
----
-source: crates/index-scheduler/src/queue/tasks_test.rs
-snapshot_kind: text
----
-### Autobatching Enabled = true
-### Processing batch None:
-[]
-----------------------------------------------------------------------
-### All Tasks:
-0 {uid: 0, batch_uid: 0, status: succeeded, details: { primary_key: Some("mouse") }, kind: IndexCreation { index_uid: "catto", primary_key: Some("mouse") }}
-1 {uid: 1, batch_uid: 1, status: canceled, canceled_by: 3, details: { primary_key: Some("sheep") }, kind: IndexCreation { index_uid: "doggo", primary_key: Some("sheep") }}
-2 {uid: 2, batch_uid: 1, status: canceled, canceled_by: 3, details: { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }, kind: IndexSwap { swaps: [IndexSwap { indexes: ("catto", "doggo") }] }}
-3 {uid: 3, batch_uid: 1, status: succeeded, details: { matched_tasks: 3, canceled_tasks: Some(2), original_filter: "test_query" }, kind: TaskCancelation { query: "test_query", tasks: RoaringBitmap<[0, 1, 2]> }}
-----------------------------------------------------------------------
-### Status:
-enqueued []
-succeeded [0,3,]
-canceled [1,2,]
-----------------------------------------------------------------------
-### Kind:
-"indexCreation" [0,1,]
-"indexSwap" [2,]
-"taskCancelation" [3,]
-----------------------------------------------------------------------
-### Index Tasks:
-catto [0,2,]
-doggo [1,2,]
-----------------------------------------------------------------------
-### Index Mapper:
-catto: { number_of_documents: 0, field_distribution: {} }
-
-----------------------------------------------------------------------
-### Canceled By:
-3 [1,2,]
-
-----------------------------------------------------------------------
-### Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [2,]
-[timestamp] [3,]
-----------------------------------------------------------------------
-### Started At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### Finished At:
-[timestamp] [0,]
-[timestamp] [1,2,3,]
-----------------------------------------------------------------------
-### All Batches:
-0 {uid: 0, details: {"primaryKey":"mouse"}, stats: {"totalNbTasks":1,"status":{"succeeded":1},"types":{"indexCreation":1},"indexUids":{"catto":1}}, }
-1 {uid: 1, details: {"primaryKey":"sheep","matchedTasks":3,"canceledTasks":2,"originalFilter":"test_query","swaps":[{"indexes":["catto","doggo"]}]}, stats: {"totalNbTasks":3,"status":{"succeeded":1,"canceled":2},"types":{"indexCreation":1,"indexSwap":1,"taskCancelation":1},"indexUids":{"doggo":1}}, }
-----------------------------------------------------------------------
-### Batch to tasks mapping:
-0 [0,]
-1 [1,2,3,]
-----------------------------------------------------------------------
-### Batches Status:
-succeeded [0,1,]
-canceled [1,]
-----------------------------------------------------------------------
-### Batches Kind:
-"indexCreation" [0,1,]
-"indexSwap" [1,]
-"taskCancelation" [1,]
-----------------------------------------------------------------------
-### Batches Index Tasks:
-catto [0,1,]
-doggo [1,]
-----------------------------------------------------------------------
-### Batches Enqueued At:
-[timestamp] [0,]
-[timestamp] [1,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Started At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### Batches Finished At:
-[timestamp] [0,]
-[timestamp] [1,]
-----------------------------------------------------------------------
-### File Store:
-
-----------------------------------------------------------------------